diff --git a/.claude/skills/openbot-screen-layout/SKILL.md b/.claude/skills/openbot-screen-layout/SKILL.md index 20cd104d6f..39442b1160 100644 --- a/.claude/skills/openbot-screen-layout/SKILL.md +++ b/.claude/skills/openbot-screen-layout/SKILL.md @@ -188,11 +188,11 @@ itself when the dialog changes something. ## Units -The root is the browser's 16px, so Tailwind's spacing, sizes and widths are their documented values -(`p-4` is 16px, `max-w-3xl` is 768px) and land on its 4px grid. Text is the exception: `styles.css` -sets every `--text-*` to Tailwind's default times 15/16, so `text-sm` is 13.125px and body text is -15px. That is the scale this app has always read at; do not size text in px or rem by hand to match -the docs. +The root is the browser's 16px, so Tailwind's spacing, sizes, widths and text sizes are their +documented values (`p-4` is 16px, `max-w-3xl` is 768px, `text-sm` is 14px) and land on its 4px grid. +Text used to be the exception, on a 15px scale of its own; it is not any more. Where a screen needs one +of the named sizes, use the five `type-*` utilities in `theme.css`; do not size text in px or rem by +hand. Even so, never hardcode a pixel constant derived from a Tailwind token, and never compute a ratio from one. Measure the element — `offsetWidth`, `clientWidth`, a `ResizeObserver` — and scale from what diff --git a/CHANGELOG.md b/CHANGELOG.md index 27745c6f84..b36f0f8cd0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -133,6 +133,67 @@ again when they come back to the end or press the arrow. A rail beside a convers per turn: hovering one previews what was asked and answered, and choosing it goes there. Nothing changes for a deployment. +### First run shows what OpenBot does, one step at a time + +- The looping intro is replaced by steps that show what OpenBot does (a browser and files of its + own, a team of Bots, and asking before it acts), then Connect your apps, Get notified, Check your + setup, and the name step, whose Get started opens the app. On a desktop the steps are listed on the + left beside a picture of each; on a phone they come one at a time. A reload, browser Back and a + connection's return all keep the person's place, which is now saved as they go. +- Check your setup asks the new `GET /api/setup` whether Intelligence, the model key, the Bot computer, + more apps and Slack and Teams work, row by row. An admin, or somebody running OpenBot alone, is + shown the page or the variables that close a gap; everybody else only that their admin sets it up. + Check again asks Intelligence and the model provider again, at most every 10 seconds. Reading the + checklist does the same once an answer has failed or is five minutes old, without waiting for it, + so the next read has the new answer. To check the Bot computer, the server sends each running + computer its network policy whenever it asks, and sees that it answers; it wakes none. +- An app connected from Connect your apps comes back to that step from the app's own sign-in page, + whether it connected, failed or was cancelled there. A connection started from Connections or an + app's admin page comes back where it did before. + +### OpenBot has a new look + +- Every screen has new colours: cool neutral greys, a near-black primary button that turns light + in dark mode, a purple focus ring, slightly rounder corners and quicker easing. Text now follows + the browser's own size, so it is a little larger than before. +- Dark mode follows the system by default. Settings → General → Appearance offers System, Light and + Dark, kept in this browser. Someone who had turned the old dark switch on stays dark. +- Every Bot, and OpenBot's own mark, is an orb with eyes that blinks and follows the pointer. It + holds still with reduced motion. +- Each Bot's orb takes its colour and pattern from the Bot, so two Bots in one colour still look + different. Violet is kept for OpenBot's own mark. +- The sidebar takes the same look, with OpenBot's orb beside its name. A row lifts to a card under + the pointer and while it is open, and you can always see where the keyboard is. While a Bot is + working, its orb in the list shows it, and so does a still blue dot. Each conversation has a "…" + button with Pin and Delete, which used to be on right-click alone. ⌘K, or Ctrl+K, opens the + sidebar and puts you in its search box. +- A conversation reads like a chat. A Bot's reply arrives one bubble per thought, and a bubble holding + a code block or a table takes the full width on a wide screen; your messages sit in a tinted bubble, + and three dots show a Bot thinking. A step is marked done only when it worked: one that was refused, + failed or declined by you is named for what it would have done ("Send your email"), and one held on + its approval card says it is waiting for you. Mail and calendar steps, through any app, are named + for what happened to yours ("Searched your email"). +- A conversation with one Bot shows that Bot's orb in its header. The orb works while the Bot does, + watches while you type, and smiles when an answer lands. Under the conversation's name, "Working · + Searching the web" says which step the Bot is on, and "Needs you" says an approval is waiting there. + A run that goes on after you allow something shows as working too. The message box is a rounded + pill with larger buttons, and it says "Message" and the Bot's name until you type. +- A Bot's computer opens in a wider pane beside the conversation. Its header shows the Bot's orb, + whose computer it is and Take control, with a dot while the conversation needs you. Under the + screen, every step the Bot took in its browser, approved clicks included, sits among the commands + it ran, newest last, marked done, waiting for you or didn't work. The pane opens by itself only + when it fits beside the conversation; on a narrow screen the computer button lights up instead. + The box for a password or a code says the value goes straight to the page and the Bot never sees + it. +- When a Bot asks before acting, its card says in plain words what it will do, in which app or + where, to whom, and what it will send, with everything else under Details. Decline and Allow once + answer it, and a quieter Always allow here saves a rule for that Bot, action and place; a step + handed to you asks "I did it myself" or "Don't do it". The same card is in the conversation, on + the Bot's page and in the list of what needs you. Once you answer, it folds to how it went + (Allowed, Sent, Done, Declined, Didn't work) and says who decided. A request for a step that is + alone in its turn now shows in the conversation too, where before it waited only on the Bot's + page. Under the message box, every conversation says that OpenBot is AI and can make mistakes. + ### Generated interfaces use A2UI 1.0 With generative UI on, a Bot builds an interface from the A2UI 1.0 basic components instead of the diff --git a/app/index.html b/app/index.html index a839d57c50..b35c822cb2 100644 --- a/app/index.html +++ b/app/index.html @@ -5,15 +5,27 @@ OpenBot + diff --git a/app/package.json b/app/package.json index b5be1f7ac8..406d4f0a40 100644 --- a/app/package.json +++ b/app/package.json @@ -28,7 +28,6 @@ "@tanstack/react-router": "^1.170.27", "@tanstack/react-table": "^9.3.1", "better-auth": "^1.6.27", - "boring-avatars": "^2.0.4", "class-variance-authority": "^0.7.1", "clsx": "^2.1.1", "embla-carousel-react": "^8.6.0", diff --git a/app/src/components/agents/abstract-avatar.tsx b/app/src/components/agents/abstract-avatar.tsx index 9ca9845070..d746511206 100644 --- a/app/src/components/agents/abstract-avatar.tsx +++ b/app/src/components/agents/abstract-avatar.tsx @@ -1,5 +1,14 @@ -import Avatar from "boring-avatars"; +import { BotOrb } from "@/components/agents/orb/bot-orb"; +/** + * A Bot's face: its orb, with the look its seed picks. The seed is the Bot's id unless a tenant + * package set one or the Bot was copied from another, so `ChannelAvatar`, which goes by id, matches it + * for every other Bot. + * The orb hides itself from assistive tech, so the Bot is announced once, by name. + * + * Clipped to its circle, so its glow never spills into the row beside it or makes a list scroll. + * At rest the glow is too faint that far out to be missed. + */ export function AbstractAvatar({ name, seed, @@ -16,10 +25,7 @@ export function AbstractAvatar({ className="inline-flex shrink-0 overflow-hidden rounded-full" style={{ height: size, width: size }} > - {/* The drawing carries its own role; hidden so the coworker is announced once, by name. */} - + ); } diff --git a/app/src/components/agents/orb/agent-orb.tsx b/app/src/components/agents/orb/agent-orb.tsx index 9fc2c7f393..4f953f922b 100644 --- a/app/src/components/agents/orb/agent-orb.tsx +++ b/app/src/components/agents/orb/agent-orb.tsx @@ -1,437 +1,52 @@ -import { - type MotionStyle, - motion, - type TargetAndTransition, - type Transition, - useReducedMotion, - useTransform, -} from "motion/react"; import { cn } from "@/lib/utils"; -import { - type AIAmplitude, - type AIState, - getAIStateMotion, - useAmplitudeValue, -} from "./ai-core"; - -const SIZE_THRESHOLD_SMALL = 50; -const SIZE_THRESHOLD_TINY = 30; -const SIZE_THRESHOLD_MEDIUM = 100; -const BLUR_MULTIPLIER_SMALL = 0.008; -const BLUR_MIN_SMALL = 1; -const BLUR_MULTIPLIER_LARGE = 0.015; -const BLUR_MIN_LARGE = 4; -const CONTRAST_MULTIPLIER_SMALL = 0.004; -const CONTRAST_MIN_SMALL = 1.2; -const CONTRAST_MULTIPLIER_LARGE = 0.008; -const CONTRAST_MIN_LARGE = 1.5; -const DOT_SIZE_MULTIPLIER_SMALL = 0.004; -const DOT_SIZE_MIN_SMALL = 0.05; -const DOT_SIZE_MULTIPLIER_LARGE = 0.008; -const DOT_SIZE_MIN_LARGE = 0.1; -const SHADOW_MULTIPLIER_SMALL = 0.004; -const SHADOW_MIN_SMALL = 0.5; -const SHADOW_MULTIPLIER_LARGE = 0.008; -const SHADOW_MIN_LARGE = 2; -const MASK_RADIUS_TINY = "0%"; -const MASK_RADIUS_SMALL = "5%"; -const MASK_RADIUS_MEDIUM = "15%"; -const MASK_RADIUS_LARGE = "25%"; -const CONTRAST_TINY = 1.1; -const CONTRAST_MULTIPLIER_FINAL = 1.2; -const CONTRAST_MIN_FINAL = 1.3; - -/** Loud audio tightens the gradient, which reads as the orb "focusing". */ -const AMPLITUDE_BLUR_FALLOFF = 0.45; -/** Amplitude adds at most 12% of extra size on top of the state scale. */ -const AMPLITUDE_SCALE_GAIN = 0.12; -/** Single lateral nudge used to signal `error`, well under 200ms. */ -const ERROR_SHAKE_KEYFRAMES = [0, -3, 3, 0]; -const ERROR_SHAKE_DURATION = 0.18; -const EASE_IN_OUT = [0.645, 0.045, 0.355, 1] as const; -const GLOW_BLUR_RATIO = 0.28; -const GLOW_MAX_OPACITY = 0.7; -/** Depth rim, as a fraction of the orb size. */ -const RIM_RATIO = 0.06; -const RIM_MIN = 1.5; -/** Sheen drift period at rest, before the state speed divides it. */ -const DRIFT_BASE_SECONDS = 12; -/** `idle` breathes: a slow, shallow scale cycle that never draws attention. */ -const BREATHE_SCALE = [1, 1.035, 1]; -const BREATHE_SECONDS = 5.5; -const SPRING_DEFAULT: Transition = { - bounce: 0.1, - duration: 0.25, - type: "spring", +import { BotOrb, type BotOrbState } from "./bot-orb"; + +/** What OpenBot is doing, wherever its own mark stands for it. */ +export type AIState = + | "idle" + | "listening" + | "thinking" + | "streaming" + | "done" + | "error"; + +/** + * Each state as the orb shows it. Streaming is still working. An error rests, and the words beside + * the mark say what went wrong. + */ +const ORB_STATE: Record = { + idle: "idle", + listening: "listening", + thinking: "thinking", + streaming: "thinking", + done: "done", + error: "idle", }; export interface AgentOrbProps { + className?: string; /** - * Live audio level, 0–1. Pass the `MotionValue` from `useAudioAmplitude` so - * the signal never re-renders React. + * Its footprint, in px, as "64px". Only px: the orb is drawn on a canvas at that size, so a share + * of its container, as "100%", has nothing to be a share of. */ - amplitude?: AIAmplitude; - /** Ambient rotation period in seconds, before the state speed multiplier. */ - animationDuration?: number; - className?: string; - colors?: { - bg?: string; - c1?: string; - c2?: string; - c3?: string; - /** Fourth mesh stop. More stops means fewer visible repeats per rotation. */ - c4?: string; - }; - size?: string; - /** Shared AI state driving speed, scale, saturation and reactivity. */ + size?: `${number}px`; state?: AIState; } -const AgentOrb: React.FC = ({ - size = "192px", +/** OpenBot's own mark: the orb, in OpenBot's violet. */ +export default function AgentOrb({ className, - colors, - animationDuration = 20, - amplitude, + size = "192px", state = "idle", -}) => { - const shouldReduceMotion = useReducedMotion(); - const amplitudeValue = useAmplitudeValue(amplitude); - const stateMotion = getAIStateMotion(state); - - /** - * Higher chroma than the original pastels. - * - * The first palette sat at chroma 0.12–0.15, which washes out once the mesh is - * blurred and the dot pattern is overlaid on top. Pushing to ~0.2 and adding a - * saturate() pass to the mesh is what makes the colour survive all that. - */ - const defaultColors = { - bg: "oklch(92% 0.03 300)", - c1: "oklch(68% 0.21 350)", // Pink - c2: "oklch(70% 0.18 210)", // Blue - c3: "oklch(66% 0.2 285)", // Violet - c4: "oklch(72% 0.19 325)", // Magenta, the fourth stop for variety - }; - - const finalColors = { ...defaultColors, ...colors }; - // The bloom stays in the orb's own palette — see the note in ai-orb-aura. - const glowColor = finalColors.c2; - - // Extract numeric value from size for calculations - const sizeValue = Number.parseInt(size.replace("px", ""), 10); - - // Responsive calculations based on size - const blurAmount = - sizeValue < SIZE_THRESHOLD_SMALL - ? Math.max(sizeValue * BLUR_MULTIPLIER_SMALL, BLUR_MIN_SMALL) // Reduced blur for small sizes - : Math.max(sizeValue * BLUR_MULTIPLIER_LARGE, BLUR_MIN_LARGE); - - const contrastAmount = - sizeValue < SIZE_THRESHOLD_SMALL - ? Math.max(sizeValue * CONTRAST_MULTIPLIER_SMALL, CONTRAST_MIN_SMALL) // Reduced contrast for small sizes - : Math.max(sizeValue * CONTRAST_MULTIPLIER_LARGE, CONTRAST_MIN_LARGE); - - const dotSize = - sizeValue < SIZE_THRESHOLD_SMALL - ? Math.max(sizeValue * DOT_SIZE_MULTIPLIER_SMALL, DOT_SIZE_MIN_SMALL) // Smaller dots for small sizes - : Math.max(sizeValue * DOT_SIZE_MULTIPLIER_LARGE, DOT_SIZE_MIN_LARGE); - - const shadowSpread = - sizeValue < SIZE_THRESHOLD_SMALL - ? Math.max(sizeValue * SHADOW_MULTIPLIER_SMALL, SHADOW_MIN_SMALL) // Reduced shadow for small sizes - : Math.max(sizeValue * SHADOW_MULTIPLIER_LARGE, SHADOW_MIN_LARGE); - - // Adjust mask radius based on size to reduce black center in small sizes - const getMaskRadius = (value: number) => { - if (value < SIZE_THRESHOLD_TINY) { - return MASK_RADIUS_TINY; - } - if (value < SIZE_THRESHOLD_SMALL) { - return MASK_RADIUS_SMALL; - } - if (value < SIZE_THRESHOLD_MEDIUM) { - return MASK_RADIUS_MEDIUM; - } - return MASK_RADIUS_LARGE; - }; - - const maskRadius = getMaskRadius(sizeValue); - - // Use more subtle contrast for very small sizes - const getFinalContrast = (value: number) => { - if (value < SIZE_THRESHOLD_TINY) { - return CONTRAST_TINY; // Very subtle contrast for tiny sizes - } - if (value < SIZE_THRESHOLD_SMALL) { - return Math.max( - contrastAmount * CONTRAST_MULTIPLIER_FINAL, - CONTRAST_MIN_FINAL, - ); // Reduced contrast for small sizes - } - return contrastAmount; - }; - - const finalContrast = getFinalContrast(sizeValue); - - // Reactivity is gated by the state preset: `thinking` barely listens so the - // orb keeps churning internally instead of throbbing with room noise. - const reactivity = shouldReduceMotion ? 0 : stateMotion.reactivity; - - const reactiveBlur = useTransform(amplitudeValue, (level) => { - const focus = 1 - level * reactivity * AMPLITUDE_BLUR_FALLOFF; - return `${blurAmount * focus}px`; - }); - - const reactiveScale = useTransform( - amplitudeValue, - (level) => stateMotion.scale + level * reactivity * AMPLITUDE_SCALE_GAIN, - ); - - const loopDuration = shouldReduceMotion - ? animationDuration - : animationDuration / stateMotion.speed; - - // Rim thickness scales with the orb, so the lit edge reads the same at 24px - // and at 240px instead of swallowing the small one. - const rim = Math.max(sizeValue * RIM_RATIO, RIM_MIN); - // The sheen drifts slower than the mesh rotates; a busier state speeds it up - // a touch without matching the rotation, which would look mechanical. - const driftDuration = (DRIFT_BASE_SECONDS / (1 + stateMotion.speed)) * 2; - - const getRootAnimate = (): TargetAndTransition => { - if (shouldReduceMotion) { - return { scale: 1, x: 0 }; - } - if (state === "error") { - return { scale: 1, x: ERROR_SHAKE_KEYFRAMES }; - } - if (stateMotion.motif === "breathe") { - return { scale: BREATHE_SCALE, x: 0 }; - } - return { scale: 1, x: 0 }; - }; - - const getRootTransition = (): Transition => { - if (shouldReduceMotion) { - return { duration: 0 }; - } - if (state === "error") { - return { duration: ERROR_SHAKE_DURATION, ease: EASE_IN_OUT }; - } - if (stateMotion.motif === "breathe") { - return { - duration: BREATHE_SECONDS, - ease: EASE_IN_OUT, - repeat: Number.POSITIVE_INFINITY, - }; - } - return SPRING_DEFAULT; - }; - +}: AgentOrbProps) { + if (!/^\d+(\.\d+)?px$/.test(size)) { + throw new Error(`AgentOrb's size is in px, as "64px", not "${size}".`); + } return ( - // The gradient disc clips its own overflow, so the bloom and the status - // motif have to live in a wrapper rather than inside it. - - {/* Bloom tinted by the state's semantic accent. */} - - - - {/* Specular sheen and depth rim. The mesh alone reads as a flat blurred - disc; a drifting highlight plus a lit top edge and shaded bottom are - what make it read as a sphere. */} - - + ); -}; - -export default AgentOrb; +} diff --git a/app/src/components/agents/orb/ai-core.tsx b/app/src/components/agents/orb/ai-core.tsx deleted file mode 100644 index a1f13a6361..0000000000 --- a/app/src/components/agents/orb/ai-core.tsx +++ /dev/null @@ -1,395 +0,0 @@ -"use client"; - -import { - type MotionValue, - useAnimationFrame, - useMotionValue, -} from "motion/react"; -import { useCallback, useEffect, useRef, useState } from "react"; - -/** - * The single state contract shared by every SmoothUI AI component. - * - * Passing the same value to an orb, a prompt input and a tool card makes the - * whole surface move as one organism instead of a set of independent widgets. - */ -export type AIState = - | "idle" - | "listening" - | "thinking" - | "streaming" - | "done" - | "error"; - -/** - * A behavioural hint each component expresses **in its own material**. - * - * Deliberately not an overlay. Bolting one shared graphic onto every orb makes - * four different materials look like the same widget wearing a costume, and it - * pushes literal iconography (a checkmark, a warning ring) onto pieces that are - * decorative by nature. Instead: a shader warps, a canvas ring loses harmonics, - * a character changes expression. Same vocabulary, different flesh. - */ -export type AIStateMotif = - | "breathe" - | "receive" - | "scan" - | "pulse" - | "ping" - | "fault"; - -/** Semantic accent applied on top of the component's own palette. */ -export type AIStateAccent = "success" | "danger" | null; - -/** Motion parameters a component reads to render a given {@link AIState}. */ -export type AIStateMotion = { - /** Semantic colour override, so status is carried by hue and not only motion. */ - accent: AIStateAccent; - /** Outer bloom strength, 0–1. */ - glow: number; - /** Palette hue rotation in degrees. Small shifts read as a mood change. */ - hueRotate: number; - /** Overall motion energy, 0–1. Scales ambient loops and displacement. */ - intensity: number; - /** Behavioural hint; each component expresses it in its own material. */ - motif: AIStateMotif; - /** Seconds between discrete pulses, for rhythmic behaviour. */ - pulseSeconds: number; - /** - * How far a shader's domain warp pushes the field, 0–1. Low values read as a - * calm surface; high values churn without the silhouette growing. - */ - turbulence: number; - /** Revolutions per second of the noise field — the slow tumble. */ - tumble: number; - /** How much external amplitude reaches the surface, 0 = ignore it. */ - reactivity: number; - /** Chroma multiplier. Below 1 desaturates. */ - saturation: number; - /** Resting scale of the surface, 1 = no change. */ - scale: number; - /** Ambient loop speed multiplier, 1 = the component's base tempo. */ - speed: number; -}; - -/** - * Per-state presets. - * - * Deliberate choices worth keeping: - * - `thinking` has `scale: 1` — internal churn only, so layout stays calm while - * the model works. Growing the surface here makes pages feel unstable. - * - `error` desaturates instead of growing, so it reads as a state change - * rather than an attention grab. - * - `done` overshoots once; components are expected to settle back to `idle`. - * - Each state owns a distinct `motif`, so the six states are told apart by - * what moves, not by how fast it moves. - */ -export const AI_STATE_MOTION: Record = { - done: { - accent: "success", - glow: 0.7, - hueRotate: 0, - intensity: 0.4, - motif: "ping", - pulseSeconds: 0.65, - reactivity: 0, - saturation: 1, - scale: 1.1, - speed: 0.8, - // The field settles almost flat — stillness is what reads as "finished". - tumble: 0.02, - turbulence: 0.08, - }, - error: { - accent: "danger", - glow: 0.25, - hueRotate: 0, - intensity: 0.5, - motif: "fault", - pulseSeconds: 0.9, - reactivity: 0, - saturation: 0.3, - scale: 0.96, - speed: 1, - // Tumble stalls while turbulence stays mid: the surface twitches in place - // instead of flowing, which is what a fault feels like. - tumble: 0, - turbulence: 0.55, - }, - idle: { - accent: null, - glow: 0.15, - hueRotate: 0, - intensity: 0.3, - motif: "breathe", - pulseSeconds: 4.5, - reactivity: 0, - saturation: 0.75, - scale: 0.94, - speed: 0.6, - tumble: 0.012, - turbulence: 0.14, - }, - listening: { - accent: null, - glow: 0.6, - hueRotate: 0, - intensity: 0.75, - motif: "receive", - pulseSeconds: 1.6, - reactivity: 1, - saturation: 1.05, - scale: 1.06, - speed: 1, - tumble: 0.03, - turbulence: 0.42, - }, - streaming: { - accent: null, - glow: 0.45, - hueRotate: -10, - intensity: 0.6, - motif: "pulse", - pulseSeconds: 1.25, - reactivity: 0.6, - saturation: 1, - scale: 1.02, - speed: 1.4, - tumble: 0.06, - turbulence: 0.5, - }, - thinking: { - accent: null, - glow: 0.35, - hueRotate: 18, - intensity: 1, - motif: "scan", - pulseSeconds: 1.1, - reactivity: 0.15, - saturation: 1, - scale: 1, - speed: 2.4, - // High churn, unchanged silhouette: the work is visible without the orb - // growing and unsettling the layout around it. - tumble: 0.14, - turbulence: 0.95, - }, -}; - -/** Semantic accents. Deliberately not tokens — orbs render outside a theme. */ -export const AI_ACCENT_COLORS: Record<"success" | "danger", string> = { - danger: "oklch(63% 0.21 25)", - success: "oklch(72% 0.17 150)", -}; - -/** - * Colour used by a state's overlay motif: the semantic accent when the state - * has one, otherwise the component's own secondary colour. - */ -export const getAIStateAccentColor = ( - state: AIState | undefined, - fallback: string, -): string => { - const accent = AI_STATE_MOTION[state ?? "idle"]?.accent; - return accent ? AI_ACCENT_COLORS[accent] : fallback; -}; - -/** Motion preset for a state, falling back to `idle` for unknown values. */ -export const getAIStateMotion = (state: AIState | undefined): AIStateMotion => - AI_STATE_MOTION[state ?? "idle"] ?? AI_STATE_MOTION.idle; - -/** - * Amplitude accepted by every reactive AI component. - * - * A `MotionValue` is the preferred form: it updates outside React, so a 60fps - * audio signal never triggers a re-render. - */ -export type AIAmplitude = number | MotionValue | undefined; - -const isMotionValue = (value: AIAmplitude): value is MotionValue => - typeof value === "object" && value !== null && "get" in value; - -/** - * Normalises the `amplitude` prop into a stable `MotionValue` so - * component internals only deal with one shape. - */ -export const useAmplitudeValue = ( - amplitude: AIAmplitude, -): MotionValue => { - const fallback = useMotionValue(0); - const numeric = typeof amplitude === "number" ? amplitude : null; - - useEffect(() => { - if (numeric !== null) { - fallback.set(numeric); - } - }, [numeric, fallback]); - - return isMotionValue(amplitude) ? amplitude : fallback; -}; - -export type AudioAmplitudeStatus = - | "idle" - | "requesting" - | "active" - | "denied" - | "unsupported"; - -export type UseAudioAmplitudeOptions = { - /** Request microphone access as soon as the hook mounts. */ - autoStart?: boolean; - /** - * Envelope smoothing, 0–1. Higher is smoother and lazier; the default keeps - * attack snappy so an orb reacts on the first syllable. - */ - smoothing?: number; - /** FFT size handed to the analyser node. Must be a power of two. */ - fftSize?: number; -}; - -export type UseAudioAmplitudeResult = { - /** Smoothed RMS level, 0–1, as a `MotionValue`. */ - amplitude: MotionValue; - status: AudioAmplitudeStatus; - start: () => Promise; - stop: () => void; -}; - -const DEFAULT_SMOOTHING = 0.55; -const DEFAULT_FFT_SIZE = 512; -/** Raw RMS rarely exceeds ~0.3 for speech, so normalise into a usable 0–1. */ -const RMS_TO_UNIT = 3.2; -/** Attack is faster than release so peaks land immediately and decay gently. */ -const ATTACK_FACTOR = 0.35; - -/** - * Reads microphone loudness as a 0–1 `MotionValue`. - * - * SSR-safe, permission-aware, and silent on failure — a denied prompt leaves - * the amplitude at 0 so the consuming component simply falls back to its - * ambient animation. - */ -export const useAudioAmplitude = ( - options: UseAudioAmplitudeOptions = {}, -): UseAudioAmplitudeResult => { - const { - autoStart = false, - smoothing = DEFAULT_SMOOTHING, - fftSize = DEFAULT_FFT_SIZE, - } = options; - - const amplitude = useMotionValue(0); - const [status, setStatus] = useState("idle"); - - const contextRef = useRef(null); - const streamRef = useRef(null); - const analyserRef = useRef(null); - const bufferRef = useRef | null>(null); - - const stop = useCallback(() => { - for (const track of streamRef.current?.getTracks() ?? []) { - track.stop(); - } - streamRef.current = null; - analyserRef.current = null; - bufferRef.current = null; - contextRef.current?.close(); - contextRef.current = null; - amplitude.set(0); - setStatus("idle"); - }, [amplitude]); - - const start = useCallback(async () => { - if (analyserRef.current) { - return; - } - - const AudioContextCtor = - typeof window === "undefined" - ? undefined - : (window.AudioContext ?? - (window as unknown as { webkitAudioContext?: typeof AudioContext }) - .webkitAudioContext); - - if (!(AudioContextCtor && navigator.mediaDevices?.getUserMedia)) { - setStatus("unsupported"); - return; - } - - setStatus("requesting"); - - try { - const stream = await navigator.mediaDevices.getUserMedia({ audio: true }); - const context = new AudioContextCtor(); - const analyser = context.createAnalyser(); - analyser.fftSize = fftSize; - context.createMediaStreamSource(stream).connect(analyser); - - streamRef.current = stream; - contextRef.current = context; - analyserRef.current = analyser; - bufferRef.current = new Float32Array(analyser.fftSize); - setStatus("active"); - } catch { - setStatus("denied"); - } - }, [fftSize]); - - useEffect(() => { - if (autoStart) { - start(); - } - return stop; - }, [autoStart, start, stop]); - - useAnimationFrame(() => { - const analyser = analyserRef.current; - const buffer = bufferRef.current; - if (!(analyser && buffer)) { - return; - } - - analyser.getFloatTimeDomainData(buffer); - - let sumOfSquares = 0; - for (const sample of buffer) { - sumOfSquares += sample * sample; - } - const rms = Math.sqrt(sumOfSquares / buffer.length); - const target = Math.min(1, rms * RMS_TO_UNIT); - - const previous = amplitude.get(); - const factor = target > previous ? smoothing * ATTACK_FACTOR : smoothing; - amplitude.set(previous + (target - previous) * (1 - factor)); - }); - - return { amplitude, start, status, stop }; -}; - -/** - * Amplitude generator for demos, docs and previews — no microphone involved. - * - * Produces a plausible speech-like envelope whose energy follows the current - * {@link AIState}, so every example can show the reactive behaviour without - * asking the visitor for permissions. - */ -export const useSimulatedAmplitude = ( - state: AIState = "idle", -): MotionValue => { - const amplitude = useMotionValue(0); - const motion = getAIStateMotion(state); - - useAnimationFrame((time) => { - const t = time / 1000; - // Three detuned sines read as organic; a single sine reads as a metronome. - const envelope = - 0.5 + - 0.3 * Math.sin(t * 2.1 * motion.speed) + - 0.14 * Math.sin(t * 5.3 * motion.speed + 1.7) + - 0.06 * Math.sin(t * 11.7 * motion.speed + 0.4); - - amplitude.set(Math.min(1, Math.max(0, envelope * motion.intensity))); - }); - - return amplitude; -}; diff --git a/app/src/components/agents/orb/bot-orb-draw.ts b/app/src/components/agents/orb/bot-orb-draw.ts new file mode 100644 index 0000000000..f3e3f2b370 --- /dev/null +++ b/app/src/components/agents/orb/bot-orb-draw.ts @@ -0,0 +1,782 @@ +/** + * The orb, every Bot's face and OpenBot's own mark, drawn on a 2D canvas: a glass sphere of + * drifting colour with two eye-lights that look, blink and smile. It comes in nine colours. A Bot's + * look, its colour and the few small ways two Bots in one colour still differ, is picked from its + * seed; OpenBot's own is violet, which no Bot is given. It also plays a finish: the check and the + * burst of rays the orb shows when something the person allowed is done. + * + * Every frame is a pure function of an `OrbPose`, so the component that animates it owns time and + * this file owns only the picture. + */ + +const TAU = Math.PI * 2; +const PI = Math.PI; + +export const clamp = (x: number, a = 0, b = 1) => (x < a ? a : x > b ? b : x); +const lerp = (a: number, b: number, u: number) => a + (b - a) * u; +const eio = (u: number) => { + const v = clamp(u); + return v < 0.5 ? 4 * v * v * v : 1 - (-2 * v + 2) ** 3 / 2; +}; +const eout = (u: number) => 1 - (1 - clamp(u)) ** 3; +const smooth = (u: number) => { + const v = clamp(u); + return v * v * (3 - 2 * v); +}; +const bump = (u: number) => (u <= 0 || u >= 1 ? 0 : Math.sin(PI * u) ** 2); + +// ---------- motion: keyed tracks with eases and closed-form springs ---------- + +/** One move of a track: to `to` from time `at`, over an ease of `ease` seconds or on a spring of [stiffness, damping]. */ +export type TrackKey = + | { at: number; to: number; ease: number } + | { at: number; to: number; spring: readonly [number, number] }; + +/** A value that starts at `from` and moves through its keys in order. */ +export type Track = { from: number; keys: TrackKey[] }; + +/** Step response of a damped spring: closed form, so it is a pure function of elapsed time. */ +function springStep(tau: number, w: number, z: number) { + if (tau <= 0) return 0; + if (z >= 1) return 1 - Math.exp(-w * tau) * (1 + w * tau); + const wd = w * Math.sqrt(1 - z * z); + return ( + 1 - + Math.exp(-z * w * tau) * + (Math.cos(wd * tau) + ((z * w) / wd) * Math.sin(wd * tau)) + ); +} + +/** + * The track's value at time t. Each key moves the value from the previous key's value to its own, + * and the responses are summed, which for springs is exactly a spring chasing a stepped target: + * anticipation, overshoot and settle all fall out of the same maths. + */ +export function track(tr: Track, t: number) { + let v = tr.from; + let prev = tr.from; + for (const k of tr.keys) { + if (t < k.at) break; + const tau = t - k.at; + const r = + "ease" in k + ? eio(tau / k.ease) + : springStep(tau, k.spring[0], k.spring[1]); + v += (k.to - prev) * r; + prev = k.to; + } + return v; +} + +/** A blink at `tau` seconds after it starts: shut fast, open slower, 0.24 s in all. */ +export function blinkCurve(tau: number) { + if (tau < 0 || tau > 0.24) return 0; + return tau < 0.07 ? eout(tau / 0.07) : 1 - eio((tau - 0.07) / 0.17); +} + +/** A nod that rings down, `tau` seconds after it starts. */ +export function nodCurve(tau: number) { + return tau >= 0 && tau < 2.2 ? Math.exp(-3 * tau) * Math.sin(8 * tau) : 0; +} + +export function mulberry32(seed: number) { + let a = seed; + return () => { + a |= 0; + a = (a + 0x6d2b79f5) | 0; + let t = Math.imul(a ^ (a >>> 15), 1 | a); + t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t; + return ((t ^ (t >>> 14)) >>> 0) / 4294967296; + }; +} + +/** + * Where it looks while nothing needs it: mostly at you, now and then a glance aside and back. + * Each look snaps over 0.14 s and then holds, so the pauses are part of the motion. + */ +export function restLook(ta: number, period: number, seed: number) { + const k = Math.floor(ta / period); + const f = ta - k * period; + const at = (n: number): [number, number, number] => { + const r = mulberry32((seed * 7919 + n * 104729) | 0); + return r() < 0.55 + ? [0, 0, 0] + : [(r() * 2 - 1) * 0.7, (r() * 2 - 1) * 0.4, (r() * 2 - 1) * 0.24]; + }; + const a = at(k - 1); + const b = at(k); + const u = eout(f / 0.14); + return [lerp(a[0], b[0], u), lerp(a[1], b[1], u), lerp(a[2], b[2], u)]; +} + +// ---------- colour ---------- + +export type Rgb = [number, number, number]; + +function ok2rgb(L: number, C: number, h: number): Rgb | null { + const hr = (h * PI) / 180; + const a = C * Math.cos(hr); + const b = C * Math.sin(hr); + const l = (L + 0.3963377774 * a + 0.2158037573 * b) ** 3; + const m = (L - 0.1055613458 * a - 0.0638541728 * b) ** 3; + const s = (L - 0.0894841775 * a - 1.291485548 * b) ** 3; + const linear = [ + 4.0767416621 * l - 3.3077115913 * m + 0.2309699292 * s, + -1.2684380046 * l + 2.6097574011 * m - 0.3413193965 * s, + -0.0041960863 * l - 0.7034186147 * m + 1.707614701 * s, + ]; + const out: number[] = []; + for (const x of linear) { + if (x < -0.002 || x > 1.002) return null; + const c = clamp(x); + out.push( + Math.round( + 255 * (c <= 0.0031308 ? 12.92 * c : 1.055 * c ** (1 / 2.4) - 0.055), + ), + ); + } + return out as Rgb; +} + +/** An oklch colour as sRGB, losing chroma until it fits rather than clipping its hue. */ +export function oklch(L: number, C: number, h: number): Rgb { + let chroma = C; + for (let i = 0; i < 40; i++) { + const c = ok2rgb(L, chroma, h); + if (c) return c; + chroma *= 0.95; + } + return ok2rgb(L, 0, h) ?? [128, 128, 128]; +} + +const rgba = (c: Rgb, a: number) => + `rgba(${c[0]},${c[1]},${c[2]},${a <= 0 ? 0 : a >= 1 ? 1 : a.toFixed(3)})`; +const mix = (a: Rgb, b: Rgb, u: number): Rgb => [ + Math.round(lerp(a[0], b[0], u)), + Math.round(lerp(a[1], b[1], u)), + Math.round(lerp(a[2], b[2], u)), +]; +const WHITE: Rgb = [255, 255, 255]; + +type Oklch = readonly [L: number, C: number, h: number]; + +/** + * One of the orb's colours, as oklch [L, C, h]: four tones for the sphere's drifting colour and its + * pale base. `main` also tints its glow and, in the dark, the landing's ping and a finish's rays; + * `deep` draws those in the light. `shade` is the tint of the sphere's shaded rim, as sRGB. + */ +type PaletteSpec = { + main: Oklch; + side: Oklch; + deep: Oklch; + pale: Oklch; + base: Oklch; + shade: readonly [r: number, g: number, b: number]; +}; + +/** + * The nine colours: violet, OpenBot's own, and the eight a Bot's look is picked from by position. + * Adding, removing or reordering one changes the colour of most Bots. + */ +const PALETTES = { + violet: { + main: [0.7, 0.14, 288], + side: [0.72, 0.15, 318], + deep: [0.47, 0.2, 296], + pale: [0.86, 0.08, 308], + base: [0.93, 0.03, 300], + shade: [40, 16, 78], + }, + ember: { + main: [0.79, 0.13, 62], + side: [0.75, 0.15, 35], + deep: [0.53, 0.17, 30], + pale: [0.89, 0.08, 52], + base: [0.94, 0.03, 50], + shade: [84, 30, 14], + }, + moss: { + main: [0.77, 0.11, 182], + side: [0.78, 0.13, 152], + deep: [0.5, 0.12, 165], + pale: [0.89, 0.07, 160], + base: [0.94, 0.03, 160], + shade: [14, 60, 40], + }, + sun: { + main: [0.81, 0.15, 72], + side: [0.85, 0.14, 92], + deep: [0.57, 0.14, 58], + pale: [0.93, 0.08, 92], + base: [0.95, 0.03, 90], + shade: [92, 58, 10], + }, + berry: { + main: [0.7, 0.15, 328], + side: [0.72, 0.16, 352], + deep: [0.48, 0.18, 345], + pale: [0.86, 0.08, 350], + base: [0.93, 0.03, 350], + shade: [80, 14, 48], + }, + ocean: { + main: [0.74, 0.13, 205], + side: [0.75, 0.13, 232], + deep: [0.53, 0.2, 262], + pale: [0.86, 0.08, 220], + base: [0.93, 0.025, 235], + shade: [14, 30, 80], + }, + // Beside ocean on the wheel, so lighter and softer: a clear sky against ocean's deep water. + sky: { + main: [0.79, 0.11, 220], + side: [0.8, 0.1, 242], + deep: [0.55, 0.15, 245], + pale: [0.9, 0.06, 228], + base: [0.95, 0.02, 230], + shade: [10, 40, 76], + }, + coral: { + main: [0.78, 0.13, 32], + side: [0.74, 0.15, 8], + deep: [0.52, 0.17, 18], + pale: [0.88, 0.08, 22], + base: [0.94, 0.03, 20], + shade: [84, 20, 24], + }, + // Nearly grey: for a team big enough to want a quiet one. + slate: { + main: [0.72, 0.045, 232], + side: [0.76, 0.04, 250], + deep: [0.48, 0.05, 265], + pale: [0.88, 0.02, 255], + base: [0.93, 0.012, 258], + shade: [30, 38, 54], + }, +} as const satisfies Record; + +export type OrbPaletteName = keyof typeof PALETTES; + +/** Every colour, OpenBot's violet first. */ +export const ORB_PALETTES = Object.keys(PALETTES) as readonly OrbPaletteName[]; + +/** The eight colours a Bot can be: all but violet, so a Bot is never mistaken for OpenBot. */ +const BOT_PALETTES = ORB_PALETTES.filter((name) => name !== "violet"); + +/** + * How one orb is drawn apart from its motion: its colour, how far round its colour masses are + * turned, in radians, a nudge to its hue in degrees, and how far apart its eyes are, as a share of + * the usual spacing. + */ +export type OrbLook = { + palette: OrbPaletteName; + turn: number; + hue: number; + eyes: number; +}; + +/** OpenBot's own look: violet, untouched. */ +export const OPENBOT_LOOK: OrbLook = { + palette: "violet", + turn: 0, + hue: 0, + eyes: 1, +}; + +/** The most a Bot's hue is nudged either way, in degrees: a touch, never enough to pass for another colour. */ +const HUE_NUDGE = 4; +/** The most a Bot's eyes are moved apart or together, as a share of the usual spacing. */ +const EYE_NUDGE = 0.08; + +/** A 32-bit FNV-1a hash of the seed, mixed with murmur3's finaliser so every character moves every bit. */ +function hashSeed(seed: string): number { + let h = 0x811c9dc5; + for (let i = 0; i < seed.length; i++) { + h ^= seed.charCodeAt(i); + h = Math.imul(h, 0x01000193); + } + h ^= h >>> 16; + h = Math.imul(h, 0x85ebca6b); + h ^= h >>> 13; + h = Math.imul(h, 0xc2b2ae35); + h ^= h >>> 16; + return h >>> 0; +} + +/** + * The look for a Bot's seed. Eight colours repeat soon in a team, so the same hash also turns the + * colour masses, nudges the hue and spaces the eyes, and two Bots in one colour still look apart. + * Stable across reloads and releases, so a Bot keeps its face. + */ +export function orbLookFor(seed: string): OrbLook { + const h = hashSeed(seed); + // The eight colours take the low three bits, and each of the rest the next eight. + const share = (from: number) => ((h >>> from) & 0xff) / 256; + return { + palette: BOT_PALETTES[h % BOT_PALETTES.length] as OrbPaletteName, + turn: share(3) * TAU, + hue: Math.round((share(11) * 2 - 1) * HUE_NUDGE), + eyes: 1 + (share(19) * 2 - 1) * EYE_NUDGE, + }; +} + +/** A colour's tones as sRGB, ready to draw. */ +export type Palette = Record; +const paletteMemo = new Map(); + +/** A colour with its saturation and hue nudged, as the orb livens when it listens and cools when it works. */ +export function palette( + name: OrbPaletteName, + sat: number, + hue: number, +): Palette { + const ks = Math.round(clamp(sat, 0.5, 1.3) * 50) / 50; + const kh = Math.round(hue * 2) / 2; + const key = `${name}|${ks}|${kh}`; + let p = paletteMemo.get(key); + if (!p) { + const spec: PaletteSpec = PALETTES[name]; + const c = ([L, C, h]: Oklch, sk = ks, hk = kh) => oklch(L, C * sk, h + hk); + p = { + main: c(spec.main), + side: c(spec.side), + deep: c(spec.deep), + pale: c(spec.pale), + base: c(spec.base, 1, kh * 0.5), + shade: [...spec.shade], + }; + paletteMemo.set(key, p); + } + return p; +} + +// ---------- the pose ---------- + +/** Everything one frame depends on. `ta` is seconds on the orb's own clock; ages are -1 when nothing happened yet. */ +export type OrbPose = { + ta: number; + /** 0 at rest, 1 listening (can overshoot a little). */ + listen: number; + think: number; + /** The landing's glow, 0 to 1. */ + done: number; + smile: number; + tilt: number; + /** Where it looks: right and down, about -1 to 1. */ + ax: number; + ay: number; + blink: number; + nod: number; + doneAge: number; + noticeAge: number; + /** Seconds into a finish (a check and a burst of rays, `finishPhase`), or -1 when none is playing. */ + finishAge: number; + /** -1 to 1, a slow breath. */ + breath: number; +}; + +// ---------- the finish ---------- + +/** How long a finish plays, in seconds, from the eyes giving way to the check until they are back. */ +export const FINISH_S = 1.2; +/** A moment in a finish when the check is whole and the rays have gone: the finish as a still. */ +export const FINISH_HELD = 0.8; +/** The short strokes that burst out round the orb when it finishes. */ +export const RAYS = 10; + +/** Where a finish is at one moment. Every part of it is a pure function of its age. */ +export type FinishPhase = { + /** How much of the check is drawn along its stroke, 0 to 1. */ + check: number; + /** The check's opacity: whole while it holds, gone as the eyes come back. */ + checkAlpha: number; + /** How far the rays have flown, 0 to 1, or -1 before they leave and once they have faded. */ + rays: number; + /** The eye-lights' opacity: they give way to the check and come back after it. */ + eyes: number; + /** The sphere's pop as the check lands, 0 to 1. */ + pop: number; +}; + +const NO_FINISH: FinishPhase = { + check: 0, + checkAlpha: 0, + rays: -1, + eyes: 1, + pop: 0, +}; + +/** + * A finish `age` seconds in: the eyes fade as the sphere pops and the check strokes on, the rays + * burst outward and fade, the check holds, then fades as the eyes come back. -1 is no finish. + */ +export function finishPhase(age: number): FinishPhase { + if (!(age >= 0 && age < FINISH_S)) return NO_FINISH; + const back = smooth((age - 0.85) / 0.3); + return { + check: eout((age - 0.06) / 0.3), + checkAlpha: 1 - back, + rays: age >= 0.1 && age < 0.75 ? (age - 0.1) / 0.65 : -1, + eyes: age < 0.85 ? 1 - smooth(age / 0.12) : back, + pop: bump(age / 0.5), + }; +} + +// ---------- the orb's pieces ---------- + +type Ctx = CanvasRenderingContext2D; + +function blob(ctx: Ctx, x: number, y: number, r: number, c: Rgb, a: number) { + if (a <= 0.003 || r <= 0) return; + const g = ctx.createRadialGradient(x, y, 0, x, y, r); + g.addColorStop(0, rgba(c, a)); + g.addColorStop(0.42, rgba(c, a * 0.78)); + g.addColorStop(0.76, rgba(c, a * 0.24)); + g.addColorStop(1, rgba(c, 0)); + ctx.fillStyle = g; + ctx.fillRect(x - r, y - r, 2 * r, 2 * r); +} + +function light(ctx: Ctx, x: number, y: number, r: number, a: number) { + if (a <= 0.003 || r <= 0) return; + const g = ctx.createRadialGradient(x, y, 0, x, y, r); + g.addColorStop(0, `rgba(255,255,255,${clamp(a).toFixed(3)})`); + g.addColorStop(0.45, `rgba(255,255,255,${clamp(a * 0.42).toFixed(3)})`); + g.addColorStop(1, "rgba(255,255,255,0)"); + ctx.fillStyle = g; + ctx.fillRect(x - r, y - r, 2 * r, 2 * r); +} + +function circle(ctx: Ctx, r: number) { + ctx.beginPath(); + ctx.arc(0, 0, r, 0, TAU); +} + +// angle, orbit, radius of the mesh's four colour masses +const MESH = [ + [3.95, 0.43, 0.88], + [5.5, 0.45, 0.74], + [0.78, 0.45, 0.86], + [2.36, 0.45, 0.88], +] as const; + +function meshPos(ta: number, fast: boolean, turn: number) { + const rot = turn + (fast ? 1.45 * ta + 1.1 : 0.22 * ta); + return MESH.map((b, i) => { + const a = + b[0] + + rot + + (fast + ? 0.5 * Math.sin(1.1 * ta + i * 1.93) + : 0.42 * Math.sin(0.29 * ta + i * 1.93)); + return [Math.cos(a) * b[1], Math.sin(a) * b[1], b[2]] as const; + }); +} + +/** + * Four soft colour masses drifting round a pale base, starting `turn` radians round; a faster swirl + * joins them while it works. + */ +function mesh( + ctx: Ctx, + R: number, + S: OrbPose, + pal: Palette, + turn: number, + lx: number, + ly: number, +) { + ctx.fillStyle = rgba(pal.base, 1); + ctx.fillRect(-R * 1.7, -R * 1.7, R * 3.4, R * 3.4); + const cols = [pal.main, pal.pale, pal.side, pal.deep]; + meshPos(S.ta, false, turn).forEach((p, i) => { + blob(ctx, p[0] * R + lx * R, p[1] * R + ly * R, p[2] * R, cols[i], 0.94); + }); + const swirl = S.think * 0.55; + if (swirl > 0.01) { + meshPos(S.ta, true, turn).forEach((p, i) => { + blob( + ctx, + p[0] * R + lx * R * 0.6, + p[1] * R + ly * R * 0.6, + p[2] * R * 0.92, + cols[i], + 0.9 * swirl, + ); + }); + } +} + +/** Lit top edge, shaded bottom: what makes the disc read as a sphere. Call inside the clip. */ +function rim(ctx: Ctx, R: number, shade: Rgb, dark: boolean) { + const g = ctx.createRadialGradient( + 0, + -R * 0.3, + R * 0.55, + 0, + -R * 0.12, + R * 1.16, + ); + g.addColorStop(0, rgba(shade, 0)); + g.addColorStop(0.72, rgba(shade, 0)); + g.addColorStop(1, rgba(shade, dark ? 0.3 : 0.24)); + ctx.fillStyle = g; + ctx.fillRect(-R * 1.3, -R * 1.3, R * 2.6, R * 2.6); + const lw = Math.max(1, R * 0.05); + const lg = ctx.createLinearGradient(0, -R, 0, R * 0.25); + lg.addColorStop(0, "rgba(255,255,255,.7)"); + lg.addColorStop(1, "rgba(255,255,255,0)"); + ctx.strokeStyle = lg; + ctx.lineWidth = lw; + circle(ctx, R - lw * 0.5); + ctx.stroke(); +} + +function glow(ctx: Ctx, R: number, amt: number, c: Rgb, dark: boolean) { + if (amt < 0.005) return; + const r = R * 1.85; + const g = ctx.createRadialGradient(0, 0, R * 0.55, 0, 0, r); + g.addColorStop(0, rgba(c, amt * (dark ? 0.8 : 0.5))); + g.addColorStop(0.38, rgba(c, amt * (dark ? 0.34 : 0.18))); + g.addColorStop(1, rgba(c, 0)); + ctx.fillStyle = g; + ctx.fillRect(-r, -r, 2 * r, 2 * r); +} + +/** How long the landing's ping rings, in seconds. */ +const PING_S = 1; + +/** One ring that leaves the orb and fades: the landing's ping. */ +function ping(ctx: Ctx, r0: number, age: number, pal: Palette, dark: boolean) { + if (!(age >= 0 && age < PING_S)) return; + const u = age / PING_S; + const r = r0 * (1.02 + (1.5 - 1.02) * eout(u)); + const a = (1 - u) ** 1.5; + ctx.strokeStyle = rgba( + dark ? mix(pal.main, WHITE, 0.2) : pal.deep, + a * (dark ? 0.75 : 0.62), + ); + ctx.lineWidth = Math.max(1, r0 * 0.07 * (1 - u * 0.6)); + circle(ctx, r); + ctx.stroke(); +} + +/** + * The finish's burst: a ring of short strokes, one straight up, that fly out round the sphere, + * shortening and fading as they go. Every other one is longer, so the ring sparkles rather than ticks. + */ +function rays(ctx: Ctx, r0: number, u: number, pal: Palette, dark: boolean) { + if (!(u >= 0 && u < 1)) return; + const travel = eout(u); + const a = (1 - u) ** 1.4; + ctx.save(); + ctx.lineCap = "round"; + ctx.strokeStyle = rgba( + dark ? mix(pal.main, WHITE, 0.25) : pal.deep, + a * (dark ? 0.9 : 0.8), + ); + ctx.lineWidth = Math.max(1, r0 * 0.09 * (1 - 0.5 * u)); + for (let i = 0; i < RAYS; i++) { + const angle = -PI / 2 + (TAU * i) / RAYS; + const length = r0 * (i % 2 === 0 ? 0.3 : 0.2) * (1 - 0.55 * u); + const inner = r0 * (1.3 + 0.32 * travel); + const dx = Math.cos(angle); + const dy = Math.sin(angle); + ctx.beginPath(); + ctx.moveTo(dx * inner, dy * inner); + ctx.lineTo(dx * (inner + length), dy * (inner + length)); + ctx.stroke(); + } + ctx.restore(); +} + +// The check's three points, as shares of the sphere's radius: in at the left, down to the elbow, up +// to the right. Its two legs are about 0.37 and 0.75 of the radius long. +const CHECK = [ + [-0.38, 0.02], + [-0.12, 0.28], + [0.42, -0.26], +] as const; + +/** The check, stroked on along its length to `drawn` (0 to 1), in the eye-lights' white. */ +function check( + ctx: Ctx, + R: number, + drawn: number, + alpha: number, + dark: boolean, +) { + if (drawn <= 0 || alpha <= 0.003) return; + const [a, b, c] = CHECK.map(([x, y]) => [x * R, y * R] as const); + const first = Math.hypot(b[0] - a[0], b[1] - a[1]); + const second = Math.hypot(c[0] - b[0], c[1] - b[1]); + const along = drawn * (first + second); + ctx.save(); + ctx.globalAlpha = clamp(alpha); + ctx.lineCap = "round"; + ctx.lineJoin = "round"; + ctx.lineWidth = Math.max(1.6, R * 0.17); + ctx.strokeStyle = "rgba(255,255,255,.97)"; + ctx.shadowColor = dark ? "rgba(8,6,20,.4)" : "rgba(20,16,33,.28)"; + ctx.shadowBlur = Math.max(1, R * 0.07); + ctx.beginPath(); + ctx.moveTo(a[0], a[1]); + if (along <= first) { + const u = along / first; + ctx.lineTo(lerp(a[0], b[0], u), lerp(a[1], b[1], u)); + } else { + const u = (along - first) / second; + ctx.lineTo(b[0], b[1]); + ctx.lineTo(lerp(b[0], c[0], u), lerp(b[1], c[1], u)); + } + ctx.stroke(); + ctx.restore(); +} + +/** The sphere's scale: it breathes, swells a little to listen and to land, and pops as a check lands. */ +function scaleOf(S: OrbPose, finish: FinishPhase) { + const L = clamp(S.listen, 0, 1.25); + return ( + 1 + + 0.015 * S.breath * (1 - S.think) + + 0.025 * L + + 0.04 * S.done + + 0.06 * finish.pop + ); +} + +function eyePath(ctx: Ctx, w: number, h: number, smile: number) { + const rx = (w / 2) * (1 + 0.12 * smile); + const ry = h / 2; + const e = 0.32 * ry * smile; + const top = -ry; + const bot = lerp(ry, -0.36 * ry, smile); + const ct = (top - 0.25 * e) / 0.75; + const cb = (bot - 0.25 * e) / 0.75; + ctx.beginPath(); + ctx.moveTo(-rx, e); + ctx.bezierCurveTo(-rx, ct, rx, ct, rx, e); + ctx.bezierCurveTo(rx, cb, -rx, cb, -rx, e); + ctx.closePath(); +} + +/** How long an orb stays perked up after the pointer comes close, in seconds. */ +const PERK_S = 1.6; + +/** The sphere: the mesh turned toward where it looks, its rim, and two eye-lights that blink and smile. */ +function sphere( + ctx: Ctx, + R: number, + S: OrbPose, + pal: Palette, + look: OrbLook, + dark: boolean, + size: number, +) { + const na = S.noticeAge; + const perk = + na >= 0 && na < PERK_S ? Math.exp(-2.6 * na) * smooth(na / 0.1) : 0; + const finish = finishPhase(S.finishAge); + const sc = scaleOf(S, finish); + ctx.save(); + ctx.translate(0, S.nod * R * 0.11); + ctx.scale(sc, sc); + circle(ctx, R); + ctx.clip(); + const yaw = clamp(S.ax, -1.1, 1.1) * 0.62; + const pitch = -clamp(S.ay, -1.1, 1.1) * 0.5; + const roll = S.tilt; + mesh(ctx, R, S, pal, look.turn, Math.sin(yaw) * 0.2, -Math.sin(pitch) * 0.16); + light(ctx, -0.42 * R, -0.5 * R, 0.5 * R, 0.26); + light(ctx, 0, R * 0.05, R * 0.9, 0.28 * S.done); + rim(ctx, R, pal.shade, dark); + const small = size < 40; + const widen = 1 + 0.16 * perk; + const open = clamp(1 - 0.94 * S.blink) * (1 - 0.3 * S.think) * widen; + const cr = Math.cos(roll); + const sr = Math.sin(roll); + const ew = Math.max(small ? 2.4 : 1.8, R * 0.16 * widen); + const eh = Math.max(small ? 3.4 : 2.4, R * 0.3); + for (const side of [-1, 1]) { + if (finish.eyes <= 0.003) break; + const lon = side * 0.35 * look.eyes; + const lat = 0.05 + 0.04 * S.smile; + const x = Math.sin(lon) * Math.cos(lat); + const y = Math.sin(lat); + const z = Math.cos(lon) * Math.cos(lat); + const y1 = y * Math.cos(pitch) + z * Math.sin(pitch); + const z1 = -y * Math.sin(pitch) + z * Math.cos(pitch); + const x2 = x * Math.cos(yaw) + z1 * Math.sin(yaw); + const sx = x2 * cr - y1 * sr; + const sy = x2 * sr + y1 * cr; + const fx = Math.sqrt(clamp(1 - sx * sx, 0.12, 1)); + const fy = Math.sqrt(clamp(1 - sy * sy, 0.2, 1)); + const h = Math.max(small ? 1.1 : 0.9, eh * open * fy); + ctx.save(); + ctx.globalAlpha = finish.eyes; + ctx.translate(sx * R * 0.84, -sy * R * 0.84 + (eh - h) * 0.28); + ctx.rotate(-roll); + ctx.shadowColor = dark ? "rgba(8,6,20,.4)" : "rgba(20,16,33,.28)"; + ctx.shadowBlur = Math.max(1, R * 0.07); + ctx.fillStyle = "rgba(255,255,255,.97)"; + eyePath(ctx, ew * fx, h, clamp(S.smile) * (1 - S.blink)); + ctx.fill(); + ctx.restore(); + } + check(ctx, R, finish.check, finish.checkAlpha, dark); + ctx.restore(); +} + +/** + * The whole orb with `look`, centred on the context's origin, for a footprint of `size` CSS px. + * Its glow, the landing's ping and a finish's rays spill past the footprint, so draw it on a canvas + * twice its size. + */ +export function drawBotOrb( + ctx: Ctx, + size: number, + S: OrbPose, + { dark, look }: { dark: boolean; look: OrbLook }, +) { + const finish = finishPhase(S.finishAge); + const pal = palette( + look.palette, + 0.9 + 0.14 * clamp(S.listen) + 0.08 * S.think + 0.12 * clamp(S.done), + look.hue - 6 * S.think, + ); + const r = (size / 2) * 0.72; + const sc = scaleOf(S, finish); + const glowAmt = clamp( + 0.18 + + 0.42 * clamp(S.listen) + + 0.22 * S.think + + 0.55 * S.done + + 0.45 * finish.pop, + ); + glow(ctx, r * sc, glowAmt * 0.85, mix(pal.main, pal.deep, 0.4), dark); + ping(ctx, r, S.doneAge, pal, dark); + rays(ctx, r, finish.rays, pal, dark); + sphere(ctx, r, S, pal, look, dark, size * 0.72); +} + +/** How long each age in a pose shows, in seconds: after that the orb draws as if it never happened. */ +const SHOWS_FOR: Partial> = { + doneAge: PING_S, + noticeAge: PERK_S, + finishAge: FINISH_S, +}; + +/** + * Whether two poses draw the same picture, every value within a thousandth of the other's. An age + * counts only while what it times still shows, so an orb that has settled poses the same picture + * frame after frame and need not draw it again. + */ +export function samePicture(a: OrbPose, b: OrbPose): boolean { + for (const key of Object.keys(a) as (keyof OrbPose)[]) { + const shows = SHOWS_FOR[key]; + const seen = (v: number) => + shows === undefined || (v >= 0 && v < shows) ? v : -1; + if (Math.abs(seen(a[key]) - seen(b[key])) >= 1e-3) return false; + } + return true; +} diff --git a/app/src/components/agents/orb/bot-orb.tsx b/app/src/components/agents/orb/bot-orb.tsx new file mode 100644 index 0000000000..d628b9cd45 --- /dev/null +++ b/app/src/components/agents/orb/bot-orb.tsx @@ -0,0 +1,542 @@ +import { useEffect, useMemo, useRef, useSyncExternalStore } from "react"; +import { cn } from "@/lib/utils"; +import { + blinkCurve, + clamp, + drawBotOrb, + FINISH_HELD, + FINISH_S, + mulberry32, + nodCurve, + OPENBOT_LOOK, + type OrbLook, + type OrbPose, + orbLookFor, + restLook, + samePicture, + type Track, + type TrackKey, + track, +} from "./bot-orb-draw"; +import { pointer, subscribeFrame, subscribePointer } from "./orb-clock"; + +/** + * What the orb is doing: resting, listening to you type, working, or landing a finished thing. + * `finished` is a moment rather than a state: something the person allowed is done, and the orb + * plays a check with a burst of rays once, over whatever it was doing, which carries on under it. + * Hold it for `FINISH_MS`, then pass the state it was in before. + */ +export type BotOrbState = + | "idle" + | "listening" + | "thinking" + | "done" + | "finished"; + +/** How long a finish plays: the check, its rays, and the eyes coming back. */ +export const FINISH_MS = FINISH_S * 1000; + +type Look = readonly [number, number]; +const LOOK_AT_YOU: Look = [0, 0.04]; +/** Up and aside, as if thinking it over. */ +const LOOK_UP: Look = [-0.36, -0.5]; +/** Down at what you are typing. */ +const LOOK_DOWN: Look = [0.1, 0.6]; +const LOOKS: Record = { + idle: LOOK_AT_YOU, + listening: LOOK_DOWN, + thinking: LOOK_UP, + done: LOOK_AT_YOU, + finished: LOOK_AT_YOU, +}; +/** Seconds between blinks, before each one's jitter. */ +const BLINK_EVERY = 4.2; +/** How far the canvas draws past the footprint on every side, as a share of the size. */ +const SPILL = 0.5; +/** Backing pixels per CSS pixel, at most: sharp on a 3× screen without a huge canvas. */ +const MAX_DPR = 3; +/** Below this footprint, in CSS px, an orb only blinks and watches: no glances, no breath, no drift. */ +const SMALL = 40; +/** + * The least time between two frames of a small orb, in ms: about thirty a second, plenty for eyes a + * few px across, where a fast screen would otherwise draw every face in a list 120 times a second. + */ +const SMALL_FRAME_MS = 30; +/** The moment on its clock a still is composed at, and where a small orb's colour holds. */ +const STILL_T = 1.6; +/** How much of its look an orb gives the pointer: all of it at rest, some while it listens or works. */ +const WATCH: Record = { + idle: 1, + listening: 0.5, + thinking: 0.35, + done: 1, + finished: 1, +}; +/** The pointer perks it up within this share of its size from its centre… */ +const NOTICE_NEAR = 0.9; +/** …and has to go past this one before it can perk it up again. */ +const NOTICE_FAR = 1.3; + +/** Every track the state drives, and when the things that play once last happened. */ +type Motion = { + listen: Track; + think: Track; + done: Track; + smile: Track; + ax: Track; + ay: Track; + doneAt: number; + noticedAt: number; + finishedAt: number; + nods: number[]; +}; + +const still = (v: number): Track => ({ from: v, keys: [] }); + +/** The motion at rest in `state`, as it is on the first frame. */ +function settled(state: BotOrbState): Motion { + const look = LOOKS[state]; + return { + listen: still(state === "listening" ? 1 : 0), + think: still(state === "thinking" ? 1 : 0), + done: still(0), + smile: still(state === "done" ? 1 : 0), + ax: still(look[0]), + ay: still(look[1]), + doneAt: -1, + noticedAt: -1, + finishedAt: state === "finished" ? 0 : -1, + nods: [], + }; +} + +/** The track restarted from its value at `now`, heading for the new key. */ +const retarget = (tr: Track, now: number, key: TrackKey): Track => ({ + from: track(tr, now), + keys: [key], +}); + +/** Moves every track toward `state` from time `now`, each on its own spring or ease. */ +function enter(m: Motion, state: BotOrbState, now: number): Motion { + const look = LOOKS[state]; + const gaze = + state === "thinking" + ? ([6.5, 0.85] as const) + : state === "listening" + ? ([15, 0.6] as const) + : ([12, 0.8] as const); + const next: Motion = { + ...m, + listen: retarget( + m.listen, + now, + state === "listening" + ? { at: now, to: 1, spring: [14, 0.5] } + : { at: now, to: 0, ease: 0.45 }, + ), + think: retarget(m.think, now, { + at: now, + to: state === "thinking" ? 1 : 0, + ease: state === "thinking" ? 0.4 : 0.45, + }), + smile: retarget(m.smile, now, { + at: now + (state === "done" ? 0.08 : 0), + to: state === "done" ? 1 : 0, + ease: state === "done" ? 0.4 : 0.8, + }), + ax: retarget(m.ax, now, { at: now, to: look[0], spring: gaze }), + ay: retarget(m.ay, now, { at: now, to: look[1], spring: gaze }), + done: retarget(m.done, now, { at: now, to: 0, ease: 0.45 }), + }; + if (state === "listening") next.noticedAt = now; + if (state === "done") { + // It lands: a glow that springs up and settles away, a ping and a nod. + next.done = { + from: track(m.done, now), + keys: [ + { at: now, to: 1, spring: [11, 0.42] }, + { at: now + 1.4, to: 0, ease: 1.2 }, + ], + }; + next.doneAt = now; + next.nods = [now + 0.1]; + } + return next; +} + +/** + * The pose at `t` seconds on the orb's clock. `seed` keeps two orbs from blinking and glancing in + * step. A small orb keeps still apart from its blinks, its colour included, so a list of them + * doesn't fidget and, once its eyes settle, has nothing new to draw. + */ +function poseAt( + m: Motion, + t: number, + seed: number, + { reduced, small }: { reduced: boolean; small: boolean }, +): OrbPose { + const listen = track(m.listen, t); + const think = clamp(track(m.think, t)); + // Left alone it glances about now and then; while it listens or works it keeps its eyes on that. + const calm = reduced || small ? 0 : (1 - clamp(listen)) * (1 - think); + const glance = restLook(t, 3.6, seed); + let blink = 0; + if (!reduced) { + const k = Math.floor(t / BLINK_EVERY); + for (const n of [k - 1, k]) { + const at = n * BLINK_EVERY + 1.3 + mulberry32(seed + n)() * 1.6; + blink = Math.max(blink, blinkCurve(t - at)); + } + } + return { + ta: small ? STILL_T : t, + listen, + think, + done: Math.max(0, track(m.done, t)), + smile: clamp(track(m.smile, t)), + tilt: 0.5 * glance[2] * calm, + ax: track(m.ax, t) + 0.6 * glance[0] * calm, + ay: track(m.ay, t) + 0.6 * glance[1] * calm, + blink, + nod: m.nods.reduce((n, at) => n + nodCurve(t - at), 0), + doneAge: m.doneAt < 0 ? -1 : t - m.doneAt, + noticeAge: m.noticedAt < 0 ? -1 : t - m.noticedAt, + // Still, a finish is its check without the burst. + finishAge: m.finishedAt < 0 ? -1 : reduced ? FINISH_HELD : t - m.finishedAt, + breath: small ? 0 : Math.sin((Math.PI * 2 * t) / 5.5), + }; +} + +/** + * An orb watching the pointer: the look it is heading for (`tx`, `ty`), where its eyes are on a + * spring (`x`, `y`, with their speeds), how much of its look it gives the pointer (`w`), whether it + * can see the pointer at all, and whether the pointer is close enough to have perked it up. `epoch` + * and `last` are the pointer's epoch and the time when it last looked. + */ +export type Gaze = { + tx: number; + ty: number; + x: number; + y: number; + vx: number; + vy: number; + w: number; + seen: boolean; + near: boolean; + epoch: number; + last: number; +}; + +export const restingGaze = (): Gaze => ({ + tx: 0, + ty: 0, + x: 0, + y: 0, + vx: 0, + vy: 0, + w: 0, + seen: false, + near: false, + epoch: -1, + last: -1, +}); + +type Box = Pick; + +/** + * Moves `gaze` on to time `t` for an orb in `state`, measuring where the orb is on screen only when + * the pointer, the scroll or the window has changed. True on the frame the pointer comes close, + * which perks the orb up; a listening orb has already noticed you, so it never perks. + */ +export function followPointer( + gaze: Gaze, + t: number, + state: BotOrbState, + measure: () => Box, +): boolean { + const dt = gaze.last < 0 ? 0 : Math.min(1 / 30, Math.max(0, t - gaze.last)); + gaze.last = t; + let perked = false; + if (gaze.epoch !== pointer.epoch) { + gaze.epoch = pointer.epoch; + const box = measure(); + const dx = pointer.x - (box.left + box.width / 2); + const dy = pointer.y - (box.top + box.height / 2); + const d = Math.hypot(dx, dy) || 1; + // Right over it, it looks straight out at you; the further away, the further round it turns. + const reach = d / (d + Math.max(48, box.width * 1.6)); + gaze.tx = (dx / d) * reach * 0.95; + gaze.ty = (dy / d) * reach; + gaze.seen = pointer.on && box.width > 0; + const near = + gaze.seen && d < box.width * (gaze.near ? NOTICE_FAR : NOTICE_NEAR); + perked = near && !gaze.near && state !== "listening"; + gaze.near = near; + } + gaze.vx += ((gaze.tx - gaze.x) * 150 - gaze.vx * 19) * dt; + gaze.vy += ((gaze.ty - gaze.y) * 150 - gaze.vy * 19) * dt; + gaze.x += gaze.vx * dt; + gaze.y += gaze.vy * dt; + gaze.w += ((gaze.seen ? WATCH[state] : 0) - gaze.w) * (1 - Math.exp(-dt * 5)); + return perked; +} + +/** Its own look, glances and all, gives way to the pointer by as much as it is watching. */ +function lookAt(pose: OrbPose, gaze: Gaze): OrbPose { + return { + ...pose, + ax: pose.ax + (gaze.x - pose.ax) * gaze.w, + ay: pose.ay + (gaze.y - pose.ay) * gaze.w, + tilt: pose.tilt + (-0.12 * gaze.x - pose.tilt) * gaze.w, + }; +} + +/** + * Sizes the canvas for a footprint of `size` and returns how to draw a pose on it with `look`, or + * null where there is no 2D context to draw with. A pose that would draw what the canvas already + * shows, in the same theme and at the same density, is skipped. + */ +function surface(canvas: HTMLCanvasElement, size: number, look: OrbLook) { + const ctx = canvas.getContext("2d"); + if (!ctx) return null; + const box = size * (1 + 2 * SPILL); + let dpr = 0; + let shown: { pose: OrbPose; dark: boolean } | null = null; + return (pose: OrbPose) => { + // Read on every draw: a window dragged to another screen, or a zoom, changes it under an orb + // that stays mounted, and an orb drawn for the old one goes soft. + const next = Math.min(MAX_DPR, window.devicePixelRatio || 1); + const dark = document.documentElement.classList.contains("dark"); + if (next === dpr && shown?.dark === dark && samePicture(shown.pose, pose)) { + return; + } + shown = { pose, dark }; + if (next !== dpr) { + dpr = next; + canvas.width = Math.round(box * dpr); + canvas.height = Math.round(box * dpr); + } + ctx.setTransform(dpr, 0, 0, dpr, 0, 0); + ctx.clearRect(0, 0, box, box); + ctx.translate(box / 2, box / 2); + drawBotOrb(ctx, size, pose, { dark, look }); + }; +} + +/** + * Calls `onChange` whenever the screen's pixel density changes, for a still orb that nothing else + * redraws. A resolution query matches only the density it names, so it is asked again after each change. + */ +function watchResolution(onChange: () => void): () => void { + if (typeof window.matchMedia !== "function") return () => {}; + let query: MediaQueryList | null = null; + const arm = () => { + query?.removeEventListener("change", fire); + query = window.matchMedia( + `(resolution: ${window.devicePixelRatio || 1}dppx)`, + ); + query.addEventListener("change", fire); + }; + const fire = () => { + arm(); + onChange(); + }; + arm(); + return () => query?.removeEventListener("change", fire); +} + +const REDUCE_MOTION = "(prefers-reduced-motion: reduce)"; + +/** + * Whether the person asks for less motion, kept current: turning it on while the app is open stills + * every orb already on screen, not only the ones drawn afterwards. + */ +function usePrefersReducedMotion(): boolean { + return useSyncExternalStore( + (onChange) => { + if (typeof window.matchMedia !== "function") return () => {}; + const query = window.matchMedia(REDUCE_MOTION); + query.addEventListener("change", onChange); + return () => query.removeEventListener("change", onChange); + }, + () => + typeof window.matchMedia === "function" && + window.matchMedia(REDUCE_MOTION).matches, + () => false, + ); +} + +let staggers = 0; +const secondsSince = (start: number) => (performance.now() - start) / 1000; + +/** + * A Bot's orb, with the look its `seed` picks; with no seed, OpenBot's own, in violet. `size` is its + * footprint in CSS px; its glow spills past it. It breathes and glances about when idle, watches + * what you type when listening, swirls while thinking, and lands with a ping and a smile when done. + * Passed `finished`, its eyes give way to a check while short rays burst out round it, and then it + * is back as it was. Wherever the pointer is, its eyes follow it: fully at rest, partly while it + * listens or works, with a small perk when the pointer comes close. An orb under 40 px only blinks + * and watches, at about thirty frames a second. + * + * Every orb on the page draws on one shared animation frame, and only while it is on screen, the + * tab is shown and its picture has changed. With reduced motion, or `still`, it is a still in the + * state's pose, drawn again only when the state or the theme changes, and its eyes stay put. + */ +export function BotOrb({ + className, + seed, + size, + state = "idle", + still = false, +}: { + className?: string; + seed?: string; + size: number; + state?: BotOrbState; + still?: boolean; +}) { + const boxRef = useRef(null); + const canvasRef = useRef(null); + const reduced = usePrefersReducedMotion() || still; + const look = useMemo( + () => (seed === undefined ? OPENBOT_LOOK : orbLookFor(seed)), + [seed], + ); + // Each orb's own, so two orbs on a page, even two of one Bot, don't blink or glance in step. + const clock = useRef<{ start: number; stagger: number } | null>(null); + clock.current ??= { start: performance.now(), stagger: ++staggers * 7 }; + const motion = useRef(null); + const shown = useRef(null); + const current = useRef(state); + const gaze = useRef(null); + gaze.current ??= restingGaze(); + + // It starts at rest in its first state, except that a first "done" still lands. After that a + // change moves every track from where it is, so a change mid-motion never jumps. A finish moves + // no track: it plays over the state it was in, and coming back to that state changes nothing. + useEffect(() => { + current.current = state; + if (!clock.current) return; + const t = secondsSince(clock.current.start); + if (state === "finished") { + motion.current = { + ...(motion.current ?? settled("idle")), + finishedAt: t, + }; + return; + } + if (shown.current === state) return; + const before = motion.current; + motion.current = + shown.current === null && state !== "done" + ? { ...settled(state), finishedAt: before?.finishedAt ?? -1 } + : enter(before ?? settled("idle"), state, t); + shown.current = state; + }, [state]); + + // A still: a composed picture in the state's pose, eyes open, its look held, no ping; finished, + // the check in place of the eyes, with no burst, for as long as it is held. Nothing draws it + // again on its own, so it is redrawn when the theme flips the root's `dark` class. + useEffect(() => { + const canvas = canvasRef.current; + const draw = reduced && canvas ? surface(canvas, size, look) : null; + if (!draw || !clock.current) return; + const { stagger } = clock.current; + const held = + state === "finished" + ? { ...settled(shown.current ?? "idle"), finishedAt: 0 } + : settled(state); + const paint = () => + draw( + poseAt(held, STILL_T, stagger, { reduced: true, small: size < SMALL }), + ); + paint(); + const theme = new MutationObserver(paint); + theme.observe(document.documentElement, { + attributes: true, + attributeFilter: ["class"], + }); + const unwatch = watchResolution(paint); + return () => { + theme.disconnect(); + unwatch(); + }; + }, [reduced, size, look, state]); + + // Alive: drawn on the shared clock while it is on screen. It counts as on screen until an + // IntersectionObserver says otherwise, and always where there is none to ask. It reads the theme + // on every frame, so a flip shows on the next one; a frame that would draw what is already there + // is skipped, so a small orb at rest with the pointer still draws nothing between blinks. + useEffect(() => { + const canvas = canvasRef.current; + const box = boxRef.current; + const draw = !reduced && canvas ? surface(canvas, size, look) : null; + if (!draw || !box || !clock.current || !gaze.current) return; + const { start, stagger } = clock.current; + const eyes = gaze.current; + const small = size < SMALL; + const measure = () => box.getBoundingClientRect(); + let ticked = Number.NEGATIVE_INFINITY; + const frame = (now: number) => { + if (small && now - ticked < SMALL_FRAME_MS) return; + ticked = now; + const t = Math.max(0, (now - start) / 1000); + const state = current.current; + if (followPointer(eyes, t, state, measure) && motion.current) { + motion.current = { ...motion.current, noticedAt: t }; + } + const pose = poseAt(motion.current ?? settled(state), t, stagger, { + reduced: false, + small, + }); + draw(lookAt(pose, eyes)); + }; + let unsubscribe: (() => void) | null = null; + const onScreen = (on: boolean) => { + if (on && !unsubscribe) unsubscribe = subscribeFrame(frame); + if (!on && unsubscribe) { + unsubscribe(); + unsubscribe = null; + } + }; + const sight = + "IntersectionObserver" in window + ? new IntersectionObserver((entries) => + // The newest entry is where it is now. + onScreen(entries.at(-1)?.isIntersecting ?? false), + ) + : null; + sight?.observe(box); + onScreen(true); + const unwatch = subscribePointer(); + return () => { + sight?.disconnect(); + onScreen(false); + unwatch(); + }; + }, [reduced, size, look]); + + const spill = size * SPILL; + return ( + + + + ); +} diff --git a/app/src/components/agents/orb/orb-clock.ts b/app/src/components/agents/orb/orb-clock.ts new file mode 100644 index 0000000000..176196bbfb --- /dev/null +++ b/app/src/components/agents/orb/orb-clock.ts @@ -0,0 +1,140 @@ +/** + * What every orb on the page shares, so a list of fifty Bots costs one animation loop and one set of + * listeners rather than fifty of each: a clock that asks the browser for one frame and draws every + * orb in it, and where the pointer is, for the orbs whose eyes follow it. + * + * Both start with their first subscriber and stop with their last, so nothing is left running or + * listening on a page with no orb on it. + */ + +type FrameCallback = (now: number) => void; + +const subscribers = new Set(); +let frame: number | null = null; +/** The document whose visibility the clock follows, while it has subscribers. */ +let watched: Document | null = null; + +function run(now: number) { + frame = null; + // A Set visits what is added during the loop and skips what is removed, so an orb that unmounts + // mid-frame is not drawn after it has gone. + for (const callback of subscribers) callback(now); + schedule(); +} + +function schedule() { + if (frame !== null || subscribers.size === 0 || document.hidden) return; + frame = window.requestAnimationFrame(run); +} + +function halt() { + if (frame !== null) window.cancelAnimationFrame(frame); + frame = null; +} + +/** A hidden tab draws nothing, and picks up again where its orbs are now once it is shown. */ +function onVisibility() { + if (document.hidden) halt(); + else schedule(); +} + +/** + * Calls `callback` once on every animation frame, with the frame's timestamp, until the returned + * function is called. Every subscriber is drawn in the same frame; no frame is asked for while the + * tab is hidden or nobody is subscribed. + */ +export function subscribeFrame(callback: FrameCallback): () => void { + subscribers.add(callback); + if (!watched) { + watched = document; + watched.addEventListener("visibilitychange", onVisibility); + } + schedule(); + return () => { + if (!subscribers.delete(callback) || subscribers.size > 0) return; + halt(); + watched?.removeEventListener("visibilitychange", onVisibility); + watched = null; + }; +} + +// ---------- the pointer ---------- + +type PointerState = { x: number; y: number; on: boolean; epoch: number }; + +const state: PointerState = { x: 0, y: 0, on: false, epoch: 0 }; + +/** + * Where the pointer is, in client px, and whether it is on the page at all. `epoch` counts every + * move, scroll and resize, so an orb measures itself again only when something has changed. Kept + * up to date only while something is subscribed with `subscribePointer`. + */ +export const pointer: Readonly = state; + +let watchers = 0; +let lift: ReturnType | undefined; +let listening: Window | null = null; + +const aim = (event: PointerEvent) => { + state.x = event.clientX; + state.y = event.clientY; + state.on = true; + state.epoch++; + clearTimeout(lift); +}; +const away = () => { + state.on = false; + state.epoch++; +}; +// A finger lifts off the glass; a mouse stays where it was. Give a tap a moment of being watched. +const lifted = (event: PointerEvent) => { + if (event.pointerType !== "mouse") lift = setTimeout(away, 1600); +}; +const left = (event: PointerEvent) => { + if (!event.relatedTarget) away(); +}; +const moved = () => { + state.epoch++; +}; + +function listen(target: Window) { + target.addEventListener("pointermove", aim, { passive: true }); + target.addEventListener("pointerdown", aim, { passive: true }); + target.addEventListener("pointerup", lifted, { passive: true }); + target.document.addEventListener("pointerout", left, { passive: true }); + target.addEventListener("blur", away); + target.addEventListener("scroll", moved, { capture: true, passive: true }); + target.addEventListener("resize", moved); +} + +function unlisten(target: Window) { + target.removeEventListener("pointermove", aim); + target.removeEventListener("pointerdown", aim); + target.removeEventListener("pointerup", lifted); + target.document.removeEventListener("pointerout", left); + target.removeEventListener("blur", away); + target.removeEventListener("scroll", moved, { capture: true }); + target.removeEventListener("resize", moved); + clearTimeout(lift); + // Nobody is following it now, so what it last knew is stale for whoever subscribes next. + away(); +} + +/** + * Keeps `pointer` up to date until the returned function is called: one set of window listeners for + * every orb that watches. + */ +export function subscribePointer(): () => void { + if (watchers++ === 0) { + listening = window; + listen(listening); + } + let subscribed = true; + return () => { + if (!subscribed) return; + subscribed = false; + if (--watchers > 0 || !listening) return; + unlisten(listening); + listening = null; + }; +} diff --git a/app/src/components/app-sidebar/app-sidebar.tsx b/app/src/components/app-sidebar/app-sidebar.tsx index 27fe657b18..2f15b783ce 100644 --- a/app/src/components/app-sidebar/app-sidebar.tsx +++ b/app/src/components/app-sidebar/app-sidebar.tsx @@ -3,7 +3,8 @@ import { useInfiniteQuery, useQuery } from "@tanstack/react-query"; import { Link, type LinkOptions, useParams } from "@tanstack/react-router"; import { AnimatePresence, motion, useReducedMotion } from "motion/react"; import type * as React from "react"; -import { useRef, useState } from "react"; +import { type RefObject, useRef, useState } from "react"; +import AgentOrb from "@/components/agents/orb/agent-orb"; import { DropdownMenu, DropdownMenuContent, @@ -25,7 +26,13 @@ import { SidebarMenuButton, SidebarMenuItem, SidebarRail, + useSidebar, } from "@/components/ui/sidebar"; +import { + Tooltip, + TooltipContent, + TooltipTrigger, +} from "@/components/ui/tooltip"; import { currentUserQueryOptions } from "@/lib/auth/queries"; import { type ChannelSummary, @@ -33,8 +40,10 @@ import { } from "@/lib/channels/queries"; import { useChannelEvents } from "@/lib/channels/use-channel-events"; import { appConfig } from "@/lib/generated/application-config"; +import { getHotkey, hotkeyLabel } from "@/lib/hotkeys/hotkeys"; +import { useHotkey } from "@/lib/hotkeys/use-hotkey"; import { EASE_OUT, ENTRANCE_SECONDS } from "@/lib/motion"; -import { navigationFor } from "@/lib/navigation"; +import { type NavigationItem, useNavigationItems } from "@/lib/navigation"; import { relativeTime } from "@/lib/relative-time"; import { type MessageListEmphasis, @@ -44,7 +53,7 @@ import { BotsNavBadge } from "../bot-profile/attention"; import { BrowserNavItem } from "../browser/browser-nav-item"; import { Button } from "../ui/button"; import { Empty, EmptyDescription, EmptyHeader, EmptyTitle } from "../ui/empty"; -import { Channel } from "./channel"; +import { Channel, LIT } from "./channel"; import { ChannelPagination } from "./channel-pagination"; import { useSignOut } from "./use-sign-out"; @@ -52,6 +61,11 @@ const appLinkOptions = { to: "/" } satisfies LinkOptions; const userMenuItemClassName = "gap-2 px-2 py-1.5"; +/** A control in the sidebar lifts to the card's fill and shadow under the pointer, as a row does. */ +const LIT_ON_HOVER = "hover:bg-sidebar-accent hover:shadow-(--shadow-card)"; + +const NEW_CHAT_KEYS = hotkeyLabel(getHotkey("new-chat").combo); + function UserAvatar() { const { data: currentUser } = useQuery(currentUserQueryOptions()); const initials = @@ -198,6 +212,61 @@ function ChannelRow({ ); } +/** + * A count beside a destination, drawn as the Bots count is so the two read as one kind of thing. + * Nothing for zero. + */ +function NavigationBadge({ count }: { count: number }) { + if (count <= 0) return null; + return ( + + {count} + + ); +} + +/** One destination in the sidebar's footer, from the shared list, with its count when it has one. */ +export function NavigationEntry({ item }: { item: NavigationItem }) { + return ( + + ( + + )} + > +
+ +
+ {item.label} + {/* What waits on you across every Bot; Bots lists them first. */} + {item.id === "bots" ? : null} + {item.badge ? : null} + + + ); +} + +/** + * Puts the cursor in the search box as soon as it is on the page, with what it holds selected so + * typing replaces the last search. On a narrow window the sidebar is a sheet that only mounts once + * it opens, and a frame later moves focus to its first control; trying again on each of the next + * few frames lands after both. + */ +function focusWhenDrawn(box: RefObject, frames = 10) { + const input = box.current; + if (input?.isConnected) { + input.focus(); + input.select(); + if (document.activeElement === input) return; + } + if (frames > 0) requestAnimationFrame(() => focusWhenDrawn(box, frames - 1)); +} + export function AppSidebar({ ...props }: React.ComponentProps) { const emphasis = useMessageListEmphasis(); const { data: currentUser } = useQuery(currentUserQueryOptions()); @@ -206,7 +275,19 @@ export function AppSidebar({ ...props }: React.ComponentProps) { // One socket for the app, opened where the roster is kept live. useChannelEvents(); const [search, setSearch] = useState(""); + const searchBox = useRef(null); const scrollRoot = useRef(null); + const { isMobile, setOpen, setOpenMobile } = useSidebar(); + /* + * ⌘K opens the sidebar, or its sheet on a narrow window, with the cursor in its search box. Bound + * here, so it works wherever this sidebar is: not on Settings or Admin, which have sidebars of + * their own, and not under a skin, which draws its own frame. + */ + useHotkey("search", () => { + if (isMobile) setOpenMobile(true); + else setOpen(true); + focusWhenDrawn(searchBox); + }); const searching = search.trim().length > 0; const visibleChannels = pinnedFirst(matchingChannels(channels.data, search)); /* @@ -217,7 +298,7 @@ export function AppSidebar({ ...props }: React.ComponentProps) { */ const animateOrder = !searching && (channels.data?.length ?? 0) <= MAX_ANIMATED_ROWS; - const navigation = navigationFor(currentUser?.role); + const navigation = useNavigationItems(); return ( @@ -225,29 +306,44 @@ export function AppSidebar({ ...props }: React.ComponentProps) { ( - {appConfig.brand.productName} + {/* OpenBot's own mark, clipped to its circle as every face below it is. */} + + + + {appConfig.brand.productName} )} /> - + + ( + + )} + /> + } + > + + + {/* An accelerator nobody is told about is not a feature. */} + + New conversation + {NEW_CHAT_KEYS} + + @@ -255,9 +351,10 @@ export function AppSidebar({ ...props }: React.ComponentProps) { - + setSearch(event.target.value)} placeholder="Search…" value={search} @@ -324,28 +421,11 @@ export function AppSidebar({ ...props }: React.ComponentProps) { + {/* In the list's own order, so a destination goes where `lib/navigation.ts` puts it. */} {navigation .filter((item) => item.section === "primary") .map((item) => ( - - ( - - )} - > -
- -
- {item.label} - {/* What waits on you across every Bot; Bots lists them first. */} - {item.id === "bots" ? : null} - - + ))} {/* Your own browser, with the Recorder in it. Drawn only where OpenBot can run one for you. */} @@ -353,11 +433,13 @@ export function AppSidebar({ ...props }: React.ComponentProps) { + } > - + {currentUser?.name || currentUser?.email} diff --git a/app/src/components/app-sidebar/channel-item-content.tsx b/app/src/components/app-sidebar/channel-item-content.tsx index ece35115c2..1a8538b810 100644 --- a/app/src/components/app-sidebar/channel-item-content.tsx +++ b/app/src/components/app-sidebar/channel-item-content.tsx @@ -26,8 +26,9 @@ export function ChannelItemContent({ pinned?: boolean; revealing?: boolean; }) { - const primaryText = `text-[0.9rem] leading-5 tracking-[-1%] ${unread ? "font-semibold" : "font-medium"}`; - const secondaryText = "text-[12px] leading-4 text-muted-foreground"; + // On the type scale: the line that leads at the sub size, the other a caption. + const primaryText = `type-sub ${unread ? "font-semibold" : "font-medium"}`; + const secondaryText = "type-caption text-muted-foreground"; return (
@@ -46,15 +47,23 @@ export function ChannelItemContent({ > {name} - - {lastMessageAt} + {/* + * The sidebar row's "…" button sits over this end of the line while it shows: under the + * pointer, with focus in the row, or with its menu open. The time and marks step aside + * for it. Under 768px the button always shows and the row keeps room for it instead. The + * preview in Settings is in no menu item, so none of this reaches it. + */} + + + {lastMessageAt} + + {unread ? ( + + ) : null} + {pinned ? ( + + ) : null} - {unread ? ( - - ) : null} - {pinned ? ( - - ) : null}
void; + onDelete: () => void; +}) { + return ( + <> + + {pinned ? : } + {pinned ? "Unpin conversation" : "Pin conversation"} + + + + Delete conversation… + + + ); +} + /** * Memoized roster row. `use-channel-events` preserves unchanged row identity, and * `content-visibility` keeps off-screen rows cheap without virtualization. * - * Right-click opens Pin and Delete. Deleting is confirmed in a dialog that names the channel, - * because the row it was invoked on is one of several identical-looking rows. + * Pin and Delete open from right-click, and from the "…" button over the row's end. Right-click + * alone was found by nobody who was not told, and reached by no keyboard. The button shows under + * the pointer and while focus is in the row, and comes straight after it in tab order. Deleting is + * confirmed in a dialog that names the channel, because the row it was invoked on is one of several + * identical-looking rows. */ export const Channel = memo(function Channel({ channelId, @@ -70,8 +128,18 @@ export const Channel = memo(function Channel({ */ const [pinProblem, setPinProblem] = useState(null); + const title = summary || lastMessage || "New conversation"; + const pin = () => { + setPinProblem(null); + setPinned.mutate( + { channelId, pinned: !pinned }, + { onError: (thrown) => setPinProblem(thrown.message) }, + ); + }; + const askToDelete = () => setConfirming(true); + return ( - <> + - { - setPinProblem(null); - setPinned.mutate( - { channelId, pinned: !pinned }, - { onError: (thrown) => setPinProblem(thrown.message) }, - ); - }} - > - {pinned ? : } - {pinned ? "Unpin conversation" : "Pin conversation"} - - setConfirming(true)} - > - - Delete conversation… - + + + + } + > + + + + + + {pinProblem ? (

{pinProblem} @@ -137,6 +212,6 @@ export const Channel = memo(function Channel({ onOpenChange={setConfirming} open={confirming} /> - + ); }); diff --git a/app/src/components/approvals/approval-card.tsx b/app/src/components/approvals/approval-card.tsx new file mode 100644 index 0000000000..f3f55adfe3 --- /dev/null +++ b/app/src/components/approvals/approval-card.tsx @@ -0,0 +1,301 @@ +import { + IconCheck, + IconChevronRight, + IconCircleCheck, + IconMail, + IconShieldCheck, + IconTerminal2, + IconWorld, + IconX, +} from "@tabler/icons-react"; +import { Fragment, type ReactNode, useId } from "react"; +import { Button } from "@/components/ui/button"; +import { + type ApprovalView, + alwaysAllowReach, + approvalOutcome, + approvalPlace, + clockTime, + decisionReceipt, + describeApproval, + type OutcomeTone, +} from "@/lib/approval-view"; +import type { ApprovalDecision, ApprovalRequest } from "@/lib/approvals"; +import { cn } from "@/lib/utils"; + +const KIND_ICON = { + mail: IconMail, + browser: IconWorld, + command: IconTerminal2, + other: IconShieldCheck, +} as const; + +const PILL_TONES: Record = { + done: "bg-status-done-soft text-status-done", + working: "bg-status-working-soft text-status-working", + needs: "bg-status-needs-soft text-status-needs", + problem: "bg-status-problem-soft text-status-problem", + neutral: "bg-muted text-muted-foreground", +}; + +/** Decline and Allow once: half the card wide each, a finger's height tall. */ +const CHOICE = + "h-(--target) min-w-0 flex-1 basis-28 rounded-full px-4 type-sub font-semibold"; + +/** + * One action a Bot wants to take, and the person's answer to it: what it is, where, who it goes to + * and what it says, then Decline and Allow once. "Always allow here" is a quiet button under the + * two, because it saves a rule for this Bot, this action and this place rather than answering only + * this request; the line under it says how far that rule reaches. A hand-off is the person's to do, + * so it asks "I did it myself" or "Don't do it". + * + * Once decided the card folds: the title, a pill saying how it ended (Allowed, Sent, Done, + * Declined, Didn't work), and a line saying who decided and when. While it waits, everything the + * request carries is one tap away under Details. + * + * `frame="card"` draws it as a card of its own, for a conversation. `frame="row"` leaves the frame + * to the list it sits in, a row of a page's `PageRows` card. + */ +export function ApprovalCard({ + request, + botName, + busy = false, + error, + onDecide, + frame = "card", + className, +}: { + request: ApprovalRequest; + /** Named where the card is shown away from the Bot's own conversation. */ + botName?: string; + busy?: boolean; + error?: string | null; + onDecide?: (choice: ApprovalDecision) => void; + frame?: "card" | "row"; + className?: string; +}) { + const view = describeApproval(request); + const waiting = request.status === "pending"; + const handOff = request.action.policy?.behaviour === "hand_off"; + const outcome = approvalOutcome(request); + const receipt = waiting ? null : decisionReceipt(request); + const place = approvalPlace(request); + const caption = [botName, view.app].filter(Boolean).join(" · "); + const Icon = KIND_ICON[view.kind]; + const reachId = useId(); + + return ( +

+
+ + + +
+

+ {handOff && waiting ? `This needs you: ${view.title}` : view.title} +

+ {caption || place ? ( +

+ {caption} + {caption && place ? " · " : null} + {/* Shown whole, however long: the person is deciding about this exact place. */} + {place ? ( + {place} + ) : null} +

+ ) : null} +
+ {outcome ? ( + + {outcome.tone === "done" ? ( + + ) : null} + {outcome.label} + + ) : null} +
+ {waiting ? ( + + ) : null} + {error ? ( +

+ {error} +

+ ) : null} + {waiting && onDecide ? ( +
+
+ + +
+ {handOff ? null : ( + <> + +

+ {alwaysAllowReach(request)} +

+ + )} +
+ ) : null} + {receipt ? ( + + {receipt} + + ) : null} +
+ ); +} + +/** The fields, the body, why the Bot asked, and everything else it would send. */ +function ApprovalContent({ + request, + view, + handOff, +}: { + request: ApprovalRequest; + view: ApprovalView; + handOff: boolean; +}) { + const detailsId = useId(); + return ( + <> + {view.fields.map((field) => ( + + +
+ + {field.label} + + + {field.value} + +
+ + ))} + {view.body ? ( + <> + + {/* Whole, not clipped: this is what the person is agreeing to send or type. */} +

+ {view.body} +

+ + ) : null} + {request.action.policy?.reason ? ( +

+ {handOff ? "Handed to you: " : ""} + {request.action.policy.reason} +

+ ) : null} + {/* Open by itself when the face above leaves out something the request would send. */} +
0 || undefined}> + + Details + + + 
+          {JSON.stringify(request.action.args, null, 2)}
+
+
+ + ); +} + +/** The line between a card's parts, short of its edges like the rows of a list. */ +function Hairline() { + return
; +} + +/** Who decided and when, under a decided card. */ +function DecidedLine({ + children, + at, + declined, +}: { + children: ReactNode; + at: string | null; + declined: boolean; +}) { + const Mark = declined ? IconX : IconCircleCheck; + return ( + <> + +
+ + {children} + {at ? ( + + ) : null} +
+ + ); +} diff --git a/app/src/components/approvals/inline-approval.tsx b/app/src/components/approvals/inline-approval.tsx index ab179e1f8b..dd0bfdecd9 100644 --- a/app/src/components/approvals/inline-approval.tsx +++ b/app/src/components/approvals/inline-approval.tsx @@ -1,18 +1,27 @@ import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query"; import type { ReactNode } from "react"; -import { Button } from "@/components/ui/button"; +import { personWasAsked } from "@/lib/approval-view"; import { - type ApprovalDecision, - approvalInboxOptions, - decideApproval, + type ApprovalRequest, + approvalInboxCacheOptions, + decideApprovalMutationOptions, } from "@/lib/approvals"; +import { ApprovalCard } from "./approval-card"; + +/** A request the person decided on a card, which keeps its folded card in the conversation. */ +const decidedByPerson = (request: ApprovalRequest) => + request.status !== "pending" && + Boolean(request.decision) && + personWasAsked(request); /** - * A pending approval, drawn in the conversation where the action it holds would have been. + * An approval, drawn in the conversation where the action it holds would have been. * - * The same request as the Approvals page, read from the same owner-scoped inbox, so a decision made - * in either place is the one decision. Once decided the server runs the action exactly once and - * continues the conversation, and this gives the line back to the tool's own renderer. + * The same request as the Bot's page, read from the same owner-scoped inbox, so a decision made in + * either place is the one decision. Once decided the server runs the action exactly once and + * continues the conversation, and this gives the line back to the tool's own renderer, under the + * folded card saying how it ended and who decided. A request the person was never asked about (a + * saved rule allowed it) leaves no card behind. */ export function InlineApproval({ toolCallId, @@ -22,89 +31,59 @@ export function InlineApproval({ fallback: ReactNode; }) { const cache = useQueryClient(); - const inbox = useQuery(approvalInboxOptions()); - const decision = useMutation({ - mutationFn: (choice: ApprovalDecision) => - decideApproval(request?.id ?? "", choice), - onSuccess: () => cache.invalidateQueries({ queryKey: ["approvals"] }), - }); + const inbox = useQuery(approvalInboxCacheOptions()); + const decision = useMutation(decideApprovalMutationOptions(cache)); const request = toolCallId ? inbox.data?.requests.find( (candidate) => candidate.action.toolCallId === toolCallId, ) : undefined; - if (request?.status !== "pending") return <>{fallback}; - const handOff = request.action.policy?.behaviour === "hand_off"; - const choices: [ApprovalDecision, string][] = handOff - ? [ - ["handled", "I did it myself"], - ["deny", "Don't do it"], - ] - : [ - ["allow_once", "Allow once"], - ["allow_always", "Always allow here"], - ["deny", "Deny"], - ]; - return ( -
-

- {handOff ? "This needs you" : "Waiting for your approval"}:{" "} - {request.action.toolRef - .replace(/^computer_|^host\//, "") - .replaceAll("_", " ")}{" "} - on {request.action.scope} -

- {request.action.policy ? ( -

{request.action.policy.reason}

- ) : null} - {decision.error ? ( -

- {decision.error.message} -

- ) : null} -
- {choices.map(([choice, label], index) => ( - - ))} + if (!request) return <>{fallback}; + if (request.status !== "pending") { + if (!decidedByPerson(request)) return <>{fallback}; + return ( +
+ + {fallback}
-
+ ); + } + return ( + decision.mutate({ id: request.id, choice })} + request={request} + /> ); } /** - * The pending approvals for a group of tool calls, for a transcript that draws those calls itself - * rather than through each tool's renderer: the browser-activity group folds every click and field - * into one collapsed line, so a card inside a step would never be seen. Drawn beside the group, - * outside the fold, one card per waiting call; nothing at all when none is waiting. + * The approval cards for tool calls a transcript draws itself rather than through each tool's + * renderer: a run of steps folded into one line, whose cards would never be seen inside the fold, + * and a server call alone in its turn, which has no renderer to draw one. Drawn beside the line, one + * card per call the person was asked about: waiting to be decided, or folded once it was. Nothing + * at all for calls nobody asked about. */ -export function PendingApprovals({ +export function ToolCallApprovals({ toolCallIds, }: { toolCallIds: readonly string[]; }) { - const inbox = useQuery(approvalInboxOptions()); - const waiting = new Set( + const inbox = useQuery(approvalInboxCacheOptions()); + const asked = new Set( (inbox.data?.requests ?? []) - .filter((request) => request.status === "pending") + .filter( + (request) => request.status === "pending" || decidedByPerson(request), + ) .flatMap((request) => request.action.toolCallId ? [request.action.toolCallId] : [], ), ); - const ids = toolCallIds.filter((id) => waiting.has(id)); + const ids = toolCallIds.filter((id) => asked.has(id)); if (ids.length === 0) return null; return ( -
+
{ids.map((id) => ( ))} diff --git a/app/src/components/approvals/waiting.tsx b/app/src/components/approvals/waiting.tsx index b101dddd51..b9b209f23e 100644 --- a/app/src/components/approvals/waiting.tsx +++ b/app/src/components/approvals/waiting.tsx @@ -1,4 +1,4 @@ -import { IconHandStop, IconMessageQuestion } from "@tabler/icons-react"; +import { IconMessageQuestion } from "@tabler/icons-react"; import { useMutation, useQuery } from "@tanstack/react-query"; import { Fragment, useState } from "react"; import { PageRows, PageSection } from "@/components/layout/page-shell"; @@ -20,37 +20,54 @@ import { decideApprovalMutationOptions, } from "@/lib/approvals"; import { queryClient } from "@/query-client"; +import { ApprovalCard } from "./approval-card"; /** - * Questions and actions waiting on the person, one row each, decided where it is shown: the answer - * box and the decision buttons sit on the row's own footer line. Given a Bot, only that Bot's. The - * same requests the conversation draws inline, so a decision made in either place is the one - * decision. + * Questions and actions waiting on the person, one row each, decided where it is shown: a question's + * answer box sits on its row's footer line, and an action is the same approval card the + * conversation draws, so a decision made in either place is the one decision. An action answered + * here stays, folded to how it went, until the page is left. Given a Bot, only that Bot's. + * + * Given a `title`, it is its own section of a page, drawn only while something is listed. */ -export function WaitingForYou({ agentId }: { agentId?: string }) { +export function WaitingForYou({ + agentId, + title, +}: { + agentId?: string; + title?: string; +}) { const inbox = useQuery(approvalInboxOptions()); const agents = useQuery(agentListQueryOptions()); const botName = (id: string) => agents.data?.find((agent) => agent.id === id)?.name ?? id; const decision = useMutation(decideApprovalMutationOptions(queryClient)); + const [decidedHere, setDecidedHere] = useState>( + () => new Set(), + ); const [answers, setAnswers] = useState>({}); const answer = useMutation(answerPersonQuestionMutationOptions(queryClient)); const ours = (botId: string) => agentId === undefined || botId === agentId; const questions = (inbox.data?.questions ?? []).filter((question) => ours(question.conversationBotId ?? question.botId), ); - const pending = (inbox.data?.requests ?? []).filter( - (request) => request.status === "pending" && ours(request.action.botId), + const requests = (inbox.data?.requests ?? []).filter( + (request) => + ours(request.action.botId) && + (request.status === "pending" || decidedHere.has(request.id)), ); - const error = inbox.error ?? decision.error ?? answer.error; - return ( + // A decision's failure is said on its own card. + const error = inbox.error ?? answer.error; + const listed = questions.length + requests.length > 0; + if (title && !listed) return null; + const list = ( <> {error ? (

{error.message}

) : null} - {questions.length + pending.length > 0 ? ( + {listed ? ( {questions.map((question, index) => ( @@ -97,103 +114,32 @@ export function WaitingForYou({ agentId }: { agentId?: string }) { ))} - {pending.map((request, index) => ( + {requests.map((request, index) => ( {questions.length + index > 0 ? : null} - - - - - - - {botName(request.action.botId)} wants to{" "} - {request.action.toolRef - .replace(/^computer_|^host\//, "") - .replaceAll("_", " ")} - - - {request.action.scope} - - {request.action.policy ? ( -

- {request.action.policy.behaviour === "hand_off" - ? "Handed to you: " - : ""} - {request.action.policy.reason} -

- ) : null} - 
-                    {JSON.stringify(request.action.args, null, 2)}
-
- - {request.action.policy?.behaviour === "hand_off" ? ( - - - - - ) : ( - - - - - - )} - + { + // Kept here, folded, once answered: the person sees how it went. + setDecidedHere((prior) => new Set(prior).add(request.id)); + decision.mutate({ id: request.id, choice }); + }} + request={request} + /> ))} ) : null} ); + return title ? {list} : list; } /** @@ -201,19 +147,5 @@ export function WaitingForYou({ agentId }: { agentId?: string }) { * waits: the sidebar's badge already said whether to look. */ export function BotNeedsYou({ agentId }: { agentId: string }) { - const inbox = useQuery(approvalInboxOptions()); - const waiting = - (inbox.data?.questions ?? []).some( - (question) => (question.conversationBotId ?? question.botId) === agentId, - ) || - (inbox.data?.requests ?? []).some( - (request) => - request.status === "pending" && request.action.botId === agentId, - ); - if (!waiting) return null; - return ( - - - - ); + return ; } diff --git a/app/src/components/bot-profile/access.tsx b/app/src/components/bot-profile/access.tsx index 6b53a54dce..755791f57d 100644 --- a/app/src/components/bot-profile/access.tsx +++ b/app/src/components/bot-profile/access.tsx @@ -76,7 +76,8 @@ function connectorTools(plugins: GrantedPlugins): Map { const connectors = new Map(); for (const tool of plugins.tools) { const key = tool.ref.split("/")[0] ?? tool.ref; - let label = readToolName(tool.toolName).label; + // What the tool does, not what it did: "Send your email", never "Sent your email". + let label = readToolName(tool.toolName, "plain").label; const prefix = `${key.toLowerCase()} `; if (label.toLowerCase().startsWith(prefix)) { const rest = label.slice(prefix.length); diff --git a/app/src/components/bot-profile/profile.tsx b/app/src/components/bot-profile/profile.tsx index fef082c983..43e5282327 100644 --- a/app/src/components/bot-profile/profile.tsx +++ b/app/src/components/bot-profile/profile.tsx @@ -7,7 +7,6 @@ import { import { useMutation, useQuery } from "@tanstack/react-query"; import { Link } from "@tanstack/react-router"; import type * as React from "react"; -import { useState } from "react"; import { AbstractAvatar } from "@/components/agents/abstract-avatar"; import { BotNeedsYou } from "@/components/approvals/waiting"; import { PageRows, PageSection } from "@/components/layout/page-shell"; @@ -40,6 +39,10 @@ import { type BotNotify, botLifecycleQueryOptions, } from "@/lib/bot-lifecycle/queries"; +import { + requestNotificationPermission, + useNotificationPermission, +} from "@/lib/notifications/permission"; import { queryClient } from "@/query-client"; import { BotActivityGlance } from "./activity"; import { BotPausedBanner } from "./pause-banner"; @@ -65,15 +68,8 @@ export function BotProfile({ const lifecycle = useQuery(botLifecycleQueryOptions(agent.id)); const pause = useMutation(setBotPausedMutationOptions(queryClient)); const notify = useMutation(setBotNotifyMutationOptions(queryClient)); - const [permission, setPermission] = useState(() => { - try { - return typeof Notification === "undefined" - ? "unsupported" - : Notification.permission; - } catch { - return "unsupported"; - } - }); + // The browser's answer, shared with onboarding's Get notified step, so a yes there shows here. + const permission = useNotificationPermission(); return ( <> @@ -186,13 +182,7 @@ export function BotProfile({
); })} @@ -106,25 +114,15 @@ function GroupFaces({ } /** - * Three bouncing dots in a small badge, ringed in the sidebar's own colour so it sits on the - * avatar as a badge rather than floating over it. The staggered negative delays start each dot at - * a different point in the same bounce, which is what makes the three read as one wave. + * A still dot in the working colour, ringed so it sits on the avatar as a badge rather than floating + * over it. Still on purpose: an orb this small only blinks, and reduced motion stills it entirely, so + * the faces working is too slight to read as busy from across the roster. The ring is the ground the + * avatar sits on, the sidebar's own unless a row lit behind it sets `--avatar-ground`. */ -function TypingBadge() { +function BusyDot() { return ( -
+ Working… - - - -
- ); -} - -function Dot({ className }: { className?: string }) { - return ( - + ); } diff --git a/app/src/components/channels/browser-activity.tsx b/app/src/components/channels/browser-activity.tsx index 4995fdf2d8..9476433d66 100644 --- a/app/src/components/channels/browser-activity.tsx +++ b/app/src/components/channels/browser-activity.tsx @@ -1,17 +1,12 @@ +import { + BROWSER_STEP_DONE as labels, + BROWSER_STEP_PLAIN as plainLabels, +} from "@/lib/computers/step-names"; +import { toolResultHeldBack } from "@/lib/plugins/tool-result"; import type { VisibleChatItem } from "./chat-messages"; type BrowserStep = Extract; -const labels: Record = { - computer_navigate: "Opened page", - computer_read: "Read page", - computer_snapshot: "Inspected page", - computer_click: "Clicked on page", - computer_type: "Filled in field", - computer_key: "Pressed key", - computer_scroll: "Scrolled page", -}; - /** Whether a tool call is one of the browser steps a tool group draws with its page link. */ export function isBrowserStep(item: BrowserStep): boolean { return Object.hasOwn(labels, item.toolCall.function.name); @@ -49,6 +44,7 @@ export function browserStepDetails(step: BrowserStep) { step.result?.startsWith("Error:") === true; return { label: labels[step.toolCall.function.name], + plain: plainLabels[step.toolCall.function.name], href: url?.href, title: navigation && typeof result.title === "string" && result.title.trim() @@ -60,3 +56,41 @@ export function browserStepDetails(step: BrowserStep) { pending: step.result === undefined, }; } + +/** + * How a browser step came out, from its result alone. + * + * ONE READING FOR BOTH PLACES A STEP IS SHOWN. The conversation's tool line (`stepDetails`) and the + * computer pane's step log (`step-log.tsx`) both read a step here, so a click the person declined + * never reads as done in one of them and declined in the other. + * + * A step the person held back is read before any of the browser's own fields, with + * `toolResultHeldBack`: declined on its card, or not asked again because the same request was still + * waiting (`alreadyWaiting`). The server says either in words for the model, sometimes inside the + * browser's JSON, and it would otherwise read as a step that worked or one that failed. `pending` is + * a step with no result yet, or only the note that it went to its approval card. Whether that is + * running, waiting for the person or cut off is for the caller to say, from the turn and the + * approvals inbox, and so is whether a request `alreadyWaiting` repeated is still waiting. + */ +export type BrowserOutcome = + | "pending" + | "declined" + | "alreadyWaiting" + | "stopped" + | "blocked" + | "notYet" + | "failed" + | "done"; + +export function browserOutcome(step: BrowserStep): BrowserOutcome { + if (step.result === undefined) return "pending"; + const result = objectOf(step.result); + if (result.waiting === true) return "pending"; + const heldBack = toolResultHeldBack(step.result); + if (heldBack) return heldBack; + // Cut off, like a call that never got its result: nothing went wrong with the page. + if (result.stopped === true) return "stopped"; + if (result.refused === true) return "blocked"; + if (result.ok === false && result.starting === true) return "notYet"; + return browserStepDetails(step).failed ? "failed" : "done"; +} diff --git a/app/src/components/channels/channel-chat.tsx b/app/src/components/channels/channel-chat.tsx index 317647c970..f162f5a89b 100644 --- a/app/src/components/channels/channel-chat.tsx +++ b/app/src/components/channels/channel-chat.tsx @@ -1012,6 +1012,11 @@ export function ChannelChat({ // Keep `seed` in state; transcriptMessages gives it up once the agent holds a user turn. }, []); + /** The Bot's own name, once its profile is in. The conversation's name may be its title instead. */ + const botName = agentProfiles?.find( + (profile) => profile.id === runtimeAgentId, + )?.name; + return ( // Activity renderers resolve their agent through this SDK context. It must match useAgent's // channel instance so an action continues this thread instead of looking for a default agent. @@ -1055,6 +1060,8 @@ export function ChannelChat({ commands={skillCommands} // Readiness is handled by `say`; deletion is the only disabled-chat state. disabled={!channel.active} + // Until the Bot's name is in, the composer keeps its own words. + {...(botName ? { placeholder: `Message ${botName}` } : {})} messages={withVoiceChats( transcriptMessages(agent.messages, seed), voiceArchive.entries, diff --git a/app/src/components/channels/chat-transcript.tsx b/app/src/components/channels/chat-transcript.tsx index ba7002775d..2bc5e34c0d 100644 --- a/app/src/components/channels/chat-transcript.tsx +++ b/app/src/components/channels/chat-transcript.tsx @@ -31,6 +31,7 @@ import { useState, } from "react"; import { Streamdown } from "streamdown"; +import { ToolCallApprovals } from "@/components/approvals/inline-approval"; import { ArgumentsArrivingContext } from "@/components/gallery/streaming"; import { Bubble, BubbleContent } from "@/components/ui/bubble"; import { @@ -67,6 +68,7 @@ import { forDisplay, REFUSAL_MARKER, toolResultFailed, + toolResultHeldBack, } from "@/lib/plugins/tool-result"; import { readTeamBotConsent } from "@/lib/team-bots"; import { cn } from "@/lib/utils"; @@ -81,9 +83,11 @@ import { } from "./chat-messages"; import { splitChatOrigin } from "./chat-origin"; import type { QueuedMessage } from "./composer"; +import { useConversationWaiting } from "./conversation-presence"; import { FollowScroll, RESUME_PX, useFollowIntent } from "./follow-intent"; import { LateRenderersContext, LateRenderersWatch } from "./late-renderers"; import { usePacedReveal } from "./paced-reveal"; +import { holdsBlock, splitBubbles } from "./split-bubbles"; import { TeamBotConsentCard } from "./team-bot-consent"; import { dropRepeatedAppCards, @@ -92,6 +96,7 @@ import { type ToolGroup, type TranscriptItem, useDrawnTool, + useHeldForPerson, } from "./tool-activity"; import { ToolRenderBoundary, toolCallFinal } from "./tool-boundary"; import { shouldDrawEarlier, useTranscriptPages } from "./transcript-pages"; @@ -139,6 +144,32 @@ type ChatTranscriptProps = { /** One shared empty array, so a screen without a queue does not hand down a new one per render. */ const EMPTY_QUEUE: readonly QueuedMessage[] = []; +/** + * A conversation's bubbles: body text, one radius on every corner and no tail, the same padding for + * the Bot and the person. Their colours are `--bubble` and `--bubble-user` (theme.css). + */ +const BUBBLE_CONTENT = "type-body rounded-(--radius-bubble) px-4 py-3"; +/** The person's words, in the accent's tint, on the right. */ +const USER_BUBBLE = "max-w-[85%] *:data-[slot=bubble-content]:bg-bubble-user"; +/* + * A Bot's thought, in a bubble of its own. STILL THE GHOST VARIANT, because a skin finds a Bot's + * words by `[data-variant="ghost"][data-align="start"]` (`skins//skin.css`); so these put back + * the fill, the radius and the padding the variant takes off, rather than choose another variant + * and leave every skin's rule matching nothing. + * + * Capped like a chat's bubble, so a short answer reads as a message rather than a page. + */ +const AGENT_BUBBLE = + "data-[variant=ghost]:max-w-[82%] *:data-[slot=bubble-content]:rounded-(--radius-bubble) *:data-[slot=bubble-content]:bg-bubble *:data-[slot=bubble-content]:px-4 *:data-[slot=bubble-content]:py-3"; +/** + * A bubble holding a code block or a table spans the column once there is room. On a phone it keeps + * the cap, and the block scrolls inside it. + */ +const WIDE_BUBBLE = "md:w-full md:data-[variant=ghost]:max-w-full"; +/** Streamdown's prose, tightened to a bubble: closer paragraphs, bullets hung outside the text. */ +const AGENT_PROSE = + "[&>div]:space-y-2.5 [&_ol]:list-outside [&_ol]:pl-5 [&_ul]:list-outside [&_ul]:pl-5 [&_li]:py-0.5"; + /** * Split a person's message into the skill they invoked and the rest of what they typed. * @@ -205,19 +236,27 @@ function RestoringTranscript() { * A person who has just asked something watches their own message sit there, and a Bot that is * thinking is indistinguishable from a Bot that failed silently — which this app has shipped before. * - * It borrows the shimmer a running tool line uses, so "working on it" reads the same whether the - * work is a tool call or a model that has not spoken yet. + * It is the Bot's bubble with three dots in it, where its first words will land, the way a chat + * shows somebody typing. The dots hold still with reduced motion (`.typing-dot` in styles.css). */ function Thinking() { return ( -

- Thinking -

+ Thinking + {[0, 1, 2].map((dot) => ( + + ))} +
); } @@ -366,8 +405,12 @@ function Queued({ * message sent by mistake rather than as a file waiting its turn. */} {text ? ( - - + + {/* Shown exactly as typed, for the same reason a sent message is. */} {text} @@ -752,23 +795,16 @@ const TranscriptMessage = memo(function TranscriptMessage({ - {/* - A Bot's message takes the whole column, not the width of its words: block content - inside it — a fenced code block, a table — should span the transcript rather than - shrink to its own text. A person's bubble keeps fitting what they said. - */} - - - {isUser ? ( - // A person's own message is shown exactly as they typed it. Rendering it as markdown - // would silently reformat what they said, and an asterisk in a sentence is not - // emphasis. The chip is the one exception, and it is not reformatting: it is drawing - // the thing that was already a chip in the composer as a chip here too, so the - // transcript shows a skill was used rather than a slash that was typed. + {isUser ? ( + + + {/* + * A person's own message is shown exactly as they typed it. Rendering it as + * markdown would silently reformat what they said, and an asterisk in a sentence + * is not emphasis. The chip is the one exception, and it is not reformatting: it is + * drawing the thing that was already a chip in the composer as a chip here too, so + * the transcript shows a skill was used rather than a slash that was typed. + */} {origin && ( @@ -792,28 +828,75 @@ const TranscriptMessage = memo(function TranscriptMessage({ said )} - ) : ( - /* - * A Bot's prose is markdown, and it arrives in pieces. - * - * Rendered with a streaming-aware renderer rather than an ordinary one: half a fenced - * code block or an unclosed bold marker is the NORMAL state for most of a run, and a - * plain markdown parser draws that as literal asterisks and backticks until the - * closing token arrives, so the answer visibly rewrites itself as it lands. This - * closes them for the duration. - */ - - {revealed} - - )} - - + + + ) : ( + + )} ); }); +/** + * A Bot's reply as separate bubbles, one per thought and 8px apart, the way a chat sends them + * (`splitBubbles`). + * + * ONE MEMOISED BUBBLE PER PART. A streamed reply changes only its last part on each chunk, so the + * parts before it compare equal and their markdown is not parsed again: the same reasoning as + * `TranscriptMessage`, one level down. The split itself is one pass over the text and runs whenever + * the text changes, which is the only time this draws at all. + */ +function AgentReply({ + mode, + text, +}: { + mode: "static" | "streaming"; + text: string; +}) { + const parts = splitBubbles(text); + return ( +
+ {parts.map((part, index) => ( + // biome-ignore lint/suspicious/noArrayIndexKey: a reply's thoughts keep their order as it streams. + + ))} +
+ ); +} + +const AgentBubble = memo(function AgentBubble({ + mode, + part, +}: { + mode: "static" | "streaming"; + part: string; +}) { + return ( + + + {/* + * A Bot's prose is markdown, and it arrives in pieces. + * + * Rendered with a streaming-aware renderer rather than an ordinary one: half a fenced code + * block or an unclosed bold marker is the NORMAL state for most of a run, and a plain + * markdown parser draws that as literal asterisks and backticks until the closing token + * arrives, so the answer visibly rewrites itself as it lands. This closes them for the + * duration. + */} + + {part} + + + + ); +}); + /** * The one shape an attachment url is allowed to have, taken from the helper that builds it rather * than written out again, so the check cannot drift from the route that serves the file. @@ -1407,6 +1490,9 @@ const ToolCallBody = memo(function ToolCallBody({ spokeAfter: boolean; }) { const renderToolCall = useRenderToolCall(); + // Held on its approval card: the same inbox the card under it reads. + const waiting = useHeldForPerson().has(toolCallId); + const conversationWaiting = useConversationWaiting(); const toolCall = useMemo( () => ({ id: toolCallId, @@ -1450,13 +1536,23 @@ const ToolCallBody = memo(function ToolCallBody({ * * What was called, shimmering until its result arrives, and then the server's own words * drawn the way a Bot's prose is drawn. + * + * WITH ITS APPROVAL CARD UNDER IT when the server held the call for the person. A run of + * steps gets its cards beside the folded line (`ToolActivity`); a call alone in its turn + * keeps this line instead, and without the card here the request waited only on the Bot's + * page. A tool the app draws itself is left alone: its own renderer places its card. */} {drawn ?? ( - + <> + + + )} @@ -1546,14 +1642,27 @@ export function ServerToolLine({ name, result, place, + waiting = false, + conversationWaiting = false, }: { name: string; result?: string; /** Where this call sits in the conversation, for an app card that may carry the task on. */ place?: CardPlace; + /** Held on its approval card for the person's answer, so it is not running yet. */ + waiting?: boolean; + /** An approval in this conversation waits on the person, which a call repeating it waits with. */ + conversationWaiting?: boolean; }) { - const running = result === undefined; - const { label, detail } = readToolName(name, running); + const heldBack = toolResultHeldBack(result); + // Said no on its card: nothing was done, and the line says so rather than naming it as done. + const declined = heldBack === "declined"; + // Not asked again, because the same request already waits on its card. Not done either way. + const alreadyWaiting = heldBack === "alreadyWaiting"; + const held = + (waiting && result === undefined) || + (alreadyWaiting && conversationWaiting); + const running = result === undefined && !held; /* * A refusal is not a result, and must not read like one. * @@ -1562,8 +1671,13 @@ export function ServerToolLine({ * any guess made here. See REFUSAL_MARKER in server/src/plugins/tools.ts. */ const answer = result === undefined ? undefined : asText(result); - const refused = answer?.startsWith(REFUSAL_MARKER) ?? false; - const failed = !refused && toolResultFailed(result); + const refused = !heldBack && (answer?.startsWith(REFUSAL_MARKER) ?? false); + const failed = !refused && !heldBack && toolResultFailed(result); + // A call that did not happen, or waits on its card, is named plainly ("Send your email"). + const { label, detail } = readToolName( + name, + refused || failed || heldBack !== null || held ? "plain" : running, + ); /* * The marker is for this component, not for the reader. Left in, a refusal reads "Blocked" in the * label and then "Refused." again in the first two words of the body, which is the same fact three @@ -1590,16 +1704,21 @@ export function ServerToolLine({ * THAT TOOL'S OTHER ANSWERS ARE FOR THE MODEL. An app nobody can name comes back as plain text * telling the Bot to say so plainly and not to browse to it — an instruction, which read by the * person looks like the Bot being told off in front of them. The line still says the check - * happened; the Bot's own reply says what it found. + * happened; the Bot's own reply says what it found. A call the person held back answers the same + * kind of thing: it tells the Bot not to try again, and the line has already said what happened. */ - const shown = name === SHOW_APP_ACCESS_TOOL ? undefined : body; + const shown = + name === SHOW_APP_ACCESS_TOOL || heldBack !== null ? undefined : body; return ( {shown ? ( diff --git a/app/src/components/channels/composer/attachment-strip.tsx b/app/src/components/channels/composer/attachment-strip.tsx index 90ba773333..b752533518 100644 --- a/app/src/components/channels/composer/attachment-strip.tsx +++ b/app/src/components/channels/composer/attachment-strip.tsx @@ -144,8 +144,8 @@ export function AttachmentStrip({ * ON THE SIZE LINE, NOT UNDER IT, AND THAT IS A LAYOUT CONSTRAINT RATHER THAN A * PREFERENCE. The tile's height is fixed so that a file and a thumbnail read as * one row. A third line does not fit: with three lines the two in-flow children - * need 65px (a 20px icon and three 15px lines) inside a 62px box. Something - * has to give, and the only choices are the icon shrinking to 17px or the text + * need 68px (a 20px icon and three 16px lines) inside a 62px box. Something + * has to give, and the only choices are the icon shrinking to 14px or the text * spilling past the border. Folding the warning onto this line keeps two. * * `truncate` is the guarantee, not the wording: `formatBytes` is unbounded, so a diff --git a/app/src/components/channels/composer/composer.tsx b/app/src/components/channels/composer/composer.tsx index f33ca00e3a..665560d0e3 100644 --- a/app/src/components/channels/composer/composer.tsx +++ b/app/src/components/channels/composer/composer.tsx @@ -13,6 +13,7 @@ import { isSegmentsEmpty, mergeAdjacentTextSegments, type Segment, + segmentsToPlainText, text, } from "prompt-area/helpers"; import { @@ -115,9 +116,9 @@ function stagedModality(attachment: Attachment) { const MAX_HEIGHT_PX = 220; /** - * Tracks the compact `text-sm` line box so PromptArea stays vertically centered in one row. + * Tracks the compact editor's 23px line box so PromptArea stays vertically centered in one row. */ -const COMPACT_MIN_HEIGHT_PX = 19; +const COMPACT_MIN_HEIGHT_PX = 23; const COMPACT_MAX_HEIGHT_PX = 96; export type ComposerProps = { @@ -243,6 +244,14 @@ export type ComposerProps = { * that can sum this; a caller with no queue leaves it out and nothing changes. */ queuedAttachmentCount?: number; + /** What the empty box says. "Message Ana" in a conversation with one Bot; "Ask anything" elsewhere. */ + placeholder?: string; + /** + * Called with the draft's plain text whenever it changes: typing, dictation, a send clearing the + * box and a failed send putting the words back alike. A conversation's header orb listens + * through it. + */ + onTextChange?: (text: string) => void; }; /** @@ -328,6 +337,8 @@ export function Composer({ droppedAttachments, // Nothing parked is the right answer for every caller without a queue, which is most of them. queuedAttachmentCount = 0, + placeholder = "Ask anything", + onTextChange, }: ComposerProps) { const [value, setValue] = useState( initialValue ? [{ type: "text", text: initialValue }] : [], @@ -1046,6 +1057,9 @@ export function Composer({ [agents, commands], ); const draft = useMemo(() => toDraft(value, staged), [staged, value]); + useEffect(() => { + onTextChange?.(segmentsToPlainText(value)); + }, [onTextChange, value]); /** * MORE FILES ON THIS STRIP THAN ONE MESSAGE MAY CARRY — a state the screening is supposed to make @@ -1337,9 +1351,10 @@ export function Composer({ const sendLabel = parking ? "Queue message" : "Send message"; const callAction = draft.isEmpty && draft.attachments.length === 0 ? voiceCall : undefined; + // The compact composer's controls are 44px, the smallest thing a finger can hit. const actionClassName = cn( "rounded-full p-0", - compact ? "size-8 self-end" : "size-7", + compact ? "size-(--target) shrink-0 self-end" : "size-7", ); const primaryAction = canStop ? ( ) : callAction ? ( ) : ( ); @@ -1484,10 +1499,10 @@ export function Composer({ data-slot="composer" className={cn( /* - * `py-3` IS LOAD-BEARING ONCE THE TEXT WRAPS. On one line `min-h-14` and the controls - * row's own `items-center` fake the vertical padding, so it read as correct for as long - * as nobody typed a paragraph. Past that the box grows to fit its content exactly and - * the glyphs sit against the border. + * THE PADDING IS LOAD-BEARING ONCE THE TEXT WRAPS. On one line the controls row's own + * `items-center` fakes the vertical padding, so it read as correct for as long as + * nobody typed a paragraph. Past that the box grows to fit its content exactly and the + * glyphs sit against the border. * * It goes on the form rather than on the editor because the editor scrolls internally * at COMPACT_MAX_HEIGHT_PX: padding inside that box would scroll away with the text, so @@ -1499,7 +1514,12 @@ export function Composer({ * started it 42px in from the frame's left edge with nothing under it. The strip is * its own row across the top now, and the three controls keep their row below it. */ - "flex min-h-14 flex-col rounded-2xl border border-border bg-card px-3 py-3 focus-within:border-ring focus-within:ring-3 focus-within:ring-ring/50", + /* + * A pill on a soft shadow, 62px tall around its 44px controls. + * The border is there but clear, so a skin can draw it in a colour of its own, and it + * takes the ring colour, with the ring around it, while the caret is inside. + */ + "flex min-h-[62px] flex-col rounded-[27px] border border-transparent bg-card p-2 shadow-(--shadow-float) focus-within:border-ring focus-within:ring-3 focus-within:ring-ring/50", className, )} onSubmit={handleFormSubmit} @@ -1518,11 +1538,11 @@ export function Composer({ * the middle of that block rather than sitting on the line the person is typing. Empty, * the row is exactly a button tall, so centred and bottom are the same pixel. */} -
+
{attachmentsEnabled ? ( ) : ( )} {/* The slot is ours: `PromptArea` passes no attribute through to its own box. */} -
+
@@ -1621,7 +1650,7 @@ export function Composer({ onChange={handleChange} onImagePaste={canAttach ? stagePastedImage : undefined} onSubmit={submitDraft} - placeholder="Ask anything" + placeholder={placeholder} ref={promptAreaRef} triggers={triggers} value={value} diff --git a/app/src/components/channels/composer/dictation-controls.tsx b/app/src/components/channels/composer/dictation-controls.tsx index 2cd2a7c6af..35976001a4 100644 --- a/app/src/components/channels/composer/dictation-controls.tsx +++ b/app/src/components/channels/composer/dictation-controls.tsx @@ -18,9 +18,12 @@ type Dictation = ReturnType; export function DictationButton({ dictation, disabled, + className, }: { dictation: Dictation; disabled: boolean; + /** Merged over the button's own: the compact composer makes it a 44px control. */ + className?: string; }) { if (!dictation.available) return null; return ( @@ -32,7 +35,7 @@ export function DictationButton({ ? "Dictate a message" : "Dictation needs a supported browser and a secure (HTTPS) connection" } - className="size-8 shrink-0 self-end rounded-full" + className={cn("size-8 shrink-0 self-end rounded-full", className)} disabled={disabled || !dictation.supported || dictation.busy} onClick={() => void dictation.session.start()} size="icon" diff --git a/app/src/components/channels/conversation-presence.tsx b/app/src/components/channels/conversation-presence.tsx new file mode 100644 index 0000000000..fcd4b1afc6 --- /dev/null +++ b/app/src/components/channels/conversation-presence.tsx @@ -0,0 +1,496 @@ +import type { Message } from "@ag-ui/core"; +import { useInfiniteQuery, useQuery } from "@tanstack/react-query"; +import { + createContext, + type ReactNode, + useCallback, + useContext, + useEffect, + useMemo, + useRef, + useState, + useSyncExternalStore, +} from "react"; +import type { BotOrbState } from "@/components/agents/orb/bot-orb"; +import { ChannelAvatar } from "@/components/channels/avatar"; +import { approvalInboxOptions } from "@/lib/approvals"; +import { channelListQueryOptions } from "@/lib/channels/queries"; +import { + type BrowserStep, + browserSteps, + type StepApproval, + stepFor, + turnWaitsOnPerson, +} from "@/lib/channels/step"; +import { cn } from "@/lib/utils"; + +/** + * What one conversation is doing, for its header: the Bot's orb works while the Bot has the turn, + * listens while the person types and lands an answer, and the line under the conversation's name + * says which step the Bot is on, or that it needs the person. + * + * The conversation route provides it and the conversation writes into it (`useReportConversation`), + * so everything on the route reads one conversation and never another's. Readers: + * + * - `useConversationOrb()`: the Bot's orb state. `ConversationOrb` draws it. + * - `useConversationStatus()`: resting, working (with the step, or null for work the server is doing + * that this tab did not start), or needs you. `ConversationStatusLine` draws it. + * - `useConversationSteps()`: the browser steps the Bot took, for the computer pane's step log. + * + * The conversation's reports live in a store outside React, and each reader subscribes to the + * fields it uses, so a keystroke reaches the orb and nothing else, and a new step reaches the line + * and not the orb. What the server says (the conversation list's busy flag, an approval waiting in + * this thread) is read here from the queries the app already keeps. + */ + +/** How long the orb holds "done": its smile springs up and settles away. */ +export const LANDING_MS = 2600; + +/** What the conversation last reported. */ +type Reported = { + /** + * The conversation reporting, null when none is mounted. Readers count a report only from the + * conversation on screen: on the way to another one, the last one's report is still here for a + * moment, until it unmounts and takes it away. + */ + channelId: string | null; + /** The Bot has the turn: asked and not answered yet, browser actions in the middle included. */ + working: boolean; + /** What `stepFor` says about the turn, null when there is none. */ + step: string | null; + /** The composer has something typed in it. */ + drafting: boolean; + /** Counts the turns that ended in an answer, rather than a failure or a Stop. */ + answers: number; + /** The browser steps in the conversation, oldest first. */ + steps: readonly ConversationStep[]; +}; + +/** A browser step, and when this tab first saw it, which is where it sits among the commands. */ +export type ConversationStep = BrowserStep & { at: number }; + +const NO_STEPS: readonly ConversationStep[] = []; + +const AT_REST: Reported = { + channelId: null, + working: false, + step: null, + drafting: false, + answers: 0, + steps: NO_STEPS, +}; + +/** + * When this tab first saw each browser step, by its call. + * + * MODULE SCOPE, SO A STEP KEEPS ITS PLACE. The command log beside these steps lives for the session, + * and a conversation opened again would otherwise stamp every step it already showed with the moment + * it came back, after commands that ran later. A step from before this tab opened is stamped when + * the conversation first showed it. Bounded, oldest out. + */ +const FIRST_SEEN = new Map(); +const MAX_FIRST_SEEN = 1000; + +function firstSeen(callId: string, now: number): number { + const seen = FIRST_SEEN.get(callId); + if (seen !== undefined) return seen; + FIRST_SEEN.set(callId, now); + while (FIRST_SEEN.size > MAX_FIRST_SEEN) { + const oldest = FIRST_SEEN.keys().next().value; + if (oldest === undefined) break; + FIRST_SEEN.delete(oldest); + } + return now; +} + +/** The same steps, unchanged: a streamed token rewrites the array and usually nothing in it. */ +function sameSteps( + current: readonly ConversationStep[], + next: readonly BrowserStep[], +): boolean { + return ( + current.length === next.length && + current.every((step, index) => { + const other = next[index]; + return ( + other !== undefined && + step.item.id === other.item.id && + step.item.result === other.item.result && + step.item.toolCall.function.arguments === + other.item.toolCall.function.arguments && + step.latest === other.latest + ); + }) + ); +} + +type Store = { + read: () => Reported; + write: (next: (current: Reported) => Reported) => void; + subscribe: (listener: () => void) => () => void; +}; + +function createStore(): Store { + let current = AT_REST; + const listeners = new Set<() => void>(); + return { + read: () => current, + write(next) { + const value = next(current); + if (value === current) return; + current = value; + for (const listener of listeners) listener(); + }, + subscribe(listener) { + listeners.add(listener); + return () => listeners.delete(listener); + }, + }; +} + +/** What the server says about this conversation. */ +type ServerFacts = { + /** The server has a turn running here that this tab did not start. */ + working: boolean; + /** An approval in this conversation is waiting on the person. */ + waiting: boolean; + /** This conversation's approvals, so a declined step stops being named before its result. */ + approvals: readonly StepApproval[]; +}; + +const NOTHING_ON_SERVER: ServerFacts = { + working: false, + waiting: false, + approvals: [], +}; + +/** The store, and the conversation on screen. */ +type Presence = { store: Store; channelId: string }; + +const StoreContext = createContext(null); +const ServerContext = createContext(NOTHING_ON_SERVER); + +const unsubscribed = () => () => {}; + +/** + * One field of what the conversation on screen reported, at rest if it has not. Re-renders only + * when that field changes. + */ +function useReported( + presence: Presence | null, + key: K, +): Reported[K] { + return useSyncExternalStore(presence?.store.subscribe ?? unsubscribed, () => { + const reported = presence?.store.read(); + return reported && reported.channelId === presence?.channelId + ? reported[key] + : AT_REST[key]; + }); +} + +/** + * `threadId` is the conversation's thread, for the approvals waiting in it; undefined while the + * conversation loads. The route stays mounted from one conversation to the next, and so does this: + * the conversation leaving takes its reports with it, and `channelId` moves what the server says. + */ +export function ConversationPresenceProvider({ + channelId, + threadId, + children, +}: { + channelId: string; + threadId: string | undefined; + children: ReactNode; +}) { + const [store] = useState(createStore); + const presence = useMemo(() => ({ store, channelId }), [store, channelId]); + const channels = useInfiniteQuery(channelListQueryOptions()); + // The conversation's one poll of the inbox. Its rows and cards read the cache this keeps fresh. + const inbox = useQuery(approvalInboxOptions()); + const row = channels.data?.find((channel) => channel.id === channelId); + // The sidebar's rule (`ChannelRow`), so the header and the conversation's row agree. + const busy = (row?.busy ?? false) || (row?.work?.length ?? 0) > 0; + const working = useUnownedBusy( + channelId, + busy, + useReported(presence, "working"), + ); + const approvals = useMemo( + () => + threadId === undefined + ? [] + : (inbox.data?.requests ?? []).filter( + (request) => request.action.threadId === threadId, + ), + [inbox.data, threadId], + ); + const waiting = approvals.some((request) => request.status === "pending"); + const facts = useMemo( + () => ({ working, waiting, approvals }), + [working, waiting, approvals], + ); + return ( + + {children} + + ); +} + +/** + * The busy flag for work this tab did not start: false while the conversation's own turn runs, and + * for the rest of that busy spell. The server's flag comes down a moment after the tab's own turn + * ends, and without this every answer would end on "Working" as the flag caught up. A run the server + * starts on its own afterwards, such as the one after Allow once, raises a new spell, and shows. + * A spell belongs to one conversation, so opening another starts again. + */ +function useUnownedBusy( + channelId: string, + busy: boolean, + ownTurn: boolean, +): boolean { + const spell = useRef({ channelId, owned: false }); + if (spell.current.channelId !== channelId) + spell.current = { channelId, owned: false }; + if (!busy) spell.current.owned = false; + else if (ownTurn) spell.current.owned = true; + return busy && !spell.current.owned; +} + +/** + * The Bot's orb state for what the conversation is doing. The Bot working outranks the person + * typing, since a correction typed mid-answer waits for the turn to end; typing cuts a landing + * short, and so does an approval waiting here, since the line then says "Needs you". Resting + * outside a conversation route. + */ +export function useConversationOrb(): BotOrbState { + const presence = useContext(StoreContext); + const server = useContext(ServerContext); + const present = useReported(presence, "channelId") !== null; + const working = useReported(presence, "working"); + const drafting = useReported(presence, "drafting"); + // Read whatever conversation it came from: it only counts, and a count from before is landed. + const answers = useSyncExternalStore( + presence?.store.subscribe ?? unsubscribed, + () => presence?.store.read().answers ?? 0, + ); + // The answer whose landing has played out. Behind `answers` while the orb holds "done". + const [landed, setLanded] = useState(answers); + useEffect(() => { + if (answers === landed) return; + // With the conversation gone the landing is over, so coming straight back does not replay it. + if (!present) { + setLanded(answers); + return; + } + const timer = setTimeout(() => setLanded(answers), LANDING_MS); + return () => clearTimeout(timer); + }, [answers, present, landed]); + if (working || server.working) return "thinking"; + if (drafting) return "listening"; + if (server.waiting) return "idle"; + return present && answers !== landed ? "done" : "idle"; +} + +/** How the conversation stands, for the line under its name. */ +export type ConversationStatus = + | { kind: "resting" } + /** `step` is what the Bot is on, or null for a turn the server is running that this tab did not start. */ + | { kind: "working"; step: string | null } + | { kind: "needs-you" }; + +/** + * How the conversation stands. An approval waiting comes first, since the Bot cannot go on with it + * until the person decides. Keystrokes don't reach it. + */ +export function useConversationStatus(): ConversationStatus { + const presence = useContext(StoreContext); + const server = useContext(ServerContext); + const working = useReported(presence, "working"); + const step = useReported(presence, "step"); + if (server.waiting) return { kind: "needs-you" }; + if (working) return { kind: "working", step }; + if (server.working) return { kind: "working", step: null }; + return { kind: "resting" }; +} + +/** + * Whether an approval in this conversation is waiting on the person. A call the server did not ask + * about again, because the same request was already waiting, waits with it while this is true. + * False outside a conversation route. + */ +export function useConversationWaiting(): boolean { + return useContext(ServerContext).waiting; +} + +/** + * The browser steps the conversation's Bot took, oldest first, each with when this tab first saw it. + * None outside a conversation route. + */ +export function useConversationSteps(): readonly ConversationStep[] { + return useReported(useContext(StoreContext), "steps"); +} + +/** + * Report a conversation's turn, draft and browser steps, for as long as it is mounted. Outside a + * conversation route there is nothing to report to, and it returns undefined. + * + * `working` is the turn, not the run, for the reason `ConversationView` gives: a browser action ends + * one run and starts another, and the orb would land in the gap. `failed` is the turn having ended + * without an answer. The step is read off `messages` on every render, as the transcript reads them, + * because the agent grows the same array in place. Hand the composer `onTextChange`, and call + * `onStop` when somebody presses Stop, so a stopped answer does not land either. Nor does a turn + * that ended on a call held for the person, which has no answer yet. + */ +export function useReportConversation({ + working, + failed, + messages, +}: { + working: boolean; + failed: boolean; + messages: readonly Message[]; +}): { onTextChange: (text: string) => void; onStop: () => void } | undefined { + const presence = useContext(StoreContext); + const { approvals } = useContext(ServerContext); + const step = presence ? stepFor(messages, working, approvals) : null; + // Read on every render for the same reason as the step. + const steps = presence ? browserSteps(messages) : NO_STEPS; + // Asked only as the turn ends, so it is not read again on every token of the answer. + const waitsOnPerson = + presence && !working ? turnWaitsOnPerson(messages) : false; + const wasWorking = useRef(false); + const stopped = useRef(false); + /* + * Every report is this conversation's, by the channel on screen when it mounted. The route keys + * the conversation on its channel, so that is the channel it stays. + */ + const store = presence?.store; + const mine = presence?.channelId ?? null; + const report = useCallback( + (change: (current: Reported) => Partial | null) => + store?.write((current) => { + // What another conversation reported is not this one's, apart from the count of answers. + const own = + current.channelId === mine + ? current + : { ...AT_REST, answers: current.answers, channelId: mine }; + const changed = change(own); + return changed ? { ...own, ...changed } : own; + }), + [store, mine], + ); + + // First, so leaving takes everything with it: an answer nobody is there to see does not land. + useEffect(() => { + if (!store) return; + report(() => null); + return () => + store.write((current) => + current.channelId === mine + ? { ...AT_REST, answers: current.answers } + : current, + ); + }, [store, mine, report]); + + useEffect(() => { + if (working && !wasWorking.current) stopped.current = false; + const answered = + wasWorking.current && + !working && + !failed && + !stopped.current && + !waitsOnPerson; + wasWorking.current = working; + report((current) => + current.working === working && !answered + ? null + : { working, answers: current.answers + (answered ? 1 : 0) }, + ); + }, [report, working, failed, waitsOnPerson]); + + useEffect(() => { + report((current) => (current.step === step ? null : { step })); + }, [report, step]); + + useEffect(() => { + report((current) => { + if (sameSteps(current.steps, steps)) return null; + const now = Date.now(); + return { + steps: steps.map((browser) => ({ + ...browser, + at: firstSeen(browser.item.id, now), + })), + }; + }); + }, [report, steps]); + + const onTextChange = useCallback( + (text: string) => { + const drafting = text.trim() !== ""; + report((current) => + current.drafting === drafting ? null : { drafting }, + ); + }, + [report], + ); + const onStop = useCallback(() => { + stopped.current = true; + }, []); + + return store ? { onTextChange, onStop } : undefined; +} + +/** + * The Bot's orb, in the state its conversation is in. Its own component, so a keystroke redraws + * the orb and nothing around it. + */ +export function ConversationOrb({ + participantIds, + size, +}: { + participantIds: string[]; + size: number; +}) { + return ( + + ); +} + +/** + * The line under the conversation's name: "Working · Searching the web" while the Bot works, + * "Working" alone for a run the server resumed, "Needs you" while an approval waits here, and + * nothing at rest. Empty, it takes no height, so the name sits level with the orb. + * + * Always mounted, so the live region is there before the first step arrives: a region added with + * its text already in it is not announced. The step shimmers as a running tool line does, and stops + * for somebody who asked for less motion. + */ +export function ConversationStatusLine({ className }: { className?: string }) { + const status = useConversationStatus(); + return ( +

+ {status.kind === "needs-you" ? ( + Needs you + ) : status.kind === "working" ? ( + <> + Working + {status.step ? ( + <> + {" · "} + {status.step} + + ) : null} + + ) : null} +

+ ); +} diff --git a/app/src/components/channels/conversation-view.tsx b/app/src/components/channels/conversation-view.tsx index f0ff124adf..ea63f06ee9 100644 --- a/app/src/components/channels/conversation-view.tsx +++ b/app/src/components/channels/conversation-view.tsx @@ -7,6 +7,7 @@ import { useRef, useState, } from "react"; +import { AiNotice } from "@/components/channels/ai-notice"; import { ChatTranscript } from "@/components/channels/chat-transcript"; import { type AgentOption, @@ -18,6 +19,7 @@ import { type QueuedMessage, reduceQueue, } from "@/components/channels/composer"; +import { useReportConversation } from "@/components/channels/conversation-presence"; import { attachmentUrl } from "@/lib/channels/attachments"; import { newId } from "../../lib/new-id"; @@ -39,6 +41,7 @@ export function ConversationView({ onSubmit, onStop, voiceCall, + placeholder, }: { messages: readonly Message[]; busy?: boolean; @@ -97,6 +100,8 @@ export function ConversationView({ /** Stop the Bot mid-answer; forwarded to turn the send button into a stop button. */ onStop?: () => void; voiceCall?: { active: boolean; supported: boolean; onStart(): void }; + /** The composer's empty-state text, e.g. "Message Ana". The composer's own when absent. */ + placeholder?: string; }) { /* * THE QUEUE LIVES HERE BECAUSE BOTH HALVES OF IT DO. @@ -154,6 +159,16 @@ export function ConversationView({ */ const [running, setRunning] = useState(false); const inFlight = pending || running; + /* + * The conversation's header follows it: the orb thinks for the turn and listens to the draft, and + * the line under the name says the step. Not while history is restoring: joining the thread is a + * run of its own, and it is not an answer. Outside a conversation route nothing is reported. + */ + const presence = useReportConversation({ + working: (busy || inFlight) && !restoring, + failed: Boolean(stopped), + messages, + }); /** * Every change to the queue goes through here, so the ref and the state can never disagree. @@ -439,7 +454,7 @@ export function ConversationView({ />
{notice} @@ -460,8 +475,17 @@ export function ConversationView({ } : undefined } - onStop={onStop} + onStop={ + onStop && presence + ? () => { + presence.onStop(); + onStop(); + } + : onStop + } onSubmit={(draft) => submit(draft, false)} + {...(presence ? { onTextChange: presence.onTextChange } : {})} + {...(placeholder ? { placeholder } : {})} /* * `inFlight` rather than the `pending` this was given. A drained turn is started from the * effect above rather than from the composer, so the composer's own send tracking knows @@ -502,6 +526,7 @@ export function ConversationView({ */ stoppable={stoppable ?? pending} /> +
); diff --git a/app/src/components/channels/group-chat.tsx b/app/src/components/channels/group-chat.tsx index 2515ce93d2..df032f2e03 100644 --- a/app/src/components/channels/group-chat.tsx +++ b/app/src/components/channels/group-chat.tsx @@ -16,6 +16,7 @@ import { } from "@/lib/groups"; import { markdownComponents } from "@/lib/markdown"; import { newId } from "@/lib/new-id"; +import { AiNotice } from "./ai-notice"; import { ChannelAvatar } from "./avatar"; import { Composer, toAgentOptions } from "./composer"; import { TeamBotConsentCard } from "./team-bot-consent"; @@ -142,6 +143,7 @@ export function GroupChat({ channelId }: { channelId: string }) { }); }} /> +
); diff --git a/app/src/components/channels/split-bubbles.ts b/app/src/components/channels/split-bubbles.ts new file mode 100644 index 0000000000..af3ab4c216 --- /dev/null +++ b/app/src/components/channels/split-bubbles.ts @@ -0,0 +1,66 @@ +/** + * A Bot's reply as the separate thoughts a chat draws as separate bubbles: one bubble per paragraph, + * 8px apart, as a person texting would send them, instead of one tall bubble holding everything. + * + * Paragraphs are split on blank lines, with three exceptions that keep markdown whole: + * - nothing is split inside a fenced code block, including one still streaming in; + * - a list stays with the line that introduces it ("Three things I could start on:"), and its items + * stay together even when the model put blank lines between them; + * - a table stays in one piece. + */ +export function splitBubbles(text: string): string[] { + const blocks: string[] = []; + let current: string[] = []; + let fenced = false; + const flush = () => { + const block = current.join("\n").trim(); + if (block) blocks.push(block); + current = []; + }; + for (const line of text.split("\n")) { + if (FENCE.test(line)) fenced = !fenced; + if (!fenced && line.trim() === "") { + flush(); + continue; + } + current.push(line); + } + flush(); + + const bubbles: string[] = []; + for (const block of blocks) { + const previous = bubbles.at(-1); + if (previous !== undefined && belongsWith(previous, block)) { + bubbles[bubbles.length - 1] = `${previous}\n\n${block}`; + } else { + bubbles.push(block); + } + } + return bubbles.length > 0 ? bubbles : [text]; +} + +/** + * Whether a bubble holds a fenced code block or a table, which wants the whole column on a wide + * screen: a line of code or a row of cells cut at a prose width scrolls sideways inside its bubble. + */ +export function holdsBlock(bubble: string): boolean { + return bubble + .split("\n") + .some((line) => FENCE.test(line) || TABLE_ROW.test(line)); +} + +const FENCE = /^\s{0,3}(```|~~~)/; +const LIST_ITEM = /^\s{0,3}([-*+]|\d+[.)])\s/; +const TABLE_ROW = /^\s*\|/; + +/** Whether a block continues the bubble before it rather than starting one of its own. */ +function belongsWith(previous: string, block: string): boolean { + const lastLine = previous.split("\n").at(-1) ?? ""; + if (LIST_ITEM.test(block)) + return LIST_ITEM.test(lastLine) || lastLine.trimEnd().endsWith(":"); + if (TABLE_ROW.test(block)) return TABLE_ROW.test(lastLine); + // An indented block under a list item is that item's continuation. + if (/^\s{2,}\S/.test(block)) + return previous.split("\n").some((line) => LIST_ITEM.test(line)); + return false; +} diff --git a/app/src/components/channels/tool-activity.tsx b/app/src/components/channels/tool-activity.tsx index 013b3a9986..571f544dc7 100644 --- a/app/src/components/channels/tool-activity.tsx +++ b/app/src/components/channels/tool-activity.tsx @@ -1,4 +1,5 @@ import { CopilotKitContext } from "@copilotkit/react-core/v2/context"; +import { useQuery } from "@tanstack/react-query"; import { readConnectPrompt, SHOW_APP_ACCESS_TOOL, @@ -13,12 +14,14 @@ import { } from "@tabler/icons-react"; import { useCallback, useContext, useMemo, useSyncExternalStore } from "react"; import { Streamdown } from "streamdown"; -import { PendingApprovals } from "@/components/approvals/inline-approval"; +import { ToolCallApprovals } from "@/components/approvals/inline-approval"; +import { approvalInboxCacheOptions } from "@/lib/approvals"; import { markdownComponents } from "@/lib/markdown"; import { connectorName, humanise, ownToolLabel, + personToolLabel, readToolName, } from "@/lib/plugins/tool-name"; import { @@ -26,10 +29,16 @@ import { forDisplay, REFUSAL_MARKER, toolResultFailed, + toolResultHeldBack, } from "@/lib/plugins/tool-result"; import { readTeamBotConsent } from "@/lib/team-bots"; -import { browserStepDetails, isBrowserStep } from "./browser-activity"; +import { + browserOutcome, + browserStepDetails, + isBrowserStep, +} from "./browser-activity"; import type { VisibleChatItem } from "./chat-messages"; +import { useConversationWaiting } from "./conversation-presence"; import { ToolLine } from "./tool-line"; type ToolItem = Extract; @@ -218,18 +227,34 @@ type StepKind = "search" | "fetch" | "browser" | "connector" | "tool"; export type StepDetails = { id: string; kind: StepKind; + /** What it is doing while it runs, and what it did once it has. */ label: string; + /** What it would do, for a step that did not happen or has not yet: "Send your email". */ + plain: string; /** What it was done to: a query, a page, a connector. Muted beside the label. */ detail?: string; href?: string; pending: boolean; failed: boolean; refused: boolean; + /** The person said no on its card. Their decision, not an issue. */ + declined: boolean; + /** + * Not done, because the same request was already waiting on its card and was not asked again. It + * waits with that one, and is not an issue. + */ + alreadyWaiting: boolean; + /** A browser step cut off before it finished, by Stop or by its computer stopping. Not an issue. */ + stopped: boolean; + /** A browser step a computer still starting turned away, so nothing was done yet. Not an issue. */ + notYet: boolean; reason?: string; /** Pages a browser step opened, or a fetch read; zero for anything else. */ pages: number; /** The connector a connector step reached, said the way a person would. */ connector?: string; + /** What it did to the person's own things, once done: "Searched your email". */ + person?: string; inputs: [string, string][]; output?: string; }; @@ -286,15 +311,26 @@ export function stepDetails(step: ToolItem): StepDetails { ); if (isBrowserStep(step)) { const browser = browserStepDetails(step); + // Read the way the computer pane's step log reads it, so the two never disagree. + const outcome = browserOutcome(step); + // A step that didn't happen is named for what it would have done, never as done. + const happened = outcome === "done" || outcome === "pending"; + const plain = browser.plain ?? "Use the browser"; + const label = happened ? (browser.label ?? "Used the browser") : plain; return { id: step.id, kind: "browser", - label: browser.label ?? "Used the browser", + label, + plain, ...(browser.title ? { detail: browser.title } : {}), ...(browser.href ? { href: browser.href } : {}), - pending: browser.pending, - failed: browser.failed, - refused: false, + pending: outcome === "pending", + failed: outcome === "failed", + refused: outcome === "blocked", + declined: outcome === "declined", + alreadyWaiting: outcome === "alreadyWaiting", + stopped: outcome === "stopped", + notYet: outcome === "notYet", ...(browser.reason ? { reason: browser.reason } : {}), pages: browser.visited ? 1 : 0, inputs, @@ -303,33 +339,50 @@ export function stepDetails(step: ToolItem): StepDetails { const name = step.toolCall.function.name; const answer = step.result === undefined ? undefined : asText(step.result); - const refused = answer?.startsWith(REFUSAL_MARKER) ?? false; + // The person holding a call back is not a result either, and its words are for the model. + const heldBack = toolResultHeldBack(step.result); + const declined = heldBack === "declined"; + const alreadyWaiting = heldBack === "alreadyWaiting"; + const refused = !heldBack && (answer?.startsWith(REFUSAL_MARKER) ?? false); const body = refused ? answer?.slice(REFUSAL_MARKER.length).trim() : answer; // This deployment's own tools answer the model, not the person; see `ServerToolLine`. const own = ownToolLabel(name) !== null; + const failed = !refused && !heldBack && toolResultFailed(step.result); const outcome = { id: step.id, pending: step.result === undefined, - failed: !refused && toolResultFailed(step.result), + failed, refused, + declined, + alreadyWaiting, + stopped: false, + notYet: false, inputs, - ...(body && !own ? { output: body } : {}), + ...(body && !own && !heldBack ? { output: body } : {}), }; + // Running, a step says what is happening now; done, what happened. One that did not happen is + // named plainly ("Send your email"), never as though it had. + const state = + failed || refused || heldBack !== null ? "plain" : outcome.pending; // Not a connector: one of this deployment's own tools, or one an external agent runs itself. // Named in words, never as an identifier. - if (!name.startsWith("mcp__")) + if (!name.startsWith("mcp__")) { + const plain = humanise(name); return { ...outcome, kind: "tool", - label: ownToolLabel(name, outcome.pending) ?? humanise(name), + label: (state === "plain" ? null : ownToolLabel(name, state)) ?? plain, + plain, pages: 0, }; + } const [, server = "", ...rest] = name.split("__"); const role = researchRoleOfTool(server, rest.join("__")); - const { label, detail } = readToolName(name); + const { label, detail } = readToolName(name, state); + const plain = readToolName(name, "plain").label; const urls = role === "fetch" ? urlsOf(args) : []; - const base = { ...outcome, label }; + const base = { ...outcome, label, plain }; if (role === "search") { const query = firstString( args.query, @@ -361,15 +414,30 @@ export function stepDetails(step: ToolItem): StepDetails { pages: Math.max(urls.length, 1), }; } + const person = personToolLabel(name); return { ...base, kind: "connector", ...(detail ? { detail } : {}), pages: 0, connector: connectorName(server), + ...(person ? { person } : {}), }; } +/** It ran and it worked: the only kind of step a summary may say was done. */ +function isDone(step: StepDetails): boolean { + return ( + !step.pending && + !step.failed && + !step.refused && + !step.declined && + !step.alreadyWaiting && + !step.stopped && + !step.notYet + ); +} + function joinPhrases(phrases: string[]): string { if (phrases.length <= 1) return phrases[0] ?? ""; return `${phrases.slice(0, -1).join(", ")} and ${phrases.at(-1)}`; @@ -384,10 +452,15 @@ function plural(count: number, one: string, many: string): string { * * Counted by kind rather than listed, because the list is what the line opens to. Research is * never attributed to a provider; a connector is named, because which system was reached is the - * person's own fact. + * person's own fact, and what it did to the person's own mail or calendar is named in their words. + * + * ONLY WHAT HAPPENED. A step that was refused, failed or declined, or never finished, is not called + * done: the first two are counted as issues beside the line, and all of them say how they stand + * when it is opened. With nothing done, the line names the first step plainly. */ export function summarise(steps: readonly StepDetails[]): string { - const of = (kind: StepKind) => steps.filter((step) => step.kind === kind); + const done = steps.filter(isDone); + const of = (kind: StepKind) => done.filter((step) => step.kind === kind); const phrases: string[] = []; if (of("search").length) phrases.push("searched the web"); const read = of("fetch").reduce((sum, step) => sum + step.pages, 0); @@ -395,9 +468,23 @@ export function summarise(steps: readonly StepDetails[]): string { const browser = of("browser"); const browsed = browser.reduce((sum, step) => sum + step.pages, 0); if (browsed) phrases.push(`browsed ${plural(browsed, "page", "pages")}`); - else if (browser.length) phrases.push("used the browser"); + else if (browser.length) { + // Named as the computer's step log names them, up to two; past that, one phrase for all. + const actions = [...new Set(browser.map((step) => step.label))]; + if (actions.length <= 2) + phrases.push(...actions.map((label) => label.toLowerCase())); + else phrases.push("used the browser"); + } + for (const person of new Set( + of("connector").flatMap((step) => (step.person ? [step.person] : [])), + )) + phrases.push(person.charAt(0).toLowerCase() + person.slice(1)); const connectors = [ - ...new Set(of("connector").map((step) => step.connector ?? "")), + ...new Set( + of("connector") + .filter((step) => !step.person) + .map((step) => step.connector ?? ""), + ), ].filter(Boolean); if (connectors.length === 1) phrases.push(`used ${connectors[0]}`); else if (connectors.length > 1) @@ -406,6 +493,7 @@ export function summarise(steps: readonly StepDetails[]): string { ); const tools = of("tool").length; if (tools) phrases.push(`used ${plural(tools, "tool", "tools")}`); + if (phrases.length === 0) return steps[0]?.plain ?? ""; const sentence = joinPhrases(phrases); return sentence.charAt(0).toUpperCase() + sentence.slice(1); } @@ -453,29 +541,85 @@ function IconStack({ kinds }: { kinds: StepKind[] }) { ); } -function StepStatus({ step, active }: { step: StepDetails; active: boolean }) { +/** + * How a step stands, beside its words, for any step that did not simply work. The computer pane's + * step log says the same words (`step-log.tsx`). + */ +export const STEP_STATUS = { + waiting: "Waiting for you", + running: "In progress", + stopped: "Not completed", + notYet: "Not done yet", + declined: "Declined", + blocked: "Blocked", + failed: "Didn't work", +} as const; + +function StepStatus({ + step, + active, + waiting, +}: { + step: StepDetails; + active: boolean; + waiting: boolean; +}) { if (step.pending) return ( - {active ? "In progress" : "Not completed"} + {waiting + ? STEP_STATUS.waiting + : active + ? STEP_STATUS.running + : STEP_STATUS.stopped} + + ); + // Waits with the request it repeated, and once that is answered it was simply not done. + if (step.alreadyWaiting) + return ( + + {waiting ? STEP_STATUS.waiting : STEP_STATUS.stopped} + + ); + if (step.stopped || step.notYet || step.declined) + return ( + + {step.stopped + ? STEP_STATUS.stopped + : step.notYet + ? STEP_STATUS.notYet + : STEP_STATUS.declined} ); if (step.refused) - return Blocked; - if (step.failed) return ( - - Didn't work + + {STEP_STATUS.blocked} ); + if (step.failed) + return ( + {STEP_STATUS.failed} + ); return null; } -function StepRow({ step, active }: { step: StepDetails; active: boolean }) { +function StepRow({ + step, + active, + waiting, +}: { + step: StepDetails; + active: boolean; + /** Held on its approval card until the person answers, or waiting with the request it repeated. */ + waiting: boolean; +}) { + // Only a step that is running now says it is happening; one stopped or held says what it would do. + const label = step.pending && (waiting || !active) ? step.plain : step.label; const line = ( - {step.label} + {label} {step.href ? ( {step.detail} ) : null} - + ); - const reason = - step.failed && step.kind === "browser" ? ( -

- {step.reason ?? "Couldn't finish this step."} -

- ) : null; + // Why a browser step did not work, or the boundary's own words for why it was refused. + const why = + step.kind !== "browser" + ? undefined + : step.failed + ? (step.reason ?? "Couldn't finish this step.") + : step.refused + ? step.reason + : undefined; + const reason = why ? ( +

{why}

+ ) : null; // A browser step says everything it has in its line. Anything else opens to what it was given // and what came back, which no other place in the transcript shows. if (step.kind === "browser" || (!step.inputs.length && !step.output)) @@ -509,10 +659,10 @@ function StepRow({ step, active }: { step: StepDetails; active: boolean }) { return (
  • - + ▸ @@ -542,6 +692,27 @@ function StepRow({ step, active }: { step: StepDetails; active: boolean }) { ); } +/** + * The tool calls the server is holding on an approval card, read from the same inbox the cards are. + * + * A held step has no result yet, the same as one still running or one a stopped turn left behind, + * and only the inbox can tell them apart. Every row asks, so it reads the cache the screen polls. + */ +export function useHeldForPerson(): ReadonlySet { + const inbox = useQuery(approvalInboxCacheOptions()); + return useMemo( + () => + new Set( + (inbox.data?.requests ?? []).flatMap((request) => + request.status === "pending" && request.action.toolCallId + ? [request.action.toolCallId] + : [], + ), + ), + [inbox.data], + ); +} + export function ToolActivity({ group, active, @@ -551,10 +722,22 @@ export function ToolActivity({ active: boolean; }) { const steps = group.steps.map(stepDetails); + const held = useHeldForPerson(); + const waitingHere = useConversationWaiting(); + // On its own card, or with the identical request it repeated while that one still waits. + const waitsForPerson = (step: StepDetails) => + held.has(step.id) || (step.alreadyWaiting && waitingHere); const running = active && steps.some((step) => step.pending); const current = steps.find((step) => step.pending); const issues = steps.filter((step) => step.failed || step.refused).length; const kinds = [...new Set(steps.map((step) => step.kind))]; + /* + * NOTHING DONE IS NOT DONE. With no step that worked, the line is named for the first one plainly + * and marked for how the group stands, so it never carries the done mark. + */ + const nothingDone = !running && !steps.some(isDone); + // Nothing done yet because a step waits on its card, which is drawn beside this line. + const waiting = nothingDone && steps.some(waitsForPerson); // The folded line draws its own steps, so a waiting approval gets its card beside it, not inside. return ( <> @@ -563,14 +746,33 @@ export function ToolActivity({ label={running && current ? `${current.label}…` : summarise(steps)} detail={issues ? plural(issues, "issue", "issues") : undefined} running={running} + failed={nothingDone && steps.some((step) => step.failed)} + refused={ + nothingDone && + !steps.some((step) => step.failed) && + steps.some((step) => step.refused) + } + declined={nothingDone && !issues && steps.some((step) => step.declined)} + notYet={ + nothingDone && + !issues && + !waiting && + !steps.some((step) => step.declined) + } + waiting={waiting && !issues && !steps.some((step) => step.declined)} >
      {steps.map((step) => ( - + ))}
    - step.toolCall.id)} /> diff --git a/app/src/components/channels/tool-line.tsx b/app/src/components/channels/tool-line.tsx index 528d1d794e..dbc533f35f 100644 --- a/app/src/components/channels/tool-line.tsx +++ b/app/src/components/channels/tool-line.tsx @@ -1,4 +1,6 @@ +import { IconCircleCheck, IconClock, IconX } from "@tabler/icons-react"; import type { ReactNode } from "react"; +import { cn } from "@/lib/utils"; /** * One line for one thing a Bot did. @@ -17,10 +19,12 @@ export function ToolLine({ running, refused, failed, + declined, notYet, + waiting, children, }: { - /** Drawn before the label, small and muted. A group of steps puts its kinds here. */ + /** Drawn after the label, small and muted. A group of steps puts its kinds here. */ icon?: ReactNode; /** What was done, in a couple of words: "Searched Slack", "Filled in", "Read the page". */ label: string; @@ -31,63 +35,121 @@ export function ToolLine({ refused?: boolean; /** It was permitted and did not work. A different request might. */ failed?: boolean; + /** The person said no on its card, so it was not done. Their decision, not a failure. */ + declined?: boolean; /** * Nothing was done yet, for an ordinary reason: a computer still starting turned the call away * before it reached anything. Muted like a success, because nothing went wrong, and labelled so it * does not read as one. */ notYet?: boolean; + /** + * Held on its approval card until the person answers. Not running, and not done: muted, with the + * card under the line saying what it would do. + */ + waiting?: boolean; /** Shown when the line is expanded. Without it the line is not expandable. */ children?: ReactNode; }) { const tone = refused - ? "text-destructive" + ? "text-status-problem" : failed - ? "text-amber-600 dark:text-amber-500" + ? "text-status-needs" : "text-muted-foreground"; - + /* + * A hairline hook, then a mark saying how it went, then what was done. Only a line that worked + * gets the check, in the done colour; a group's own kinds come after the label, where they no + * longer compete with the outcome. + */ + const Mark = + refused || failed || declined + ? IconX + : running || notYet || waiting + ? IconClock + : IconCircleCheck; + const markTone = + refused || failed + ? tone + : running || notYet || waiting || declined + ? "text-muted-foreground" + : "text-status-done"; const text = ( - - {icon ? {icon} : null} - - {refused - ? `Blocked: ${label}` - : failed - ? `${label}, didn't work` - : notYet - ? `${label}, not done yet` - : label} + + + + + + + {/* Cut with an ellipsis rather than run past its row: a summary can be long. */} + + {declined + ? `You declined: ${label}` + : refused + ? `Blocked: ${label}` + : failed + ? `${label}, didn't work` + : notYet + ? `${label}, not done yet` + : waiting + ? `${label}, waiting for you` + : label} + + {detail ? {detail} : null} - {detail ? {detail} : null} + {icon ? ( + + {icon} + + ) : null} ); - if (!children) return
    {text}
    ; + if (!children) return
    {text}
    ; return ( -
    +
    {/* * Native
    owns expansion because SDK tool renderers can be re-invoked independently - * of this component's React state. + * of this component's React state. The summary is the line, a finger's height tall, and the + * chevron after the label says it opens. */} - + + {text} ▸ - {text} {/* + * Indented to the label, past the hook and the mark, so the detail reads as belonging to it. * Headings are cut down to the size of the surrounding text. A tool result is markdown a * server wrote for a model, so its `#` is a section marker rather than a page title. */} -
    +
    {children}
    diff --git a/app/src/components/computer/activity-log.tsx b/app/src/components/computer/activity-log.tsx index 417dc47b41..ffb75fbd74 100644 --- a/app/src/components/computer/activity-log.tsx +++ b/app/src/components/computer/activity-log.tsx @@ -9,33 +9,35 @@ * This session only. The record is the audit trail, which is on the server and survives a reload; * this is a window, and it is allowed to be empty when a tab is opened on a conversation that * already happened. + * + * The pane draws these among the Bot's browser steps, in one list (`step-log.tsx`). What is here is + * what only a command log can say: what each call is called, how it ended, and what it printed. */ -import { IconFile, IconFolder, IconTerminal2 } from "@tabler/icons-react"; -import { useSyncExternalStore } from "react"; -import { Empty, EmptyHeader, EmptyTitle } from "@/components/ui/empty"; -import { - activityFor, - type ComputerActivity, - subscribeToActivity, -} from "@/lib/computers/activity"; +import type { ComputerActivity } from "@/lib/computers/activity"; import { CommandOutput } from "./command-output"; -const ICONS = { - command: IconTerminal2, - read_file: IconFile, - write_file: IconFile, - list_files: IconFolder, +/** + * What the line says was done, in the words the call's own line in the conversation uses + * (`computer-tools.tsx`). The subject beside it says what it was done to. + */ +export const ACTIVITY_LABELS = { + command: "Ran a command", + read_file: "Read file", + write_file: "Saved file", + list_files: "Listed files", } as const; -/** What the line says was done. The subject beside it says what it was done to. */ -const LABELS = { - command: "Ran", - read_file: "Read", - write_file: "Saved", - list_files: "Listed", -} as const; +/** How a call ended: a boundary refused it, it exited non-zero, or it worked. */ +export function activityStatus( + entry: ComputerActivity, +): "done" | "blocked" | "failed" { + if (entry.refused === true) return "blocked"; + if (typeof entry.exitCode === "number" && entry.exitCode !== 0) + return "failed"; + return "done"; +} -function timeOf(at: number): string { +export function timeOf(at: number): string { return new Date(at).toLocaleTimeString(undefined, { hour: "2-digit", minute: "2-digit", @@ -43,84 +45,22 @@ function timeOf(at: number): string { }); } -function Entry({ entry }: { entry: ComputerActivity }) { - const Icon = ICONS[entry.kind]; - const failed = - entry.refused === true || - (typeof entry.exitCode === "number" && entry.exitCode !== 0); - - return ( -
  • -
    - - - {LABELS[entry.kind]} - - {/* - The command wraps rather than truncating. In the transcript one line per call is what keeps - it readable; here the whole point is to see exactly what ran, and a long command with its - middle cut out is the one thing this pane must not do. - */} - - {entry.subject} - - - {timeOf(entry.at)} - -
    - - {entry.refused === true ? ( -

    - {entry.output || "A boundary refused it."} -

    - ) : ( -
    - -
    - )} -
    • - ); -} - -export function ActivityLog({ computerId }: { computerId: string }) { - const entries = useSyncExternalStore( - subscribeToActivity, - () => activityFor(computerId), - () => activityFor(computerId), - ); - - if (entries.length === 0) { +/** What came back: the refusal's reason, or what the call printed. */ +export function ActivityOutput({ entry }: { entry: ComputerActivity }) { + if (entry.refused === true) return ( - - - - Nothing yet. Commands the Bot runs and files it reads appear here. - - - +

    + {entry.output || "A boundary refused it."} +

    ); - } - - // Newest first: what somebody watching wants is what just happened, without scrolling for it. - const newestFirst = [...entries].reverse(); - return ( -
      - {newestFirst.map((entry) => ( - - ))} -
    +
    + +
    ); } diff --git a/app/src/components/computer/computer-controls.tsx b/app/src/components/computer/computer-controls.tsx index 328ef03d2f..8f262cae2a 100644 --- a/app/src/components/computer/computer-controls.tsx +++ b/app/src/components/computer/computer-controls.tsx @@ -7,29 +7,39 @@ import { useComputerControl } from "@/lib/computers/use-control"; export function ComputerControlButton({ computerId, onTakeControl, + inline = false, }: { computerId: string; onTakeControl?: () => void; + /** + * The button alone, for a row with no room under it such as the computer pane's header. How a + * change is going is then the screen's to say (`ComputerView`'s `wheelInHeader`). + */ + inline?: boolean; }) { const { control, busy, problem, change, someoneElse } = useComputerControl(computerId); // Someone else's wheel is not the person's to hand back, and not theirs to take from under them either. const human = control?.holder === "human" && !someoneElse; + const button = ( + + ); + if (inline) return button; return (
    - + {button} {control?.transitioning ? ( Finishing the current action… @@ -48,15 +58,20 @@ export function ComputerControlButton({ * The header's way into the Computer: an icon that opens and closes it, with a dot when the * Computer wants the person. Taking and handing back control happen inside the Computer, where the * person can see what they are taking over; the Computer opens itself once for each new prompt. + * + * A quieter dot, `unseen`, says the Bot has been using its computer while it stayed shut, on a + * screen with no room to open it beside the conversation. A prompt for the person outranks it. */ export function ComputerChatControls({ computerId, open, onOpenChange, + unseen = false, }: { computerId?: string; open: boolean; onOpenChange: (open: boolean) => void; + unseen?: boolean; }) { const { control, someoneElse } = useComputerControl( computerId ?? "", @@ -93,7 +108,13 @@ export function ComputerChatControls({ {needsYou ? ( + ) : unseen && !open ? ( + ) : null} diff --git a/app/src/components/computer/computer-panel.tsx b/app/src/components/computer/computer-panel.tsx index d9dc715094..bba1c639e3 100644 --- a/app/src/components/computer/computer-panel.tsx +++ b/app/src/components/computer/computer-panel.tsx @@ -1,21 +1,73 @@ -import { ActivityLog } from "./activity-log"; -import { ComputerView } from "./computer-view"; +import type { Ref } from "react"; +import { + ConversationOrb, + useConversationStatus, +} from "@/components/channels/conversation-presence"; +import { ComputerView, type ComputerViewHandle } from "./computer-view"; +import { StepLog } from "./step-log"; /** The current live computer, separate from screenshots kept in historical tool results. */ export function ComputerViewPanel({ agentId, name, + wheelInHeader = false, + screen, }: { agentId: string; name?: string; + /** Take control is in the pane's header, which opens the screen full size through `screen`. */ + wheelInHeader?: boolean; + screen?: Ref; }) { return (
    - +

    Activity

    - +
    ); } + +/** Small enough for the pane's 48px header, so it blinks and watches and otherwise keeps still. */ +const TITLE_ORB_SIZE = 28; + +/** + * The computer pane's title: the Bot's orb, in the state its conversation is in, whose computer this + * is, and a dot while the conversation needs the person. "Computer" until the Bot's name arrives. + * + * The title alone, which the skins dress as the pane's tab. Take control sits beside it, in the + * header's actions. + */ +export function ComputerPaneTitle({ + agentId, + name, +}: { + agentId: string; + name: string | undefined; +}) { + const needsYou = useConversationStatus().kind === "needs-you"; + return ( + <> + + + {name ? `${name}'s computer` : "Computer"} + + {needsYou ? ( + + ) : null} + + ); +} diff --git a/app/src/components/computer/computer-view.tsx b/app/src/components/computer/computer-view.tsx index c50e055579..f2319ea445 100644 --- a/app/src/components/computer/computer-view.tsx +++ b/app/src/components/computer/computer-view.tsx @@ -1,6 +1,14 @@ import type { BotBrowserApiState } from "@openbot/shared/bot-browser"; +import { IconEyeOff, IconLock, IconShieldCheck } from "@tabler/icons-react"; import { useQuery } from "@tanstack/react-query"; -import { type ReactNode, useEffect, useRef, useState } from "react"; +import { + type ReactNode, + type Ref, + useEffect, + useImperativeHandle, + useRef, + useState, +} from "react"; import { createPortal } from "react-dom"; import { botBrowserKeys, @@ -311,6 +319,18 @@ type Props = { toolCallId?: string; /** How long to wait for a first frame before calling it a problem. Shortened by tests. */ startupGraceMs?: number; + /** + * Take control is in the pane's header, so the bar under the screen says who has control and + * what went wrong, without a second button. The header asks for the full-size view through `ref`. + */ + wheelInHeader?: boolean; + ref?: Ref; +}; + +/** What a header holding the wheel can ask of the screen under it. */ +export type ComputerViewHandle = { + /** The readable full-size view, where a person who has just taken control drives. */ + openFullSize: () => void; }; export function ComputerView({ @@ -325,6 +345,8 @@ export function ComputerView({ finished, toolCallId, startupGraceMs = STARTUP_GRACE_MS, + wheelInHeader = false, + ref, }: Props) { const [shot, setShot] = useState(null); /** @@ -338,6 +360,11 @@ export function ComputerView({ const [expanded, setExpanded] = useState(false); /** The full-size view's live screen, which the bar hands the keyboard back to once an address goes. */ const screen = useRef(null); + useImperativeHandle( + ref, + () => ({ openFullSize: () => setExpanded(true) }), + [], + ); const [streamProblem, setStreamProblem] = useState(null); /** A dropped live stream being retried on its own. A badge, never the overlay. */ const [reconnecting, setReconnecting] = useState(false); @@ -375,6 +402,7 @@ export function ComputerView({ const { control, busy: changingControl, + problem: controlProblem, change: changeControl, refresh: refreshControl, someoneElse, @@ -762,8 +790,13 @@ export function ComputerView({ Audit records that a secret was supplied, not the value. */} {control?.secretWanted ? ( + /* + * The password box: what the page wants, a masked field, and the promise that the value + * goes to the page and nowhere else. "The page", not a site's name: the value goes to the + * element the Bot pointed at, and the frame drawn here can lag behind or show another page. + */
    { event.preventDefault(); if (!secret || sendingSecret) return; @@ -777,11 +810,24 @@ export function ComputerView({ await refreshControl?.(); }} > - -
    +
    + + + + +
    + {/* + Ringed in the accent from the start, since this is what the Bot is waiting for, and + ringed again outside while the field has the caret, as every other field is. + */} +
    + +
    +

    + + It goes straight to the page. {name ?? "The Bot"} never sees it. +

    + {secretProblem ? ( +

    + {secretProblem} +

    + ) : null} +
    -

    - This goes straight to the page. It isn't shown in the - conversation, and the Bot never sees it. -

    - {secretProblem ? ( -

    {secretProblem}

    - ) : null} ) : null} @@ -824,10 +881,21 @@ export function ComputerView({ ? "Someone else has control." : "The Bot has control."} - setExpanded(true)} - /> + {wheelInHeader ? null : ( + setExpanded(true)} + /> + )} + {/* The header's button has no room under it, so what went wrong is said here. */} + {wheelInHeader && controlProblem ? ( + + {controlProblem} + + ) : null}
    ) : null} diff --git a/app/src/components/computer/desktop-illustration.tsx b/app/src/components/computer/desktop-illustration.tsx index a0321c9a73..d93188d814 100644 --- a/app/src/components/computer/desktop-illustration.tsx +++ b/app/src/components/computer/desktop-illustration.tsx @@ -10,8 +10,17 @@ import { cn } from "@/lib/utils"; * Built the way Tailark builds its workspace illustrations — theme tokens, thin borders, skeleton * bars where text would be — so it needs no asset, follows the theme, and never shows a stale * product screenshot. Decorative only: hidden from the tree and inert to the pointer. + * + * `still` holds the cursor where it rests, as reduced motion does: onboarding passes it while the + * picture is off screen or its tab is hidden, so a loop nobody can see does not keep running. */ -export function DesktopIllustration({ className }: { className?: string }) { +export function DesktopIllustration({ + className, + still = false, +}: { + className?: string; + still?: boolean; +}) { return ( ); } @@ -70,8 +79,8 @@ const CLICK_EASE = [ * Percentages rather than pixels, so the same journey fits whatever size the illustration is * drawn at. */ -function PointerCursor() { - const reducedMotion = useReducedMotion(); +function PointerCursor({ still }: { still: boolean }) { + const reducedMotion = useReducedMotion() || still; return ( ; + waiting?: boolean; +}; + +/** + * The browser steps and the command log as one list, oldest first. Stable, so steps seen at the + * same moment, such as a conversation's history arriving at once, keep the conversation's order. + */ +export function stepLogEntries( + steps: readonly ConversationStep[], + activity: readonly ComputerActivity[], + context: StepLogContext, +): StepLogEntry[] { + const browser = steps.map((step): StepLogEntry => { + const status = browserState(step, context); + const details = stepDetails(step.item); + return { + kind: "browser", + id: step.item.id, + at: step.at, + status, + // Only a step that ran, or is running, is named as done; one waiting or cut off says what it would do. + label: + status === "done" || status === "running" + ? details.label + : details.plain, + ...(details.detail ? { detail: details.detail } : {}), + ...(details.href ? { href: details.href } : {}), + ...(status === "failed" + ? { reason: details.reason ?? "Couldn't finish this step." } + : {}), + }; + }); + const commands = activity.map( + (entry): StepLogEntry => ({ + kind: "activity", + id: entry.id, + at: entry.at, + status: activityStatus(entry), + activity: entry, + }), + ); + return [...browser, ...commands].sort((a, b) => a.at - b.at); +} + +const MARKS = { + done: { Icon: IconCircleCheck, tone: "text-status-done" }, + waiting: { Icon: IconClock, tone: "text-status-needs" }, + running: { Icon: IconClock, tone: "text-muted-foreground" }, + stopped: { Icon: IconClock, tone: "text-muted-foreground" }, + notYet: { Icon: IconClock, tone: "text-muted-foreground" }, + declined: { Icon: IconX, tone: "text-muted-foreground" }, + blocked: { Icon: IconX, tone: "text-status-problem" }, + failed: { Icon: IconX, tone: "text-status-problem" }, +} as const; + +/** The colour a state's words take: amber for the person's turn, red for a problem, else grey. */ +function toneOf(status: StepState): string { + if (status === "waiting") return "text-status-needs"; + if (status === "blocked" || status === "failed") return "text-status-problem"; + return "text-muted-foreground"; +} + +function StepRow({ + entry, + children, +}: { + entry: StepLogEntry; + /** What the step was done to, after its words. */ + children?: ReactNode; +}) { + const { Icon, tone } = MARKS[entry.status]; + const label = + entry.kind === "browser" + ? entry.label + : ACTIVITY_LABELS[entry.activity.kind]; + return ( +
  • +
    + + + {label} + + {children} + + {entry.status === "done" ? null : ( + + {STEP_STATUS[entry.status]} + + )} + {/* + A command's time is when it ran. A browser step's is only when this tab first saw it, + which for a conversation opened later is not when it happened, so it is not shown. + */} + {entry.kind === "activity" ? ( + + {timeOf(entry.at)} + + ) : null} + +
    + {entry.kind === "browser" && entry.reason ? ( +

    + {entry.reason} +

    + ) : null} + {entry.kind === "activity" ? ( +
    + +
    + ) : null} +
    • + ); +} + +function BrowserSubject({ + entry, +}: { + entry: Extract; +}) { + if (entry.href) + return ( +     + {entry.detail ?? entry.href} + + ); + return entry.detail ? ( + + {entry.detail} + + ) : null; +} + +export function StepLog({ computerId }: { computerId: string }) { + const steps = useConversationSteps(); + const activity = useSyncExternalStore( + subscribeToActivity, + () => activityFor(computerId), + () => activityFor(computerId), + ); + const working = useConversationStatus().kind === "working"; + const held = useHeldForPerson(); + const waiting = useConversationWaiting(); + const entries = stepLogEntries(steps, activity, { working, held, waiting }); + + if (entries.length === 0) { + return ( + + + + Nothing yet. Pages the Bot opens and commands it runs appear here. + + + + ); + } + + return ( +
      + {entries.map((entry) => + entry.kind === "browser" ? ( + + + + ) : ( + + {/* + The command wraps rather than truncating. In the transcript one line per call is what + keeps it readable; here the whole point is to see exactly what ran, and a long command + with its middle cut out is the one thing this pane must not do. + */} + + {entry.activity.subject} + + + ), + )} +
    + ); +} diff --git a/app/src/components/computer/use-auto-watch-screen.ts b/app/src/components/computer/use-auto-watch-screen.ts index b3195ba956..fb16195e2d 100644 --- a/app/src/components/computer/use-auto-watch-screen.ts +++ b/app/src/components/computer/use-auto-watch-screen.ts @@ -1,4 +1,4 @@ -import { useEffect, useRef } from "react"; +import { useEffect, useRef, useState } from "react"; import { onComputerActivity, onComputerActivityWithdrawn, @@ -14,11 +14,16 @@ import { * * The person's pane comes first. It does not open over something they opened (`busy`), it does not * reopen what they are already watching, and a run they closed stays closed for the rest of it. + * + * NOR OVER THE CONVERSATION. Where there is no room for the pane beside it (`docked` false), the + * pane would be a sheet covering what the person is reading, so it stays shut and `unseen` lights + * the header's computer button instead, until they open it or the call that lit it is taken back. */ export function useAutoWatchScreen({ botId, watching, busy, + docked, open, close, }: { @@ -27,9 +32,11 @@ export function useAutoWatchScreen({ watching: boolean; /** The pane is showing something else the person asked for, which auto-open must not take. */ busy: boolean; + /** The pane would open beside the conversation, rather than as a sheet over it. */ + docked: boolean; open: () => void; close: () => void; -}): { dismiss: () => void } { +}): { dismiss: () => void; unseen: boolean } { /** The run the Bot is in, so a dismissal can name the one it applies to. */ const run = useRef(null); const dismissed = useRef(null); @@ -40,24 +47,44 @@ export function useAutoWatchScreen({ * a page some earlier call loaded holds that call's announcement rather than a later one's. */ const openedFor = useRef(null); + /** The announcement that lit the button instead of opening the pane, or null when it is out. */ + const litFor = useRef(null); + const [unseen, setUnseen] = useState(false); // Read inside subscriptions that outlive the render they were made in. - const latest = useRef({ watching, busy, open, close }); - latest.current = { watching, busy, open, close }; + const latest = useRef({ watching, busy, docked, open, close }); + latest.current = { watching, busy, docked, open, close }; useEffect(() => { + // A light for one Bot is not the next conversation's. + litFor.current = null; + setUnseen(false); if (!botId) return; const stopActivity = onComputerActivity((activity) => { if (activity.botId !== botId) return; run.current = activity.epoch; if (dismissed.current === activity.epoch) return; - const { watching: isWatching, busy: isBusy, open: show } = latest.current; + const { + watching: isWatching, + busy: isBusy, + docked: beside, + open: show, + } = latest.current; if (isWatching || isBusy) return; + if (!beside) { + litFor.current = activity.announcement; + setUnseen(true); + return; + } openedFor.current = activity.announcement; show(); }); const stopWithdrawals = onComputerActivityWithdrawn((withdrawal) => { if (withdrawal.botId !== botId) return; + if (litFor.current === withdrawal.announcement) { + litFor.current = null; + setUnseen(false); + } if (openedFor.current !== withdrawal.announcement) return; openedFor.current = null; latest.current.close(); @@ -68,10 +95,18 @@ export function useAutoWatchScreen({ }; }, [botId]); + // Seen: the person opened the screen, by the lit button or any other way. + useEffect(() => { + if (!watching) return; + litFor.current = null; + setUnseen(false); + }, [watching]); + return { dismiss: () => { dismissed.current = run.current; openedFor.current = null; }, + unseen, }; } diff --git a/app/src/components/layout/detail-panel.tsx b/app/src/components/layout/detail-panel.tsx index 8795daf96d..d1d3fa150e 100644 --- a/app/src/components/layout/detail-panel.tsx +++ b/app/src/components/layout/detail-panel.tsx @@ -1,6 +1,12 @@ import { IconX } from "@tabler/icons-react"; import { motion, useReducedMotion } from "motion/react"; -import { type ReactNode, useLayoutEffect, useRef, useState } from "react"; +import { + type ReactNode, + useEffect, + useLayoutEffect, + useRef, + useState, +} from "react"; import { Button } from "@/components/ui/button"; import { Sheet, SheetContent, SheetTitle } from "@/components/ui/sheet"; import { EASE_OUT } from "@/lib/motion"; @@ -13,6 +19,9 @@ import { EASE_OUT } from "@/lib/motion"; * * The pane is animated by width and the content inside it is given that width outright, so the * content is laid out once and the pane reveals it. + * + * Whether it would be a sheet is the owner's business too, open or closed: something that opens + * the pane by itself can keep from covering the page it sits beside (`onOverlayChange`). */ const ANIMATION_DURATION_SECONDS = 0.3; @@ -32,17 +41,23 @@ export function DetailPanel({ open, onClose, title, + actions, detail, detailWidth = DEFAULT_DETAIL_WIDTH, + onOverlayChange, children, }: { open: boolean; onClose: () => void; /** Rendered at the left of the detail pane's header row, beside the close button. */ title?: ReactNode; + /** Controls for the detail, at the right of its header row before the close button. */ + actions?: ReactNode; detail?: ReactNode; /** Open width of the detail pane, in pixels. */ detailWidth?: number; + /** Told whether the detail would be a sheet over the page, and again whenever that changes. */ + onOverlayChange?: (overlay: boolean) => void; children: ReactNode; }) { // Reduced motion keeps the fade, which explains the change, and drops the movement. @@ -61,6 +76,12 @@ export function DetailPanel({ observer.observe(element); return () => observer.disconnect(); }, [detailWidth]); + // Read when the answer changes, so an owner passing a new function each render is not told again. + const reportOverlay = useRef(onOverlayChange); + reportOverlay.current = onOverlayChange; + useEffect(() => { + reportOverlay.current?.(overlay); + }, [overlay]); return (
    @@ -76,9 +97,13 @@ export function DetailPanel({ className="gap-0" style={{ width: `min(${detailWidth}px, 100vw)`, maxWidth: "100vw" }} > - - {title ?? "Details"} - + {/* The sheet's own close button sits at the top right, past the actions. */} +
    + + {title ?? "Details"} + + {actions} +
    {open ? detail : null}
    @@ -111,7 +136,8 @@ export function DetailPanel({ > {title}
    -
    +
    + {actions} + ) : undefined + } + icon={IconPlug} + title="No apps to connect yet" + > + {isAdmin + ? "Add apps for everyone in Plugins, then connect your own here or in Connections." + : "When an admin adds apps, you can connect them in Connections."} + + ) : ( + +
      + {rows?.map((app, at) => ( +
    • + {at > 0 ? : null} + { + setProblem(null); + connect.mutate(app.key); + }} + /> +
      • + ))} +
    +     + )} +
    + ); +} + +/** One app: its mark, its name, and Connected, Connect, or why neither. */ +function AppRow({ + app, + botName, + busy, + connecting, + checking, + onConnect, +}: { + app: AppAccess; + botName: string; + busy: boolean; + connecting: boolean; + checking: boolean; + onConnect: () => void; +}) { + const note = noteFor(app, botName); + return ( + + + + + + {app.displayName} + {note ? {note} : null} + + + {checking ? ( + Checking… + ) : app.state === "ready" ? ( + + + Connected + + ) : connectsHere(app) ? ( + + ) : null} + + + ); +} diff --git a/app/src/components/onboarding/demo-stage.tsx b/app/src/components/onboarding/demo-stage.tsx new file mode 100644 index 0000000000..20635caef1 --- /dev/null +++ b/app/src/components/onboarding/demo-stage.tsx @@ -0,0 +1,107 @@ +import { useReducedMotion } from "motion/react"; +import { type ReactNode, useEffect, useRef, useState } from "react"; +import { IllustrationFrame } from "./illustration-frame"; + +/** How long one beat of a demo holds before the next. */ +export const DEMO_BEAT_MS = 2400; + +/** What a demo draws each beat with. */ +export type DemoBeat = { + /** Which beat, from 0 to `beats - 1`. Reduced motion holds `rest`. */ + beat: number; + /** Moving now: on screen, in a shown tab, and motion allowed. Anything animated stops without it. */ + playing: boolean; + /** The reader asked for less motion. */ + reduced: boolean; +}; + +/** + * Whether an element is on screen in a shown tab, kept current. + * + * On screen until an IntersectionObserver says otherwise, and always where there is none to ask, + * so a browser without one still plays rather than freezing on its first beat. + */ +function useOnScreen() { + const ref = useRef(null); + const [visible, setVisible] = useState(true); + useEffect(() => { + const element = ref.current; + let intersecting = true; + const update = () => setVisible(intersecting && !document.hidden); + const observer = + typeof IntersectionObserver === "undefined" + ? null + : new IntersectionObserver(([entry]) => { + intersecting = entry?.isIntersecting ?? false; + update(); + }); + if (element) observer?.observe(element); + document.addEventListener("visibilitychange", update); + update(); + return () => { + observer?.disconnect(); + document.removeEventListener("visibilitychange", update); + }; + }, []); + return [ref, visible] as const; +} + +/** + * The picture a step shows, and the clock it plays on. + * + * A PICTURE, AND SAID TO BE ONE. Everything inside sits in {@link IllustrationFrame}: nothing in it + * can be clicked, typed in or tabbed to, and a screen reader hears `label` once in its place. The + * buttons drawn on the approval card are a drawing of the decision, not the decision. + * + * ONE CLOCK PER DEMO, AND ONLY WHILE SOMEBODY CAN SEE IT. The beat advances every `beatMs` while + * the demo is on screen in a shown tab. Scrolled away, in a background tab, or with the reader's + * reduced-motion setting on, it stops, and `playing` tells the orbs and loops inside to stop too. + * With reduced motion it holds `rest`, the beat that says the most as a still. + */ +export function DemoStage({ + kind, + label, + beats = 1, + rest = beats - 1, + beatMs = DEMO_BEAT_MS, + className = "", + children, +}: { + /** Which demo, for tests and styles. */ + kind: string; + /** What the picture shows, read in place of everything drawn inside it. */ + label: string; + beats?: number; + rest?: number; + beatMs?: number; + className?: string; + children: (now: DemoBeat) => ReactNode; +}) { + const reduced = useReducedMotion() === true; + const [ref, visible] = useOnScreen(); + const [beat, setBeat] = useState(0); + const playing = !reduced && visible; + useEffect(() => { + if (!playing || beats < 2) return; + const timer = setInterval( + () => setBeat((value) => (value + 1) % beats), + beatMs, + ); + return () => clearInterval(timer); + }, [playing, beats, beatMs]); + const shown = reduced ? rest : beat; + + return ( +
    + + {children({ beat: shown, playing, reduced })} + +
    + ); +} diff --git a/app/src/components/onboarding/demos/approval-demo.tsx b/app/src/components/onboarding/demos/approval-demo.tsx new file mode 100644 index 0000000000..497195f77b --- /dev/null +++ b/app/src/components/onboarding/demos/approval-demo.tsx @@ -0,0 +1,113 @@ +import { IconCheck, IconMail } from "@tabler/icons-react"; +import { BotOrb } from "@/components/agents/orb/bot-orb"; +import { buttonVariants } from "@/components/ui/button"; +import { cn } from "@/lib/utils"; +import { DemoStage } from "../demo-stage"; + +/** The three answers an approval offers, in the words the real card uses. */ +export const APPROVAL_CHOICES = [ + "Decline", + "Allow once", + "Always allow here", +] as const; + +/** + * When a Bot asks: a Bot about to send an email waits for an answer, gets Allow once, and + * sends it. + * + * A DRAWING OF AN APPROVAL CARD, FOR THIS STEP ONLY. The real card is the conversation's, and it + * decides a real request; this one decides nothing, so it is drawn here rather than borrowed. The + * buttons say what the real card says, Allow once first and loudest, Always allow here the quietest. + * + * Three beats: waiting for you, Allow once pressed, sent. With reduced motion it waits, because the + * question is the point. + */ +export function ApprovalDemo() { + return ( + + {({ beat, playing }) => { + const pressed = beat >= 1; + const sent = beat === 2; + return ( +
    +
    + + {/* The team step's Research Analyst, with the same face. */} + + +
    +

    + Research Analyst wants to send an email +

    +

    + To ana@example.com +

    +
    + + {sent ? "Sent" : "Needs you"} + +
    +
    + +

    + Hi Ana, Friday at 7 works for all eight of us. I'll book the + table. +

    +
    +
    + {sent ? ( +

    + + Allowed once +

    + ) : ( + <> + + {APPROVAL_CHOICES[0]} + + + {APPROVAL_CHOICES[1]} + + + {APPROVAL_CHOICES[2]} + + + )} +
    +
    + ); + }} +     + ); +} diff --git a/app/src/components/onboarding/demos/computer-demo.tsx b/app/src/components/onboarding/demos/computer-demo.tsx new file mode 100644 index 0000000000..1ed0808dfa --- /dev/null +++ b/app/src/components/onboarding/demos/computer-demo.tsx @@ -0,0 +1,37 @@ +import { BotOrb } from "@/components/agents/orb/bot-orb"; +import { DesktopIllustration } from "@/components/computer/desktop-illustration"; +import { ComputerPlaceholder } from "@/components/computer/placeholder"; +import { DemoStage } from "../demo-stage"; + +/** + * A browser and files of its own: the desktop a Bot works on, its browser and its folder of files, + * with its pointer on an errand, and the orb working in the corner. + * + * The computer panel's own drawing of a Bot's desktop, so the picture matches what it shows + * when a Bot is using it. Two beats only to swap the caption; the pointer keeps its own loop, held + * still while the demo is off screen. + */ +export function ComputerDemo() { + return ( + + {({ beat, playing }) => ( +
    + + +
    + + + {beat === 0 ? "Reading the page" : "Saving it to a file"} + +
    +
    + )} +     + ); +} diff --git a/app/src/components/onboarding/demos/self-host-demo.tsx b/app/src/components/onboarding/demos/self-host-demo.tsx new file mode 100644 index 0000000000..0f33d0e154 --- /dev/null +++ b/app/src/components/onboarding/demos/self-host-demo.tsx @@ -0,0 +1,94 @@ +import { WEB_HARNESSES } from "@openbot/shared/harnesses"; +import { AnimatePresence, motion } from "motion/react"; +import type { ReactNode } from "react"; +import { HarnessMark } from "@/components/agents/harness-mark"; +import { appConfig } from "@/lib/generated/application-config"; +import { LaptopIllustration } from "../laptop-illustration"; +import { OrganizationFlow } from "../organization-flow"; +import { DemoStage } from "../demo-stage"; + +/** Every framework the New Bot wizard offers by name; "an agent you already run" is the sentence. */ +export const FRAMEWORKS = WEB_HARNESSES.filter( + (harness) => harness.id !== "byo-url", +); + +const PRODUCT = appConfig.brand.productName; + +/** The three things somebody running OpenBot themselves is deciding, one picture each. */ +const PANELS: Array<{ id: string; caption: string; picture: () => ReactNode }> = + [ + { + id: "local", + caption: "On this computer", + picture: () => , + }, + { + id: "agents", + caption: "With the agents you already have", + picture: () => ( +
    + {FRAMEWORKS.map((harness) => ( + + + {harness.name} + + ))} +
    + ), + }, + { + id: "organization", + caption: "For your organization", + picture: () => , + }, + ]; + +function Panel({ panel }: { panel: (typeof PANELS)[number] }) { + return ( +
    +

    + {panel.caption} +

    + {panel.picture()} +
    + ); +} + +/** + * Running it yourself: on this computer, with the agents you already have, and for your + * organization. The three local-evaluation slides as one picture that turns through them. + * With reduced motion the three sit one above the other. + */ +export function SelfHostDemo() { + return ( + harness.name).join(", ")}; and your organization signing in with Google, Microsoft or Okta`} + rest={-1} + > + {({ beat, reduced }) => + reduced ? ( + PANELS.map((panel) => ) + ) : ( + + + {PANELS[beat] ? : null} + + + ) + } + + ); +} diff --git a/app/src/components/onboarding/demos/team-demo.tsx b/app/src/components/onboarding/demos/team-demo.tsx new file mode 100644 index 0000000000..b4d1a7c211 --- /dev/null +++ b/app/src/components/onboarding/demos/team-demo.tsx @@ -0,0 +1,130 @@ +import { useQuery } from "@tanstack/react-query"; +import { BotOrb } from "@/components/agents/orb/bot-orb"; +import { type AgentProfile, agentListQueryOptions } from "@/lib/agents/queries"; +import { cn } from "@/lib/utils"; +import { DemoStage } from "../demo-stage"; + +/** What a card needs. Examples carry these and nothing more. */ +type TeamCard = Pick & { + /** What it does, in a few words. */ + role: string; + /** Invented, to show what a team looks like: dimmed and labelled, never read as a Bot here. */ + example?: boolean; +}; + +/** + * Stand-ins for a deployment with fewer than three public Bots to show. Invented names on purpose: + * they illustrate a team without claiming any of these exist here. Their seeds are picked to draw + * three different colours: berry, ocean and coral. + */ +const EXAMPLES: TeamCard[] = [ + { + id: "example-research", + name: "Research Analyst", + avatarSeed: "example-research", + role: "Finds and compares", + }, + { + id: "example-data", + name: "Data Analyst", + avatarSeed: "data-analyst", + role: "Reads the numbers", + }, + { + id: "example-support", + name: "Support Desk", + avatarSeed: "example-support-agent", + role: "Answers customers", + }, +]; + +/** The Bots a team card shows: real public Bots first, examples topping up a sparse deployment. */ +export function teamCards(agents: AgentProfile[] | undefined): TeamCard[] { + const shared = + agents?.filter((agent) => !agent.mine && agent.visibility === "public") ?? + []; + return [ + ...shared.slice(0, 3).map((agent) => ({ + id: agent.id, + name: agent.name, + avatarSeed: agent.avatarSeed, + role: agent.title, + })), + // Slicing past the end is [], so three or more real Bots take no examples at all. + ...EXAMPLES.slice(shared.length).map((example) => ({ + ...example, + example: true, + })), + ]; +} + +/** + * Your team of Bots: three Bots, each its own orb in its own colour, and a place for one of yours. + * One at a time, a Bot picks something up and works on it, so the cards read as a team. + */ +export function TeamDemo() { + const { data: agents } = useQuery(agentListQueryOptions()); + const cards = teamCards(agents); + return ( + `${card.name} (${card.example ? "example" : "available"})`, + ) + .join(", ")}, and one of your own`} + rest={-1} + > + {({ beat, playing }) => ( + <> + {cards.map((card, at) => { + const working = at === beat; + return ( +
    + + + +
    +

    {card.name}

    +

    + {card.example ? "Example" : card.role || "Ready"} +

    +
    + + + Working + +
    + ); + })} +
    + +

    + Your own Bot +

    +
    + + )} +     + ); +} diff --git a/app/src/components/onboarding/demos/welcome-demo.tsx b/app/src/components/onboarding/demos/welcome-demo.tsx new file mode 100644 index 0000000000..eeca941845 --- /dev/null +++ b/app/src/components/onboarding/demos/welcome-demo.tsx @@ -0,0 +1,147 @@ +import { IconArrowUp, IconMicrophone, IconPlus } from "@tabler/icons-react"; +import { useEffect, useState } from "react"; +import { BotOrb } from "@/components/agents/orb/bot-orb"; +import { cn } from "@/lib/utils"; +import { DemoStage } from "../demo-stage"; + +/** The request the composer types, a thing somebody would really hand off. */ +export const WELCOME_REQUEST = + "Find somewhere near the office for a team dinner on Friday, for eight."; + +/** How long each character takes to land. Linear: a typewriter that eases reads as hesitating. */ +const CHARACTER_MS = 38; + +/** + * Types `text` in while `active`, one character at a time, starting again each time it turns on. + * Off, or with nothing moving, it is the whole text at once. + */ +function useTyped(text: string, active: boolean): string { + const [shown, setShown] = useState(active ? 0 : text.length); + useEffect(() => { + if (!active) { + setShown(text.length); + return; + } + setShown(0); + const timer = setInterval( + () => + setShown((count) => { + if (count >= text.length) { + clearInterval(timer); + return count; + } + return count + 1; + }), + CHARACTER_MS, + ); + return () => clearInterval(timer); + }, [text, active]); + return text.slice(0, shown); +} + +/** + * Welcome's picture: OpenBot's orb listening while a request is typed, then working on it. + * + * A DRAWING OF THE COMPOSER, NOT THE COMPOSER. The real one reads its text once, when it mounts, + * so it cannot be typed into from outside; and a live editor in a picture would be one more thing + * to tab to. This is its outline: the same card, the same buttons, and the words arriving. + * + * Three beats: typing, handed off and working, done. With reduced motion it is the typed request + * with the orb listening. + */ +export function WelcomeDemo() { + return ( + + {({ beat, playing, reduced }) => ( + + )} + + ); +} + +function WelcomeScene({ + beat, + playing, + reduced, +}: { + beat: number; + playing: boolean; + reduced: boolean; +}) { + const typing = beat === 0; + const typed = useTyped(WELCOME_REQUEST, typing && playing); + const sent = !typing; + const state = typing ? "listening" : beat === 1 ? "thinking" : "done"; + return ( + <> + +
    +
    +

    + {sent && !reduced ? WELCOME_REQUEST : typed} + {typing && playing ? ( + + ) : null} +

    +
    + + + +
    + + + + 0 || sent + ? "bg-primary text-primary-foreground" + : "bg-muted text-muted-foreground", + )} + > + + +
    +
    +
    +

    + + + {beat === 2 ? "Found three places" : "Working on it"} + +

    +
    + + ); +} diff --git a/app/src/components/onboarding/intro-animation.tsx b/app/src/components/onboarding/intro-animation.tsx deleted file mode 100644 index 2ed60fa3b1..0000000000 --- a/app/src/components/onboarding/intro-animation.tsx +++ /dev/null @@ -1,51 +0,0 @@ -import { AnimatePresence, motion, useReducedMotion } from "motion/react"; -import { type ReactNode, useEffect, useState } from "react"; - -export type IntroScene = { id: string; label: string; content: ReactNode }; - -/** A looping introduction instead of more wizard steps. */ -export function IntroAnimation({ scenes }: { scenes: IntroScene[] }) { - const reducedMotion = useReducedMotion(); - const [scene, setScene] = useState(0); - useEffect(() => { - if (reducedMotion || scenes.length < 2) return; - const timer = window.setInterval( - () => setScene((current) => (current + 1) % scenes.length), - 4000, - ); - return () => window.clearInterval(timer); - }, [reducedMotion, scenes.length]); - - // Keep every scene available without auto-updating content when motion is reduced. - if (reducedMotion) { - return ( -
    - {scenes.map((item) => ( -
    {item.content}
    - ))} -
    - ); - } - - return ( -
    -
    - - - {scenes[scene]?.content} - - -
    -
    - ); -} diff --git a/app/src/components/onboarding/notify-step.tsx b/app/src/components/onboarding/notify-step.tsx new file mode 100644 index 0000000000..c7fb86341c --- /dev/null +++ b/app/src/components/onboarding/notify-step.tsx @@ -0,0 +1,172 @@ +import { useState } from "react"; +import AgentOrb from "@/components/agents/orb/agent-orb"; +import { attentionSummary } from "@/components/bot-profile/attention"; +import { appConfig } from "@/lib/generated/application-config"; +import { + requestNotificationPermission, + useNotificationPermission, +} from "@/lib/notifications/permission"; +import { StepIntro } from "./onboarding-frame"; +import type { OnboardingStep } from "./onboarding-wizard"; + +const PRODUCT = appConfig.brand.productName; + +/** How long "Notifications on" shows before the step moves on by itself. */ +export const GRANTED_BEAT_MS = 1_000; + +/** + * The notification the preview shows: the one this web app raises today, word for word. + * + * `AttentionNotifier` (`bot-profile/attention.tsx`) titles it " needs you" and says what is + * waiting with `attentionSummary`, which is called here so the body cannot drift from it. A Bot with + * a question and an approval waiting is the common case it fires for. + */ +export function notificationPreview(botName: string): { + title: string; + body: string; +} { + return { + title: `${botName} needs you`, + body: attentionSummary({ + agentId: "preview", + name: botName, + questions: 1, + approvals: 1, + handoffs: 0, + unread: 0, + paused: false, + notify: "all", + }), + }; +} + +/** + * Onboarding's Get notified step: a preview of the notification, and the button that asks the + * browser for permission. + * + * Its buttons are the frame's, so this returns the whole step. "Turn on notifications" asks the + * browser, and shows a spinner while its prompt is up. A yes turns the button green, "Notifications + * on", for a beat and then moves on by itself. A no, a prompt closed without an answer, or a + * browser with no notifications at all says so on the step, and the button becomes Continue. "Not + * now" moves on and asks nothing. The step is optional, so the button keeps its place when Not now + * goes. + */ +export function useNotifyStep(botName: string): OnboardingStep { + const permission = useNotificationPermission(); + const [asking, setAsking] = useState(false); + const [dismissed, setDismissed] = useState(false); + const granted = permission === "granted"; + // Answered, or nothing to ask: the button only moves on from here. + const done = + permission === "denied" || permission === "unsupported" || dismissed; + + /* + * Why the button now says Continue, said just above it in the frame's notice slot rather than + * beside the preview: on a small screen the preview pushes anything under it out of view, and the + * person would see the button change with no reason. + */ + const said = + permission === "denied" + ? "Notifications are blocked for this site. Allow them in your browser's site settings to turn them on." + : permission === "unsupported" + ? "This browser can't show notifications." + : dismissed && permission === "default" + ? "You can turn them on later from a Bot's page." + : null; + + return { + id: "notify", + title: "Get notified", + intro: () => ( + +

    + Allow notifications so {PRODUCT} can tell you when a Bot needs you. +

    +     + ), + demo: () => , + notice: said ? ( +

    + {said} +

    + ) : undefined, + optional: true, + primaryLabel: granted + ? "Notifications on" + : done + ? "Continue" + : "Turn on notifications", + primaryDone: granted, + primaryPending: asking, + onPrimary: + granted || done + ? undefined + : (next) => { + setAsking(true); + void requestNotificationPermission().then((answer) => { + setAsking(false); + if (answer === "granted") setTimeout(next, GRANTED_BEAT_MS); + // The prompt closed with neither answer: the browser still says "default". + else if (answer === "default") setDismissed(true); + }); + }, + secondary: granted || done ? undefined : { label: "Not now" }, + footnote: `Notifications show while ${PRODUCT} is open in a browser tab.`, + }; +} + +/** + * A ghost phone, so the card reads as something arriving on a screen: taller than it is wide, its + * top fading out rather than drawn, the notification over it and four app tiles under it. + */ +function NotifyPreview({ botName }: { botName: string }) { + const preview = notificationPreview(botName); + return ( +
    +
    +
    +
    + + + + + + + {preview.title} + + + now + + + + {preview.body} + + +
    +
    + {[0, 1, 2, 3].map((tile) => ( + + ))} +
    +
    +
    + ); +} diff --git a/app/src/components/onboarding/onboarding-frame.tsx b/app/src/components/onboarding/onboarding-frame.tsx new file mode 100644 index 0000000000..8383db7578 --- /dev/null +++ b/app/src/components/onboarding/onboarding-frame.tsx @@ -0,0 +1,399 @@ +import { IconCheck, IconChevronLeft } from "@tabler/icons-react"; +import { type ReactNode, useLayoutEffect, useRef } from "react"; +import AgentOrb from "@/components/agents/orb/agent-orb"; +import { buttonVariants } from "@/components/ui/button"; +import { appConfig } from "@/lib/generated/application-config"; +import { cn } from "@/lib/utils"; + +/** + * The frame every onboarding step sits in: one DOM, laid out two ways by CSS. + * + * FROM `lg` UP, TWO PANES. On the left, the steps as a list with the one on screen marked, then + * the step's heading and sentence, then Continue and Back. On the right, filling the rest of the + * screen, the step's picture. Steps already reached can be picked from the list; the ones ahead + * cannot. + * + * ONE SCREEN TALL ON A DESKTOP, AND THE LIST IS WHAT GIVES. Nine steps, a heading, a few lines and + * the buttons are taller than a laptop's window on the longer steps, which used to cut Continue and + * Back off at the bottom. So the frame is the window's height, the step's words and its buttons + * keep their full size, and the step list takes what is left: on a short screen it scrolls within + * itself, with the step on screen kept in view. Its rows are spaced so that every step still fits + * whole in a 1280 by 800 window; the list scrolls only below that, and only a window too short for + * the words and the buttons themselves scrolls the page. + * + * BELOW `lg`, ONE COLUMN, for a phone: a round back chevron at the top left, the steps as dashes + * across the top, the heading and its picture, and the buttons pinned to the bottom, full width, + * with a fade above them where the step scrolls under. + * + * The frame owns where things are, never what a step says. Back, the list and the buttons sit in + * the same place on every step, so moving through onboarding never moves the control somebody is + * about to press. Back is drawn twice, once for each layout, and only one is ever displayed: each + * sits where its layout's reading order puts it, so the keyboard meets it where the eye does. + * + * AN OPTIONAL STEP HAS A SECOND BUTTON. "Not now" goes beside the primary (under it on a phone, the + * same size and shape, outlined), and its place is kept when it goes (`reserveSecondary`), so the + * primary does not move mid-step. A footnote, where a step has one, sits just above the buttons. + */ +export function OnboardingFrame({ + titles, + index, + reached, + onBack, + onPick, + primary, + secondary, + reserveSecondary = false, + footnote, + notice, + intro, + demo, + demoFirst = false, +}: { + /** Every step's name, in order. */ + titles: string[]; + /** Zero-based. */ + index: number; + /** The furthest step reached: every step up to it can be picked from the list. */ + reached: number; + onBack: () => void; + onPick: (index: number) => void; + /** Null when the step's own form carries its button. */ + primary: { + label: string; + onClick: () => void; + disabled?: boolean; + /** Drawn green with a check: what the button asked for has happened. */ + done?: boolean; + /** Waiting on an answer, such as the browser's prompt: a spinner beside the label. */ + pending?: boolean; + } | null; + /** A way past the step, such as Not now or Skip intro. */ + secondary?: { label: string; onClick: () => void }; + /** Keep the second button's place while it has none, so the primary doesn't move. */ + reserveSecondary?: boolean; + /** A line about the step, just above the buttons. */ + footnote?: ReactNode; + /** Said just above the buttons, such as a save that failed. */ + notice?: ReactNode; + /** The heading, its sentence, and anything that belongs with them. */ + intro: ReactNode; + /** The picture, or the step's own rows. */ + demo?: ReactNode; + /** Below `lg`, the picture above the heading. */ + demoFirst?: boolean; +}) { + const total = titles.length; + const shape = + "h-[52px] w-full rounded-full px-5 type-sub font-semibold lg:h-10 lg:w-auto lg:min-w-32 lg:rounded-lg"; + return ( +
    + {/* + * On a desktop the list's row is the one that shrinks (`minmax(10rem,auto)` above, against + * the words' and the buttons' `auto`), down to room for the product's name and a step or two. + */} +
    + {/* + * The phone's chrome. Back is not drawn on the first step, rather than drawn disabled: there + * is nowhere behind it, and a control that does nothing is one more thing a screen reader + * stops on. + */} +
    + {index > 0 ? ( + + ) : null} + +
    + +
    + +
    + {intro} +
    + + {/* + * On a desktop the picture fills its column by stretching, at most a screen tall, rather than + * being set to the screen's height: a set height spanning every row would hand its height to + * the empty last row before the step list got its own. No margin there either, which would + * stop it short of the bottom. + */} + {demo ? ( +
    +
    {demo}
    +
    + ) : null} + + {/* + * On a phone, a step taller than the screen scrolls under the buttons rather than being cut + * at them: the fade above says there is more. On a desktop the buttons follow the sentence. + */} +
    +
    + {notice} + {footnote ? ( +

    + {footnote} +

    + ) : null} +
    + {index > 0 ? ( + + ) : null} + {primary ? ( + + ) : null} + {secondary ? ( + + ) : reserveSecondary ? ( +
    + ) : null} +
    +
    +
    +
    + ); +} + +/** + * The steps as a list, on a desktop: each with a mark, done ones ticked, the one on screen filled. + * A step already reached is a button back to it; one ahead is only its name. + */ +function StepList({ + titles, + index, + reached, + onPick, +}: { + titles: string[]; + index: number; + reached: number; + onPick: (index: number) => void; +}) { + // On a screen too short for every step, the list scrolls: keep the one on screen in it, before + // the first paint, so arriving on a late step never shows the list's top for a frame. + const list = useRef(null); + useLayoutEffect(() => { + list.current?.children[index]?.scrollIntoView?.({ block: "nearest" }); + }, [index]); + return ( + + ); +} + +/** + * Where somebody is, as short dashes with the current one filled, on a phone. + * + * A progress bar to assistive technology, read as "Step 3 of 7": the dashes are a picture of that + * sentence, so they are hidden and the sentence is what is announced. + */ +export function StepDashes({ index, total }: { index: number; total: number }) { + return ( +
    + {Array.from({ length: total }, (_, at) => ( + + ))} +
    + ); +} + +/** + * A step's heading and sentence, the way every step draws them: centred on a phone, flush left in + * the desktop's left pane. `children` is the sentence; `after` is a control that belongs with it. + */ +export function StepIntro({ + title, + children, + after, +}: { + title: ReactNode; + children?: ReactNode; + after?: ReactNode; +}) { + return ( +
    +

    + {title} +

    + {children ? ( +
    + {children} +
    + ) : null} + {after ?
    {after}
    : null} +
    + ); +} diff --git a/app/src/components/onboarding/onboarding-wizard.tsx b/app/src/components/onboarding/onboarding-wizard.tsx new file mode 100644 index 0000000000..0867836c4b --- /dev/null +++ b/app/src/components/onboarding/onboarding-wizard.tsx @@ -0,0 +1,277 @@ +import { AnimatePresence, MotionConfig, motion } from "motion/react"; +import { type ReactNode, useEffect, useRef, useState } from "react"; +import { OnboardingFrame } from "./onboarding-frame"; + +/** + * One step: an id that outlives the list changing around it, its name in the step list, its words + * and its picture, and what its buttons say and do. Ids are this app's own fixed words, which is + * what lets them sit in a selector unescaped. + * + * TWO HALVES, BECAUSE A DESKTOP SHOWS THEM SIDE BY SIDE. `intro` is the step's heading (an `h1`), + * its sentence and any control that belongs with them; `demo` is what it shows. From `lg` up the + * intro sits under the step list on the left and the demo fills the right; below, the demo follows + * the intro, or comes first where `demoFirst` says so (the name step, under its orb). + * + * The buttons are the frame's, so a step describes them rather than drawing them: the primary's + * words where the step has its own action ("Turn on notifications"), what that action does given + * the move to the next step, and, on an optional step, a second button such as "Not now" that + * moves on and writes nothing. A step whose own form carries its button (`ownAction`) gets no + * primary at all. + */ +export type OnboardingStep = { + id: string; + /** The step's name in the step list: short, a few words. */ + title: string; + intro: () => ReactNode; + demo?: () => ReactNode; + /** Below `lg`, the demo above the intro rather than after it. */ + demoFirst?: boolean; + /** The primary's words here. "Continue" when absent. */ + primaryLabel?: string; + /** What the primary does, given the move to the next step. Moving on when absent. */ + onPrimary?: (next: () => void) => void; + /** The primary reads as done, green with a check. */ + primaryDone?: boolean; + /** The primary is waiting on something, such as the browser's permission prompt. */ + primaryPending?: boolean; + primaryDisabled?: boolean; + /** The step's own form has the button that moves on, so the frame draws none. */ + ownAction?: boolean; + /** A way past it, beside the primary. Moves on when it has no action of its own. */ + secondary?: { label: string; onClick?: (next: () => void) => void }; + /** + * An optional step keeps the second button's place even while it has none, so the primary stays + * put for the whole step: Get notified drops Not now once the browser has answered. + */ + optional?: boolean; + /** A line about the step, above the buttons. */ + footnote?: ReactNode; + /** Something the step has to say right above its buttons, such as why the button became Next. */ + notice?: ReactNode; +}; + +/** + * A step crossfades with a short travel toward the way the journey is going: the chrome around it + * stays still, so only the content moves, and never by more than 24 px. + */ +const variants = { + initial: (direction: number) => ({ x: 24 * direction, opacity: 0 }), + active: { x: 0, opacity: 1 }, + exit: (direction: number) => ({ x: -24 * direction, opacity: 0 }), +}; + +/** A pane that crossfades as the step changes, keyed on the step so the old one can leave. */ +function Pane({ + stepId, + direction, + children, + attribute, +}: { + stepId: string | undefined; + direction: number; + children: ReactNode; + attribute: "data-step" | "data-step-demo"; +}) { + return ( + + + {children} + + + ); +} + +/** + * The steps, moved through one at a time inside {@link OnboardingFrame}. + * + * WHERE SOMEBODY IS, BY ID. The list grows after it first draws: the self-hosting step joins once + * the server says this is somebody trying OpenBot alone, and that answer can arrive while they are + * on step two. Held as an index, their place would slide; held as an id, the step they are on stays + * the step they are on, and only the list around it changes. + * + * WHO HOLDS THE ID. The route does, through `stepId` and `onMove`, because the step lives in the + * address (`?step=`) so that a reload, browser Back and a consent's return all land on the right + * one. Without `onMove` the wizard keeps the id itself, from `initialStepId`. + * + * HOW FAR SOMEBODY HAS COME. Every step up to the furthest one they have been on can be picked + * from the desktop's step list; the ones after it cannot, because nobody has seen them yet. + * + * FOCUS FOLLOWS THE STEP. A change of step replaces the heading, the picture and the buttons, so + * the element that had focus either vanished or now means something else. Focus moves to the new + * step's heading, which a screen reader then reads, and the first draw leaves focus alone. Motion + * respects the reader's reduced-motion setting: the movement becomes a fade. + */ +export function OnboardingWizard({ + steps, + stepId: heldStepId, + onMove, + initialStepId, + notice, +}: { + steps: OnboardingStep[]; + /** The step to show, when the caller holds it. */ + stepId?: string; + /** Asked to move to another step, when the caller holds it. */ + onMove?: (to: string, way: "forward" | "back") => void; + initialStepId?: string; + /** Said above the buttons on every step, such as a save that failed. */ + notice?: ReactNode; +}) { + const [own, setOwn] = useState(() => initialStepId ?? steps[0]?.id ?? ""); + const current = heldStepId ?? own; + const container = useRef(null); + + // A step that left the list (the server's answer changed) falls back to the first, never to nothing. + const found = steps.findIndex((step) => step.id === current); + const index = found === -1 ? 0 : found; + const step = steps[index]; + + /* + * The furthest step reached, by id, so a step joining the list (the self-hosting one, once the + * server answers) does not shift it. Every step up to it can be picked from the list. + */ + const visited = useRef(new Set()); + if (step) visited.current.add(step.id); + const reachedAt = steps.reduce( + (furthest, candidate, at) => + visited.current.has(candidate.id) ? Math.max(furthest, at) : furthest, + index, + ); + + // The way the journey is going, for the motion: toward a later step, or back to an earlier one. + const shown = useRef(index); + const way = useRef(1); + if (index !== shown.current) { + way.current = index > shown.current ? 1 : -1; + shown.current = index; + } + const direction = way.current; + + const go = (to: number) => { + const target = steps[to]; + if (!target || to === index) return; + if (onMove) onMove(target.id, to > index ? "forward" : "back"); + else setOwn(target.id); + }; + /* + * On from the step this was handed to, not from whichever step is current when it runs: a step + * that moves on after a beat (Get notified, once the browser said yes) must not skip a step if + * the person pressed Back or Continue during that beat. + */ + const showing = useRef(step?.id); + showing.current = step?.id; + const from = step?.id; + const next = () => { + if (showing.current !== from) return; + go(index + 1); + }; + + /* + * Keyed on the id, not the step object: the list is rebuilt on every render of the route, and a + * fresh object would pull focus back to the heading each time anything on the page changed, such + * as the checklist's answer arriving while somebody tabs through it. The first draw is skipped, + * so arriving on a page leaves focus where the browser put it. + */ + const stepId = step?.id; + const arrivedOn = useRef(stepId); + const moved = useRef(false); + useEffect(() => { + if (!stepId) return; + if (!moved.current && stepId === arrivedOn.current) return; + moved.current = true; + // The new heading starts at the top, on the page and in the desktop's picture pane. + window.scrollTo?.({ top: 0 }); + const frame = container.current; + const demo = frame?.querySelector("[data-onboarding-demo]"); + if (demo) demo.scrollTop = 0; + const heading = frame?.querySelector( + `[data-step="${stepId}"] h1`, + ); + if (!heading) return; + // Focusable by script only, so it is not a stop when tabbing through the page. + heading.tabIndex = -1; + heading.focus({ preventScroll: true }); + }, [stepId]); + + return ( + +
    + + {step.demo()} + + ) : null + } + demoFirst={step?.demoFirst === true} + footnote={step?.footnote} + index={index} + intro={ + + {step?.intro()} + + } + notice={ + step?.notice || notice ? ( + <> + {step?.notice} + {notice} + + ) : undefined + } + onBack={() => go(index - 1)} + onPick={(at) => { + if (at <= reachedAt) go(at); + }} + primary={ + step?.ownAction + ? null + : { + label: step?.primaryLabel ?? "Continue", + disabled: + step?.primaryDisabled === true || + step?.primaryPending === true, + done: step?.primaryDone === true, + pending: step?.primaryPending === true, + onClick: () => { + if (step?.onPrimary) return step.onPrimary(next); + next(); + }, + } + } + reached={reachedAt} + reserveSecondary={step?.optional === true} + secondary={ + step?.secondary + ? { + label: step.secondary.label, + onClick: () => + step.secondary?.onClick + ? step.secondary.onClick(next) + : next(), + } + : undefined + } + titles={steps.map((candidate) => candidate.title)} + /> +
    +     + ); +} diff --git a/app/src/components/onboarding/setup-checklist.tsx b/app/src/components/onboarding/setup-checklist.tsx new file mode 100644 index 0000000000..70624db64f --- /dev/null +++ b/app/src/components/onboarding/setup-checklist.tsx @@ -0,0 +1,269 @@ +import { + IconApps, + IconBrain, + IconDeviceDesktop, + IconMessages, + IconSparkles, +} from "@tabler/icons-react"; +import { useMutation, useQuery } from "@tanstack/react-query"; +import { Fragment } from "react"; +import { PageRows } from "@/components/layout/page-shell"; +import { Button } from "@/components/ui/button"; +import { + Item, + ItemActions, + ItemContent, + ItemDescription, + ItemFooter, + ItemMedia, + ItemTitle, +} from "@/components/ui/item"; +import { Separator } from "@/components/ui/separator"; +import { appConfig } from "@/lib/generated/application-config"; +import { recheckSetupMutationOptions } from "@/lib/setup/mutations"; +import { + type SetupFix, + type SetupRow, + type SetupRowId, + setupChecklistQueryOptions, +} from "@/lib/setup/queries"; +import { cn } from "@/lib/utils"; +import { queryClient } from "@/query-client"; + +const ICONS: Record = { + intelligence: IconBrain, + model: IconSparkles, + computer: IconDeviceDesktop, + apps: IconApps, + chat: IconMessages, +}; + +const PRODUCT = appConfig.brand.productName; + +/** + * What this deployment has, and what still needs setting up: one row each, with a pill saying + * Ready, Not checked or Needs setup. + * + * The rows, their sentences and their fixes are the server's (see `lib/setup/queries.ts`): this + * draws them, and holds the one thing only a browser can do, which is asking it to check again. + * An admin is handed the page to open or the variables to set; a member is handed neither, because + * the server never sends them one. + * + * NOTHING HERE BLOCKS ANYTHING. A required gap is drawn plainly, in red, and the step around it still + * lets the person carry on: they cannot set a server's environment from a browser, and keeping them + * out of the app would not set it either. + */ +export function SetupChecklist({ + openPage, +}: { + /** + * Opens one of the app's own screens. Onboarding has to be finished before any other page opens, + * so it carries the page to its last step and opens it from there. + */ + openPage: (target: { label: string; path: string }) => void; +}) { + const checklist = useQuery(setupChecklistQueryOptions()); + const recheck = useMutation(recheckSetupMutationOptions(queryClient)); + + /* + * Check again asks the server to ask Intelligence, the model provider and the computers again, + * and waits for their answers. A plain refetch is answered from what they last said, and asks + * again only for the read after it, once an answer failed or is a few minutes old. A checklist + * that could not be read at all is read again first. + */ + const checking = checklist.isFetching || recheck.isPending; + const checkAgain = ( + + ); + + return ( +
    + {recheck.error ? ( +

    + Couldn't check again. Try again. +

    + ) : null} + {checklist.isPending ? null : checklist.error ? ( + <> +

    + Couldn't check your setup. Check again, or carry on and look later. +

    + {checkAgain} + + ) : ( + <> +

    + {summaryOf(checklist.data.rows)} +

    + +
      + {checklist.data.rows.map((row, at) => ( +
    • + {at > 0 ? : null} + +
      • + ))} +
    +     + {checkAgain} + + )} +
    + ); +} + +/** One line over the list, so the state of the whole list is said before its rows are read. */ +export function summaryOf(rows: SetupRow[]): string { + const gaps = rows.filter((row) => row.status === "needs_setup"); + const unchecked = rows.filter((row) => row.status === "not_checked").length; + const notChecked = + unchecked === 0 + ? null + : unchecked === 1 + ? "1 thing isn't checked yet." + : `${unchecked} things aren't checked yet.`; + if (gaps.length === 0) return notChecked ?? "Everything is set up."; + const required = gaps.filter((row) => row.required).length; + const things = + gaps.length === 1 ? "1 thing needs" : `${gaps.length} things need`; + const needs = + required > 0 + ? `${things} setup, and ${required === 1 ? "1 is" : `${required} are`} required.` + : `${things} setup. ${PRODUCT} works without ${gaps.length === 1 ? "it" : "them"}.`; + return notChecked ? `${needs} ${notChecked}` : needs; +} + +/** The pill's words and colours: done green, a gap that blocks red, everything else amber. */ +function pillFor(row: SetupRow) { + if (row.status === "ready") + return { label: "Ready", tone: "bg-status-done-soft text-status-done" }; + if (row.status === "not_checked") + return { + label: "Not checked", + tone: "bg-status-needs-soft text-status-needs", + }; + return { + label: "Needs setup", + tone: row.required + ? "bg-status-problem-soft text-status-problem" + : "bg-status-needs-soft text-status-needs", + }; +} + +function SetupItem({ + row, + openPage, +}: { + row: SetupRow; + openPage: (target: { label: string; path: string }) => void; +}) { + const Icon = ICONS[row.id]; + const pill = pillFor(row); + return ( + + + + + + {/* `line-clamp-none` makes the title a block, so the tag keeps its own space. */} + + {row.title} + {row.required ? ( + + Required + + ) : null} + + + {row.detail} + + + + + {pill.label} + + + {!row.ready && row.fix ? ( + + + + ) : null} + + ); +} + +const LINK = + "rounded-sm font-medium text-brand underline-offset-2 outline-none hover:underline focus-visible:underline focus-visible:ring-3 focus-visible:ring-ring/50"; + +/** How to close one gap: a page to open, or the variables to set and a guide. */ +function Fix({ + fix, + openPage, +}: { + fix: SetupFix; + openPage: (target: { label: string; path: string }) => void; +}) { + const page = (target: { label: string; path: string }) => ( + { + event.preventDefault(); + openPage(target); + }} + > + {target.label} + + ); + const guide = (url: string) => ( + + Setup guide + + ); + + if (fix.kind === "page") { + return ( +

    + Open {page(fix)}. {fix.docsUrl ? guide(fix.docsUrl) : null} +

    + ); + } + + return ( +

    + Set{" "} + {fix.names.map((name, at) => ( + + {at > 0 ? (at === fix.names.length - 1 ? " and " : ", ") : null} + + {name} + + + ))} + , then restart {PRODUCT}. + {fix.page ? <> Or add one in {page(fix.page)}. : null}{" "} + {guide(fix.docsUrl)} +

    + ); +} diff --git a/app/src/components/settings/profile.tsx b/app/src/components/settings/profile.tsx index 3f4069632d..d26a6d1b55 100644 --- a/app/src/components/settings/profile.tsx +++ b/app/src/components/settings/profile.tsx @@ -1,7 +1,8 @@ import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query"; import { useId, useState } from "react"; -import AgentOrb from "@/components/agents/orb/agent-orb"; +import { BotOrb } from "@/components/agents/orb/bot-orb"; import { PageSection } from "@/components/layout/page-shell"; +import { StepIntro } from "@/components/onboarding/onboarding-frame"; import { Button } from "@/components/ui/button"; import { Input } from "@/components/ui/input"; import { authKeys, currentUserQueryOptions } from "@/lib/auth/queries"; @@ -43,7 +44,9 @@ export function ProfileNameForm({ if (name.trim() && !save.isPending) save.mutate(name.trim()); }} > - + @@ -95,6 +103,12 @@ export function ProfileSettings() { ); } +/** + * Onboarding's last step, under the orb: the person's name, and Get started, which saves it. + * + * Its heading and form are the step's words; the orb above them is its picture, which the + * onboarding frame draws beside it on a desktop and above it on a phone ({@link OnboardingNameOrb}). + */ export function OnboardingNameStep({ onSaved, }: { @@ -102,19 +116,40 @@ export function OnboardingNameStep({ }) { const { data: user } = useQuery(currentUserQueryOptions()); return ( -
    - -

    - Let's get to it. -

    -

    + + } + title="Let's get to it." + > +

    What should we call you? You can change your name later in Settings.

    - -
    + + ); +} + +/** + * OpenBot's orb, listening while somebody types their name. `finished` plays its finish once: the + * check and its burst of rays, which is how onboarding ends. + */ +export function OnboardingNameOrb({ + finished = false, + size = 64, +}: { + finished?: boolean; + size?: number; +}) { + return ( + + + ); } diff --git a/app/src/components/theme-provider.tsx b/app/src/components/theme-provider.tsx index 9806bb8119..1a5fd3dbed 100644 --- a/app/src/components/theme-provider.tsx +++ b/app/src/components/theme-provider.tsx @@ -1,41 +1,87 @@ import { createContext, type ReactNode, + useCallback, useContext, useEffect, useState, } from "react"; import { - applyDarkTheme, - parseStoredDarkTheme, + applyTheme, + LEGACY_THEME_STORAGE_KEY, + parseThemePreference, + type ResolvedTheme, + resolveTheme, + SYSTEM_DARK_QUERY, THEME_STORAGE_KEY, + type ThemePreference, } from "@/lib/theme"; type ThemeContextValue = { - dark: boolean; - setDark: (dark: boolean) => void; + /** What the person chose: follow the system, or always light or dark. */ + preference: ThemePreference; + /** What is drawn right now. */ + resolved: ResolvedTheme; + setPreference: (preference: ThemePreference) => void; }; const ThemeContext = createContext(null); +/** Storage throws in some privacy modes; a theme is not worth taking the app down for. */ +function readStored(key: string): string | null { + try { + return window.localStorage.getItem(key); + } catch { + return null; + } +} + +function systemDarkNow(): boolean { + return window.matchMedia?.(SYSTEM_DARK_QUERY).matches ?? false; +} + export function ThemeProvider({ children }: { children: ReactNode }) { - const [dark, setDark] = useState(() => - parseStoredDarkTheme(window.localStorage.getItem(THEME_STORAGE_KEY)), + const [preference, setPreferenceState] = useState(() => + parseThemePreference( + readStored(THEME_STORAGE_KEY), + readStored(LEGACY_THEME_STORAGE_KEY), + ), ); + const [systemDark, setSystemDark] = useState(systemDarkNow); + + // The system's setting, live: it changes at sunset without anybody reloading. + useEffect(() => { + const query = window.matchMedia?.(SYSTEM_DARK_QUERY); + if (!query) return; + const follow = () => setSystemDark(query.matches); + follow(); + query.addEventListener("change", follow); + return () => query.removeEventListener("change", follow); + }, []); + const resolved = resolveTheme(preference, systemDark); useEffect(() => { - applyDarkTheme(dark, { - setStoredValue: (key, value) => window.localStorage.setItem(key, value), + applyTheme(resolved, { toggleRootClass: (name, force) => document.documentElement.classList.toggle(name, force), setRootColorScheme: (scheme) => { document.documentElement.style.colorScheme = scheme; }, }); - }, [dark]); + }, [resolved]); + + // Written only when the person chooses, so following the system stays the default until they do. + const setPreference = useCallback((next: ThemePreference) => { + setPreferenceState(next); + try { + window.localStorage.setItem(THEME_STORAGE_KEY, next); + } catch { + // The choice holds for this visit; it just isn't remembered. + } + }, []); return ( - + {children} ); diff --git a/app/src/lib/approval-view.ts b/app/src/lib/approval-view.ts new file mode 100644 index 0000000000..41dd4cabc1 --- /dev/null +++ b/app/src/lib/approval-view.ts @@ -0,0 +1,339 @@ +import type { ApprovalRequest } from "./approvals"; +import { BROWSER_STEP_PLAIN } from "./computers/step-names"; +import { connectorName, humanise, readToolName } from "./plugins/tool-name"; + +/** + * How an approval request reads on its card: a title in plain words, the app it is for, the fields a + * person checks before saying yes (To, Subject), and the body. Built only from what the request + * carries; anything it cannot place stays in the card's Details. + */ +export type ApprovalView = { + title: string; + app: string | null; + kind: "mail" | "browser" | "command" | "other"; + fields: { label: string; value: string }[]; + body: string | null; + /** + * How many of the request's own arguments the card's face leaves out: past the fields it has + * room for, a second body, or values that are not plain text. Plumbing (ids, refs) is not + * counted. When any are left out, Details opens by itself, so nothing being agreed to is hidden. + */ + unshown: number; +}; + +const RECIPIENT = + /^(to|recipients?|recipient_?emails?|to_?emails?|to_?address(es)?|email_?to|send_?to)$/i; +const SUBJECT = /^(subject|title)$/i; +const BODY = /^(body|text|message|content|message_?body|html|plain_?text)$/i; +/** Plumbing the person cannot act on: page references, ids and the content's digest. */ +const HIDDEN = + /^(ref|snapshot_?id|element_?ref|id|thread_?id|tool_?call_?id|bot_?id|actor_?id|owner_?user_?id|grant_?id|content_?digest)$/i; +/** Argument names that need a few words to read as a question somebody can answer. */ +const LABELS: Record = { submit: "Submit after typing" }; +/** A command on the person's computer also names its own kind and the folder's write switch. */ +const HOST_HIDDEN = /^(kind|writable)$/i; +const MAX_FIELDS = 4; + +/** The ref the server's private-share check asks under (server/src/proactive/private-share.ts). */ +const PRIVATE_SHARE = "openbot/share_private_information"; +/** Refs OpenBot itself owns, which name no app. */ +const OWN_PREFIXES = new Set(["host", "bot", "openbot"]); +/** OpenBot's own tools, in words somebody chose. */ +const OWN_TITLES: Record = { + "bot/message_bot": "Ask another Bot", +}; + +const words = (text: string) => + text + .replace(/([a-z])([A-Z])/g, "$1 $2") + .split(/[\s_\-.]+/) + .filter(Boolean) + .map((word) => word.toLowerCase()); + +const sentence = (parts: string[]) => { + const text = parts.join(" "); + return text ? text[0].toUpperCase() + text.slice(1) : text; +}; + +const display = (value: unknown): string | null => { + if (typeof value === "string") return value.trim() || null; + if (typeof value === "boolean") return value ? "Yes" : "No"; + if (typeof value === "number") return String(value); + if ( + Array.isArray(value) && + value.length > 0 && + value.every((entry) => typeof entry === "string") + ) + return value.join(", "); + return null; +}; + +const record = (value: unknown): Record | null => + value && typeof value === "object" && !Array.isArray(value) + ? (value as Record) + : null; + +/** + * The app before the `/` of an app's tool ref (`gmail/send_email`), the prefix OpenBot's own tools + * carry (`host/`, `bot/`), or null for the computer's. A rule may write an app's ref as + * `mcp/gmail/send_email`, and that `mcp/` names no app. + */ +function serverOf(toolRef: string): string | null { + const ref = toolRef.replace(/^mcp\//, ""); + const slash = ref.indexOf("/"); + return slash > 0 ? ref.slice(0, slash) : null; +} + +/** + * A send to other people, as the private-share check asks about it. + * + * The check asks before the send itself, so its request carries the audience as `: + * ` and a preview of what goes out, not the tool's own arguments. Read as the message + * it is: the app, who it goes to, and what it says. The content digest is how the server matches a + * later send to this answer; it means nothing to the person and is left out. + */ +function describeShare(args: Record): ApprovalView { + const audience = display(args.audience) ?? ""; + const split = audience.indexOf(": "); + const server = split > 0 ? audience.slice(0, split) : ""; + const to = split > 0 ? audience.slice(split + 2) : audience; + // With no recipients in its arguments the check names the tool instead, which is nobody. + const named = to && !(server && to.startsWith(`${server}/`)); + const mail = /mail|outlook|inbox/i.test(server); + return { + title: mail ? "Send email" : "Send a message", + app: server ? connectorName(server) : null, + kind: mail ? "mail" : "other", + fields: named ? [{ label: "To", value: to }] : [], + body: display(args.preview), + unshown: 0, + }; +} + +/** + * Where an action happens, as a person reads it, or null where the card already says it: the site a + * browser step acts on, the folder a command runs in. The scope is the server's matching key, so it + * is shown only where it is a place. An app's own id, a share's recipients and the Bot a hand-off + * goes to are already on the card as the app and the To or Bot field. + */ +export function approvalPlace( + request: Pick, +): string | null { + const { toolRef, scope, target } = request.action; + if (toolRef === PRIVATE_SHARE) return null; + const server = serverOf(toolRef); + if (server === "host") return display(record(target)?.displayName); + if (server === "bot" || scope === server) return null; + // The computer's own words for "no page" and "its workspace" are no place a person names. + if (!scope || ["*", "browser", "workspace"].includes(scope)) return null; + return scope; +} + +/** Whether the person was asked about this request, rather than a saved rule deciding it. */ +export function personWasAsked( + request: Pick, +): boolean { + return ( + Boolean(request.action.policy) || request.action.toolRef === PRIVATE_SHARE + ); +} + +export function describeApproval( + request: Pick, +): ApprovalView { + const { toolRef } = request.action; + const args = record(request.action.args); + if (toolRef === PRIVATE_SHARE && args) return describeShare(args); + const server = serverOf(toolRef); + const tool = server + ? toolRef.replace(/^mcp\//, "").slice(server.length + 1) + : toolRef; + /* + * An app's tool is named the way its line in the conversation names it ("Send your email"), with + * the app beside it, so the card and the step above it say the same thing. + */ + const ownTitle = OWN_TITLES[toolRef]; + const title = + ownTitle ?? + (server && !OWN_PREFIXES.has(server) + ? readToolName(`mcp__${server}__${tool}`, "plain").label + : server + ? humanise(tool) + : (BROWSER_STEP_PLAIN[tool] ?? + sentence(words(tool.replace(/^computer_/, ""))))); + const app = + server && !OWN_PREFIXES.has(server) ? connectorName(server) : null; + + const kind: ApprovalView["kind"] = /^computer_/.test(tool) + ? "browser" + : server === "host" + ? "command" + : server !== "bot" && /mail|message|send|reply|forward/i.test(tool) + ? "mail" + : "other"; + + const fields: ApprovalView["fields"] = []; + let body: string | null = null; + let unshown = 0; + if (args) { + const entries = Object.entries(args); + const hidden = (key: string) => + HIDDEN.test(key) || (server === "host" && HOST_HIDDEN.test(key)); + const shownKeys = new Set(); + // The first key matching, under `label` or, without one, under its own name. + const pick = (pattern: RegExp, label?: string) => { + for (const [key, value] of entries) { + if (!pattern.test(key)) continue; + const shown = display(value); + if (shown) { + fields.push({ label: label ?? sentence(words(key)), value: shown }); + shownKeys.add(key); + } + return; + } + }; + pick(RECIPIENT, "To"); + pick(/^cc$/i, "Cc"); + pick(/^bcc$/i, "Bcc"); + pick(SUBJECT); + for (const [key, value] of entries) { + if (!BODY.test(key)) continue; + body = display(value); + if (body) { + shownKeys.add(key); + break; + } + } + for (const [key, value] of entries) { + if (fields.length >= MAX_FIELDS) break; + if ([RECIPIENT, SUBJECT, BODY].some((test) => test.test(key))) continue; + if (hidden(key) || /^b?cc$/i.test(key)) continue; + const shown = display(value); + if (shown) { + fields.push({ + label: LABELS[key] ?? sentence(words(key)), + value: shown, + }); + shownKeys.add(key); + } + } + unshown = entries.filter( + ([key, value]) => + !shownKeys.has(key) && + !hidden(key) && + value !== null && + value !== undefined && + !(typeof value === "string" && !value.trim()), + ).length; + } + + return { title: title || toolRef, app, kind, fields, body, unshown }; +} + +/** + * What "Always allow here" lets this Bot do from then on. The server saves the rule for this Bot, + * this action and this place (`approvals/store.ts`), whatever the arguments, so a card that leads + * with one call's details says how much further its rule reaches: not only this command, every + * command in the folder; not only this email, every email from the app. + */ +export function alwaysAllowReach( + request: Pick, +): string { + const { toolRef, scope, target } = request.action; + if (toolRef === PRIVATE_SHARE) + return "This Bot won't ask you again before sharing private information with these people."; + const server = serverOf(toolRef); + if (server === "bot") + return "This Bot won't ask you again before asking that Bot, whatever it asks."; + const where = + server === "host" + ? `in ${display(record(target)?.displayName) ?? "this folder"}` + : server && !OWN_PREFIXES.has(server) + ? `in ${connectorName(server)}` + : !scope || scope === "*" + ? "anywhere" + : scope === "browser" + ? "in its browser" + : scope === "workspace" + ? "on its computer" + : scope.includes("/") + ? `in ${scope}` + : `on ${scope}`; + return `This Bot won't ask you again before doing this ${where}, whatever the details.`; +} + +/** Who decided, in the person's words, for the line under a decided card. */ +export function decisionReceipt( + request: Pick, +): string | null { + switch (request.decision) { + case "allow_once": + return "Allowed once by you"; + case "allow_always": + return "Always allowed here by you"; + case "deny": + return "Declined by you"; + case "handled": + return "You did it yourself"; + default: + return null; + } +} + +/** A pill's colour: the four status colours, or neutral for an outcome that is nobody's problem. */ +export type OutcomeTone = "done" | "working" | "needs" | "problem" | "neutral"; + +/** + * How the action ended, as the pill on a decided card shows it. Nothing while it waits. + * + * The person's own decline comes first, whatever the status: once the Bot has been told, the server + * finishes a declined request with the decline as its error, and that is not the action failing. + */ +export function approvalOutcome( + request: Pick, +): { label: string; tone: OutcomeTone } | null { + if (request.status === "pending") return null; + if (request.decision === "deny") + return { label: "Declined", tone: "neutral" }; + switch (request.status) { + case "completed": + if (request.result?.error) return { label: "Didn't work", tone: "needs" }; + // A share the person allowed sends nothing itself: the Bot is told to send it again. + if (request.action.toolRef === PRIVATE_SHARE) + return { label: "Allowed", tone: "done" }; + return { + label: + request.decision !== "handled" && + describeApproval(request).kind === "mail" && + /send|reply|forward/i.test(request.action.toolRef) + ? "Sent" + : "Done", + tone: "done", + }; + case "denied": + return { label: "Not done", tone: "neutral" }; + case "approved": + case "consumed": + return { label: "Allowed", tone: "working" }; + default: + return null; + } +} + +const timeFormat = new Intl.DateTimeFormat(undefined, { + hour: "numeric", + minute: "2-digit", +}); +const dayFormat = new Intl.DateTimeFormat(undefined, { + month: "short", + day: "numeric", +}); + +/** "2:59 PM" today, "Oct 20" before that: the stamp on a decided card. */ +export function clockTime(iso: string, now = new Date()): string { + const at = new Date(iso); + if (!Number.isFinite(at.getTime())) return ""; + return at.toDateString() === now.toDateString() + ? timeFormat.format(at) + : dayFormat.format(at); +} diff --git a/app/src/lib/approvals.ts b/app/src/lib/approvals.ts index 720777cdf9..95beaba797 100644 --- a/app/src/lib/approvals.ts +++ b/app/src/lib/approvals.ts @@ -25,28 +25,53 @@ export const BEHAVIOUR_LABELS: Record = { ask: "Ask before taking action", hand_off: "Hand off to you", }; +/** One action a Bot asked to take, as the inbox returns it: waiting, decided, or done. */ +export type ApprovalRequest = { + id: string; + /** "pending", "approved", "denied", "consumed" (running), or "completed". */ + status: string; + createdAt: string; + action: { + botId: string; + toolRef: string; + effect: string; + scope: string; + threadId: string; + toolCallId?: string; + args: unknown; + target: unknown; + policy?: { + behaviour: "allow" | "ask" | "hand_off" | "deny"; + source: string; + reason: string; + }; + }; + /** + * What was chosen: by the person, or `allow_always` by a saved rule as the request opened. Null + * while it waits, and for a request withdrawn before anyone answered. + */ + decision?: ApprovalDecision | null; + /** ISO-8601, when it was decided. */ + decidedAt?: string | null; + /** ISO-8601, when the action finished running. */ + completedAt?: string | null; + /** How running it went. The content stays on the server. */ + result?: { error?: string } | null; +}; +/** A question a Bot asked mid-run and is waiting on the person to answer. */ +export type PersonQuestion = { + id: string; + botId: string; + /** The Bot whose conversation this belongs to: for a hand-off, the Bot that handed it on. */ + conversationBotId?: string; + threadId: string; + question: string; + why?: string; + createdAt: string; +}; export type ApprovalInboxData = { enabled: boolean; - requests: { - id: string; - status: string; - createdAt: string; - action: { - botId: string; - toolRef: string; - effect: string; - scope: string; - threadId: string; - toolCallId?: string; - args: unknown; - target: unknown; - policy?: { - behaviour: "allow" | "ask" | "hand_off" | "deny"; - source: string; - reason: string; - }; - }; - }[]; + requests: ApprovalRequest[]; rules: ApprovalRuleRow[]; teamRules: ApprovalRuleRow[]; preferences?: { @@ -61,17 +86,10 @@ export type ApprovalInboxData = { }; /** The member's command policy after the team cap. */ hostCommands?: HostCommandPolicy; - questions: { - id: string; - botId: string; - /** The Bot whose conversation this belongs to: for a hand-off, the Bot that handed it on. */ - conversationBotId?: string; - threadId: string; - question: string; - why?: string; - createdAt: string; - }[]; + questions: PersonQuestion[]; }; +/** How often a screen showing approvals asks for them again. */ +const INBOX_POLL_MS = 10_000; export const approvalInboxOptions = () => queryOptions({ queryKey: ["approvals"], @@ -81,7 +99,21 @@ export const approvalInboxOptions = () => }); return response.json(); }, - refetchInterval: 10_000, + refetchInterval: INBOX_POLL_MS, + }); +/** + * The same inbox, for a reader drawn once per tool call, with no poll of its own. + * + * TanStack Query polls once per observer, and every tool call in a conversation reads the inbox, so + * a reply with thirty calls asked for it sixty times every ten seconds. The screen polls it once + * (`ConversationPresenceProvider`, the Bot page), and a row that mounts between two polls reads what + * the last one brought. + */ +export const approvalInboxCacheOptions = () => + queryOptions({ + ...approvalInboxOptions(), + refetchInterval: false, + staleTime: INBOX_POLL_MS, }); export const decideApproval = (id: string, decision: ApprovalDecision) => client(`/api/approvals/${encodeURIComponent(id)}/decision`, { @@ -210,12 +242,48 @@ export const revokeTeamApprovalRuleMutationOptions = ( mutationFn: revokeTeamApprovalRule, onSettled: settle(queryClient), }); -/** Allow once, always allow, deny, or "I did it myself" on one waiting action. */ +/** + * Allow once, always allow, decline, or "I did it myself" on one waiting action. + * + * THE ONE WRITE HERE THAT ALSO PATCHES THE INBOX, before reading it again. The card folds as soon + * as the decision is saved, and stays folded if that read fails: left pending, it showed its buttons + * again and a second press was refused as "already decided". Only what the server just answered is + * written: the decision it accepted, the status it returned (or the one `decide` sets for it), and + * the time. Everything else waits for the read. + */ export const decideApprovalMutationOptions = (queryClient: QueryClient) => mutationOptions({ - mutationFn: ({ id, choice }: { id: string; choice: ApprovalDecision }) => - decideApproval(id, choice), - onSuccess: settle(queryClient), + mutationFn: async ({ + id, + choice, + }: { + id: string; + choice: ApprovalDecision; + }): Promise<{ status?: string }> => + // Saved once this resolves. A body that does not parse loses only the status, not the answer. + (await decideApproval(id, choice)).json().catch(() => ({})), + onSuccess: async (saved, { id, choice }) => { + queryClient.setQueryData( + approvalInboxOptions().queryKey, + (inbox) => + inbox && { + ...inbox, + requests: inbox.requests.map((request) => + request.id === id + ? { + ...request, + decision: choice, + decidedAt: new Date().toISOString(), + status: + saved.status ?? + (choice === "deny" ? "denied" : "approved"), + } + : request, + ), + }, + ); + await settle(queryClient)(); + }, }); /** Answer a question a Bot asked mid-run; the run resumes with the answer. */ export const answerPersonQuestionMutationOptions = (queryClient: QueryClient) => diff --git a/app/src/lib/channels/step.ts b/app/src/lib/channels/step.ts new file mode 100644 index 0000000000..c5e6c1cc22 --- /dev/null +++ b/app/src/lib/channels/step.ts @@ -0,0 +1,188 @@ +import type { Message } from "@ag-ui/core"; +import { researchRoleOfTool } from "@openbot/shared/research"; +import { isBrowserStep } from "@/components/channels/browser-activity"; +import { + toVisibleChatItems, + type VisibleChatItem, +} from "@/components/channels/chat-messages"; +import { + connectorName, + ownToolLabel, + personToolLabel, + readToolName, +} from "@/lib/plugins/tool-name"; + +/** + * What the line under a conversation's name may say about the step its Bot is on, where the step's + * own line has no words for it: a plain phrase about the person and their things, in the present, + * with no ellipsis. Never a tool's identifier, never a server key. + * + * Research, the person's mail and calendar, and this deployment's own tools are named by + * `tool-name.ts`, in the running words their lines use, so the header and the transcript say the + * same thing at the same moment. + */ +export const STEP_PHRASES = { + thinking: "Thinking it through", + browser: "Working in your browser", + memory: "Checking memory", +} as const; + +/** What the line says while the Bot works and no step of its own is running. */ +export const THINKING = STEP_PHRASES.thinking; + +/** + * Tools the line does not name, because they are not work the person is waiting on: a question put + * to the person, and an interface drawn for them to read or fill in (`askForm`, `showBarChart`). + * While one runs the line says whatever the turn was doing before it. + */ +const UNNAMED_TOOLS = new Set(["ask_person"]); + +/** + * The line's phrase for a running tool call, or null for one it does not name. + * + * Read from what the tool does to the person's things: the web, their mail and calendar through any + * app, their browser, their memory. An app's call that cannot be named that way is named by the app + * ("Using Notion"), since "Calendar list events" is still an identifier. Anything else is passed + * over, like the tools above. + */ +export function stepPhrase(name: string): string | null { + if (UNNAMED_TOOLS.has(name)) return null; + // A gallery interface: `askForm`, `askChoice`, `showBarChart`, `showTable`. + if (/^(ask|show)[A-Z]/.test(name)) return null; + const own = ownToolLabel(name, true); + if (own) return own; + if (name.startsWith("computer_")) return STEP_PHRASES.browser; + + const parts = name.split("__"); + const isMcp = parts.length >= 3 && parts[0] === "mcp"; + const server = isMcp ? (parts[1] ?? "") : ""; + const tool = isMcp ? parts.slice(2).join("__") : name; + if (isMcp && researchRoleOfTool(server, tool)) + return readToolName(name, true).label; + const personal = personToolLabel(name, "running"); + if (personal) return personal; + const words = `${server} ${tool}` + .replace(/([a-z\d])([A-Z])/g, "$1 $2") + .toLowerCase() + .split(/[^a-z\d]+/); + if (words.some((word) => MEMORY_WORDS.has(word))) return STEP_PHRASES.memory; + return isMcp ? `Using ${connectorName(server)}` : null; +} + +const MEMORY_WORDS = new Set(["memory", "memories", "remember", "recall"]); + +/** + * What `stepFor` needs to know about an approval: the call it holds and how it was decided. The + * approvals inbox's requests fit it as they are. + */ +export type StepApproval = { + status: string; + decision?: string | null; + action: { toolCallId?: string }; +}; + +/** A call the person declined, or did themselves, does not run here, whatever its result says yet. */ +function wontRun(approval: StepApproval): boolean { + return ( + approval.status === "denied" || + approval.decision === "deny" || + approval.decision === "handled" + ); +} + +/** + * What the Bot is doing right now, in a phrase for the line under the conversation's name, or null + * when it has not got the turn. + * + * The newest step still waiting on its result, named by `stepPhrase`: "Searching your email", never + * "Gmail search messages". Steps the line does not name are passed over. Only this turn's steps + * count: a call from an earlier turn that never got its result is not running, it was cut off. With + * no step running the Bot is thinking, or writing what it found. + */ +export function stepFor( + messages: readonly Message[], + working: boolean, + approvals: readonly StepApproval[] = [], +): string | null { + if (!working) return null; + const items = toVisibleChatItems(messages); + const turn = items.slice(turnStart(items)); + const running = turn + .flatMap((item) => + item.kind === "tool" && + item.result === undefined && + !approvals.some( + (approval) => + approval.action.toolCallId === item.toolCall.id && wontRun(approval), + ) + ? [stepPhrase(item.toolCall.function.name)] + : [], + ) + .findLast((phrase) => phrase !== null); + return running ?? THINKING; +} + +/** + * Whether the turn the Bot is on stopped to wait for the person: a call in it has no result, or only + * the note that it went to its approval card. Read once the turn is over, that turn ended on + * something the person has to answer (an approval, a sign-in), not on an answer of its own. + */ +export function turnWaitsOnPerson(messages: readonly Message[]): boolean { + const items = toVisibleChatItems(messages); + return items + .slice(turnStart(items)) + .some( + (item) => + item.kind === "tool" && + (item.result === undefined || wentToItsCard(item.result)), + ); +} + +/** The browser's note for a computer call it sent to its approval card instead of running it. */ +function wentToItsCard(result: string): boolean { + try { + const parsed: unknown = JSON.parse(result); + return ( + !!parsed && + typeof parsed === "object" && + (parsed as { waiting?: unknown }).waiting === true + ); + } catch { + return false; + } +} + +/** Where the turn the Bot is on began: after the person's last message, their words or files. */ +function turnStart(items: readonly VisibleChatItem[]): number { + return ( + items.findLastIndex( + (item) => + item.kind === "attachments" || + (item.kind === "text" && item.role === "user"), + ) + 1 + ); +} + +/** One browser step in a conversation, as the transcript has it. */ +export type BrowserStep = { + item: Extract; + /** It is in the turn the Bot is on, so without a result it may still be running. */ + latest: boolean; +}; + +/** + * The browser steps a conversation's Bot took, oldest first, for the computer pane's step log. + * + * READ FROM THE MESSAGES, BECAUSE NOTHING ELSE SEES THEM ALL. A click or a typed field the person + * has to approve runs on the server once they allow it, and never reaches a browser handler that + * could note it down; its call and its result are in the conversation either way. + */ +export function browserSteps(messages: readonly Message[]): BrowserStep[] { + const items = toVisibleChatItems(messages); + const start = turnStart(items); + return items.flatMap((item, index) => + item.kind === "tool" && isBrowserStep(item) + ? [{ item, latest: index >= start }] + : [], + ); +} diff --git a/app/src/lib/computers/step-names.ts b/app/src/lib/computers/step-names.ts new file mode 100644 index 0000000000..ca42eb887b --- /dev/null +++ b/app/src/lib/computers/step-names.ts @@ -0,0 +1,29 @@ +/** + * What each browser step on a Bot's computer is called, in one place, so the conversation's step + * line, the computer's step log and an approval card for the same step all say the same words. + */ + +/** A step that happened. */ +export const BROWSER_STEP_DONE: Record = { + computer_navigate: "Opened page", + computer_read: "Read page", + computer_snapshot: "Inspected page", + computer_click: "Clicked on page", + computer_type: "Filled in field", + computer_key: "Pressed key", + computer_scroll: "Scrolled page", +}; + +/** + * The same steps as what they would do: waiting on the person, declined, blocked or cut off. Whole + * phrases, because they also sit inside sentences ("Allow it to fill in a field, or decline"). + */ +export const BROWSER_STEP_PLAIN: Record = { + computer_navigate: "Open a page", + computer_read: "Read the page", + computer_snapshot: "Inspect the page", + computer_click: "Click on the page", + computer_type: "Fill in a field", + computer_key: "Press a key", + computer_scroll: "Scroll the page", +}; diff --git a/app/src/lib/hotkeys/hotkeys.ts b/app/src/lib/hotkeys/hotkeys.ts index 831f6ab8c7..6c42228521 100644 --- a/app/src/lib/hotkeys/hotkeys.ts +++ b/app/src/lib/hotkeys/hotkeys.ts @@ -9,7 +9,8 @@ * browser window before the page ever sees the keystroke, and `preventDefault` is ignored for * reserved shortcuts, so no entry here can use one. Plain Shift+letter combos work everywhere, * at the cost that they are also just how capital letters are typed — which is why `useHotkey` - * ignores them while focus is in anything editable. + * ignores them while focus is in anything editable. Cmd/Ctrl+K is not reserved, and a modified + * combo fires from inside a field too, so search reaches the roster from the composer. */ export type HotkeyCombo = { @@ -36,6 +37,12 @@ export const HOTKEYS = [ description: "Start a new chat from anywhere in the app.", combo: { key: "n", shift: true }, }, + { + id: "search", + label: "Search conversations", + description: "Open the sidebar and search your conversations.", + combo: { key: "k", mod: true }, + }, ] as const satisfies readonly Hotkey[]; export type HotkeyId = (typeof HOTKEYS)[number]["id"]; @@ -103,3 +110,8 @@ export function formatHotkey(combo: HotkeyCombo): string[] { parts.push(combo.key.toUpperCase()); return parts; } + +/** The combo as one short label, for a tooltip: "⇧N" on a Mac, "Shift+N" everywhere else. */ +export function hotkeyLabel(combo: HotkeyCombo): string { + return formatHotkey(combo).join(isMac ? "" : "+"); +} diff --git a/app/src/lib/navigation.ts b/app/src/lib/navigation.ts index e7d8391153..d399316203 100644 --- a/app/src/lib/navigation.ts +++ b/app/src/lib/navigation.ts @@ -17,7 +17,8 @@ import { currentUserQueryOptions } from "@/lib/auth/queries"; * missing from that skin, with nothing failing to say so. * * `section` says where OpenBot's sidebar puts each entry: `primary` destinations sit in its footer, - * `account` ones in the menu under the person's name. A skin may group them its own way. + * `account` ones in the menu under the person's name. A skin may group them its own way. The sidebar + * draws each section in the order of this list, so a new destination goes where it is written. * * Bots, Capabilities and Connections are each somebody's own, which is why they sit beside each * other and not inside Admin. Capabilities are your skills and what your Bots remember about you; @@ -32,6 +33,12 @@ export type NavigationItem = { section: "primary" | "account"; /** Admin routes are guarded on the server; the entry is hidden from people who cannot open them. */ adminOnly?: boolean; + /** + * How many things wait on the person there, drawn beside the entry; nothing is drawn for zero. A + * count is live and this list is not, so `useNavigationItems` sets it from whatever query counts + * them, and an entry and its count are added in this file together. + */ + badge?: number; }; export const NAVIGATION: readonly NavigationItem[] = [ @@ -80,7 +87,10 @@ export function navigationFor( return NAVIGATION.filter((item) => !item.adminOnly || role === "admin"); } -/** `navigationFor` the signed-in person. */ +/** + * `navigationFor` the signed-in person. OpenBot's sidebar draws this, so an entry's `badge` is set + * here. + */ export function useNavigationItems(): readonly NavigationItem[] { const { data: user } = useQuery(currentUserQueryOptions()); return navigationFor(user?.role); diff --git a/app/src/lib/notifications/permission.ts b/app/src/lib/notifications/permission.ts new file mode 100644 index 0000000000..8d314f6eae --- /dev/null +++ b/app/src/lib/notifications/permission.ts @@ -0,0 +1,77 @@ +import { useSyncExternalStore } from "react"; + +/** + * Whether this browser may show OpenBot's notifications: the browser's own answer, never a guess. + * + * WHAT A NOTIFICATION IS HERE. The web app raises one when a Bot starts needing the person + * (`AttentionNotifier` in `bot-profile/attention.tsx`), from the page itself. So it shows while + * OpenBot is open in a tab, and not once the tab is closed: there is no web push. Onboarding's Get + * notified step says so rather than promise more. + * + * ONE ANSWER FOR THE WHOLE PAGE. Onboarding's step and a Bot's page both offer to turn them on, + * and a press on either has to show on the other, so the state lives here once and both read it + * through `useNotificationPermission`. + */ +export type NotificationPermissionState = + | "granted" + | "denied" + | "default" + /** No Notification API, as in Safari on iPhone outside a Home Screen app. */ + | "unsupported"; + +/** What the browser says now. A browser that throws on reading it is one that cannot show them. */ +export function notificationPermission(): NotificationPermissionState { + try { + return typeof Notification === "undefined" + ? "unsupported" + : Notification.permission; + } catch { + return "unsupported"; + } +} + +const listeners = new Set<() => void>(); +let current: NotificationPermissionState | null = null; + +function read(): NotificationPermissionState { + current ??= notificationPermission(); + return current; +} + +function set(next: NotificationPermissionState) { + current = next; + for (const listener of listeners) listener(); +} + +/** + * Asks the browser, which shows its own prompt the first time and answers at once after that. + * Resolves with the answer; never throws. + */ +export async function requestNotificationPermission(): Promise { + if (read() === "unsupported") return "unsupported"; + let answer: NotificationPermissionState; + try { + answer = await Notification.requestPermission(); + } catch { + answer = "unsupported"; + } + set(answer); + return answer; +} + +/** Reads the browser's answer again, for a "Check again" after somebody changed a site setting. */ +export function refreshNotificationPermission() { + set(notificationPermission()); +} + +function subscribe(listener: () => void) { + listeners.add(listener); + return () => { + listeners.delete(listener); + }; +} + +/** The browser's answer, kept current across every component that draws it. */ +export function useNotificationPermission(): NotificationPermissionState { + return useSyncExternalStore(subscribe, read, () => "unsupported" as const); +} diff --git a/app/src/lib/plugins/mutations.ts b/app/src/lib/plugins/mutations.ts index 02d3bacf34..c26b9579ae 100644 --- a/app/src/lib/plugins/mutations.ts +++ b/app/src/lib/plugins/mutations.ts @@ -390,10 +390,11 @@ export function registerOAuthClientMutationOptions(queryClient: QueryClient) { /** * Start a consent flow, and say which screen it started from. * - * `returnTo` decides where the vendor's callback puts somebody down, because two screens offer this: - * a person's own connected-accounts page, and the connector's admin page where an administrator - * verifies the setup they have just finished. Sending an administrator to their personal settings - * afterwards is the round trip the inline row exists to remove. + * `returnTo` decides where the vendor's callback puts somebody down, because three screens offer this: + * a person's own connected-accounts page, the connector's admin page where an administrator + * verifies the setup they have just finished, and onboarding's Connect your apps step, which is the + * only page somebody who has not finished onboarding may come back to. Sending an administrator to + * their personal settings afterwards is the round trip the inline row exists to remove. * * A name rather than a URL. The server narrows it to a known set before signing it into the state, * so this parameter cannot become an open redirect however it is called. @@ -430,7 +431,7 @@ export function registerOAuthClientMutationOptions(queryClient: QueryClient) { * so a caller has to say what it draws for it. */ export function connectAccountMutationOptions( - returnTo: "settings" | "admin" = "settings", + returnTo: "settings" | "admin" | "onboarding" = "settings", ) { return mutationOptions({ mutationFn: async (serverId: string): Promise => { diff --git a/app/src/lib/plugins/tool-name.ts b/app/src/lib/plugins/tool-name.ts index 0fcb5630c1..5b2b66a941 100644 --- a/app/src/lib/plugins/tool-name.ts +++ b/app/src/lib/plugins/tool-name.ts @@ -22,9 +22,19 @@ export type ToolName = { detail?: string; }; -export function readToolName(name: string, running = false): ToolName { - const own = ownToolLabel(name, running); - if (own) return { label: own }; +/** + * How a call stands, for the words it is named in: still running ("Searching your email"), done + * ("Searched your email"), or neither (`"plain"`: "Search your email"), for a call that was refused, + * failed or declined, so its line never says something happened that did not. + */ +export type ToolState = boolean | "plain"; + +export function readToolName( + name: string, + running: ToolState = false, +): ToolName { + const own = ownToolLabel(name, running === true); + if (own) return { label: running === "plain" ? humanise(name) : own }; const parts = name.split("__"); if (parts.length < 3 || parts[0] !== "mcp") return { label: name }; @@ -38,7 +48,22 @@ export function readToolName(name: string, running = false): ToolName { * their answer. Which provider is granted is an administrator's fact, on the Bot's access page. */ const role = researchRoleOfTool(server, tool); - if (role) return { label: RESEARCH_ROLE_LABELS[role] }; + if (role) + return { + label: + running === true ? RESEARCH_RUNNING[role] : RESEARCH_ROLE_LABELS[role], + }; + /* + * The person's own mail and calendar are named for what happened to them, whichever app reached + * them, with the app beside the words so two mailboxes are still told apart. See `personalAction` + * for which calls qualify. + */ + const personal = personalAction(server, tool); + if (personal) + return { + label: PERSONAL_WORDS[personal][stateOf(running)], + detail: connectorName(server), + }; /* * A BROKERED APP IS NAMED ONCE, AS THE APP, BESIDE THE ACTION. Composio publishes * `__`, so its app leads every action and reads as a vendor's prefix @@ -111,6 +136,105 @@ const OWN_TOOLS: Record = { }, }; +/** Research while it runs. Done, or refused, it keeps `RESEARCH_ROLE_LABELS`. */ +const RESEARCH_RUNNING = { + search: "Searching the web", + fetch: "Reading web pages", +} as const; + +type PersonalAction = + | "searchEmail" + | "readEmail" + | "sendEmail" + | "checkCalendar"; +type PersonalState = "running" | "done" | "plain"; + +const PERSONAL_WORDS: Record> = { + searchEmail: { + running: "Searching your email", + done: "Searched your email", + plain: "Search your email", + }, + readEmail: { + running: "Reading your email", + done: "Read your email", + plain: "Read your email", + }, + sendEmail: { + running: "Sending your email", + done: "Sent your email", + plain: "Send your email", + }, + checkCalendar: { + running: "Checking your calendar", + done: "Checked your calendar", + plain: "Check your calendar", + }, +}; + +function stateOf(running: ToolState): PersonalState { + return running === "plain" ? "plain" : running ? "running" : "done"; +} + +/** + * A call to the person's own mail or calendar named for what happened to them, or null for any + * other call, which keeps its app's own name. A summary uses this to say "searched your email" + * rather than count a connector. + */ +export function personToolLabel( + name: string, + state: PersonalState = "done", +): string | null { + const parts = name.split("__"); + if (parts.length < 3 || parts[0] !== "mcp") return null; + const action = personalAction(parts[1] ?? "", parts.slice(2).join("__")); + return action ? PERSONAL_WORDS[action][state] : null; +} + +/* + * WHICH CALLS ARE THE PERSON'S MAIL OR CALENDAR, BY THEIR WORDS, FOR ANY APP. A mail app is one the + * server names (`composio-gmail`, `arcade-Outlook`) or the tool leads with (`gmail_search_messages` + * on a server called `google-workspace`); a calendar is a calendar app, or a mail app's events. + * + * Only what can be said truthfully is renamed. A call that changes something (a draft, a label, a + * new event) keeps its own name, because "sent your email" for a draft or "checked your calendar" + * for a new event would say what did not happen; so does a mail app's call that is not about mail + * (contacts, labels, the profile). The rest are named for looking, reading or sending. + */ +const MAIL_APPS = ["gmail", "outlook", "mail", "email", "inbox"]; +const CALENDAR_APPS = ["calendar", "googlecalendar", "gcal"]; +const CALENDAR_WORDS = ["calendar", "calendars", "event", "events"]; +const MAIL_WORDS = [ + ...["email", "emails", "mail", "mails", "inbox"], + ...["message", "messages", "thread", "threads"], +]; +const CHANGES = [ + ...["create", "update", "delete", "remove", "add", "move", "trash"], + ...["modify", "patch", "insert", "archive", "mark", "draft", "drafts"], +]; +const SENDS = ["send", "reply", "forward"]; +const LOOKS = ["search", "list", "find", "query"]; +const READS = ["read", "get", "fetch"]; + +function personalAction(server: string, tool: string): PersonalAction | null { + const appWords = wordsOf(brokeredApp(server) ?? server); + const toolWords = wordsOf(tool); + const leading = toolWords[0] ?? ""; + const has = (words: string[], any: string[]) => + any.some((word) => words.includes(word)); + const mailApp = has(appWords, MAIL_APPS) || MAIL_APPS.includes(leading); + const calendarApp = + has(appWords, CALENDAR_APPS) || CALENDAR_APPS.includes(leading); + if (!(mailApp || calendarApp) || has(toolWords, CHANGES)) return null; + if (calendarApp || has(toolWords, CALENDAR_WORDS)) + return has(toolWords, [...LOOKS, ...READS]) ? "checkCalendar" : null; + if (has(toolWords, SENDS)) return "sendEmail"; + if (!has(toolWords, MAIL_WORDS)) return null; + if (has(toolWords, LOOKS)) return "searchEmail"; + if (has(toolWords, READS)) return "readEmail"; + return null; +} + /** One of this deployment's own tools in words, or null for any other name. */ export function ownToolLabel(name: string, running = false): string | null { const own = OWN_TOOLS[name]; diff --git a/app/src/lib/plugins/tool-result.ts b/app/src/lib/plugins/tool-result.ts index 88f6281045..7af8bc0c0e 100644 --- a/app/src/lib/plugins/tool-result.ts +++ b/app/src/lib/plugins/tool-result.ts @@ -129,6 +129,54 @@ export function toolResultFailed(result: string | undefined): boolean { const answer = asText(result); return ( answer.startsWith("The vendor reported an error:") || - answer.startsWith("That tool could not be called") + answer.startsWith("That tool could not be called") || + // The model SDK's own answer when a call names a tool the run does not hold. + answer.includes("tried to call unavailable tool") ); } + +/** + * The server's reason for not carrying a call out, wherever the call's door put it. + * + * The same reason arrives in three shapes: on its own, once the person has answered a card; after + * {@link REFUSAL_MARKER}, when a connector refuses a call the person has already answered; and in a + * computer tool's JSON, as `error` from the server's computer or `reason` from the browser's. A + * computer's JSON that did not fail carries no reason, and reads as empty. + */ +function reasonNotDone(result: string): string { + const answer = asText(result).trim(); + if (answer.startsWith(REFUSAL_MARKER)) + return answer.slice(REFUSAL_MARKER.length).trim(); + if (!answer.startsWith("{")) return answer; + try { + const parsed: unknown = JSON.parse(answer); + if (!parsed || typeof parsed !== "object") return ""; + const { ok, error, reason } = parsed as Record; + if (ok !== false) return ""; + if (typeof error === "string") return error; + return typeof reason === "string" ? reason : ""; + } catch { + return ""; + } +} + +/** + * Whether a call was not carried out because of the person: `"declined"` when they said no on its + * card, `"alreadyWaiting"` when an identical request was still waiting for their answer, so this + * one was not asked again. Null for anything else. + * + * Read from the first words of `DECLINED_BY_PERSON` and `ALREADY_WAITING` in + * `server/src/approvals/service.ts`, declared at both ends like {@link REFUSAL_MARKER}. Neither is a + * result, and neither is a boundary: a line must never say a send went out when the person said + * no, call it blocked, or say they declined a card they have not answered yet. + */ +export function toolResultHeldBack( + result: string | undefined, +): "declined" | "alreadyWaiting" | null { + if (result === undefined) return null; + const reason = reasonNotDone(result); + if (reason.startsWith("The person declined this action")) return "declined"; + if (reason.startsWith("An identical request is already waiting")) + return "alreadyWaiting"; + return null; +} diff --git a/app/src/lib/setup/mutations.ts b/app/src/lib/setup/mutations.ts new file mode 100644 index 0000000000..7d2b4a1925 --- /dev/null +++ b/app/src/lib/setup/mutations.ts @@ -0,0 +1,21 @@ +import { mutationOptions, type QueryClient } from "@tanstack/react-query"; +import { client } from "@/lib/client"; +import { type SetupChecklist, setupKeys } from "./queries"; + +/** + * "Check again": the server asks Intelligence, the model provider and the computers again, then + * answers. Asking them is the server's job, and it holds the keys; the browser only says when. + * + * Invalidated rather than written into the cache, as every write here is. The read that follows is + * answered from what the check just learned, so it costs a round trip and no second check. + */ +export function recheckSetupMutationOptions(queryClient: QueryClient) { + return mutationOptions({ + mutationFn: (): Promise => + client("/api/setup/check", "checklist", { + method: "POST", + fallback: "Couldn't check your setup.", + }), + onSuccess: () => queryClient.invalidateQueries({ queryKey: setupKeys.all }), + }); +} diff --git a/app/src/lib/setup/queries.ts b/app/src/lib/setup/queries.ts new file mode 100644 index 0000000000..f4774a9d35 --- /dev/null +++ b/app/src/lib/setup/queries.ts @@ -0,0 +1,70 @@ +import { queryOptions } from "@tanstack/react-query"; +import { client } from "@/lib/client"; + +/** + * The setup checklist, as the server decided it for the person asking. See + * server/src/setup/checklist.ts. + * + * Every sentence and every fix comes from the server, already shaped for its reader: an admin is + * told which page to open or which variable to set, and a member is told only that it works or that + * their admin sets it up. The browser draws what it is given and never decides who may see a fix. + * Nothing on it is a secret's value: a key is reported as there or not there. + */ +export type SetupRowId = + | "intelligence" + | "model" + | "computer" + | "apps" + | "chat"; + +export type SetupFix = + | { + kind: "env"; + names: string[]; + docsUrl: string; + page?: { label: string; path: string }; + } + | { kind: "page"; label: string; path: string; docsUrl?: string }; + +/** + * `ready` works, as far as the server can tell. `needs_setup` does not, and the row says why. + * `not_checked` is set up, but nobody has asked whether it works, such as a model key added since + * the server last asked the provider. + */ +export type SetupStatus = "ready" | "needs_setup" | "not_checked"; + +export type SetupRow = { + id: SetupRowId; + /** Every row is the deployment's, which an admin sets up. */ + scope: "deployment"; + title: string; + /** True only where the product cannot answer without it. */ + required: boolean; + status: SetupStatus; + /** `status === "ready"`. */ + ready: boolean; + /** What it gives you when ready, or what happens without it. */ + detail: string; + fix: SetupFix | null; +}; + +export type SetupChecklist = { + /** Who the server answered as: an admin (or somebody running OpenBot alone) or a member. */ + audience: "admin" | "member"; + rows: SetupRow[]; +}; + +export const setupKeys = { + all: ["setup"] as const, + checklist: () => ["setup", "checklist"] as const, +}; + +export function setupChecklistQueryOptions() { + return queryOptions({ + queryKey: setupKeys.checklist(), + queryFn: (): Promise => + client("/api/setup", "checklist", { + fallback: "Couldn't check your setup.", + }), + }); +} diff --git a/app/src/lib/theme.ts b/app/src/lib/theme.ts index 59f663febd..1c613de78b 100644 --- a/app/src/lib/theme.ts +++ b/app/src/lib/theme.ts @@ -1,18 +1,66 @@ -export const THEME_STORAGE_KEY = "openbot-theme"; +/** + * Light, dark, or whatever the system is set to. The default is the system's setting, and it follows + * the system live, so a computer that turns dark at sunset turns OpenBot dark with it. + * + * `index.html` repeats `parseThemePreference` and `resolveTheme` inline, before the first paint, so the app never flashes + * the wrong theme while the bundle loads. The two read the same keys; a test holds them together. + */ -export function parseStoredDarkTheme(value: string | null) { - return value === "dark"; +/** Where the person's choice is kept: "system", "light" or "dark". */ +export const THEME_STORAGE_KEY = "openbot-appearance"; + +/** + * Where OpenBot kept its old switch, as "dark" or "light". It wrote "light" on every load whether or + * not anybody chose it, so only "dark" there means somebody asked for it; "light" there is the old + * default and now means "follow the system". + */ +export const LEGACY_THEME_STORAGE_KEY = "openbot-theme"; + +/** The media query the system's setting arrives on. */ +export const SYSTEM_DARK_QUERY = "(prefers-color-scheme: dark)"; + +export type ThemePreference = "system" | "light" | "dark"; +export type ResolvedTheme = "light" | "dark"; + +export const THEME_PREFERENCES: readonly ThemePreference[] = [ + "system", + "light", + "dark", +]; + +/** + * The person's choice, from what is stored under the two keys. Anything not a choice, including + * nothing at all, is "system". + */ +export function parseThemePreference( + stored: string | null, + legacy: string | null = null, +): ThemePreference { + if (stored === "system" || stored === "light" || stored === "dark") + return stored; + return legacy === "dark" ? "dark" : "system"; +} + +/** The theme to draw: the choice, or the system's when the choice is to follow it. */ +export function resolveTheme( + preference: ThemePreference, + systemDark: boolean, +): ResolvedTheme { + if (preference === "system") return systemDark ? "dark" : "light"; + return preference; } type ThemeEffects = { - setStoredValue: (key: string, value: string) => void; toggleRootClass: (name: string, force: boolean) => void; - setRootColorScheme: (scheme: "dark" | "light") => void; + setRootColorScheme: (scheme: ResolvedTheme) => void; }; -export function applyDarkTheme(dark: boolean, effects: ThemeEffects) { - effects.setStoredValue(THEME_STORAGE_KEY, dark ? "dark" : "light"); - effects.toggleRootClass("dark", dark); - // `index.html` sets this inline before paint, and an inline style outranks the palette. - effects.setRootColorScheme(dark ? "dark" : "light"); +/** + * Draw a theme: the `dark` class the tokens hang off, and the colour scheme the browser draws its + * own controls and scrollbars in. `index.html` sets the scheme inline before paint, and an inline + * style outranks the stylesheet, so this has to set it too. + */ +export function applyTheme(theme: ResolvedTheme, effects: ThemeEffects) { + effects.toggleRootClass("dark", theme === "dark"); + effects.setRootColorScheme(theme); } diff --git a/app/src/lib/utils.ts b/app/src/lib/utils.ts index 365058cebd..ee2cf32355 100644 --- a/app/src/lib/utils.ts +++ b/app/src/lib/utils.ts @@ -1,5 +1,31 @@ import { type ClassValue, clsx } from "clsx"; -import { twMerge } from "tailwind-merge"; +import { extendTailwindMerge } from "tailwind-merge"; + +/** + * The five sizes of the type scale (`theme.css`). + * + * Each sets a font size, a line height and tracking. `tailwind-merge` only knows Tailwind's own + * utilities, so without this it keeps `text-sm` beside `type-body` and the stylesheet's order picks + * `text-sm`. Registered as font sizes, a `type-*` given later replaces an earlier `text-*` size, and + * drops the `leading-*` that came with it. + */ +export const TYPE_SCALE_CLASSES = [ + "type-display", + "type-nav", + "type-body", + "type-sub", + "type-caption", +] as const; + +const twMerge = extendTailwindMerge({ + extend: { + classGroups: { + "font-size": [ + { type: TYPE_SCALE_CLASSES.map((name) => name.slice("type-".length)) }, + ], + }, + }, +}); export function cn(...inputs: ClassValue[]) { return twMerge(clsx(inputs)); diff --git a/app/src/routes/__root.tsx b/app/src/routes/__root.tsx index 932492d23b..f05bcfe1fd 100644 --- a/app/src/routes/__root.tsx +++ b/app/src/routes/__root.tsx @@ -37,9 +37,9 @@ export const Route = createRootRouteWithContext()({ * app-wide guard could not be installed by a leaf: several composers can be mounted at once, so a * `document` listener owned by one of them would be installed once per composer and torn down by * whichever unmounted first. The leaf approach also cannot reach every surface even in principle. - * The onboarding poster (`_authed/onboarding.tsx`) wraps its decorative composer in - * `pointer-events-none` on purpose, so that composer can never receive a drop at all: the event - * goes straight past it to the document. So do the transcript, the sidebar, the page margin, and + * Onboarding's pictures (`components/onboarding/illustration-frame.tsx`), its drawing of a + * composer among them, are `pointer-events-none` on purpose, so nothing in them can ever receive a + * drop at all: the event goes straight past them to the document. So do the transcript, the sidebar, the page margin, and * every screen that will ever be added without thinking about drag and drop. The root is the one * place where "once" is a fact rather than a hope: `RootComponent` mounts once per app. * diff --git a/app/src/routes/_authed/_app/bot.tsx b/app/src/routes/_authed/_app/bot.tsx index f5c41b5ab9..5935643c08 100644 --- a/app/src/routes/_authed/_app/bot.tsx +++ b/app/src/routes/_authed/_app/bot.tsx @@ -13,6 +13,7 @@ import { agentKeys, agentListQueryOptions, } from "@/lib/agents/queries"; +import { approvalInboxOptions } from "@/lib/approvals"; import { tryClient } from "@/lib/client"; import { useActiveBot } from "@/lib/copilot/active-bot"; import { useBotThread } from "@/lib/copilot/bot-thread"; @@ -142,6 +143,8 @@ function RouteComponent() { function BotChat({ agentId, name }: { agentId: string; name: string }) { // Tool calls here act on this Bot's own computer. useActiveBot(agentId); + // This screen's one poll of the inbox. The approval cards in the chat read the cache it keeps fresh. + useQuery(approvalInboxOptions()); /* * Minted by this deployment rather than by the chat. `history` reports whether Intelligence * still recognised the thread this browser remembered from a previous visit; when it did not, diff --git a/app/src/routes/_authed/_app/channel/$channelId.tsx b/app/src/routes/_authed/_app/channel/$channelId.tsx index 5cc23c2d17..fb93815e40 100644 --- a/app/src/routes/_authed/_app/channel/$channelId.tsx +++ b/app/src/routes/_authed/_app/channel/$channelId.tsx @@ -2,18 +2,32 @@ import { IconSettings } from "@tabler/icons-react"; import { useQuery } from "@tanstack/react-query"; import { createFileRoute, Navigate } from "@tanstack/react-router"; import { motion, useReducedMotion } from "motion/react"; +import { useRef, useState } from "react"; import { z } from "zod"; import { AgentProfile } from "@/components/agents/agent-profile"; import { BotPausedBanner } from "@/components/bot-profile/pause-banner"; import { ChannelAvatar } from "@/components/channels/avatar"; import { ChannelChat } from "@/components/channels/channel-chat"; import { ConversationActions } from "@/components/channels/conversation-actions"; -import { ComputerChatControls } from "@/components/computer/computer-controls"; -import { ComputerViewPanel } from "@/components/computer/computer-panel"; +import { + ConversationOrb, + ConversationPresenceProvider, + ConversationStatusLine, +} from "@/components/channels/conversation-presence"; +import { + ComputerChatControls, + ComputerControlButton, +} from "@/components/computer/computer-controls"; +import { + ComputerPaneTitle, + ComputerViewPanel, +} from "@/components/computer/computer-panel"; +import type { ComputerViewHandle } from "@/components/computer/computer-view"; import { useAutoWatchScreen } from "@/components/computer/use-auto-watch-screen"; import { DetailPanel } from "@/components/layout/detail-panel"; import { SidebarToggle } from "@/components/layout/sidebar-toggle"; import { Button } from "@/components/ui/button"; +import { agentListQueryOptions } from "@/lib/agents/queries"; import { useMarkOpenChannelRead } from "@/lib/channels/mark-read"; import { type AgentChannel, channelQueryOptions } from "@/lib/channels/queries"; @@ -28,15 +42,35 @@ const EASE_OUT = [0.23, 1, 0.32, 1] as const; const HEADING_ENTRANCE_SECONDS = 0.18; const HEADING_ENTRANCE_OFFSET = "translateY(4px)"; -/** Shared detail pane width for the live screen view. */ -const SCREEN_PANEL_WIDTH = 400; +/** The live screen's pane: wide enough for a readable screen and the steps under it. */ +const SCREEN_PANEL_WIDTH = 440; + +/** A conversation's Bot in its header: 40px, the smallest orb that is fully animated. */ +const HEADER_ORB_SIZE = 40; export const Route = createFileRoute("/_authed/_app/channel/$channelId")({ validateSearch: chatSearchSchema, component: RouteComponent, }); +/** + * The conversation's presence wraps everything on the route, so its header and the pane beside it + * read the same conversation; `ChannelChat` reports into it. + */ function RouteComponent() { + const { channelId } = Route.useParams(); + const channel = useQuery(channelQueryOptions(channelId)); + return ( + + + + ); +} + +function Conversation() { const { channelId } = Route.useParams(); const { settings, watch } = Route.useSearch(); const channel = useQuery(channelQueryOptions(channelId)); @@ -46,14 +80,36 @@ function RouteComponent() { const isWatching = watch === true; /** Channel routing currently supports one coworker. */ const agentId = channel.data?.agentIds[0]; + /* + * The Bot's own name, for its computer. The conversation's name is what the conversation is about, + * and a screen badged "Q3 budget" says nothing about whose computer it is. + */ + const botName = useQuery(agentListQueryOptions()).data?.find( + (agent) => agent.id === agentId, + )?.name; + /** Whether the pane would be a sheet over the conversation, as `DetailPanel` last measured. */ + const [overlaying, setOverlaying] = useState(true); + /** The screen in the pane, which the header's Take control opens full size. */ + const screen = useRef(null); + /* + * A group is on its way to its own route (`ChannelBody`) and keeps its small faces until then. + * Anything else, a conversation still loading included, has room for one Bot's orb, so the header + * does not grow once the conversation arrives. + */ + const group = + channel.data !== undefined && channel.data.agentIds.length !== 1; useMarkOpenChannelRead(channelId); - // Browser activity may auto-open the screen once per run unless this run was dismissed. + /* + * Browser activity may auto-open the screen once per run unless this run was dismissed, and only + * where it opens beside the conversation. On a narrow screen it lights the computer button. + */ const autoWatch = useAutoWatchScreen({ botId: agentId, watching: isWatching, busy: isSettingsOpen, + docked: !overlaying, open: () => navigate({ search: (previous) => ({ @@ -83,12 +139,35 @@ function RouteComponent() { show(null)} open={(isSettingsOpen || isWatching) && agentId !== undefined} - detailWidth={isWatching ? SCREEN_PANEL_WIDTH : undefined} - title={isWatching ? "Computer" : undefined} + /* + * The screen's width unless the Bot's settings are open, closed included, so whether it would + * overlay is measured for the pane that opens by itself. + */ + detailWidth={isSettingsOpen ? undefined : SCREEN_PANEL_WIDTH} + onOverlayChange={setOverlaying} + title={ + isWatching && agentId !== undefined ? ( + + ) : undefined + } + actions={ + isWatching && agentId !== undefined ? ( + screen.current?.openFullSize()} + /> + ) : undefined + } detail={ agentId === undefined ? null : isWatching ? ( // Manual watch remains active even when there is no current browser action. - + ) : ( - + {group ? ( + + ) : ( + + )} - - {channel.data?.name ?? "Channel"} - + {/* The name, and under it what the Bot is doing. At rest the name sits level with the orb. */} +
    + + {channel.data?.name ?? "Channel"} + + {group ? null : } +
    show(open ? "watch" : null)} + unseen={autoWatch.unseen} /> +
    + ) : undefined; + const intro: OnboardingStep[] = [ + welcome, + computer, + team, + approval, + ...(capabilities?.localEvaluation ? [selfhost] : []), + ].map((step) => ({ + ...step, + primaryDisabled: !capabilities, + notice: capabilitiesFailed, + secondary: { label: "Skip intro", onClick: () => move("apps", "forward") }, + })); + const steps: OnboardingStep[] = [ + ...intro, + /* + * Two optional steps, before the checklist. Continue and Not now both move on, and Not now + * writes nothing; the checklist after them says what is left to set up. + */ + { + id: "apps", + title: "Connect apps", + intro: () => ( + +

    + Connect the accounts your Bots use as you. You can connect more + later in Connections. +

    +     + ), + demo: () => ( + leaveForConsent("apps", url, app)} + openPage={(target) => move("name", "forward", target.path)} + returned={returnedApp} + /> + ), + secondary: { label: "Not now" }, + optional: true, + }, + notify, + { + id: "setup", + title: "Check your setup", + intro: () => ( + +

    See what's ready and what still needs setting up.

    +     + ), + demo: () => ( + move("name", "forward", target.path)} + /> + ), + }, + { + id: "name", + title: "Your name", + intro: () => , + demo: () => , + demoFirst: true, + ownAction: true, + notice: openLabel ? ( +

    + {openLabel} opens once you get started. +

    + ) : undefined, + }, + ]; + + // A self-hosting step in the address waits for the answer that puts it in the list, rather + // than drawing step 1 for a moment and then jumping. + const waiting = askingCapabilities && current === "selfhost"; + + // The rewrite described at `returned`, once the list of steps is known. The order as one string, + // so the effect runs when the list changes rather than on every render that rebuilds it. + const order = steps.map((step) => step.id).join(" "); + React.useEffect(() => { + if (returned === null || askingCapabilities) return; + dropConsentDeparture(); + setReturned(null); + const ids = order.split(" "); + const at = ids.indexOf(returned); + const before = departure?.from ?? ids[at - 1]; + if (at === -1 || !isStepId(before)) return; + /* + * Through the history itself, flushed between the two: the router queues history writes and + * folds a replace followed by a push into one push, which left the arrival entry as it was. + */ + const href = (step: StepId) => + router.buildLocation({ to: "/onboarding", search: { step } }).href; + router.history.replace(href(before), {}); + router.history.flush(); + const state: Record = { onboardingFrom: before }; + router.history.push(href(returned), state); + }, [returned, askingCapabilities, order, router, departure]); return ( // Outside `_app` on purpose: no sidebar and no chrome until onboarding is done. // The fade runs only after the completion is saved, so a failed save never fades a page the - // person still needs — and navigation waits for the fade, so the home screen never pops in - // over a half-faded wizard. + // person still needs, and navigation waits for the fade, so the next screen never pops in over + // a half-faded onboarding. { - if (leaving) { - navigate({ to: "/" }); - } - }} - transition={{ duration: 0.5, ease: "easeInOut" }} + transition={{ duration: reduced ? 0 : 0.3, ease: "easeOut" }} > -
    - - {/* The frame follows each pane's height, so the buttons glide instead of jumping. */} - 0 ? bounds.height : "auto" }} - className="overflow-hidden" - > -
    - - - {step === "intro" && ( - - )} - {step === "organization" && } - {step === "name" && ( - { - await complete.mutateAsync(undefined); - setLeaving(true); - }} - /> - )} - - - - {capabilitiesError && !capabilities ? ( -
    -

    - We couldn't load your setup. Try again to continue. -

    - -
    - ) : null} - - {complete.error ? ( -

    - {complete.error.message} -

    - ) : null} - - - {!last && ( - - )} - {step !== "intro" && ( - - )} - -
    -     -     -
    + {waiting ? null : ( + move(to, way, search.open)} + stepId={current} + steps={steps} + /> + )}     ); } diff --git a/app/src/routes/_authed/settings/index.tsx b/app/src/routes/_authed/settings/index.tsx index 9d52eb0016..c0c6febc3d 100644 --- a/app/src/routes/_authed/settings/index.tsx +++ b/app/src/routes/_authed/settings/index.tsx @@ -16,17 +16,30 @@ import { ItemDescription, ItemTitle, } from "@/components/ui/item"; +import { + Select, + SelectContent, + SelectItem, + SelectTrigger, + SelectValue, +} from "@/components/ui/select"; import { Separator } from "@/components/ui/separator"; -import { Switch } from "@/components/ui/switch"; import { formatHotkey, HOTKEYS } from "@/lib/hotkeys/hotkeys"; +import { THEME_PREFERENCES, type ThemePreference } from "@/lib/theme"; import { SkinPicker } from "@/skins/skin-picker"; export const Route = createFileRoute("/_authed/settings/")({ component: RouteComponent, }); +const APPEARANCE: Record = { + system: "System", + light: "Light", + dark: "Dark", +}; + function RouteComponent() { - const { dark, setDark } = useTheme(); + const { preference, setPreference } = useTheme(); /* * The measurements that used to be written out here now live in `PageShell`, which Skills, Admin @@ -44,19 +57,32 @@ function RouteComponent() { + {/* Three choices, so a select like the message list's below, not a switch. */} - Dark theme + Appearance - Use the dark appearance across OpenBot. + System follows your device's light or dark setting. - + diff --git a/app/src/skins/opendots/sidebar.tsx b/app/src/skins/opendots/sidebar.tsx index 7011d53dca..22599a8ef4 100644 --- a/app/src/skins/opendots/sidebar.tsx +++ b/app/src/skins/opendots/sidebar.tsx @@ -207,8 +207,6 @@ function BotsSection({ className={cn( navItem, "min-h-[42px] gap-[9px] px-2.5 py-1.5 text-[12px] text-(--opendots-dot-ink) hover:bg-(--opendots-hover)", - // The row's only drawing is the avatar, which keeps its own size. - "[&_svg]:size-full", active && "bg-(--opendots-active) hover:bg-(--opendots-active)", )} diff --git a/app/src/skins/opendots/skin.css b/app/src/skins/opendots/skin.css index caaf719ffa..0eaf931ccd 100644 --- a/app/src/skins/opendots/skin.css +++ b/app/src/skins/opendots/skin.css @@ -201,12 +201,24 @@ line-height: 1.75; } -/* A Bot's message is plain prose on the canvas, a shade softer than the ink. */ +/* + * A Bot's message is plain prose on the canvas, a shade softer than the ink. OpenBot draws each of + * a Bot's thoughts in a bubble of its own, so the bubble's fill, padding, corners and cap come off. + */ +:root[data-skin="opendots"] + [data-opendots-main] + [data-slot="bubble"][data-align="start"][data-variant="ghost"] { + max-width: 100%; +} + :root[data-skin="opendots"] [data-opendots-main] [data-slot="bubble"][data-align="start"][data-variant="ghost"] > [data-slot="bubble-content"] { color: var(--opendots-assistant-ink); + background: transparent; + padding: 0; + border-radius: 0; font-size: 14px; line-height: 1.75; } @@ -260,7 +272,7 @@ /* * Where the conversation column is no wider than its composer, OpenDots still keeps the composer - * and its notices off the edges of the pane; OpenBot's runs them flush. 972px is the widest window + * and its notices 16px off the edges of the pane, a little more than OpenBot's own 12px. 972px is the widest window * at which the 672px column fills the pane beside the rail and sidebar. */ @media (max-width: 972px) { diff --git a/app/src/skins/openduet/header.tsx b/app/src/skins/openduet/header.tsx index 85e81c0618..1880ffc8ad 100644 --- a/app/src/skins/openduet/header.tsx +++ b/app/src/skins/openduet/header.tsx @@ -9,7 +9,7 @@ import { } from "@tanstack/react-router"; import { useEffect, useState } from "react"; import AgentOrb from "@/components/agents/orb/agent-orb"; -import type { AIState } from "@/components/agents/orb/ai-core"; +import type { AIState } from "@/components/agents/orb/agent-orb"; import { currentUserQueryOptions } from "@/lib/auth/queries"; import { type ChannelWork, diff --git a/app/src/skins/openduet/skin.css b/app/src/skins/openduet/skin.css index 333cfcfea7..a01840ba07 100644 --- a/app/src/skins/openduet/skin.css +++ b/app/src/skins/openduet/skin.css @@ -279,7 +279,8 @@ /* * A conversation's composer floats inset from the column's edges, as OpenDuet's does, with its - * focus ring off the screen's edge. OpenBot draws it in a column with no side padding. + * focus ring off the screen's edge. OpenBot's own column insets it by the same 12px now; the frame + * keeps its rule so the inset holds whatever the default does. */ [data-skin="openduet"] [data-openduet-frame] [data-slot="composer-column"] { padding-inline: 12px; diff --git a/app/src/skins/openmuse/frame.tsx b/app/src/skins/openmuse/frame.tsx index 5108ce9f4b..c996720ee9 100644 --- a/app/src/skins/openmuse/frame.tsx +++ b/app/src/skins/openmuse/frame.tsx @@ -10,7 +10,12 @@ import { } from "@tabler/icons-react"; import { useInfiniteQuery, useQuery } from "@tanstack/react-query"; import { Link, useParams, useRouterState } from "@tanstack/react-router"; -import { type ReactNode, useEffect, useState } from "react"; +import { + type ReactNode, + useEffect, + useState, + useSyncExternalStore, +} from "react"; import { AbstractAvatar } from "@/components/agents/abstract-avatar"; import AgentOrb from "@/components/agents/orb/agent-orb"; import { BotsNavBadge } from "@/components/bot-profile/attention"; @@ -53,6 +58,25 @@ const TABS: { id: MuseTab; label: string; icon: MuseIcon }[] = [ { id: "apps", label: "Apps", icon: IconTriangleSquareCircle }, ]; +const WIDE = "(min-width: 900px)"; + +/** + * Whether the window is wide enough for the header's larger face, kept current as it is resized. + * The face is drawn on a canvas at a size in px, so the header picks the size here rather than in + * its classes. + */ +function useWide(): boolean { + return useSyncExternalStore( + (onChange) => { + const query = window.matchMedia(WIDE); + query.addEventListener("change", onChange); + return () => query.removeEventListener("change", onChange); + }, + () => window.matchMedia(WIDE).matches, + () => false, + ); +} + /** * OpenMuse's shell around every OpenBot page: one centred column, the agent at the top, a bell for * what waits on you, and a floating pill of five tabs. The page itself is OpenBot's, untouched — @@ -89,6 +113,8 @@ export function OpenMuseFrame({ children }: { children: ReactNode }) { const userId = user.data?.id; const chosenId = useChosenBotId(userId); const [sheet, setSheet] = useState(null); + // The centre face, in px: 44 on a phone, 52 from 900px wide, where the header grows. + const face = useWide() ? 52 : 44; /* * The Bot this screen is about, when it is about one: the Bot of a one-to-one conversation, the Bot @@ -164,15 +190,15 @@ export function OpenMuseFrame({ children }: { children: ReactNode }) { * The Bot's own OpenBot avatar, the one its page and the Bots roster draw. With no Bot * yet, the deployment's own mark stands in. */} - + {bot ? ( ) : ( - + )} diff --git a/app/src/styles.css b/app/src/styles.css index 46f20924b4..f73ab2b140 100644 --- a/app/src/styles.css +++ b/app/src/styles.css @@ -62,6 +62,36 @@ transform: rotate(90deg); } +/* + * The Bot thinking, before it has said anything: three dots in its bubble, each rising a little and + * brightening, one after another. Only `opacity` and `transform` animate. + */ +.typing-dot { + opacity: 0.4; + animation: typing-dot 1.1s ease-in-out infinite; +} + +@keyframes typing-dot { + 0%, + 60%, + 100% { + opacity: 0.4; + transform: translateY(0); + } + 30% { + opacity: 0.85; + transform: translateY(-2px); + } +} + +/* Held still with reduced motion, and a little darker, so the three dots still read as a mark. */ +@media (prefers-reduced-motion: reduce) { + .typing-dot { + animation: none; + opacity: 0.6; + } +} + /* * A generated interface is never allowed to occupy nothing. * @@ -121,6 +151,22 @@ } } +/* + * The compact composer's editor. PromptArea draws its placeholder as an overlay beside an editor + * that is empty, so the box keeps one line's height of its own at any text size, and a long + * placeholder ("Message" and a long Bot name) ends in an ellipsis on that line instead of wrapping + * under the controls. + */ +.composer-editor > .relative { + min-height: 1lh; +} +.composer-editor > .relative > [aria-hidden="true"] { + max-width: 100%; + overflow: hidden; + white-space: nowrap; + text-overflow: ellipsis; +} + /* * A conversation's newest rows rendering in as it opens. The delay is set per row as * `--arrive-delay`; see diff --git a/app/src/theme.css b/app/src/theme.css index 8e30caa8b0..6fb52b7d38 100644 --- a/app/src/theme.css +++ b/app/src/theme.css @@ -1,76 +1,162 @@ @custom-variant dark (&:is(.dark *)); +/* + * OpenBot's visual system, in the variables every screen already reads. Neutral greys with a cool + * bias; a near-black primary that turns light in the dark; one accent, `--brand`, kept for focus, + * selection and links and never for headings (shadcn's `--accent` stays the neutral hover fill); it + * reaches the focus ring today. Four status colours for how work stands, and the glass and shadow + * below, are here for the screens that adopt them. Every colour here has a dark value in `.dark` below, and + * `tests/visual-system.test.ts` fails if one is missing or a colour that carries words falls under + * 4.5:1 on a ground it sits on. + */ :root { color-scheme: light; - --background: oklch(0.985 0 0); - --foreground: oklch(0.145 0 0); - --card: oklch(1 0 0); - --card-foreground: oklch(0.145 0 0); - --popover: oklch(1 0 0); - --popover-foreground: oklch(0.145 0 0); - --primary: oklch(0.205 0 0); - --primary-foreground: oklch(0.985 0 0); - --secondary: oklch(0.97 0 0); - --secondary-foreground: oklch(0.205 0 0); - --muted: oklch(0.96 0 0); - --muted-foreground: oklch(0.5 0 0); - --accent: oklch(0.97 0 0); - --accent-foreground: oklch(0.205 0 0); - --destructive: oklch(0.577 0.245 27.325); - --success: oklch(0.6 0.118 184.704); - --border: oklch(0.922 0 0); - --input: oklch(0.922 0 0); - --ring: oklch(0.708 0 0); + --background: #f6f6f8; + --foreground: #18181b; + --card: #ffffff; + --card-foreground: #18181b; + --popover: #ffffff; + --popover-foreground: #18181b; + --primary: #111114; + --primary-foreground: #ffffff; + --secondary: #f1f1f4; + --secondary-foreground: #18181b; + --muted: #f1f1f4; + --muted-foreground: #6b6b75; + --accent: #f1f1f4; + --accent-foreground: #18181b; + --destructive: #b91c1c; + --success: #147a3a; + --border: rgb(24 24 27 / 9%); + --input: rgb(24 24 27 / 12%); + --ring: #6430ab; --chart-1: oklch(0.87 0 0); --chart-2: oklch(0.556 0 0); --chart-3: oklch(0.439 0 0); --chart-4: oklch(0.371 0 0); --chart-5: oklch(0.269 0 0); - --radius: 0.5rem; - --sidebar: oklch(0.97 0 0); - --sidebar-foreground: oklch(0.145 0 0); - --sidebar-primary: oklch(0.205 0 0); - --sidebar-primary-foreground: oklch(0.5 0 0); - --sidebar-accent: oklch(0.95 0 0); - --sidebar-accent-foreground: oklch(0.205 0 0); - --sidebar-border: oklch(0.922 0 0); - --sidebar-ring: oklch(0.708 0 0); + --radius: 0.55rem; + --sidebar: #f1f1f4; + --sidebar-foreground: #18181b; + --sidebar-primary: #6430ab; + --sidebar-primary-foreground: #ffffff; + /* A row lifts to the card's white on the sidebar's grey, the way a selected segment does. */ + --sidebar-accent: #ffffff; + --sidebar-accent-foreground: #18181b; + --sidebar-border: rgb(24 24 27 / 9%); + --sidebar-ring: #6430ab; + + /* The one accent, and what sits on it. */ + --brand: #6430ab; + --brand-foreground: #ffffff; + /* How work stands: ready for review, needs you, working, couldn't finish. Only on status. */ + --status-done: #147a3a; + --status-done-soft: #e6f4ea; + --status-needs: #a34a06; + --status-needs-soft: #fdf3e1; + --status-working: #1d4ed8; + --status-working-soft: #eaf0fd; + --status-problem: #b91c1c; + --status-problem-soft: #fbeaea; + /* + * A conversation's bubbles: a Bot's in a grey a shade darker than the page, the person's in a + * tint of the accent. One radius on every corner and no tail. `--radius-bubble` sits in + * Tailwind's `--radius-*` namespace like `--shadow-*` below, so it is not registered either: use + * it as `rounded-(--radius-bubble)`. + */ + --bubble: #eeeef1; + --bubble-user: #e7e3f7; + --radius-bubble: 1.25rem; + /* + * Not registered with Tailwind: `--shadow-*` is its namespace, and a theme key of the same name + * would point at itself. Use them as `shadow-(--shadow-float)`. + */ + --shadow-float: + 0 0 0 0.5px rgb(1 5 7 / 10%), 0 1px 1.5px rgb(1 5 7 / 6%), + 0 4px 12px -4px rgb(1 5 7 / 12%); + --shadow-card: 0 0 0 1px var(--border), 0 6px 18px -12px rgb(1 5 7 / 18%); + --shadow-raised: 0 0 0 1px var(--border), 0 10px 26px -14px rgb(1 5 7 / 20%); + /* + * Glass: the chrome that floats over a screen lets what scrolls under it show through, frosted. + * Translucent enough to read as a layer, opaque enough that text under it never competes with the + * controls on it. `--veil` is the same material laid over a whole screen, for a sheet that the + * screen stays visible behind. The `glass` and `veil` utilities below draw them. + */ + --glass: rgb(255 255 255 / 80%); + --veil: rgb(246 246 248 / 74%); + --glass-edge: inset 0 0.5px 0 rgb(255 255 255 / 80%); + --shadow-glass: var(--glass-edge), var(--shadow-float); + /* The smallest thing a finger can hit. */ + --target: 44px; + /* + * Motion. `--ease-out` and `--ease-in` are Tailwind's own names, so `ease-out` and `ease-in` draw + * these curves; `--ease-out` is the one `lib/motion.ts` already uses. + */ + --ease-out: cubic-bezier(0.23, 1, 0.32, 1); + --ease-in: cubic-bezier(0.7, 0, 0.84, 0); + --dur-tap: 120ms; + --dur-swap: 200ms; + --dur-enter: 250ms; + --dur-sheet: 350ms; } +/* Near-black and neutral, the same roles, and the primary action turns light. */ .dark { color-scheme: dark; - --background: oklch(0.145 0 0); - --foreground: oklch(0.985 0 0); - --card: oklch(0.205 0 0); - --card-foreground: oklch(0.985 0 0); - --popover: oklch(0.205 0 0); - --popover-foreground: oklch(0.985 0 0); - --primary: oklch(0.922 0 0); - --primary-foreground: oklch(0.205 0 0); - --secondary: oklch(0.269 0 0); - --secondary-foreground: oklch(0.985 0 0); - --muted: oklch(0.269 0 0); - --muted-foreground: oklch(0.708 0 0); - --accent: oklch(0.269 0 0); - --accent-foreground: oklch(0.985 0 0); - --destructive: oklch(0.704 0.191 22.216); - --success: oklch(0.704 0.14 182.503); - --border: oklch(1 0 0 / 10%); - --input: oklch(1 0 0 / 15%); - --ring: oklch(0.556 0 0); + --background: #111113; + --foreground: #f4f4f5; + --card: #1c1c1f; + --card-foreground: #f4f4f5; + --popover: #1c1c1f; + --popover-foreground: #f4f4f5; + --primary: #f4f4f5; + --primary-foreground: #111114; + --secondary: #26262b; + --secondary-foreground: #f4f4f5; + --muted: #26262b; + --muted-foreground: #8e8e98; + --accent: #26262b; + --accent-foreground: #f4f4f5; + --destructive: #f87171; + --success: #4ade80; + --border: rgb(255 255 255 / 10%); + --input: rgb(255 255 255 / 15%); + --ring: #b49ae8; --chart-1: oklch(0.87 0 0); --chart-2: oklch(0.556 0 0); --chart-3: oklch(0.439 0 0); --chart-4: oklch(0.371 0 0); --chart-5: oklch(0.269 0 0); - --sidebar: oklch(0.205 0 0); - --sidebar-foreground: oklch(0.985 0 0); - --sidebar-primary: oklch(0.488 0.243 264.376); - --sidebar-primary-foreground: oklch(0.985 0 0); - --sidebar-accent: oklch(0.269 0 0); - --sidebar-accent-foreground: oklch(0.985 0 0); - --sidebar-border: oklch(1 0 0 / 10%); - --sidebar-ring: oklch(0.556 0 0); + --sidebar: #18181b; + --sidebar-foreground: #f4f4f5; + --sidebar-primary: #b49ae8; + --sidebar-primary-foreground: #1a1033; + --sidebar-accent: #26262b; + --sidebar-accent-foreground: #f4f4f5; + --sidebar-border: rgb(255 255 255 / 10%); + --sidebar-ring: #b49ae8; + + --brand: #b49ae8; + /* Dark text on the light lilac: white on it was 2.4:1. */ + --brand-foreground: #1a1033; + --status-done: #4ade80; + --status-done-soft: #14301f; + --status-needs: #fbbf24; + --status-needs-soft: #3a2a0c; + --status-working: #7aa2ff; + --status-working-soft: #18233d; + --status-problem: #f87171; + --status-problem-soft: #3a1717; + --bubble: #26262b; + --bubble-user: #2f2845; + --shadow-float: + 0 0 0 0.5px rgb(255 255 255 / 10%), 0 4px 12px -4px rgb(0 0 0 / 50%); + --shadow-card: 0 0 0 1px var(--border); + --shadow-raised: 0 0 0 1px var(--border), 0 10px 26px -14px rgb(0 0 0 / 60%); + --glass: rgb(28 28 31 / 82%); + --veil: rgb(17 17 19 / 72%); + --glass-edge: inset 0 0.5px 0 rgb(255 255 255 / 8%); + --shadow-glass: var(--glass-edge), var(--shadow-float); } * { @@ -102,23 +188,11 @@ body { */ /* - * Text is sized on a 15px base while everything else is on Tailwind's 16px rem. - * - * The root stays at the browser's 16px so spacing, sizes and radii land on Tailwind's 4px grid - * (`p-4` is 16px, `h-8` is 32px). Only type keeps the scale it had under the old 15px root: each - * size below is Tailwind's default multiplied by 15/16, so `text-sm` is still 13.125px. Line - * heights are Tailwind's own unitless ratios and scale with them. `body` reads `--text-base`, so - * unclassed text is 15px too. + * No `--text-*` here. Text follows the browser's own size, 16px by default, on Tailwind's own scale, + * like the spacing and the radii; OpenBot used to set every size to 15/16 of Tailwind's. The type + * scale below is in rem too, so it follows the person's text size. */ @theme inline { - --text-xs: 0.703125rem; /* 11.25px */ - --text-sm: 0.8203125rem; /* 13.125px */ - --text-base: 0.9375rem; /* 15px */ - --text-lg: 1.0546875rem; /* 16.875px */ - --text-xl: 1.171875rem; /* 18.75px */ - --text-2xl: 1.40625rem; /* 22.5px */ - --text-3xl: 1.7578125rem; /* 28.125px */ - --text-4xl: 2.109375rem; /* 33.75px */ --font-heading: var(--font-sans); --font-sans: "Inter Variable", sans-serif; --color-sidebar-ring: var(--sidebar-ring); @@ -160,6 +234,83 @@ body { --radius-2xl: calc(var(--radius) * 1.8); --radius-3xl: calc(var(--radius) * 2.2); --radius-4xl: calc(var(--radius) * 2.6); + --color-brand: var(--brand); + --color-brand-foreground: var(--brand-foreground); + --color-status-done: var(--status-done); + --color-status-done-soft: var(--status-done-soft); + --color-status-needs: var(--status-needs); + --color-status-needs-soft: var(--status-needs-soft); + --color-status-working: var(--status-working); + --color-status-working-soft: var(--status-working-soft); + --color-status-problem: var(--status-problem); + --color-status-problem-soft: var(--status-problem-soft); + --color-bubble: var(--bubble); + --color-bubble-user: var(--bubble-user); +} + +/* + * The type scale: five sizes, in rem so they follow the person's text size. Inter's tracking tightens + * as it grows. Weights are not part of a size: 600 for titles, 500 for row titles, chips and + * buttons, 400 otherwise. `cn` knows these as font sizes (`lib/utils.ts`), so one given later + * replaces a primitive's `text-*` and the `leading-*` that came with it. + * + * type-display 24/28 screen titles + * type-nav 18/22 navigation rows, Settings rows + * type-body 17/23 messages, task and card titles + * type-sub 15/20 status lines, chips, the composer, section headers, buttons + * type-caption 13/16 captions, labels, the footer, timestamps, tags + */ +@utility type-display { + font-size: 1.5rem; + line-height: 1.75rem; + letter-spacing: -0.018em; +} +@utility type-nav { + font-size: 1.125rem; + line-height: 1.375rem; + letter-spacing: -0.011em; +} +@utility type-body { + font-size: 1.0625rem; + line-height: 1.4375rem; + letter-spacing: -0.011em; +} +@utility type-sub { + font-size: 0.9375rem; + line-height: 1.25rem; + letter-spacing: -0.006em; +} +@utility type-caption { + font-size: 0.8125rem; + line-height: 1rem; + letter-spacing: 0; +} +/* + * Frosted chrome over a scrolling screen. The blur is what makes the translucency readable: without + * it, a message passing under the composer would sit legibly behind its placeholder. A browser that + * cannot blur gets the opaque card instead, never a see-through bar. + */ +@utility glass { + background-color: var(--card); + @supports (backdrop-filter: blur(1px)) or + (-webkit-backdrop-filter: blur(1px)) { + background-color: var(--glass); + -webkit-backdrop-filter: blur(22px) saturate(180%); + backdrop-filter: blur(22px) saturate(180%); + } +} +@utility veil { + background-color: var(--background); + @supports (backdrop-filter: blur(1px)) or + (-webkit-backdrop-filter: blur(1px)) { + background-color: var(--veil); + -webkit-backdrop-filter: blur(28px) saturate(160%); + backdrop-filter: blur(28px) saturate(160%); + } +} +/* Inter's single-storey a (cv11) and flat-topped 3 (ss03). On `body`, so every screen has them. */ +@utility face { + font-feature-settings: "cv11", "ss03"; } @layer base { @@ -167,7 +318,7 @@ body { @apply border-border outline-ring/50; } body { - @apply bg-background text-foreground text-base; + @apply bg-background text-foreground face; } html { @apply font-sans; diff --git a/app/tests/ai-notice.test.tsx b/app/tests/ai-notice.test.tsx new file mode 100644 index 0000000000..226bae3098 --- /dev/null +++ b/app/tests/ai-notice.test.tsx @@ -0,0 +1,84 @@ +import { + afterAll, + afterEach, + beforeAll, + beforeEach, + expect, + test, +} from "bun:test"; +import { GlobalRegistrator } from "@happy-dom/global-registrator"; +import { QueryClient, QueryClientProvider } from "@tanstack/react-query"; +import { cleanup, render } from "@testing-library/react"; +import { ConversationView } from "@/components/channels/conversation-view"; +import { GroupChat } from "@/components/channels/group-chat"; +import { groupKeys } from "@/lib/groups"; +import { appConfig } from "@/lib/generated/application-config"; +import { settleReactWork } from "./settle-react-work"; + +/** + * Every conversation says, under its composer, that its Bots are AI and can be wrong, in the name + * this OpenBot is configured with. Small and muted, in the caption size. + */ + +const originalFetch = globalThis.fetch; +beforeAll(() => GlobalRegistrator.register({ url: "http://localhost/" })); +beforeEach(() => { + globalThis.fetch = Object.assign( + async () => Response.json({ error: "not served" }, { status: 404 }), + { preconnect: originalFetch.preconnect }, + ); +}); +afterEach(() => { + cleanup(); + globalThis.fetch = originalFetch; +}); +afterAll(async () => { + await settleReactWork(); + GlobalRegistrator.unregister(); +}); + +const notice = `${appConfig.brand.productName} is AI and can make mistakes.`; + +/** The notice, checked to come after the conversation's composer. */ +function under(container: HTMLElement) { + const line = Array.from(container.querySelectorAll("p")).find( + (paragraph) => paragraph.textContent === notice, + ); + if (!line) throw new Error("no notice"); + const composer = container.querySelector('form[data-slot="composer"]'); + if (!composer) throw new Error("no composer"); + expect( + composer.compareDocumentPosition(line) & Node.DOCUMENT_POSITION_FOLLOWING, + ).toBeTruthy(); + expect(composer.contains(line)).toBe(false); + return line; +} + +test("a conversation with one Bot says so under its composer", () => { + const view = render( + {}} />, + ); + const line = under(view.container); + const column = view.container.querySelector('[data-slot="composer-column"]'); + expect(line.parentElement).toBe(column as HTMLElement); + expect(line.className).toContain("type-caption"); + expect(line.className).toContain("text-muted-foreground"); +}); + +test("a conversation with several Bots says so under its composer too", () => { + const client = new QueryClient({ + defaultOptions: { queries: { retry: false } }, + }); + client.setQueryData(groupKeys.detail("g-1"), { + id: "g-1", + title: "Launch", + bots: [{ id: "inbox", name: "Inbox" }], + messages: [], + } as never); + const view = render( + + + , + ); + expect(under(view.container).className).toContain("type-caption"); +}); diff --git a/app/tests/app-navigation.test.tsx b/app/tests/app-navigation.test.tsx index 273257d4db..03b443edd9 100644 --- a/app/tests/app-navigation.test.tsx +++ b/app/tests/app-navigation.test.tsx @@ -16,13 +16,20 @@ import { } from "@tanstack/react-router"; import { cleanup, render, waitFor, within } from "@testing-library/react"; import userEvent from "@testing-library/user-event"; -import { AppSidebar } from "@/components/app-sidebar/app-sidebar"; -import { SidebarProvider } from "@/components/ui/sidebar"; +import { + AppSidebar, + NavigationEntry, +} from "@/components/app-sidebar/app-sidebar"; +import { SidebarMenu, SidebarProvider } from "@/components/ui/sidebar"; import { agentKeys } from "@/lib/agents/queries"; import { authKeys } from "@/lib/auth/queries"; import { botLifecycleKeys } from "@/lib/bot-lifecycle/queries"; import { channelKeys } from "@/lib/channels/queries"; -import { NAVIGATION, navigationFor } from "@/lib/navigation"; +import { + NAVIGATION, + type NavigationItem, + navigationFor, +} from "@/lib/navigation"; import { userPreferencesQueryOptions } from "@/lib/settings/message-list"; import { settleReactWork } from "./settle-react-work"; @@ -30,7 +37,8 @@ import { settleReactWork } from "./settle-react-work"; * OpenBot's destinations live in one list (`lib/navigation.ts`) that the sidebar and every skin draw * from. This holds the sidebar to that list exactly: the same entries, in the same order, at the * same addresses, with Admin only for admins. A destination added to the list and not drawn, or - * drawn here and not added to the list, fails. + * drawn here and not added to the list, fails. An entry's badge, a count of what waits there, comes + * from the list too. */ class OfflineWebSocket extends EventTarget { @@ -194,3 +202,41 @@ test("Admin is in the list for admins only", () => { "settings", ]); }); + +test("an entry's badge counts what waits there, and draws nothing at zero", async () => { + const connections = NAVIGATION.find((item) => item.id === "connections"); + if (!connections) throw new Error("No Connections entry"); + const counted = (badge: number): NavigationItem => ({ + ...connections, + badge, + }); + const router = createRouter({ + routeTree: createRootRoute({ + component: () => ( + + + + + + + ), + }), + history: createMemoryHistory({ initialEntries: ["/"] }), + }); + const queryClient = new QueryClient(); + clients.push(queryClient); + const view = render( + + + , + ); + + const [three, zero] = await view.findAllByRole("link", { + name: /Connections/, + }); + expect( + within(three).getByRole("img", { name: "3 things need you" }).textContent, + ).toBe("3"); + expect(within(zero).queryByRole("img")).toBeNull(); + expect(zero.textContent).toBe("Connections"); +}); diff --git a/app/tests/approval-inbox-poll.test.tsx b/app/tests/approval-inbox-poll.test.tsx new file mode 100644 index 0000000000..e2b1ff618b --- /dev/null +++ b/app/tests/approval-inbox-poll.test.tsx @@ -0,0 +1,171 @@ +import { + afterAll, + afterEach, + beforeAll, + beforeEach, + expect, + test, +} from "bun:test"; +import type { Message } from "@ag-ui/core"; +import { CopilotKitContext } from "@copilotkit/react-core/v2/context"; +import { GlobalRegistrator } from "@happy-dom/global-registrator"; +import { + environmentManager, + focusManager, + isServer, + QueryClient, + QueryClientProvider, +} from "@tanstack/react-query"; +import { cleanup, render, waitFor } from "@testing-library/react"; +import { ChatTranscript } from "@/components/channels/chat-transcript"; +import { ConversationPresenceProvider } from "@/components/channels/conversation-presence"; +import { PAGE_ROWS } from "@/components/channels/transcript-pages"; +import { settleReactWork } from "./settle-react-work"; + +/** + * A conversation asks for its approvals once a poll, however many tool calls it has. + * + * TanStack Query polls once for every component reading a query, and every tool call reads the + * inbox to know whether it is held on a card. A reply with thirty calls asked thirty times and more + * every ten seconds. The conversation polls it once, and its rows read what that brings. + */ + +const POLL_MS = 10_000; +let asked = 0; +let realFetch: typeof fetch; +let realSetInterval: typeof setInterval; +let realClearInterval: typeof clearInterval; +/** Every ten-second interval started, run by hand below rather than waited for. */ +const polls = new Map void>(); +let nextPoll = 1_000_000; + +beforeAll(() => { + GlobalRegistrator.register({ url: "http://localhost/" }); + realFetch = globalThis.fetch; + globalThis.fetch = Object.assign( + async (input: RequestInfo | URL) => { + const url = String(input); + if (url.endsWith("/api/approvals")) { + asked += 1; + return Response.json({ + enabled: true, + requests: [], + rules: [], + teamRules: [], + questions: [], + }); + } + if (url.includes("/api/channels")) + return Response.json({ channels: [], nextCursor: null }); + return Response.json({}); + }, + { preconnect: realFetch.preconnect }, + ); +}); + +beforeEach(() => { + realSetInterval = globalThis.setInterval; + realClearInterval = globalThis.clearInterval; + globalThis.setInterval = (( + callback: () => void, + delay?: number, + ...rest: unknown[] + ) => { + if (delay !== POLL_MS) return realSetInterval(callback, delay, ...rest); + nextPoll += 1; + polls.set(nextPoll, callback); + return nextPoll; + }) as typeof setInterval; + globalThis.clearInterval = ((id?: number) => { + if (id === undefined || !polls.delete(id)) realClearInterval(id); + }) as typeof clearInterval; + // TanStack decides it is on a server when it loads before the DOM does, and a server never polls. + environmentManager.setIsServer(() => false); + focusManager.setFocused(true); +}); + +afterEach(() => { + cleanup(); + globalThis.setInterval = realSetInterval; + globalThis.clearInterval = realClearInterval; + polls.clear(); + environmentManager.setIsServer(() => isServer); + focusManager.setFocused(undefined); + asked = 0; +}); + +afterAll(async () => { + await settleReactWork(); + globalThis.fetch = realFetch; + GlobalRegistrator.unregister(); +}); + +/** No renderer is registered for a connector tool, so each call draws the transcript's own line. */ +const noRenderers = { + copilotkit: { + renderToolCalls: [], + subscribe: () => ({ unsubscribe() {} }), + }, + executingToolCallIds: new Set(), +} as never; + +/** A reply that said something, searched, and went on, thirty times: thirty rows with a call each. */ +const reply: Message[] = [ + { id: "user-1", role: "user", content: "What changed in Q3?" }, + ...Array.from({ length: 30 }, (_, index): Message[] => [ + { + id: `assistant-${index}`, + role: "assistant", + content: `Looking at part ${index + 1}.`, + toolCalls: [ + { + id: `call-${index}`, + type: "function", + function: { + name: "mcp__exa__web_search_exa", + arguments: JSON.stringify({ query: `Q3 part ${index + 1}` }), + }, + }, + ], + }, + { + id: `result-${index}`, + role: "tool", + toolCallId: `call-${index}`, + content: '"Results"', + }, + ]).flat(), +]; + +/** One poll's worth of time: each interval fires in turn, as staggered timers do. */ +async function onePoll() { + for (const poll of [...polls.values()]) { + poll(); + await new Promise((resolve) => setTimeout(resolve, 0)); + } +} + +test("a conversation with thirty tool calls asks for its approvals once a poll", async () => { + const client = new QueryClient({ + defaultOptions: { queries: { retry: false } }, + }); + const view = render( + + + + + + + , + ); + // It opens at its newest page of rows (`PAGE_ROWS`), each a message or its call: still many calls. + await waitFor(() => + expect(view.getAllByText("Search the web")).toHaveLength(PAGE_ROWS / 2), + ); + await waitFor(() => expect(asked).toBeGreaterThan(0)); + await settleReactWork(); + + asked = 0; + for (let poll = 0; poll < 3; poll += 1) await onePoll(); + expect(asked).toBe(3); +}); diff --git a/app/tests/approval-view.test.ts b/app/tests/approval-view.test.ts new file mode 100644 index 0000000000..174e5108c5 --- /dev/null +++ b/app/tests/approval-view.test.ts @@ -0,0 +1,330 @@ +import { expect, test } from "bun:test"; +import { + alwaysAllowReach, + approvalOutcome, + approvalPlace, + clockTime, + decisionReceipt, + describeApproval, + personWasAsked, +} from "@/lib/approval-view"; +import type { ApprovalRequest } from "@/lib/approvals"; + +const action = ( + toolRef: string, + args: unknown, + extra: Partial = {}, +): Pick => ({ + action: { + botId: "inbox", + toolRef, + effect: "write", + scope: "*", + threadId: "t", + args, + target: {}, + ...extra, + }, +}); + +test("an app's send tool reads as the action, with the app named apart", () => { + const view = describeApproval( + action("gmail/GMAIL_SEND_EMAIL", { + recipient_email: "ana@example.com", + subject: "Ship date", + body: "Hi Ana", + thread_id: "abc", + }), + ); + expect(view.title).toBe("Send your email"); + expect(view.app).toBe("Gmail"); + expect(view.kind).toBe("mail"); + expect(view.fields).toEqual([ + { label: "To", value: "ana@example.com" }, + { label: "Subject", value: "Ship date" }, + ]); + expect(view.body).toBe("Hi Ana"); +}); + +test("an app reached through a broker is named as the app, never the broker", () => { + const view = describeApproval( + action("composio-gmail/GMAIL_SEND_EMAIL", { to: "sam@example.com" }), + ); + expect(view.title).toBe("Send your email"); + expect(view.app).toBe("Gmail"); +}); + +test("any other app's tool is named in words, with the app beside it", () => { + const view = describeApproval( + action("notion/create_page", { title: "Launch notes" }), + ); + expect(view.title).toBe("Create page"); + expect(view.app).toBe("Notion"); + expect(view.kind).toBe("other"); + expect(view.fields).toEqual([{ label: "Title", value: "Launch notes" }]); +}); + +test("a browser action keeps what it acts on and drops the page plumbing", () => { + const request = action( + "computer_click", + { element: "Pay now", ref: "e7", snapshotId: 3 }, + { scope: "shop.example" }, + ); + const view = describeApproval(request); + // Named as its step in the conversation names it, so the card and the line above it agree. + expect(view.title).toBe("Click on the page"); + expect(view.app).toBeNull(); + expect(view.kind).toBe("browser"); + expect(view.fields).toEqual([{ label: "Element", value: "Pay now" }]); + expect(approvalPlace(request)).toBe("shop.example"); +}); + +test("a command on the person's computer names the command and its folder, not the grant", () => { + const request = action( + "host/run_command", + { + kind: "run_command", + botId: "inbox", + actorId: "user-1", + grantId: "grant-9", + command: "ls -la", + }, + { + scope: "grant-9", + target: { grantId: "grant-9", displayName: "Projects" }, + }, + ); + const view = describeApproval(request); + expect(view.title).toBe("Run command"); + expect(view.kind).toBe("command"); + expect(view.fields).toEqual([{ label: "Command", value: "ls -la" }]); + expect(approvalPlace(request)).toBe("Projects"); +}); + +test("handing work to another Bot says so, and the Bot it goes to is a field, not a place", () => { + const request = action( + "bot/message_bot", + { bot: "knowledge", message: "Find the March invoice." }, + { scope: "knowledge" }, + ); + const view = describeApproval(request); + expect(view.title).toBe("Ask another Bot"); + expect(view.kind).toBe("other"); + expect(view.fields).toEqual([{ label: "Bot", value: "knowledge" }]); + expect(view.body).toBe("Find the March invoice."); + expect(approvalPlace(request)).toBeNull(); +}); + +test("a scope that is only the app's own id, or anything, is no place", () => { + expect( + approvalPlace(action("notion/create_page", {}, { scope: "notion" })), + ).toBeNull(); + expect(approvalPlace(action("notion/create_page", {}))).toBeNull(); + // The computer's own "its workspace" is not a place either. + expect( + approvalPlace(action("computer_run_command", {}, { scope: "workspace" })), + ).toBeNull(); +}); + +test("who decided, in the person's words", () => { + expect(decisionReceipt({ decision: "allow_once" })).toBe( + "Allowed once by you", + ); + expect(decisionReceipt({ decision: "allow_always" })).toBe( + "Always allowed here by you", + ); + expect(decisionReceipt({ decision: "deny" })).toBe("Declined by you"); + expect(decisionReceipt({ decision: "handled" })).toBe("You did it yourself"); + expect(decisionReceipt({ decision: null })).toBeNull(); +}); + +test("how it ended: allowed, sent, done, declined, or did not work", () => { + const mail = action("gmail/GMAIL_SEND_EMAIL", {}).action; + const ended = ( + request: Partial>, + ) => + approvalOutcome({ + status: "completed", + result: null, + decision: "allow_once", + action: mail, + ...request, + }); + expect(ended({ status: "pending", decision: null })).toBeNull(); + expect(ended({ status: "approved" })).toEqual({ + label: "Allowed", + tone: "working", + }); + expect(ended({})).toEqual({ label: "Sent", tone: "done" }); + expect(ended({ result: { error: "boom" } })).toEqual({ + label: "Didn't work", + tone: "needs", + }); + // Marked done by the person: nothing was sent by the Bot. + expect(ended({ decision: "handled" })).toEqual({ + label: "Done", + tone: "done", + }); + expect( + approvalOutcome({ + status: "completed", + result: null, + decision: "allow_once", + action: action("computer_click", {}).action, + }), + ).toEqual({ label: "Done", tone: "done" }); +}); + +test("a decline stays a decline once the server has finished it with the decline as its error", () => { + const mail = action("gmail/GMAIL_SEND_EMAIL", {}).action; + for (const status of ["denied", "completed"]) + expect( + approvalOutcome({ + status, + decision: "deny", + result: { error: "The person declined this action." }, + action: mail, + }), + ).toEqual({ label: "Declined", tone: "neutral" }); +}); + +test("a time today is a clock time; an earlier one is a date", () => { + const now = new Date(2026, 9, 21, 15, 0); + expect(clockTime(new Date(2026, 9, 21, 14, 59).toISOString(), now)).toMatch( + /2:59/, + ); + expect(clockTime(new Date(2026, 9, 20, 9, 0).toISOString(), now)).toMatch( + /20/, + ); + expect(clockTime("not a date", now)).toBe(""); +}); + +test("a send the private-share check asks about reads as the email it is", () => { + const view = describeApproval({ + action: { + toolRef: "openbot/share_private_information", + args: { + audience: "composio-gmail: ana@hearthandco.example", + recipients: 0, + contentDigest: "9f2c", + preview: "Hi Ana, our kiln is down until Monday.", + }, + }, + } as never); + expect(view).toEqual({ + title: "Send email", + app: "Gmail", + kind: "mail", + fields: [{ label: "To", value: "ana@hearthandco.example" }], + body: "Hi Ana, our kiln is down until Monday.", + unshown: 0, + }); +}); + +test("a share the person allowed is allowed, not sent: the Bot sends it again itself", () => { + expect( + approvalOutcome({ + status: "completed", + result: null, + decision: "allow_once", + action: { + toolRef: "openbot/share_private_information", + args: { audience: "composio-gmail: ana@hearthandco.example" }, + }, + } as never), + ).toEqual({ label: "Allowed", tone: "done" }); +}); + +test("a send put to the person by the share check counts as asked, and its recipients are not a place", () => { + const share = { + action: { + toolRef: "openbot/share_private_information", + scope: "external:ana@hearthandco.example", + args: {}, + }, + } as never; + expect(personWasAsked(share)).toBe(true); + expect(approvalPlace(share)).toBeNull(); + expect( + personWasAsked({ + action: { toolRef: "gmail/send_reply", scope: "x", args: {} }, + } as never), + ).toBe(false); +}); + +test("typing into a page reads as the step does, and a yes-or-no argument reads as Yes or No", () => { + const view = describeApproval( + action( + "computer_type", + { ref: "e3", text: "OpenBot", submit: true }, + { scope: "en.wikipedia.org" }, + ), + ); + expect(view.title).toBe("Fill in a field"); + expect(view.body).toBe("OpenBot"); + expect(view.fields).toEqual([{ label: "Submit after typing", value: "Yes" }]); +}); + +test("the face counts what it leaves for Details: a second body, objects, fields past its room", () => { + expect( + describeApproval( + action("gmail/GMAIL_SEND_EMAIL", { + to: "ana@example.com", + subject: "Ship date", + body: "Hi Ana", + thread_id: "abc", + }), + ).unshown, + ).toBe(0); + const view = describeApproval( + action("outlook/send_mail", { + to: [{ email: "ana@example.com" }], + subject: "Ship date", + text: "Hi", + html: 'Hi', + a: "1", + b: "2", + c: "3", + d: "4", + id: "msg-1", + }), + ); + // The recipients (an object), the html after the text, and the fifth plain field; never the id. + expect(view.fields.map((field) => field.label)).toEqual([ + "Subject", + "A", + "B", + "C", + ]); + expect(view.unshown).toBe(3); +}); + +test("Always allow here says how far its rule reaches, past this one call", () => { + const reach = ( + toolRef: string, + extra: Partial = {}, + ) => alwaysAllowReach(action(toolRef, {}, extra)); + expect( + reach("host/run_command", { target: { displayName: "Projects" } }), + ).toBe( + "This Bot won't ask you again before doing this in Projects, whatever the details.", + ); + expect(reach("gmail/GMAIL_SEND_EMAIL", { scope: "gmail" })).toBe( + "This Bot won't ask you again before doing this in Gmail, whatever the details.", + ); + expect(reach("computer_click", { scope: "shop.example" })).toBe( + "This Bot won't ask you again before doing this on shop.example, whatever the details.", + ); + expect(reach("computer_run", { scope: "workspace" })).toBe( + "This Bot won't ask you again before doing this on its computer, whatever the details.", + ); + expect(reach("computer_write_file", { scope: "notes/drafts" })).toBe( + "This Bot won't ask you again before doing this in notes/drafts, whatever the details.", + ); + expect(reach("bot/message_bot", { scope: "research" })).toBe( + "This Bot won't ask you again before asking that Bot, whatever it asks.", + ); + expect(reach("openbot/share_private_information")).toBe( + "This Bot won't ask you again before sharing private information with these people.", + ); +}); diff --git a/app/tests/approvals-rules.test.tsx b/app/tests/approvals-rules.test.tsx index cb4302a54b..06554117c3 100644 --- a/app/tests/approvals-rules.test.tsx +++ b/app/tests/approvals-rules.test.tsx @@ -369,7 +369,11 @@ test("a pending change is drawn in the conversation where the action was, and de , ); await view.findByRole("region", { name: "Waiting for your approval" }); - expect(view.getByText(/This needs you: click on shop.example/)).toBeTruthy(); + expect(view.getByText("This needs you: Click on the page")).toBeTruthy(); + expect(view.getByText("shop.example")).toBeTruthy(); + // A hand-off is the person's to do: nothing offers to let the Bot do it. + expect(view.queryByRole("button", { name: "Allow once" })).toBeNull(); + expect(view.getByRole("button", { name: "Don't do it" })).toBeTruthy(); // Another call's line is left to its own renderer. expect(view.getByText("Filled in field")).toBeTruthy(); expect(view.queryByText("Clicked on page")).toBeNull(); diff --git a/app/tests/auto-watch-screen.test.tsx b/app/tests/auto-watch-screen.test.tsx index aec164cecd..ba44733426 100644 --- a/app/tests/auto-watch-screen.test.tsx +++ b/app/tests/auto-watch-screen.test.tsx @@ -30,16 +30,20 @@ afterAll(async () => { GlobalRegistrator.unregister(); }); -function paneFor(options: { watching?: boolean; busy?: boolean } = {}) { +function paneFor( + options: { watching?: boolean; busy?: boolean; docked?: boolean } = {}, +) { const opened: number[] = []; const closed: number[] = []; let watching = options.watching ?? false; + const docked = options.docked ?? true; const view = renderHook( (props: { watching: boolean }) => useAutoWatchScreen({ botId: "bot-1", watching: props.watching, busy: options.busy ?? false, + docked, open: () => { opened.push(1); watching = true; @@ -56,6 +60,14 @@ function paneFor(options: { watching?: boolean; busy?: boolean } = {}) { return { opened, closed, + /** Whether the header's computer button is lit for activity nobody is watching. */ + unseen: () => view.result.current.unseen, + /** What opening the pane by hand does. */ + watch: () => + act(() => { + watching = true; + view.rerender({ watching }); + }), /** What closing the pane by hand does: the pane shuts, and this run stops auto-opening. */ dismiss: () => act(() => { @@ -164,3 +176,70 @@ describe("another Bot's computer", () => { expect(pane.opened).toHaveLength(0); }); }); + +/* + * On a screen too narrow for the pane beside the conversation, opening it would cover the + * conversation with a sheet. The header's computer button lights up instead, and the person opens + * it when they want to watch. + */ +describe("a screen with no room beside the conversation", () => { + test("lights the computer button instead of covering the conversation", () => { + const pane = paneFor({ docked: false }); + expect(pane.unseen()).toBe(false); + + announce(); + + expect(pane.opened).toHaveLength(0); + expect(pane.unseen()).toBe(true); + }); + + test("puts the light out once the person opens the screen", () => { + const pane = paneFor({ docked: false }); + announce(); + + pane.watch(); + + expect(pane.unseen()).toBe(false); + }); + + test("puts the light out when the call that lit it is taken back", () => { + const pane = paneFor({ docked: false }); + const announcement = announce(); + + act(() => withdrawComputerActivity("bot-1", announcement)); + + expect(pane.unseen()).toBe(false); + expect(pane.closed).toHaveLength(0); + }); + + test("lights nothing for a run the person dismissed", () => { + const pane = paneFor({ docked: false }); + announce(); + pane.watch(); + pane.dismiss(); + + announce(); + + expect(pane.unseen()).toBe(false); + expect(pane.opened).toHaveLength(0); + }); + + test("lights nothing beside a pane opened for something else", () => { + const pane = paneFor({ docked: false, busy: true }); + + announce(); + + expect(pane.unseen()).toBe(false); + }); +}); + +describe("a screen with room beside the conversation", () => { + test("opens rather than lighting the button", () => { + const pane = paneFor({ docked: true }); + + announce(); + + expect(pane.opened).toHaveLength(1); + expect(pane.unseen()).toBe(false); + }); +}); diff --git a/app/tests/bot-access-apps.test.tsx b/app/tests/bot-access-apps.test.tsx index 657f5f4d5e..30b791e425 100644 --- a/app/tests/bot-access-apps.test.tsx +++ b/app/tests/bot-access-apps.test.tsx @@ -152,6 +152,8 @@ test("each app is named by its own name and says it is ready, with its tools", a const gmail = await row(view, "composio-gmail"); expect(gmail.getByText("Gmail")).toBeTruthy(); expect(gmail.getByText(/^Ready to use\./)).toBeTruthy(); + // What each tool does, never what it did. + expect(gmail.getByText(/Send your email, Read your email$/)).toBeTruthy(); expect(gmail.getByText("2 tools")).toBeTruthy(); expect(view.queryByText(/Composio Gmail/)).toBeNull(); expect(gmail.queryByRole("button")).toBeNull(); diff --git a/app/tests/bot-needs-you.test.tsx b/app/tests/bot-needs-you.test.tsx index 487ce5270f..f18a3720ba 100644 --- a/app/tests/bot-needs-you.test.tsx +++ b/app/tests/bot-needs-you.test.tsx @@ -5,6 +5,7 @@ import { cleanup, fireEvent, render, waitFor } from "@testing-library/react"; import type { ReactNode } from "react"; import { BotNeedsYou } from "@/components/approvals/waiting"; import { SuggestionsInbox } from "@/components/suggestions/proactive-panel"; +import { queryClient } from "@/query-client"; import { settleReactWork } from "./settle-react-work"; /** @@ -55,11 +56,22 @@ function serving() { global.fetch = (async (input: RequestInfo | URL, init?: RequestInit) => { const url = String(input); if (init?.method && init.method !== "GET") { - sent.push({ - path: url, - method: init.method, - body: init.body ? JSON.parse(String(init.body)) : undefined, - }); + const body = init.body ? JSON.parse(String(init.body)) : undefined; + sent.push({ path: url, method: init.method, body }); + // A decision is read back from the inbox, as the server's is. + const decided = url.match(/^\/api\/approvals\/([^/]+)\/decision$/); + if (decided) { + const status = body.decision === "deny" ? "denied" : "approved"; + inbox = { + ...inbox, + requests: (inbox.requests as { id: string }[]).map((row) => + row.id === decided[1] + ? { ...row, status, decision: body.decision } + : row, + ), + }; + return Response.json({ id: decided[1], status }); + } return Response.json({ ok: true }); } if (url === "/api/approvals") return Response.json(inbox); @@ -119,8 +131,8 @@ test("the target of a waiting action is shown whole, however long", async () => serving(); const view = draw(); const scope = await view.findByText(long.action.scope); - expect(scope.className).toContain("line-clamp-none"); - expect(scope.className).toContain("break-all"); + expect(scope.className).toContain("[overflow-wrap:anywhere]"); + expect(scope.closest(".truncate, [class*='line-clamp-']")).toBeNull(); }); test("only what this Bot waits on is shown, and it can be decided here", async () => { @@ -151,7 +163,12 @@ test("only what this Bot waits on is shown, and it can be decided here", async ( expect(view.queryByText(/Which wiki\?/)).toBeNull(); expect(view.getAllByRole("button", { name: "Allow once" })).toHaveLength(1); expect(view.getByRole("button", { name: "Always allow here" })).toBeTruthy(); - expect(view.getByRole("button", { name: "Deny" })).toBeTruthy(); + expect(view.getByRole("button", { name: "Decline" })).toBeTruthy(); + // What the action is and where, in words, and who asks: this page shows every Bot's requests. + expect(view.getByText("Click on the page")).toBeTruthy(); + expect(view.getByText("shop.example").parentElement?.textContent).toBe( + "Expenses · shop.example", + ); expect(view.getByRole("button", { name: "Send answer" })).toBeTruthy(); fireEvent.click(view.getByRole("button", { name: "Allow once" })); await waitFor(() => @@ -163,6 +180,28 @@ test("only what this Bot waits on is shown, and it can be decided here", async ( ); }); +test("an action decided on the Bot's page stays there, folded to who decided", async () => { + inbox = { ...base, requests: [request("r1", "expenses")], questions: [] }; + serving(); + // The app's own cache, which the decision writes to, as it is on the page. + const view = render( + + + , + ); + try { + fireEvent.click(await view.findByRole("button", { name: "Decline" })); + expect(await view.findByText("Declined by you")).toBeTruthy(); + expect(view.getByText("Declined")).toBeTruthy(); + expect(view.getByText("Needs you")).toBeTruthy(); + expect(view.queryByRole("button", { name: "Allow once" })).toBeNull(); + expect(view.queryByRole("button", { name: "Decline" })).toBeNull(); + } finally { + cleanup(); + queryClient.clear(); + } +}); + test("a hand-off is marked done by the person, never allowed on their behalf", async () => { inbox = { ...base, diff --git a/app/tests/bot-orb.test.tsx b/app/tests/bot-orb.test.tsx new file mode 100644 index 0000000000..2ca0690592 --- /dev/null +++ b/app/tests/bot-orb.test.tsx @@ -0,0 +1,794 @@ +import { + afterAll, + afterEach, + beforeAll, + describe, + expect, + test, +} from "bun:test"; +import { GlobalRegistrator } from "@happy-dom/global-registrator"; +import { cleanup, render } from "@testing-library/react"; +import { AbstractAvatar } from "@/components/agents/abstract-avatar"; +import AgentOrb, { type AIState } from "@/components/agents/orb/agent-orb"; +import { + BotOrb, + type BotOrbState, + followPointer, + restingGaze, +} from "@/components/agents/orb/bot-orb"; +import { + drawBotOrb, + FINISH_HELD, + FINISH_S, + finishPhase, + OPENBOT_LOOK, + ORB_PALETTES, + type OrbLook, + type OrbPose, + oklch, + orbLookFor, + palette, + RAYS, + samePicture, + track, +} from "@/components/agents/orb/bot-orb-draw"; +import { subscribePointer } from "@/components/agents/orb/orb-clock"; +import { settleReactWork } from "./settle-react-work"; + +beforeAll(() => GlobalRegistrator.register({ url: "http://localhost/" })); +afterEach(() => cleanup()); +afterAll(async () => { + await settleReactWork(); + await GlobalRegistrator.unregister(); +}); + +const REST: OrbPose = { + ta: 1.6, + listen: 0, + think: 0, + done: 0, + smile: 0, + tilt: 0, + ax: 0, + ay: 0.04, + blink: 0, + nod: 0, + doneAge: -1, + noticeAge: -1, + finishAge: -1, + breath: 0, +}; + +/** + * Just enough of a 2D context to run the drawing: it counts every stroke, counts the frames drawn, + * each of which starts by clearing the canvas, and keeps where each soft blob and each eye is drawn + * and in what colour. + */ +function fakeContext() { + const marks: string[] = []; + const gradient = { + addColorStop: (_: number, colour: string) => marks.push(colour), + }; + const ctx = { + strokes: 0, + draws: 0, + marks, + stroke() { + this.strokes++; + }, + clearRect() { + this.draws++; + }, + translate(x: number, y: number) { + marks.push(`at ${x.toFixed(2)},${y.toFixed(2)}`); + }, + createLinearGradient: () => gradient, + createRadialGradient(x: number, y: number) { + marks.push(`blob ${x.toFixed(2)},${y.toFixed(2)}`); + return gradient; + }, + }; + return new Proxy(ctx, { + get: (target, key) => + key in target ? target[key as keyof typeof target] : () => {}, + set: (target, key, value) => { + (target as Record)[key] = value; + return true; + }, + }); +} + +/** One frame of an orb 60px across in `pose`, drawn on a fake context. */ +const drawn = (pose: OrbPose, look: OrbLook = OPENBOT_LOOK) => { + const ctx = fakeContext(); + drawBotOrb(ctx as unknown as CanvasRenderingContext2D, 60, pose, { + dark: false, + look, + }); + return ctx; +}; + +/** Every stroke one frame draws: the rim, the landing's ping, and a finish's rays and check. */ +const strokesOf = (pose: OrbPose) => drawn(pose).strokes; + +/** The Bots a fresh OpenBot ships with, whose ids are their seeds. */ +const TEAM = [ + "expense-review", + "feedback-digest", + "interview-notes", + "meeting-follow-ups", + "onboarding-buddy", + "oncall-handover", + "release-notes", + "research-desk", + "ticket-triage", +]; + +describe("the orb's motion", () => { + test("an eased key arrives and a spring overshoots, then both settle on the target", () => { + const eased = { from: 0, keys: [{ at: 1, to: 1, ease: 0.4 }] }; + expect(track(eased, 0.5)).toBe(0); + expect(track(eased, 1.2)).toBeGreaterThan(0); + expect(track(eased, 2)).toBe(1); + + const sprung = { + from: 0, + keys: [{ at: 0, to: 1, spring: [14, 0.5] as const }], + }; + const peak = Math.max( + ...Array.from({ length: 60 }, (_, i) => track(sprung, i / 60)), + ); + expect(peak).toBeGreaterThan(1); + expect(track(sprung, 5)).toBeCloseTo(1, 4); + }); + + test("a later key moves on from where the earlier one left it", () => { + const tr = { + from: 0, + keys: [ + { at: 0, to: 1, ease: 0.4 }, + { at: 1, to: 0.25, ease: 0.4 }, + ], + }; + expect(track(tr, 0.9)).toBe(1); + expect(track(tr, 3)).toBe(0.25); + }); +}); + +describe("the colours", () => { + test("every palette's base is pale and its shade deep", () => { + for (const name of ORB_PALETTES) { + const pal = palette(name, 1, 0); + expect(Math.min(...pal.base)).toBeGreaterThan(200); + expect(Math.max(...pal.deep)).toBeLessThan(230); + expect(Math.max(...pal.shade)).toBeLessThan(100); + } + }); + + test("a colour outside sRGB loses chroma rather than turning into another hue", () => { + const [r, g, b] = oklch(0.6, 0.5, 262); + expect(b).toBeGreaterThan(r); + expect(b).toBeGreaterThan(g); + }); + + test("a seed always gets the same look", () => { + for (const seed of TEAM) { + expect(orbLookFor(seed)).toEqual(orbLookFor(seed)); + expect(ORB_PALETTES).toContain(orbLookFor(seed).palette); + } + // Pinned, so a change to the hash, which would change every Bot's face, can't land unnoticed. + expect(orbLookFor("research-desk").palette).toBe("ocean"); + expect(orbLookFor("ticket-triage").palette).toBe("sun"); + expect(ORB_PALETTES).toContain(orbLookFor("").palette); + }); + + test("nine palettes, and a team of nine seeds uses more than four of them", () => { + expect(ORB_PALETTES).toHaveLength(9); + expect(new Set(ORB_PALETTES).size).toBe(9); + const used = TEAM.map((seed) => orbLookFor(seed).palette); + expect(new Set(used).size).toBeGreaterThan(4); + }); + + test("no Bot is violet, which is OpenBot's own, and every other colour is given", () => { + const seeds = [ + ...TEAM, + "", + ...Array.from({ length: 400 }, (_, i) => `bot-${i}`), + ]; + const given = new Set(seeds.map((seed) => orbLookFor(seed).palette)); + expect(given.has("violet")).toBe(false); + expect([...given].sort()).toEqual( + ORB_PALETTES.filter((name) => name !== "violet").sort(), + ); + expect(OPENBOT_LOOK.palette).toBe("violet"); + }); + + test("every Bot in the default team is drawn unlike every other, even two in one colour", () => { + const apart = (a: number, b: number) => { + const d = Math.abs(a - b) % (2 * Math.PI); + return Math.min(d, 2 * Math.PI - d); + }; + let sharing = 0; + for (const [i, a] of TEAM.entries()) { + for (const b of TEAM.slice(i + 1)) { + const one = orbLookFor(a); + const other = orbLookFor(b); + expect(drawn(REST, one).marks).not.toEqual(drawn(REST, other).marks); + if (one.palette !== other.palette) continue; + sharing++; + // Their colours sit at least a sixth of a turn apart, which shows at any size. + expect(apart(one.turn, other.turn)).toBeGreaterThan(Math.PI / 3); + } + } + // Eight colours can't go round nine Bots, so the team has Bots that share one. + expect(sharing).toBeGreaterThan(0); + }); + + test("each palette is a colour of its own", () => { + const bases = ORB_PALETTES.map((name) => palette(name, 1, 0).main.join()); + expect(new Set(bases).size).toBe(ORB_PALETTES.length); + }); +}); + +describe("the drawing", () => { + test("has no grilles: at rest its only stroke is the sphere's rim", () => { + expect(strokesOf(REST)).toBe(1); + }); + + test("two poses draw the same picture once what their ages time is over, not while it plays", () => { + expect(samePicture(REST, { ...REST })).toBe(true); + expect(samePicture(REST, { ...REST, ax: REST.ax + 0.01 })).toBe(false); + // A ping long rung, a perk long faded and a finish long over: the orb at rest. + expect( + samePicture(REST, { ...REST, doneAge: 4, noticeAge: 9, finishAge: 30 }), + ).toBe(true); + expect( + samePicture({ ...REST, doneAge: 0.2 }, { ...REST, doneAge: 0.3 }), + ).toBe(false); + }); +}); + +describe("a finish", () => { + test("the eyes give way to a check while the rays fly, then the eyes come back", () => { + expect(finishPhase(-1)).toMatchObject({ check: 0, rays: -1, eyes: 1 }); + + const burst = finishPhase(0.4); + expect(burst.eyes).toBe(0); + expect(burst.check).toBe(1); + expect(burst.rays).toBeGreaterThan(0); + expect(burst.rays).toBeLessThan(1); + + // Held: the check whole, the burst over. + expect(finishPhase(FINISH_HELD)).toMatchObject({ + check: 1, + checkAlpha: 1, + rays: -1, + eyes: 0, + }); + + expect(finishPhase(FINISH_S)).toEqual(finishPhase(-1)); + }); + + test("the check strokes on along its length", () => { + const early = finishPhase(0.12).check; + expect(early).toBeGreaterThan(0); + expect(early).toBeLessThan(finishPhase(0.24).check); + }); + + test("draws its rays and its check over the orb, and nothing once it is over", () => { + const rest = strokesOf(REST); + expect(strokesOf({ ...REST, finishAge: 0.4 })).toBe(rest + RAYS + 1); + expect(strokesOf({ ...REST, finishAge: FINISH_HELD })).toBe(rest + 1); + expect(strokesOf({ ...REST, finishAge: FINISH_S })).toBe(rest); + }); +}); + +/** Every canvas draws on `ctx` until `restore`, as if each had its own 2D context. */ +function withContext(ctx: ReturnType) { + const getContext = HTMLCanvasElement.prototype.getContext; + HTMLCanvasElement.prototype.getContext = (() => + ctx) as unknown as typeof getContext; + return () => { + HTMLCanvasElement.prototype.getContext = getContext; + }; +} + +/** + * Animation frames that run only when the test says, as the browser would at its next paint, at + * the time the test gives or now: `requested` counts every frame asked for, `pending` the ones not + * yet run or cancelled. + */ +function manualFrames() { + const queue = new Map(); + let next = 1; + let requested = 0; + const request = window.requestAnimationFrame; + const cancel = window.cancelAnimationFrame; + window.requestAnimationFrame = (callback) => { + requested++; + queue.set(next, callback); + return next++; + }; + window.cancelAnimationFrame = (id) => { + queue.delete(id); + }; + return { + get requested() { + return requested; + }, + get pending() { + return queue.size; + }, + run(now = performance.now()) { + const due = [...queue.values()]; + queue.clear(); + for (const callback of due) callback(now); + }, + restore() { + window.requestAnimationFrame = request; + window.cancelAnimationFrame = cancel; + }, + }; +} + +/** An `IntersectionObserver` that reports only what the test tells it to. */ +class Sight { + static all: Sight[] = []; + readonly root = null; + readonly rootMargin = ""; + readonly thresholds = []; + targets: Element[] = []; + + constructor(private readonly callback: IntersectionObserverCallback) { + Sight.all.push(this); + } + + observe(target: Element) { + this.targets.push(target); + } + + unobserve(target: Element) { + this.targets = this.targets.filter((t) => t !== target); + } + + disconnect() { + this.targets = []; + } + + takeRecords() { + return []; + } + + static show(target: Element, isIntersecting: boolean) { + for (const sight of Sight.all) { + if (!sight.targets.includes(target)) continue; + sight.callback( + [{ target, isIntersecting } as IntersectionObserverEntry], + sight as unknown as IntersectionObserver, + ); + } + } +} + +/** Runs `body` with `IntersectionObserver` set to `observer`, or missing when it is undefined. */ +function withObserver(observer: typeof Sight | undefined, body: () => void) { + const original = globalThis.IntersectionObserver; + if (observer) { + globalThis.IntersectionObserver = + observer as unknown as typeof IntersectionObserver; + } else { + Reflect.deleteProperty(globalThis, "IntersectionObserver"); + } + try { + body(); + } finally { + globalThis.IntersectionObserver = original; + Sight.all = []; + } +} + +const orbOf = (container: HTMLElement) => + container.querySelector("[data-state]") as HTMLElement; + +describe("", () => { + test("holds its footprint, hides from assistive tech, and draws on a canvas twice its size", () => { + const view = render(); + const orb = view.container.firstElementChild as HTMLElement; + expect(orb.getAttribute("aria-hidden")).toBe("true"); + expect(orb.dataset.state).toBe("listening"); + expect(orb.dataset.palette).toBe("violet"); + expect(orb.style.width).toBe("40px"); + const canvas = orb.querySelector("canvas") as HTMLCanvasElement; + expect(canvas.style.width).toBe("80px"); + expect(canvas.style.left).toBe("-20px"); + }); + + test("is a block, so a small face never sits on a line's baseline and drops out of its circle", () => { + const view = render(); + const orb = view.container.firstElementChild as HTMLElement; + expect(orb.classList.contains("block")).toBe(true); + expect(orb.classList.contains("inline-block")).toBe(false); + }); + + test("still, it draws once and again only when the theme changes, never on a frame loop", async () => { + const ctx = fakeContext(); + const frames = manualFrames(); + const restore = withContext(ctx); + try { + render(); + expect(ctx.draws).toBe(1); + document.documentElement.classList.add("dark"); + await new Promise((resolve) => setTimeout(resolve, 0)); + expect(ctx.draws).toBe(2); + expect(frames.requested).toBe(0); + + cleanup(); + render(); + expect(frames.requested).toBe(1); + } finally { + cleanup(); + document.documentElement.classList.remove("dark"); + frames.restore(); + restore(); + } + }); + + test("still, finished is the check without the burst", () => { + const strokes = (state?: "finished") => { + const ctx = fakeContext(); + const restore = withContext(ctx); + try { + render( + , + ); + cleanup(); + return ctx.strokes; + } finally { + restore(); + } + }; + expect(strokes("finished")).toBe(strokes() + 1); + }); + + test("follows a change of state", () => { + const view = render(); + expect(orbOf(view.container).dataset.state).toBe("idle"); + view.rerender(); + expect(orbOf(view.container).dataset.state).toBe("thinking"); + view.rerender(); + expect(orbOf(view.container).dataset.state).toBe("finished"); + }); + + test("fifty orbs share one animation frame", () => { + withObserver(undefined, () => { + const ctx = fakeContext(); + const frames = manualFrames(); + const restore = withContext(ctx); + try { + render( +
    + {Array.from({ length: 50 }, (_, i) => `bot-${i}`).map((id) => ( + + ))} +
    , + ); + expect(frames.requested).toBe(1); + frames.run(); + expect(ctx.draws).toBe(50); + expect(frames.requested).toBe(2); + + // The last one gone, the clock stops asking for frames. + cleanup(); + expect(frames.pending).toBe(0); + } finally { + cleanup(); + frames.restore(); + restore(); + } + }); + }); + + const pointAt = (x: number, y: number) => + window.dispatchEvent( + new PointerEvent("pointermove", { clientX: x, clientY: y }), + ); + + /** + * One orb `size` px across, its corner at (100, 100) on screen. `play` runs its frames from `from` + * to `to` seconds after it mounts, `fps` a second, calling `each` before every frame, and says how + * many of them it drew. + */ + function onScreen(size: number) { + const ctx = fakeContext(); + const frames = manualFrames(); + const restore = withContext(ctx); + const view = render(); + orbOf(view.container).getBoundingClientRect = () => + ({ left: 100, top: 100, width: size, height: size }) as DOMRect; + const start = performance.now(); + return { + play(from: number, to: number, fps = 60, each?: (t: number) => void) { + const before = ctx.draws; + for (let i = Math.ceil(from * fps); i < to * fps; i++) { + each?.(i / fps); + frames.run(start + (i / fps) * 1000); + } + return ctx.draws - before; + }, + done() { + cleanup(); + frames.restore(); + restore(); + }, + }; + } + + test("a small orb at rest with the pointer still draws nothing once it settles, and draws again when the pointer moves", () => { + withObserver(undefined, () => { + const orb = onScreen(28); + try { + pointAt(400, 114); + // It turns to the pointer, settles and blinks once, all before 3.2s. + expect(orb.play(0, 3.2)).toBeGreaterThan(0); + // Its next blink is 5.5s in at the soonest. + expect(orb.play(3.2, 4.4)).toBe(0); + // A theme flip is drawn once, though nothing else has moved. + document.documentElement.classList.add("dark"); + expect(orb.play(4.4, 4.8)).toBe(1); + pointAt(114, 400); + expect(orb.play(4.8, 5.4)).toBeGreaterThan(0); + } finally { + document.documentElement.classList.remove("dark"); + orb.done(); + } + }); + }); + + test("a small orb draws about thirty frames a second on a 120Hz screen, a large one every frame", () => { + withObserver(undefined, () => { + // A pointer circling the orb, so its eyes always have somewhere new to look. + const circle = (t: number) => + pointAt( + 114 + 200 * Math.cos(2 * Math.PI * t), + 114 + 200 * Math.sin(2 * Math.PI * t), + ); + const small = onScreen(28); + try { + const drawn = small.play(0, 1, 120, circle); + expect(drawn).toBeGreaterThanOrEqual(28); + expect(drawn).toBeLessThanOrEqual(31); + } finally { + small.done(); + } + const large = onScreen(56); + try { + expect(large.play(0, 1, 120, circle)).toBe(120); + } finally { + large.done(); + } + }); + }); + + test("an orb off screen is not drawn", () => { + withObserver(Sight, () => { + const ctx = fakeContext(); + const frames = manualFrames(); + const restore = withContext(ctx); + try { + const view = render(); + const orb = orbOf(view.container); + Sight.show(orb, false); + expect(frames.pending).toBe(0); + frames.run(); + expect(ctx.draws).toBe(0); + + Sight.show(orb, true); + expect(frames.pending).toBe(1); + frames.run(); + expect(ctx.draws).toBe(1); + } finally { + cleanup(); + frames.restore(); + restore(); + } + }); + }); + + test("nothing draws while the tab is hidden", () => { + withObserver(undefined, () => { + const ctx = fakeContext(); + const frames = manualFrames(); + const restore = withContext(ctx); + const hide = (hidden: boolean) => { + Object.defineProperty(document, "hidden", { + configurable: true, + get: () => hidden, + }); + document.dispatchEvent(new Event("visibilitychange")); + }; + try { + render(); + expect(frames.pending).toBe(1); + hide(true); + expect(frames.pending).toBe(0); + frames.run(); + expect(ctx.draws).toBe(0); + + hide(false); + expect(frames.pending).toBe(1); + frames.run(); + expect(ctx.draws).toBe(1); + } finally { + Reflect.deleteProperty(document, "hidden"); + cleanup(); + frames.restore(); + restore(); + } + }); + }); + + test("renders with no canvas context", () => { + const frames = manualFrames(); + try { + const view = render( + , + ); + expect(view.container.querySelector("canvas")).not.toBeNull(); + // Nothing to draw on, so it asks for no frames. + expect(frames.requested).toBe(0); + } finally { + cleanup(); + frames.restore(); + } + }); +}); + +describe("its eyes", () => { + /** An orb 40 px across, centred at (120, 120). */ + const rect = { left: 100, top: 100, width: 40, height: 40 }; + const measure = () => rect; + const moveTo = (x: number, y: number) => + window.dispatchEvent( + new PointerEvent("pointermove", { clientX: x, clientY: y }), + ); + /** A second of frames at 30 a second, from `from` seconds on, and whether any of them perked up. */ + const watchFor = ( + gaze: ReturnType, + state: BotOrbState, + from = 0, + ) => { + let perked = false; + for (let i = 1; i <= 30; i++) { + perked = followPointer(gaze, from + i / 30, state, measure) || perked; + } + return perked; + }; + + test("turn to the pointer, fully at rest and partly while it works", () => { + const stop = subscribePointer(); + try { + moveTo(600, 120); + const idle = restingGaze(); + const thinking = restingGaze(); + watchFor(idle, "idle"); + watchFor(idle, "idle", 1); + watchFor(thinking, "thinking"); + watchFor(thinking, "thinking", 1); + // Far to its right: it looks right, level. + expect(idle.x).toBeGreaterThan(0.6); + expect(Math.abs(idle.y)).toBeLessThan(0.05); + expect(idle.w).toBeCloseTo(1, 2); + expect(thinking.w).toBeCloseTo(0.35, 2); + } finally { + stop(); + } + }); + + test("perk up once when the pointer comes close, and let go only well past it", () => { + const stop = subscribePointer(); + try { + moveTo(600, 120); + const gaze = restingGaze(); + expect(watchFor(gaze, "idle")).toBe(false); + + // Within 0.9 of its size of its centre. + moveTo(150, 120); + expect(watchFor(gaze, "idle", 1)).toBe(true); + expect(watchFor(gaze, "idle", 2)).toBe(false); + + // Out to 1.2 of its size: still close, no second perk. + moveTo(168, 120); + expect(watchFor(gaze, "idle", 3)).toBe(false); + moveTo(150, 120); + expect(watchFor(gaze, "idle", 4)).toBe(false); + + // Past 1.3 it lets go, and the next approach perks again. + moveTo(180, 120); + watchFor(gaze, "idle", 5); + moveTo(150, 120); + expect(watchFor(gaze, "idle", 6)).toBe(true); + } finally { + stop(); + } + }); + + test("a listening orb has already noticed you, and stops watching once the pointer leaves", () => { + const stop = subscribePointer(); + try { + moveTo(150, 120); + const gaze = restingGaze(); + expect(watchFor(gaze, "listening")).toBe(false); + expect(gaze.w).toBeGreaterThan(0.4); + + document.dispatchEvent( + new PointerEvent("pointerout", { bubbles: true, relatedTarget: null }), + ); + watchFor(gaze, "listening", 1); + watchFor(gaze, "listening", 2); + expect(gaze.w).toBeLessThan(0.01); + } finally { + stop(); + } + }); +}); + +describe("a Bot's face and OpenBot's mark", () => { + test("a Bot's avatar is named once and drawn as its orb", () => { + const view = render( + , + ); + const avatar = view.getByRole("img", { name: "Research desk" }); + expect(view.queryAllByRole("img")).toHaveLength(1); + expect(avatar.style.width).toBe("36px"); + const orb = orbOf(avatar); + expect(orb.getAttribute("aria-hidden")).toBe("true"); + expect(orb.dataset.palette).toBe(orbLookFor("research-desk").palette); + expect(orb.dataset.seed).toBe("research-desk"); + expect(orb.style.width).toBe("36px"); + expect(avatar.querySelector("canvas")).not.toBeNull(); + expect(avatar.querySelector("svg")).toBeNull(); + }); + + test("two Bots with different seeds can differ in colour, the same seed never does", () => { + const paletteOf = (seed: string) => { + const view = render(); + const palette = orbOf(view.container).dataset.palette; + cleanup(); + return palette; + }; + // Chosen to land on different colours: one pair in eight shares one. + expect(paletteOf("research-desk")).not.toBe(paletteOf("expense-review")); + expect(paletteOf("research-desk")).toBe(paletteOf("research-desk")); + }); + + test("OpenBot's mark is violet and maps every state", () => { + const states: Record = { + idle: "idle", + listening: "listening", + thinking: "thinking", + streaming: "thinking", + done: "done", + error: "idle", + }; + for (const [state, shown] of Object.entries(states)) { + const view = render(); + const orb = orbOf(view.container); + expect(orb.dataset.palette).toBe("violet"); + expect(orb.dataset.state).toBe(shown); + expect(orb.style.width).toBe("64px"); + cleanup(); + } + const view = render(); + expect(orbOf(view.container).style.width).toBe("192px"); + }); + + test("OpenBot's mark is drawn at a size in px, and refuses a share of its container", () => { + const view = render(); + expect(orbOf(view.container).style.width).toBe("27.5px"); + // Read as a number, "100%" would draw a 100px orb whatever its container is. + // @ts-expect-error Only a size in px type-checks. + expect(() => AgentOrb({ size: "100%" })).toThrow( + `AgentOrb's size is in px, as "64px", not "100%".`, + ); + }); +}); diff --git a/app/tests/bot-route-default-agent.test.tsx b/app/tests/bot-route-default-agent.test.tsx index 72c3f0ee26..4169e60072 100644 --- a/app/tests/bot-route-default-agent.test.tsx +++ b/app/tests/bot-route-default-agent.test.tsx @@ -398,3 +398,25 @@ test("/bot?watch=true restores the live Computer panel on reload", async () => { "general-assistant", ); }); + +/* + * The approval cards in the chat read the inbox without polling it, one reader per tool call, so + * the screen polls it once for all of them. + */ +test("/bot keeps the approvals its chat's cards read fresh, with one poll", async () => { + const queryClient = queryClientWithAgents([GENERAL_ASSISTANT]); + queryClient.setQueryData(["approvals"], { + enabled: true, + requests: [], + rules: [], + teamRules: [], + questions: [], + }); + const view = renderBot(queryClient); + await view.findByTestId("copilot-chat"); + const polling = ( + queryClient.getQueryCache().find({ queryKey: ["approvals"] })?.observers ?? + [] + ).filter((observer) => observer.options.refetchInterval); + expect(polling).toHaveLength(1); +}); diff --git a/app/tests/channel-avatar.test.tsx b/app/tests/channel-avatar.test.tsx new file mode 100644 index 0000000000..e59b93b133 --- /dev/null +++ b/app/tests/channel-avatar.test.tsx @@ -0,0 +1,60 @@ +import { afterAll, afterEach, beforeAll, expect, test } from "bun:test"; +import { GlobalRegistrator } from "@happy-dom/global-registrator"; +import { cleanup, render } from "@testing-library/react"; +import { ChannelAvatar } from "@/components/channels/avatar"; +import { settleReactWork } from "./settle-react-work"; + +/* + * A conversation's face in the sidebar. While the conversation is busy its Bot's orb works, and a + * still dot says so as well: an orb this small only blinks, and with reduced motion it does not move + * at all, so the orb alone would not read as busy from across the roster. + */ + +beforeAll(() => GlobalRegistrator.register({ url: "http://localhost/" })); +afterEach(() => cleanup()); +afterAll(async () => { + await settleReactWork(); + GlobalRegistrator.unregister(); +}); + +/** The state each orb in the avatar is drawn in, back to front. */ +function orbStates(container: HTMLElement) { + return [...container.querySelectorAll("[data-state]")].map((orb) => + orb.getAttribute("data-state"), + ); +} + +test("a busy conversation's orb works, and a still dot says it is working", () => { + const view = render(); + + expect(orbStates(view.container)).toEqual(["thinking"]); + expect(view.getByText("Working…").className).toContain("sr-only"); + // Nothing in the indicator moves, with or without reduced motion. + expect(view.container.querySelector("[class*='animate-']")).toBeNull(); +}); + +test("a quiet conversation's orb rests and has no dot", () => { + const view = render(); + + expect(orbStates(view.container)).toEqual(["idle"]); + expect(view.queryByText("Working…")).toBeNull(); +}); + +test("a state passed in reaches every face, a group's included", () => { + const one = render( + , + ); + expect(orbStates(one.container)).toEqual(["listening"]); + one.unmount(); + + const group = render( + , + ); + expect(orbStates(group.container)).toEqual(["done", "done", "done"]); + // The dot is about the conversation, so it stays whatever the faces show. + expect(group.getByText("Working…")).toBeTruthy(); +}); diff --git a/app/tests/channel-pagination.test.tsx b/app/tests/channel-pagination.test.tsx index 55b6300c5b..3dc3acedd6 100644 --- a/app/tests/channel-pagination.test.tsx +++ b/app/tests/channel-pagination.test.tsx @@ -225,7 +225,7 @@ test("message emphasis updates mounted rows from the account preferences cache", const view = renderSidebar(); const agent = await view.findByText("Recent conversation"); const thread = view.getByText("New conversation"); - expect(thread.className).toContain("text-[0.9rem]"); + expect(thread.className).toContain("type-sub"); act(() => clients[0]?.setQueryData(userPreferencesQueryOptions("user").queryKey, { messageListEmphasis: "agent", @@ -234,7 +234,7 @@ test("message emphasis updates mounted rows from the account preferences cache", skin: "openbot", }), ); - await waitFor(() => expect(agent.className).toContain("text-[0.9rem]")); + await waitFor(() => expect(agent.className).toContain("type-sub")); expect(thread.className).toContain("text-muted-foreground"); }); diff --git a/app/tests/channel-work-row.test.tsx b/app/tests/channel-work-row.test.tsx index a37164c9f9..d4a7ab4b62 100644 --- a/app/tests/channel-work-row.test.tsx +++ b/app/tests/channel-work-row.test.tsx @@ -56,7 +56,7 @@ test("says who was asked, with their avatar and no clock", () => { // The avatar is decoration beside a label that already names the Bot, so it is announced once. const avatar = row.querySelector("[aria-hidden='true'] [role='img']"); expect(avatar?.getAttribute("aria-label")).toBe("Ana"); - expect(row.querySelector("svg")).not.toBeNull(); + expect(row.querySelector("canvas")).not.toBeNull(); }); test("prefers the name and avatar the person knows the Bot by", () => { diff --git a/app/tests/chat-step.test.tsx b/app/tests/chat-step.test.tsx new file mode 100644 index 0000000000..8c3f641c73 --- /dev/null +++ b/app/tests/chat-step.test.tsx @@ -0,0 +1,310 @@ +import { + afterAll, + afterEach, + beforeAll, + describe, + expect, + test, +} from "bun:test"; +import type { Message } from "@ag-ui/core"; +import { GlobalRegistrator } from "@happy-dom/global-registrator"; +import { QueryClient, QueryClientProvider } from "@tanstack/react-query"; +import { act, cleanup, render } from "@testing-library/react"; +import { useState } from "react"; +import { + ConversationPresenceProvider, + useConversationOrb, + useConversationStatus, + useReportConversation, +} from "@/components/channels/conversation-presence"; +import { readToolName } from "@/lib/plugins/tool-name"; +import { type StepApproval, stepFor, THINKING } from "@/lib/channels/step"; +import { settleReactWork } from "./settle-react-work"; + +/** + * The line under a conversation's name says what its Bot is doing right now: the step it is + * running, in the words that step's own line uses, or that it is thinking, and nothing once the + * turn is over. + */ + +beforeAll(() => GlobalRegistrator.register({ url: "http://localhost/" })); +afterEach(cleanup); +afterAll(async () => { + await settleReactWork(); + GlobalRegistrator.unregister(); +}); + +const asked = (id: string, content: string): Message => ({ + id, + role: "user", + content, +}); + +/** The Bot calling a tool, answered when `result` is given. */ +function called(id: string, name: string, result?: string): Message[] { + const call: Message = { + id: `${id}-call`, + role: "assistant", + content: "", + toolCalls: [ + { + id, + type: "function", + function: { name, arguments: JSON.stringify({ query: "Q3 budget" }) }, + }, + ], + }; + return result === undefined + ? [call] + : [ + call, + { id: `${id}-result`, role: "tool", toolCallId: id, content: result }, + ]; +} + +const SEARCH = "mcp__exa__web_search_exa"; +const FETCH = "mcp__exa__web_fetch_exa"; + +test("a declined email stops being named before the tool result arrives", () => { + const declined: StepApproval = { + status: "denied", + decision: "deny", + action: { toolCallId: "send" }, + }; + const messages = [ + asked("u1", "Reply to Ana"), + ...called("send", "mcp__google-workspace__gmail_send_reply"), + ]; + expect(stepFor(messages, true)).toBe("Sending your email"); + expect(stepFor(messages, true, [declined])).toBe(THINKING); + // Done by the person themselves, it does not run here either. + expect( + stepFor(messages, true, [ + { + status: "approved", + decision: "handled", + action: { toolCallId: "send" }, + }, + ]), + ).toBe(THINKING); + // Allowed, it is the step the Bot is on. + expect( + stepFor(messages, true, [ + { + status: "approved", + decision: "allow_once", + action: { toolCallId: "send" }, + }, + ]), + ).toBe("Sending your email"); +}); + +describe("the step the line names", () => { + test("a step still running is named as its line names it", () => { + const messages = [asked("u1", "What changed?"), ...called("s", SEARCH)]; + expect(stepFor(messages, true)).toBe("Searching the web"); + expect(stepFor(messages, true)).toBe(readToolName(SEARCH, true).label); + }); + + test("the newest running step wins over an earlier one", () => { + const messages = [ + asked("u1", "What changed?"), + ...called("s", SEARCH), + ...called("f", FETCH), + ]; + expect(stepFor(messages, true)).toBe("Reading web pages"); + expect(stepFor(messages, true)).toBe(readToolName(FETCH, true).label); + }); + + test("a step that has its result is not running", () => { + const messages = [ + asked("u1", "What changed?"), + ...called("s", SEARCH), + ...called("f", FETCH, '"Page text"'), + ]; + expect(stepFor(messages, true)).toBe("Searching the web"); + }); + + test("working with no step running is thinking", () => { + expect(stepFor([asked("u1", "Hi")], true)).toBe(THINKING); + const answered = [ + asked("u1", "What changed?"), + ...called("s", SEARCH, '"Results"'), + ]; + expect(stepFor(answered, true)).toBe(THINKING); + }); + + test("a call an earlier turn left unanswered is not this turn's step", () => { + const messages = [ + asked("u1", "What changed?"), + ...called("s", SEARCH), + asked("u2", "Never mind, what's on today?"), + ]; + expect(stepFor(messages, true)).toBe(THINKING); + }); + + test("the person's mail and calendar are named for what happens to them, through any app", () => { + const ask = asked("u1", "Anything from Priya?"); + const workspace = (tool: string) => [ + ask, + ...called("g", `mcp__google-workspace__${tool}`), + ]; + expect(stepFor(workspace("gmail_search_messages"), true)).toBe( + "Searching your email", + ); + expect(stepFor(workspace("gmail_read_message"), true)).toBe( + "Reading your email", + ); + expect(stepFor(workspace("calendar_list_events"), true)).toBe( + "Checking your calendar", + ); + expect(stepFor(workspace("gmail_send_reply"), true)).toBe( + "Sending your email", + ); + // A brokered Gmail is the same mailbox, in the same words its line uses. + expect( + stepFor( + [ask, ...called("c", "mcp__composio-gmail__GMAIL_FETCH_EMAILS")], + true, + ), + ).toBe("Reading your email"); + }); + + test("the browser, memory and the deployment's own tools say what they do", () => { + const ask = asked("u1", "Book it"); + expect(stepFor([ask, ...called("b", "computer_navigate")], true)).toBe( + "Working in your browser", + ); + expect(stepFor([ask, ...called("m", "recall_personal_memory")], true)).toBe( + "Checking memory", + ); + expect(stepFor([ask, ...called("a", "show_app_access")], true)).toBe( + "Checking app access", + ); + }); + + test("an app's call it cannot name by what it does is named by the app, never its identifier", () => { + const messages = [ + asked("u1", "Do the thing"), + ...called("x", "mcp__acme__frobnicate_widgets"), + ]; + expect(stepFor(messages, true)).toBe("Using Acme"); + // A mail call that changes something keeps its own name on its line, so here it is the app. + expect( + stepFor( + [ + asked("u1", "Draft it"), + ...called("d", "mcp__google-workspace__gmail_create_draft"), + ], + true, + ), + ).toBe("Using Google Workspace"); + }); + + test("a question for the person and a drawn interface are not named; the step before them is", () => { + const messages = [ + asked("u1", "What changed?"), + ...called("s", SEARCH), + ...called("q", "ask_person"), + ...called("f", "askForm"), + ]; + expect(stepFor(messages, true)).toBe("Searching the web"); + expect( + stepFor([asked("u1", "Hi"), ...called("c", "showBarChart")], true), + ).toBe(THINKING); + }); + + test("no phrase carries an ellipsis or a tool's identifier", () => { + for (const name of [ + "mcp__google-workspace__gmail_search_messages", + "mcp__google-workspace__calendar_list_events", + "mcp__google-workspace__gmail_create_draft", + "mcp__acme__frobnicate_widgets", + "make_page", + "show_app_access", + ]) { + const step = + stepFor([asked("u1", "Go"), ...called("x", name)], true) ?? ""; + expect(step).not.toContain("…"); + expect(step).not.toMatch( + /_|google-workspace|Gmail search|Calendar list|Frobnicate/, + ); + } + expect(THINKING).toBe("Thinking it through"); + }); + + test("with the turn over there is nothing to say, running step or not", () => { + const messages = [asked("u1", "What changed?"), ...called("s", SEARCH)]; + expect(stepFor(messages, false)).toBeNull(); + expect(stepFor([], false)).toBeNull(); + }); +}); + +test("a keystroke does not reach the status line, and a new step does not reach the orb", () => { + let report: ReturnType; + let nextMessages: (messages: Message[]) => void = () => {}; + const renders = { status: 0, orb: 0 }; + + function Reporter() { + const [messages, setMessages] = useState([ + asked("u1", "What changed?"), + ]); + nextMessages = setMessages; + report = useReportConversation({ working: true, failed: false, messages }); + return null; + } + function StatusProbe() { + renders.status++; + const status = useConversationStatus(); + return ( + + {status.kind === "working" ? status.step : status.kind} + + ); + } + function OrbProbe() { + renders.orb++; + return {useConversationOrb()}; + } + + const client = new QueryClient({ + defaultOptions: { queries: { retry: false, staleTime: Infinity } }, + }); + client.setQueryData(["approvals"], { + enabled: true, + requests: [], + rules: [], + teamRules: [], + questions: [], + }); + client.setQueryData(["channels", "list"], { + pages: [{ channels: [], nextCursor: null }], + pageParams: [""], + }); + const view = render( + + + + + + + , + ); + const probe = (name: string) => + view.container.querySelector(`[data-probe='${name}']`)?.textContent; + expect(probe("status")).toBe(THINKING); + expect(probe("orb")).toBe("thinking"); + + const atStart = { ...renders }; + act(() => report?.onTextChange("A correction")); + act(() => report?.onTextChange("A correction, typed on")); + expect(renders.status).toBe(atStart.status); + + const afterTyping = { ...renders }; + act(() => + nextMessages([asked("u1", "What changed?"), ...called("s", SEARCH)]), + ); + expect(probe("status")).toBe("Searching the web"); + expect(renders.status).toBeGreaterThan(afterTyping.status); + expect(renders.orb).toBe(afterTyping.orb); +}); diff --git a/app/tests/computer-calm-start.test.tsx b/app/tests/computer-calm-start.test.tsx index a1c9c63b5a..fba11e65ae 100644 --- a/app/tests/computer-calm-start.test.tsx +++ b/app/tests/computer-calm-start.test.tsx @@ -21,8 +21,8 @@ test("a call turned away by a computer still starting reads as not done yet, not />, ); const line = view.getByText("Clicked, not done yet"); - expect(line.closest(".text-amber-600")).toBeNull(); - expect(line.closest(".text-destructive")).toBeNull(); + expect(line.closest(".text-status-needs")).toBeNull(); + expect(line.closest(".text-status-problem")).toBeNull(); expect(view.queryByText(/didn't work/)).toBeNull(); }); diff --git a/app/tests/computer-pane.test.tsx b/app/tests/computer-pane.test.tsx new file mode 100644 index 0000000000..ca84e3009c --- /dev/null +++ b/app/tests/computer-pane.test.tsx @@ -0,0 +1,466 @@ +import { + afterAll, + afterEach, + beforeAll, + describe, + expect, + test, +} from "bun:test"; +import { CopilotKitProvider } from "@copilotkit/react-core/v2"; +import { GlobalRegistrator } from "@happy-dom/global-registrator"; +import { + type InfiniteData, + QueryClient, + QueryClientProvider, +} from "@tanstack/react-query"; +import { + type AnyRoute, + createMemoryHistory, + createRootRoute, + createRoute, + createRouter, + Outlet, + RouterProvider, +} from "@tanstack/react-router"; +import { + act, + cleanup, + fireEvent, + render, + waitFor, + within, +} from "@testing-library/react"; +import { ConversationPresenceProvider } from "@/components/channels/conversation-presence"; +import { ComputerPaneTitle } from "@/components/computer/computer-panel"; +import { ComputerView } from "@/components/computer/computer-view"; +import { SidebarProvider } from "@/components/ui/sidebar"; +import { type AgentProfile, agentKeys } from "@/lib/agents/queries"; +import type { ApprovalInboxData } from "@/lib/approvals"; +import { authKeys } from "@/lib/auth/queries"; +import { + type AgentChannel, + type ChannelPage, + channelKeys, +} from "@/lib/channels/queries"; +import { reportComputerActivity } from "@/lib/copilot/computer-activity"; +import { Route as ChannelRoute } from "@/routes/_authed/_app/channel/$channelId"; +import { settleReactWork } from "./settle-react-work"; + +/** + * The pane beside a conversation that shows its Bot's computer: a header naming whose computer it + * is, with the Bot's orb, Take control and a dot while the conversation needs the person; the + * screen; and under it the steps. It is 440px wide beside the conversation, opens itself only when + * it has room there, and otherwise lights the header's computer button. + */ + +let width = 1280; +class Observer implements ResizeObserver { + constructor(private readonly callback: ResizeObserverCallback) {} + observe(target: Element) { + this.callback( + [ + { + target, + contentRect: new DOMRect(0, 0, width, 800), + borderBoxSize: [], + contentBoxSize: [], + devicePixelContentBoxSize: [], + }, + ], + this, + ); + } + unobserve() {} + disconnect() {} +} + +class OfflineWebSocket extends EventTarget { + readyState = 0; + onclose = null; + onerror = null; + onmessage = null; + onopen = null; + close() {} + send() {} +} + +const originalFetch = globalThis.fetch; +const originalObserver = globalThis.ResizeObserver; +const originalWebSocket = globalThis.WebSocket; +beforeAll(() => { + GlobalRegistrator.register({ url: "http://localhost/" }); + globalThis.ResizeObserver = Observer; + globalThis.WebSocket = OfflineWebSocket as unknown as typeof WebSocket; +}); +afterEach(() => { + cleanup(); + width = 1280; + globalThis.fetch = originalFetch; +}); +afterAll(async () => { + await settleReactWork(); + globalThis.ResizeObserver = originalObserver; + globalThis.WebSocket = originalWebSocket; + GlobalRegistrator.unregister(); +}); + +const THREAD = "t-1"; + +function inboxWith(threads: string[]): ApprovalInboxData { + return { + enabled: true, + requests: threads.map((threadId, index) => ({ + id: `r-${index}`, + status: "pending", + createdAt: "2026-10-28T00:00:00.000Z", + action: { + botId: "ana", + toolRef: "computer/click", + effect: "write", + scope: "external", + threadId, + toolCallId: `call-${index}`, + args: {}, + target: {}, + }, + })), + rules: [], + teamRules: [], + questions: [], + }; +} + +function emptyList(): InfiniteData { + return { pages: [{ channels: [], nextCursor: null }], pageParams: [""] }; +} + +describe("the pane's title", () => { + function renderTitle( + name: string | undefined, + inbox: ApprovalInboxData = inboxWith([]), + ) { + const client = new QueryClient({ + defaultOptions: { queries: { retry: false, staleTime: Infinity } }, + }); + client.setQueryData(channelKeys.list(), emptyList()); + client.setQueryData(["approvals"], inbox); + return render( + + +

    + +

    +     +     , + ); + } + + test("names whose computer it is, beside that Bot's orb", () => { + const view = renderTitle("Ana"); + const heading = view.getByRole("heading"); + expect(heading.textContent).toBe("Ana's computer"); + const orb = heading.querySelector("[data-palette]"); + expect(orb).not.toBeNull(); + // The conversation's orb: at rest while nothing is happening. + expect(orb?.getAttribute("data-state")).toBe("idle"); + }); + + test("is the Computer until the Bot's name arrives", () => { + expect(renderTitle(undefined).getByRole("heading").textContent).toBe( + "Computer", + ); + }); + + test("carries a dot while an approval waits in this conversation, and only this one", () => { + const waiting = renderTitle("Ana", inboxWith([THREAD])); + const dot = waiting.getByRole("img", { name: "Needs you" }); + expect(dot.className).toContain("bg-status-needs"); + cleanup(); + + const elsewhere = renderTitle("Ana", inboxWith(["another-thread"])); + expect(elsewhere.queryByRole("img", { name: "Needs you" })).toBeNull(); + }); +}); + +const ANA: AgentProfile = { + id: "ana", + name: "Ana", + title: "Research", + roleDescription: "Looks things up.", + avatarSeed: "ana", + visibility: "private", + endpoint: null, + builtIn: true, + hasAuth: false, + hasCallbackToken: false, + isolated: false, + isolationActive: false, + isolationOffReason: null, + hidden: false, + pinned: false, + systemOwned: false, + canManage: true, + mine: true, +}; +const CHANNEL: AgentChannel = { + id: "c-1", + // A conversation is named for what it is about; its computer is named for its Bot. + name: "Q3 budget", + agentIds: ["ana"], + threadId: THREAD, + active: true, + lastMessageAt: null, +}; + +const NativeResponse = globalThis.Response; +const control = { + holder: "bot", + since: "2026-10-28T00:00:00.000Z", + requested: false, + transitioning: false, + resumeSnapshotRequired: false, + recoveryRequired: false, +}; +function sse(events: unknown[]) { + return new NativeResponse( + events.map((event) => `data: ${JSON.stringify(event)}\n\n`).join(""), + { headers: { "content-type": "text/event-stream" } }, + ); +} +async function served(input: Parameters[0]) { + const url = new URL( + typeof input === "string" + ? input + : input instanceof URL + ? input.href + : input.url, + "http://localhost", + ); + if (url.pathname === "/api/agents") + return NativeResponse.json({ agents: [ANA] }); + if (url.pathname === "/api/plugins/for/ana") + return NativeResponse.json({ skills: [], tools: [] }); + if (url.pathname.endsWith("/info")) + return NativeResponse.json({ + version: "fixture", + agents: { ana: { description: "Fixture", capabilities: {} } }, + mode: "sse", + telemetryDisabled: true, + }); + if (url.pathname.endsWith("/connect")) + return sse([ + { type: "RUN_STARTED", threadId: THREAD, runId: "join" }, + { type: "MESSAGES_SNAPSHOT", messages: [] }, + { type: "RUN_FINISHED", threadId: THREAD, runId: "join" }, + ]); + if (url.pathname.endsWith(`/threads/${THREAD}/messages`)) + return NativeResponse.json({ messages: [] }); + if (url.pathname === "/api/voice/sessions") + return NativeResponse.json({ sessions: [], nextCursor: null }); + if (url.pathname === "/api/approvals") + return NativeResponse.json(inboxWith([])); + if (url.pathname.endsWith("/control")) return NativeResponse.json(control); + return NativeResponse.json({ error: "not served" }, { status: 404 }); +} + +/** The conversation route as the app registers it, with its channel and Bot already loaded. */ +function renderConversation(path: string) { + globalThis.fetch = Object.assign(served, { + preconnect: originalFetch.preconnect, + }); + const root = createRootRoute({ + component: () => ( + + + + ), + }); + const authed = createRoute({ + id: "_authed", + getParentRoute: () => root, + component: Outlet, + }); + const app = createRoute({ + id: "_app", + getParentRoute: () => authed, + component: Outlet, + }); + const channel = createRoute({ + path: "channel/$channelId", + getParentRoute: () => app, + component: (ChannelRoute as AnyRoute).options.component, + validateSearch: (ChannelRoute as AnyRoute).options.validateSearch, + }); + const client = new QueryClient({ + defaultOptions: { queries: { retry: false, staleTime: Infinity } }, + }); + client.setQueryData(authKeys.currentUser(), { + id: "maya", + email: "maya@example.com", + role: "user", + onboarding: null, + }); + client.setQueryData(agentKeys.list(false), [ANA]); + client.setQueryData(channelKeys.detail("c-1"), CHANNEL); + client.setQueryData(channelKeys.list(), emptyList()); + client.setQueryData(["approvals"], inboxWith([])); + const router = createRouter({ + routeTree: root.addChildren([ + authed.addChildren([app.addChildren([channel])]), + ]), + history: createMemoryHistory({ initialEntries: [path] }), + }); + const view = render( + + + + + , + ); + return { view, router }; +} + +describe("the Bot's computer beside a conversation", () => { + test("is 440px wide, with whose computer it is and Take control in its header", async () => { + const { view } = renderConversation("/channel/c-1?watch=true"); + const pane = await waitFor(() => { + const found = view.container.querySelector( + '[data-slot="detail-pane"]', + ); + if (!found) throw new Error("No pane yet"); + return found; + }); + expect(pane.style.width).toBe("440px"); + const header = pane.querySelector( + '[data-slot="detail-pane-header"]', + ); + const title = header?.querySelector('[data-slot="detail-pane-title"]'); + await waitFor(() => expect(title?.textContent).toBe("Ana's computer")); + const take = await within(header as HTMLElement).findByRole("button", { + name: "Take control", + }); + expect(take.closest('[data-slot="detail-pane-title"]')).toBeNull(); + // Take control is in the header alone, not again under the screen. + expect( + within(pane).getAllByRole("button", { name: "Take control" }), + ).toHaveLength(1); + // The screen's badge is the Bot's name, not the conversation's. + await waitFor(() => + expect(within(pane).getByText("The Bot has control.")).toBeTruthy(), + ); + expect(within(pane).queryByText("Q3 budget")).toBeNull(); + expect( + within(pane).getByRole("heading", { name: "Activity" }), + ).toBeTruthy(); + }); + + test("opens itself beside the conversation when the Bot starts using it", async () => { + const { view, router } = renderConversation("/channel/c-1"); + await view.findByRole("button", { name: "Open Computer" }); + act(() => { + reportComputerActivity("ana"); + }); + await waitFor(() => + expect(router.state.location.search).toMatchObject({ watch: true }), + ); + }); + + test("does not cover a narrow conversation: it lights the computer button instead", async () => { + width = 700; + const { view, router } = renderConversation("/channel/c-1"); + const button = await view.findByRole("button", { name: "Open Computer" }); + act(() => { + reportComputerActivity("ana"); + }); + await waitFor(() => + expect(within(button).getByRole("img", { name: "In use" })).toBeTruthy(), + ); + expect(router.state.location.search).not.toMatchObject({ watch: true }); + expect(view.queryByRole("dialog")).toBeNull(); + + // Opened by hand, it is a sheet, and the light goes out. + fireEvent.click(button); + expect( + await view.findByRole("dialog", { name: "Ana's computer" }), + ).toBeTruthy(); + await waitFor(() => + expect(view.queryByRole("img", { name: "In use" })).toBeNull(), + ); + }); +}); + +describe("the screen in the pane", () => { + // Without an observer the preview counts as on screen, so it polls for its frame. + let originalIntersectionObserver: typeof IntersectionObserver | undefined; + let originalDecode: typeof HTMLImageElement.prototype.decode; + beforeAll(() => { + originalIntersectionObserver = globalThis.IntersectionObserver; + originalDecode = HTMLImageElement.prototype.decode; + Reflect.deleteProperty(globalThis, "IntersectionObserver"); + HTMLImageElement.prototype.decode = () => Promise.resolve(); + }); + afterAll(() => { + if (originalIntersectionObserver) + globalThis.IntersectionObserver = originalIntersectionObserver; + HTMLImageElement.prototype.decode = originalDecode; + }); + + function screenServer(state: Record) { + globalThis.fetch = Object.assign( + async (input: Parameters[0]) => { + const path = String(input); + if (path.endsWith("/control")) + return NativeResponse.json({ ...control, ...state }); + if (path.endsWith("/screenshot")) + return NativeResponse.json({ + base64: "AAAA", + width: 320, + height: 200, + capturedAt: new Date(0).toISOString(), + url: "https://accounts.example.com/login", + }); + return NativeResponse.json({ error: "not served" }, { status: 404 }); + }, + { preconnect: originalFetch.preconnect }, + ); + } + function renderScreen(wheelInHeader = false) { + const client = new QueryClient({ + defaultOptions: { queries: { retry: false } }, + }); + return render( + + + , + ); + } + + test("asks for a secret for the page, and says it goes only there, never naming a site it can't vouch for", async () => { + screenServer({ secretWanted: "the code sent to your phone" }); + const view = renderScreen(); + expect( + await view.findByText("The page needs the code sent to your phone"), + ).toBeTruthy(); + expect( + view.getByText("It goes straight to the page. Ana never sees it."), + ).toBeTruthy(); + const field = view.getByLabelText( + /The page needs the code sent to your phone/, + ); + expect(field.getAttribute("type")).toBe("password"); + expect(view.getByRole("button", { name: "Send to the page" })).toBeTruthy(); + }); + + test("with Take control in the header, under the screen it only says who has control", async () => { + screenServer({}); + const view = renderScreen(true); + expect(await view.findByText("The Bot has control.")).toBeTruthy(); + expect(view.queryByRole("button", { name: "Take control" })).toBeNull(); + }); +}); diff --git a/app/tests/conversation-presence.test.tsx b/app/tests/conversation-presence.test.tsx new file mode 100644 index 0000000000..f8077287f7 --- /dev/null +++ b/app/tests/conversation-presence.test.tsx @@ -0,0 +1,550 @@ +import { + afterAll, + afterEach, + beforeAll, + beforeEach, + describe, + expect, + test, +} from "bun:test"; +import type { Message } from "@ag-ui/core"; +import { CopilotKitProvider } from "@copilotkit/react-core/v2"; +import { GlobalRegistrator } from "@happy-dom/global-registrator"; +import { + type InfiniteData, + QueryClient, + QueryClientProvider, +} from "@tanstack/react-query"; +import { act, cleanup, render, waitFor } from "@testing-library/react"; +import userEvent from "@testing-library/user-event"; +import { type ReactNode, useState } from "react"; +import { ChannelChat } from "@/components/channels/channel-chat"; +import { Composer } from "@/components/channels/composer/composer"; +import { + ConversationPresenceProvider, + ConversationStatusLine, + useConversationOrb, +} from "@/components/channels/conversation-presence"; +import { ConversationView } from "@/components/channels/conversation-view"; +import type { AgentProfile } from "@/lib/agents/queries"; +import type { ApprovalInboxData } from "@/lib/approvals"; +import { + type AgentChannel, + type ChannelPage, + type ChannelSummary, + channelKeys, +} from "@/lib/channels/queries"; +import { THINKING } from "@/lib/channels/step"; +import { queryClient } from "@/query-client"; +import { settleReactWork } from "./settle-react-work"; + +/** + * A conversation with one Bot shows that Bot's orb in its header, with a line under the name: the + * orb works while the Bot has the turn and listens while the person types; the line says "Working" + * and the step, or "Needs you" while an approval waits there, and nothing at rest. The conversation + * list's busy flag counts as working too, so a run the server resumes after Allow once still shows. + */ + +const originalFetch = globalThis.fetch; +beforeAll(() => GlobalRegistrator.register({ url: "http://localhost/" })); +// Nothing here is served unless a test says so: what the header reads is seeded into the cache. +beforeEach(() => { + globalThis.fetch = Object.assign( + async () => Response.json({ error: "not served" }, { status: 404 }), + { preconnect: originalFetch.preconnect }, + ); +}); +afterEach(() => { + cleanup(); + queryClient.clear(); + globalThis.fetch = originalFetch; +}); +afterAll(async () => { + await settleReactWork(); + GlobalRegistrator.unregister(); +}); + +const CHANNEL = "c-1"; +const THREAD = "t-1"; + +const asked = (id: string, content: string): Message => ({ + id, + role: "user", + content, +}); +function searching(result?: string): Message[] { + const call: Message = { + id: "s-call", + role: "assistant", + content: "", + toolCalls: [ + { + id: "s", + type: "function", + function: { + name: "mcp__exa__web_search_exa", + arguments: JSON.stringify({ query: "Q3 budget" }), + }, + }, + ], + }; + return result === undefined + ? [call] + : [ + call, + { id: "s-result", role: "tool", toolCallId: "s", content: result }, + ]; +} +const RUNNING = [asked("u1", "What changed?"), ...searching()]; +const ANSWERED = [asked("u1", "What changed?"), ...searching('"Results"')]; + +function summary(busy: boolean): ChannelSummary { + return { + id: CHANNEL, + name: "Q3 budget", + agentIds: ["ana"], + threadId: THREAD, + active: true, + lastMessageAt: null, + summary: null, + lastMessage: null, + lastMessageAgentId: null, + createdAt: "2026-10-28T00:00:00.000Z", + pinned: false, + lastReadAt: null, + busy, + }; +} +function listOf(busy: boolean): InfiniteData { + return { + pages: [{ channels: [summary(busy)], nextCursor: null }], + pageParams: [""], + }; +} +function inboxWith( + requests: { threadId: string; status: string }[], +): ApprovalInboxData { + return { + enabled: true, + requests: requests.map(({ threadId, status }, index) => ({ + id: `r-${index}`, + status, + createdAt: "2026-10-28T00:00:00.000Z", + action: { + botId: "ana", + toolRef: "google-workspace/gmail_send_reply", + effect: "write", + scope: "external", + threadId, + toolCallId: `call-${index}`, + args: {}, + target: {}, + }, + })), + rules: [], + teamRules: [], + questions: [], + }; +} + +/** The header's orb state, read as the header's orb reads it. */ +function OrbProbe() { + return {useConversationOrb()}; +} + +/** + * The route's presence with the header's two readers and whatever `show` puts under them, so + * swapping the conversation changes that and nothing else. + */ +function renderConversation( + first: ReactNode, + { + busy = false, + inbox = inboxWith([]), + }: { busy?: boolean; inbox?: ApprovalInboxData } = {}, +) { + let show: (screen: ReactNode) => void = () => {}; + function Screen() { + const [screen, setScreen] = useState(first); + show = setScreen; + return screen; + } + const client = new QueryClient({ + defaultOptions: { queries: { retry: false, staleTime: Infinity } }, + }); + client.setQueryData(channelKeys.list(), listOf(busy)); + client.setQueryData(["approvals"], inbox); + // The transcript draws a lone tool call through CopilotKit, so the conversation needs its provider. + const view = render( + + + + + + + + + , + ); + return { + view, + show: (screen: ReactNode) => act(() => show(screen)), + // The cache tells its readers on a timer of its own, so the update is waited out. + setBusy: (next: boolean) => + act(async () => { + client.setQueryData(channelKeys.list(), listOf(next)); + await new Promise((resolve) => setTimeout(resolve, 0)); + }), + line: () => + view.container.querySelector("[aria-live='polite']")?.textContent ?? null, + orb: () => + view.container.querySelector("[data-probe='orb']")?.textContent ?? null, + }; +} + +function Chat({ + busy = false, + messages, + stopped, +}: { + busy?: boolean; + messages: Message[]; + stopped?: string; +}) { + return ( + {}} + pending={busy} + {...(stopped ? { stopped } : {})} + /> + ); +} + +describe("the header's orb and the line under the name", () => { + test("rest with nothing to say, in a live region that is already there", () => { + const { view, line, orb } = renderConversation( + , + ); + expect(view.container.querySelector("[aria-live='polite']")).not.toBeNull(); + expect(line()).toBe(""); + expect(orb()).toBe("idle"); + }); + + test("say Working and the step while the Bot works, and nothing once the turn is over", () => { + const { show, line, orb } = renderConversation( + , + ); + expect(line()).toBe("Working · Searching the web"); + expect(orb()).toBe("thinking"); + + show(); + expect(line()).toBe(`Working · ${THINKING}`); + + show(); + expect(line()).toBe(""); + // An answer lands before the orb rests. + expect(orb()).toBe("done"); + }); + + test("a stopped turn does not land", () => { + const { show, orb } = renderConversation(); + show(); + expect(orb()).toBe("idle"); + }); + + test("a turn that ends on a call held for the person does not land", () => { + // The server holds the search for the person: the turn ends with the call and no result. + const held = renderConversation(); + held.show(); + expect(held.orb()).toBe("idle"); + cleanup(); + + // The browser sends a click to its card, stops the run, and notes that it went there. + const clicking = (result?: string): Message[] => [ + asked("u1", "Pay the invoice"), + { + id: "c-call", + role: "assistant", + content: "", + toolCalls: [ + { + id: "click", + type: "function", + function: { + name: "computer_click", + arguments: JSON.stringify({ ref: "e1" }), + }, + }, + ], + }, + ...(result === undefined + ? [] + : [ + { + id: "c-result", + role: "tool" as const, + toolCallId: "click", + content: result, + }, + ]), + ]; + const sent = renderConversation(); + sent.show(); + expect(sent.orb()).toBe("idle"); + }); + + test("Needs you holds off the landing of a turn that did answer", () => { + const { show, line, orb } = renderConversation( + , + { inbox: inboxWith([{ threadId: THREAD, status: "pending" }]) }, + ); + show(); + expect(line()).toBe("Needs you"); + expect(orb()).toBe("idle"); + }); + + test("say Needs you while an approval waits in this conversation, and only this one", () => { + const waiting = renderConversation(, { + inbox: inboxWith([{ threadId: THREAD, status: "pending" }]), + }); + expect(waiting.line()).toBe("Needs you"); + cleanup(); + + const elsewhere = renderConversation(, { + inbox: inboxWith([ + { threadId: "another-thread", status: "pending" }, + { threadId: THREAD, status: "approved" }, + ]), + }); + expect(elsewhere.line()).toBe(""); + }); + + test("Needs you comes before the step the Bot is on", () => { + const { line } = renderConversation(, { + inbox: inboxWith([{ threadId: THREAD, status: "pending" }]), + }); + expect(line()).toBe("Needs you"); + }); + + test("a run the server resumed after Allow once still shows as working", async () => { + const { line, orb, setBusy } = renderConversation( + , + { busy: true }, + ); + expect(line()).toBe("Working"); + expect(orb()).toBe("thinking"); + + await setBusy(false); + expect(line()).toBe(""); + expect(orb()).toBe("idle"); + }); + + test("the busy flag of the conversation's own turn does not outlast the turn", async () => { + const { show, line, setBusy } = renderConversation( + , + { busy: true }, + ); + expect(line()).toBe("Working · Searching the web"); + + // The turn is over here a moment before the server's flag comes down. + show(); + expect(line()).toBe(""); + await setBusy(false); + expect(line()).toBe(""); + + // A run the server starts afterwards is its own, and shows. + await setBusy(true); + expect(line()).toBe("Working"); + }); + + test("a busy spell this tab owned in one conversation does not hide the next one's", () => { + const client = new QueryClient({ + defaultOptions: { queries: { retry: false, staleTime: Infinity } }, + }); + // Both conversations are busy: this tab's own turn in the first, the server in the second. + client.setQueryData(channelKeys.list(), { + pages: [ + { + channels: [summary(true), { ...summary(true), id: "c-2" }], + nextCursor: null, + }, + ], + pageParams: [""], + }); + client.setQueryData(["approvals"], inboxWith([])); + let open: (channelId: string) => void = () => {}; + function Route() { + const [channelId, setChannelId] = useState(CHANNEL); + open = setChannelId; + return ( + + + {channelId === CHANNEL ? : null} + + ); + } + const view = render( + + + + + , + ); + const line = () => + view.container.querySelector("[aria-live='polite']")?.textContent; + expect(line()).toBe("Working · Searching the web"); + + act(() => open("c-2")); + expect(line()).toBe("Working"); + }); + + test("the orb listens while the person types, and rests when the box is empty again", async () => { + const { view, orb, line } = renderConversation( + , + ); + const user = userEvent.setup({ document: view.container.ownerDocument }); + const box = view.getByRole("textbox", { name: "Message" }); + await user.type(box, "One more thing"); + await waitFor(() => expect(orb()).toBe("listening")); + expect(line()).toBe(""); + + await user.clear(box); + await waitFor(() => expect(orb()).toBe("idle")); + }); + + test("leaving the conversation mid-turn rests the orb and empties the line", () => { + const { show, line, orb } = renderConversation( + , + ); + expect(orb()).toBe("thinking"); + show(null); + expect(line()).toBe(""); + expect(orb()).toBe("idle"); + }); +}); + +describe("the composer's placeholder", () => { + test("names the Bot it was given, and otherwise keeps its own", () => { + const named = render( + {}} + placeholder="Message Ana" + />, + ); + expect(named.getByText("Message Ana")).toBeTruthy(); + cleanup(); + + for (const compact of [true, false]) { + const plain = render( {}} />); + expect(plain.getByText("Ask anything")).toBeTruthy(); + cleanup(); + } + }); +}); + +/* + * A real conversation with one Bot, drawn as the route draws it under its header, against a server + * that answers just enough for it to mount. + */ +describe("a conversation with one Bot", () => { + const ANA: AgentProfile = { + id: "ana", + name: "Ana", + title: "Research", + roleDescription: "Looks things up.", + avatarSeed: "ana", + visibility: "private", + endpoint: null, + builtIn: true, + hasAuth: false, + hasCallbackToken: false, + isolated: false, + isolationActive: false, + isolationOffReason: null, + hidden: false, + pinned: false, + systemOwned: false, + canManage: true, + mine: true, + }; + const channel: AgentChannel = { + id: CHANNEL, + name: "Q3 budget", + agentIds: ["ana"], + threadId: THREAD, + active: true, + lastMessageAt: null, + }; + const NativeResponse = globalThis.Response; + const sse = (events: unknown[]) => + new NativeResponse( + events.map((event) => `data: ${JSON.stringify(event)}\n\n`).join(""), + { headers: { "content-type": "text/event-stream" } }, + ); + async function served(input: Parameters[0]) { + const url = new URL( + typeof input === "string" + ? input + : input instanceof URL + ? input.href + : input.url, + "http://localhost", + ); + if (url.pathname === "/api/agents") + return NativeResponse.json({ agents: [ANA] }); + if (url.pathname === "/api/plugins/for/ana") + return NativeResponse.json({ skills: [], tools: [] }); + if (url.pathname.endsWith("/info")) + return NativeResponse.json({ + version: "fixture", + agents: { ana: { description: "Fixture", capabilities: {} } }, + mode: "sse", + telemetryDisabled: true, + }); + if (url.pathname.endsWith("/connect")) + return sse([ + { type: "RUN_STARTED", threadId: THREAD, runId: "join" }, + { type: "MESSAGES_SNAPSHOT", messages: [] }, + { type: "RUN_FINISHED", threadId: THREAD, runId: "join" }, + ]); + if (url.pathname.endsWith(`/threads/${THREAD}/messages`)) + return NativeResponse.json({ messages: [] }); + if (url.pathname === "/api/voice/sessions") + return NativeResponse.json({ sessions: [], nextCursor: null }); + if (url.pathname === "/api/approvals") + return NativeResponse.json(inboxWith([])); + return NativeResponse.json({ error: "not served" }, { status: 404 }); + } + + test("asks for a message to the Bot by name, and its orb listens while the person types", async () => { + globalThis.fetch = Object.assign(served, { + preconnect: originalFetch.preconnect, + }); + queryClient.setQueryData(channelKeys.list(), listOf(false)); + const view = render( + + + + + + + + , + ); + expect(await view.findByText("Message Ana")).toBeTruthy(); + const orb = () => + view.container.querySelector("[data-probe='orb']")?.textContent; + expect(orb()).toBe("idle"); + + const user = userEvent.setup({ document: view.container.ownerDocument }); + await user.type( + view.getByRole("textbox", { name: "Message" }), + "Is the budget final?", + ); + await waitFor(() => expect(orb()).toBe("listening")); + }); +}); diff --git a/app/tests/detail-panel.test.tsx b/app/tests/detail-panel.test.tsx index cf887a65ed..9faa9fe0fb 100644 --- a/app/tests/detail-panel.test.tsx +++ b/app/tests/detail-panel.test.tsx @@ -6,6 +6,7 @@ import { fireEvent, render, waitFor, + within, } from "@testing-library/react"; import { useEffect } from "react"; import { DetailPanel } from "@/components/layout/detail-panel"; @@ -92,6 +93,61 @@ test("narrow container overlays details without remounting the chat draft", asyn expect(mounts).toBe(1); }); +test("says whether it overlays, so an owner can keep from covering the page", async () => { + const reports: boolean[] = []; + const view = render( + Browser preview

    } + detailWidth={440} + onClose={() => {}} + onOverlayChange={(overlay) => reports.push(overlay)} + open={false} + title="Computer" + > +

    Chat

    +     , + ); + // Closed or open, measured against the width it opens at: 440 beside a 400px page. + await resize(view.container, 840); + expect(reports.at(-1)).toBe(false); + await resize(view.container, 839); + expect(reports.at(-1)).toBe(true); + await resize(view.container, 1200); + expect(reports.at(-1)).toBe(false); + // Told when it changes, not on every resize. + const told = reports.length; + await resize(view.container, 1300); + expect(reports).toHaveLength(told); +}); + +test("the owner's actions sit beside the close button, beside the page and in a sheet", async () => { + const props = { + title: "Computer", + actions: , + onClose: () => {}, + detail:

    Browser preview

    , + children:

    Chat

    , + }; + const view = render(); + await resize(view.container, 1000); + const header = view.container.querySelector( + '[data-slot="detail-pane-header"]', + ); + expect( + header?.querySelector('[data-slot="detail-pane-title"]')?.textContent, + ).toBe("Computer"); + // In the header, and outside the title the skins dress. + const docked = view.getByRole("button", { name: "Take control" }); + expect(header?.contains(docked)).toBe(true); + expect(docked.closest('[data-slot="detail-pane-title"]')).toBeNull(); + + await resize(view.container, 560); + const sheet = await view.findByRole("dialog", { name: "Computer" }); + expect( + within(sheet).getByRole("button", { name: "Take control" }), + ).toBeTruthy(); +}); + test("closing narrow details invokes the owner and unmounts detail work", async () => { let closed = 0; const props = { diff --git a/app/tests/hotkey-layouts.test.ts b/app/tests/hotkey-layouts.test.ts index 8815d3289f..092c6d41ae 100644 --- a/app/tests/hotkey-layouts.test.ts +++ b/app/tests/hotkey-layouts.test.ts @@ -1,17 +1,22 @@ import { expect, test } from "bun:test"; -import { getHotkey, matchesHotkey } from "@/lib/hotkeys/hotkeys"; +import { formatHotkey, getHotkey, matchesHotkey } from "@/lib/hotkeys/hotkeys"; /** - * The New chat shortcut on a keyboard layout that does not write Latin letters. + * The New chat and Search shortcuts on a keyboard layout that does not write Latin letters. * - * Settings shows it as Shift+N, and `matchesHotkey` compared `KeyboardEvent.key`. That is right for - * a layout that moves the letter, such as Dvorak: the key a person presses when told "N" is the one - * that writes an N. A layout that writes another script has no key that writes an N at + * Settings shows New chat as Shift+N, and `matchesHotkey` compared `KeyboardEvent.key`. That is + * right for a layout that moves the letter, such as Dvorak: the key a person presses when told "N" + * is the one that writes an N. A layout that writes another script has no key that writes an N at * all. Shift and the N key write "Т" on Russian and "Ν" (Greek capital nu, not a Latin N) on Greek, - * so the shortcut Settings lists never fired for anybody with one of those layouts selected. + * so the shortcut Settings lists never fired for anybody with one of those layouts selected. Search's + * K key writes "л" on Russian and "κ" on Greek, so ⌘K goes by the physical key the same way. */ const combo = getHotkey("new-chat").combo; +const search = getHotkey("search").combo; +/** ⌘ on a Mac, Ctrl everywhere else: whichever one the combo's own keycaps name. */ +const mod = formatHotkey(search)[0] === "⌘" ? "metaKey" : "ctrlKey"; +const otherMod = mod === "metaKey" ? "ctrlKey" : "metaKey"; /** A keydown as a browser reports it: `key` from the layout, `code` from the physical key. */ function keydown( @@ -76,3 +81,39 @@ test("the modifiers are still exact on the physical key", () => { matchesHotkey(keydown("Shift", "ShiftLeft", { shiftKey: true }), combo), ).toBe(false); }); + +test("⌘K or Ctrl+K searches on a layout that writes another script", () => { + const pressed = [ + ["Russian", keydown("л", "KeyK", { [mod]: true })], + ["Greek", keydown("κ", "KeyK", { [mod]: true })], + ["Latin", keydown("k", "KeyK", { [mod]: true })], + ] as const; + + expect( + pressed.map(([layout, event]) => [layout, matchesHotkey(event, search)]), + ).toEqual([ + ["Russian", true], + ["Greek", true], + ["Latin", true], + ]); +}); + +test("search needs exactly its modifier on the K key", () => { + // The K key alone writes the letter, and Shift or the other modifier makes it another combo. + expect(matchesHotkey(keydown("л", "KeyK"), search)).toBe(false); + expect( + matchesHotkey( + keydown("Л", "KeyK", { [mod]: true, shiftKey: true }), + search, + ), + ).toBe(false); + expect( + matchesHotkey( + keydown("κ", "KeyK", { [mod]: true, [otherMod]: true }), + search, + ), + ).toBe(false); + expect(matchesHotkey(keydown("т", "KeyN", { [mod]: true }), search)).toBe( + false, + ); +}); diff --git a/app/tests/lone-step-approval.test.tsx b/app/tests/lone-step-approval.test.tsx new file mode 100644 index 0000000000..44b33dc420 --- /dev/null +++ b/app/tests/lone-step-approval.test.tsx @@ -0,0 +1,360 @@ +import { afterAll, afterEach, beforeAll, expect, test } from "bun:test"; +import type { Message } from "@ag-ui/core"; +import { CopilotKitContext } from "@copilotkit/react-core/v2/context"; +import { GlobalRegistrator } from "@happy-dom/global-registrator"; +import { QueryClient, QueryClientProvider } from "@tanstack/react-query"; +import { + cleanup, + fireEvent, + render, + waitFor, + within, +} from "@testing-library/react"; +import { ChatTranscript } from "@/components/channels/chat-transcript"; +import { approvalInboxOptions } from "@/lib/approvals"; +import { settleReactWork } from "./settle-react-work"; + +/** + * A gated call that is the only step of its turn keeps its approval card in the conversation. + * + * A run of steps folds into one line with the cards for its waiting calls beside it + * (`ToolActivity`), but a single server call keeps a line of its own, and that line had no card: + * the request waited on the Bot's page while the conversation showed a step that never finished. + * Once decided the card folds to how it went and who decided, and stays. + */ + +let approvals: unknown[] = []; +let asked = 0; +let inboxDown = false; +const decisions: { path: string; method?: string; body?: BodyInit | null }[] = + []; +let realFetch: typeof fetch; + +beforeAll(() => { + GlobalRegistrator.register({ url: "http://localhost/" }); + realFetch = globalThis.fetch; + globalThis.fetch = Object.assign( + async (input: RequestInfo | URL, init?: RequestInit) => { + if (String(input).endsWith("/decision")) { + decisions.push({ + path: String(input), + method: init?.method, + body: init?.body, + }); + // Decided as the server decides it: the inbox reads the decision back from now on. + const { decision } = JSON.parse(String(init?.body)); + const status = decision === "deny" ? "denied" : "approved"; + const id = String(input).split("/").at(-2); + approvals = approvals.map((request) => + (request as { id: string }).id === id + ? { ...(request as object), status, decision } + : request, + ); + return Response.json({ id, status }); + } + if (String(input).endsWith("/api/approvals")) { + asked += 1; + if (inboxDown) + return Response.json({ error: "Inbox unavailable" }, { status: 503 }); + return Response.json({ + enabled: true, + requests: approvals, + rules: [], + teamRules: [], + questions: [], + }); + } + return Response.json({}); + }, + { preconnect: realFetch.preconnect }, + ); +}); + +afterEach(() => { + cleanup(); + approvals = []; + asked = 0; + inboxDown = false; + decisions.length = 0; +}); + +afterAll(async () => { + await settleReactWork(); + globalThis.fetch = realFetch; + GlobalRegistrator.unregister(); +}); + +const SEND = "mcp__composio-gmail__GMAIL_SEND_EMAIL"; + +/** The person's answer, then the Bot's one call: the send, still waiting on its result. */ +const turn: Message[] = [ + { id: "user-1", role: "user", content: "Double-wall boxes" }, + { + id: "assistant-1", + role: "assistant", + content: "", + toolCalls: [ + { + id: "toolu_send", + type: "function", + function: { + name: SEND, + arguments: JSON.stringify({ + recipient_email: "sam@marlowceramics.com", + body: "Hi Sam, please use the double-wall boxes.", + }), + }, + }, + ], + }, +]; + +const send = ( + toolCallId: string, + overrides: Record = {}, + behaviour: "ask" | "hand_off" = "ask", +) => ({ + id: `approval-${toolCallId}`, + status: "pending", + createdAt: "2026-10-22T20:04:43Z", + action: { + botId: "inbox", + toolRef: "gmail/GMAIL_SEND_EMAIL", + effect: "write", + scope: "gmail", + threadId: "thread", + toolCallId, + args: { + recipient_email: "sam@marlowceramics.com", + subject: "Re: Which box for the Hearth order?", + body: "Hi Sam, please use the double-wall boxes.", + }, + target: {}, + policy: { + behaviour, + source: "rule", + reason: + behaviour === "hand_off" + ? "Emails to suppliers are yours to send." + : "Emails ask first.", + }, + }, + decision: null, + decidedAt: null, + completedAt: null, + result: null, + ...overrides, +}); + +/** + * The transcript asks CopilotKit for a renderer per call. None is registered for a connector tool, + * which is the case under test: the call is drawn by the transcript's own line. + */ +const noRenderers = { + copilotkit: { + renderToolCalls: [], + subscribe: () => ({ unsubscribe() {} }), + }, + executingToolCallIds: new Set(), +} as never; + +function draw(messages: Message[] = turn) { + const client = new QueryClient({ + defaultOptions: { queries: { retry: false } }, + }); + const view = render( + + + + + , + ); + return { ...view, client }; +} + +const card = (view: ReturnType) => + view.findByRole("region", { name: "Waiting for your approval" }); + +test("a gated call alone in its turn draws its approval card under its own line", async () => { + approvals = [send("toolu_send")]; + const view = draw(); + const waiting = within(await card(view)); + expect(waiting.getByRole("button", { name: "Allow once" })).toBeTruthy(); + expect(waiting.getByRole("button", { name: "Decline" })).toBeTruthy(); + // How far the rule it saves reaches, read with the button: every email from Gmail, not this one. + expect( + waiting + .getByRole("button", { name: "Always allow here" }) + .getAttribute("aria-describedby"), + ).toBe( + waiting.getByText( + "This Bot won't ask you again before doing this in Gmail, whatever the details.", + ).id, + ); + // Everything the request would send fits on the face, so Details stays shut. + expect(waiting.getByText("Details").closest("details")?.open).toBe(false); + // What will happen, to whom, and what it says, in plain words. + expect(waiting.getByText("Send your email")).toBeTruthy(); + expect(waiting.getByText("Gmail")).toBeTruthy(); + expect(waiting.getByText("sam@marlowceramics.com")).toBeTruthy(); + expect(waiting.getByText("Re: Which box for the Hearth order?")).toBeTruthy(); + expect( + waiting.getByText("Hi Sam, please use the double-wall boxes."), + ).toBeTruthy(); + // Held on its card, the line waits for the person rather than claiming to send. + expect(view.getByText("Send your email, waiting for you")).toBeTruthy(); + expect(view.queryByText("Sending your email")).toBeNull(); +}); + +test("a request for some other call draws nothing under this one", async () => { + approvals = [send("toolu_other")]; + const view = draw(); + await waitFor(() => expect(asked).toBeGreaterThan(0)); + await settleReactWork(); + expect(view.getByText("Sending your email")).toBeTruthy(); + expect(view.queryByRole("button", { name: "Allow once" })).toBeNull(); +}); + +test("a decline folds the card to who decided, and it stays folded when the inbox refresh fails", async () => { + approvals = [send("toolu_send")]; + const view = draw(); + const key = approvalInboxOptions().queryKey; + const waiting = await card(view); + inboxDown = true; + fireEvent.click(within(waiting).getByRole("button", { name: "Decline" })); + await waitFor(() => + expect(view.client.getQueryState(key)?.status).toBe("error"), + ); + expect(decisions).toEqual([ + { + path: "/api/approvals/approval-toolu_send/decision", + method: "POST", + body: '{"decision":"deny"}', + }, + ]); + await waitFor(() => + expect( + view.queryAllByRole("region", { name: "Waiting for your approval" }), + ).toHaveLength(0), + ); + expect(view.queryByRole("button", { name: "Allow once" })).toBeNull(); + expect(view.queryByRole("button", { name: "Decline" })).toBeNull(); + expect(view.getByText("Declined")).toBeTruthy(); + expect(view.getByText("Declined by you")).toBeTruthy(); + expect(view.client.getQueryData(key)?.requests[0]).toMatchObject({ + id: "approval-toolu_send", + status: "denied", + decision: "deny", + decidedAt: expect.any(String), + }); +}); + +test("a request with more than its face shows opens Details by itself", async () => { + approvals = [ + send("toolu_send", { + action: { + ...send("toolu_send").action, + args: { + recipient_email: "sam@marlowceramics.com", + subject: "Re: Which box for the Hearth order?", + body: "Hi Sam, please use the double-wall boxes.", + html: 'Pay here', + }, + }, + }), + ]; + const view = draw(); + const waiting = within(await card(view)); + expect(waiting.getByText("Details").closest("details")?.open).toBe(true); +}); + +test("an allowed send folds to Allowed, then to Sent once it has run", async () => { + approvals = [send("toolu_send")]; + const view = draw(); + fireEvent.click( + within(await card(view)).getByRole("button", { name: "Allow once" }), + ); + expect(await view.findByText("Allowed once by you")).toBeTruthy(); + approvals = [ + send("toolu_send", { + status: "completed", + decision: "allow_once", + decidedAt: "2026-10-22T20:05:00Z", + completedAt: "2026-10-22T20:05:02Z", + }), + ]; + await view.client.invalidateQueries({ + queryKey: approvalInboxOptions().queryKey, + }); + expect(await view.findByText("Sent")).toBeTruthy(); + expect(view.getByText("Allowed once by you")).toBeTruthy(); + // Folded: the message and the buttons are gone. + expect( + view.queryByText("Hi Sam, please use the double-wall boxes."), + ).toBeNull(); + expect(view.queryByRole("button", { name: "Always allow here" })).toBeNull(); +}); + +test("an action that ran and failed says it didn't work", async () => { + approvals = [ + send("toolu_send", { + status: "completed", + decision: "allow_always", + decidedAt: "2026-10-22T20:05:00Z", + result: { error: "Gmail refused the message." }, + }), + ]; + const view = draw(); + expect(await view.findByText("Didn't work")).toBeTruthy(); + expect(view.getByText("Always allowed here by you")).toBeTruthy(); +}); + +test("a hand-off asks the person to do it, and never offers to let the Bot", async () => { + approvals = [send("toolu_send", {}, "hand_off")]; + const view = draw(); + const waiting = within(await card(view)); + expect(waiting.getByText("This needs you: Send your email")).toBeTruthy(); + expect( + waiting.getByText("Handed to you: Emails to suppliers are yours to send."), + ).toBeTruthy(); + expect(waiting.queryByRole("button", { name: "Allow once" })).toBeNull(); + expect( + waiting.queryByRole("button", { name: "Always allow here" }), + ).toBeNull(); + expect(waiting.getByRole("button", { name: "Don't do it" })).toBeTruthy(); + fireEvent.click(waiting.getByRole("button", { name: "I did it myself" })); + expect(await view.findByText("You did it yourself")).toBeTruthy(); + expect(decisions.map((decision) => decision.body)).toEqual([ + '{"decision":"handled"}', + ]); +}); + +test("a send nobody was asked about, because a saved rule allowed it, leaves no card", async () => { + approvals = [ + send("toolu_send", { + status: "completed", + decision: "allow_always", + action: { ...send("toolu_send").action, policy: undefined }, + }), + ]; + const view = draw(); + await waitFor(() => expect(asked).toBeGreaterThan(0)); + await settleReactWork(); + expect(view.queryByText("Always allowed here by you")).toBeNull(); +}); + +test("a send the person declined is never named as sent", async () => { + const declined: Message[] = [ + ...turn, + { + id: "result-1", + role: "tool", + toolCallId: "toolu_send", + content: + "The person declined this action when you asked for their approval, so it was not done.", + }, + ]; + const view = draw(declined); + expect(await view.findByText("You declined: Send your email")).toBeTruthy(); + expect(view.queryByText(/Sent your email/)).toBeNull(); +}); diff --git a/app/tests/onboarding-frame.test.tsx b/app/tests/onboarding-frame.test.tsx new file mode 100644 index 0000000000..0207a0042f --- /dev/null +++ b/app/tests/onboarding-frame.test.tsx @@ -0,0 +1,644 @@ +import { afterAll, afterEach, beforeAll, expect, test } from "bun:test"; +import { GlobalRegistrator } from "@happy-dom/global-registrator"; +import { + act, + cleanup, + fireEvent, + render, + waitFor, + within, +} from "@testing-library/react"; +import { DemoStage } from "@/components/onboarding/demo-stage"; +import { StepIntro } from "@/components/onboarding/onboarding-frame"; +import { + type OnboardingStep, + OnboardingWizard, +} from "@/components/onboarding/onboarding-wizard"; +import { settleReactWork } from "./settle-react-work"; + +/** + * The frame every onboarding step sits in. One DOM, laid out two ways: on a phone, Back at the top + * left, the steps as dashes and the buttons pinned to the bottom; from `lg` up, the step list and + * the step's words on the left with Continue and Back under them, and its picture on the right. + * happy-dom applies no stylesheet, so both layouts' parts are in the document here, and each test + * reads the one it is about. + */ + +beforeAll(() => GlobalRegistrator.register({ url: "http://localhost:3010" })); +afterEach(cleanup); +afterAll(async () => { + await settleReactWork(); + GlobalRegistrator.unregister(); +}); + +const plain = (id: string, title: string): OnboardingStep => ({ + id, + title, + intro: () => About {title}., + demo: () =>

    Picture of {title}

    , +}); + +const STEPS = [ + plain("one", "One"), + plain("two", "Two"), + plain("three", "Three"), +]; + +function draw(props: Partial[0]> = {}) { + return render(); +} + +const progress = (view: ReturnType) => + view.getByRole("progressbar", { name: "Onboarding progress" }); + +const dashes = (view: ReturnType) => + Array.from(progress(view).querySelectorAll("span")); + +const footer = (view: ReturnType) => { + const found = view.container.querySelector("footer"); + if (!found) throw new Error("no footer"); + return found; +}; + +/** The primary and the second button, in the footer, without the desktop's Back. */ +const footerButtons = (view: ReturnType) => + Array.from(footer(view).querySelectorAll("button")).filter( + (button) => !button.hasAttribute("data-onboarding-back"), + ); + +const primary = (view: ReturnType) => { + const found = footer(view).querySelector( + "[data-onboarding-primary]", + ); + if (!found) throw new Error("no primary"); + return found; +}; + +/** The phone's Back, at the top left. */ +const phoneBack = (view: ReturnType) => + view.container.querySelector( + '[data-onboarding-back="phone"]', + ); + +/** The desktop's Back, beside Continue. */ +const desktopBack = (view: ReturnType) => + view.container.querySelector( + '[data-onboarding-back="desktop"]', + ); + +const stepList = (view: ReturnType) => + view.getByRole("navigation", { name: "Onboarding steps" }); + +test("the first step has no Back, and the dashes say step 1", () => { + const view = draw(); + + expect(phoneBack(view)).toBeNull(); + expect(desktopBack(view)).toBeNull(); + expect(progress(view).getAttribute("aria-valuetext")).toBe("Step 1 of 3"); + expect(dashes(view)).toHaveLength(3); + expect(dashes(view).map((dash) => dash.hasAttribute("data-current"))).toEqual( + [true, false, false], + ); +}); + +test("on a phone, Back sits in the top bar and the button is pinned below the step", () => { + const view = draw({ initialStepId: "two" }); + + expect(phoneBack(view)?.closest("header")).not.toBeNull(); + expect(progress(view).closest("header")).not.toBeNull(); + expect(primary(view).textContent).toBe("Continue"); + // Below the step's words and its picture in document order, which is where it sits on screen. + const heading = view.getByRole("heading", { name: "Two" }); + const picture = view.getByText("Picture of Two"); + for (const before of [heading, picture]) { + expect( + before.compareDocumentPosition(primary(view)) & + Node.DOCUMENT_POSITION_FOLLOWING, + ).toBeTruthy(); + } + // No step here is optional, so nothing offers to skip it. + expect(view.queryByRole("button", { name: "Not now" })).toBeNull(); +}); + +test("the step's words and its picture sit in their own panes, words first", () => { + const view = draw({ initialStepId: "two" }); + + const intro = view.container.querySelector("[data-onboarding-intro]"); + const demo = view.container.querySelector("[data-onboarding-demo]"); + expect(intro?.textContent).toContain("About Two."); + expect(demo?.textContent).toBe("Picture of Two"); + expect( + intro && + demo && + intro.compareDocumentPosition(demo) & Node.DOCUMENT_POSITION_FOLLOWING, + ).toBeTruthy(); + // The desktop's two columns: the picture fills the right, held in view. + expect(demo?.className).toContain("lg:col-start-2"); + expect(demo?.className).toContain("lg:sticky"); +}); + +test("a step with its picture first puts it above the heading on a phone", () => { + const view = draw({ + initialStepId: "two", + steps: [STEPS[0], { ...plain("two", "Two"), demoFirst: true }, STEPS[2]], + }); + + const intro = view.container.querySelector("[data-onboarding-intro]"); + const demo = view.container.querySelector("[data-onboarding-demo]"); + expect(demo?.className).toContain("row-start-2"); + expect(intro?.className).toContain("row-start-3"); + // On a desktop it is still the right-hand pane, and the words still head the left. + expect(intro?.className).toContain("lg:row-start-2"); + expect(demo?.className).toContain("lg:col-start-2"); + // Without the phone's margin under it, which would stop the stretched pane short of the bottom. + expect(demo?.className).toContain("lg:my-0"); +}); + +test("Continue moves on, the dashes follow, and focus lands on the new heading", async () => { + const view = draw(); + + fireEvent.click(primary(view)); + + const heading = await view.findByRole("heading", { name: "Two" }); + expect(progress(view).getAttribute("aria-valuetext")).toBe("Step 2 of 3"); + expect(dashes(view).map((dash) => dash.hasAttribute("data-current"))).toEqual( + [false, true, false], + ); + await waitFor(() => expect(document.activeElement).toBe(heading)); + expect(phoneBack(view)).not.toBeNull(); +}); + +test("Back returns, and hides again on the first step", async () => { + const view = draw({ initialStepId: "two" }); + + fireEvent.click(phoneBack(view) as HTMLButtonElement); + + await view.findByRole("heading", { name: "One" }); + expect(progress(view).getAttribute("aria-valuetext")).toBe("Step 1 of 3"); + expect(phoneBack(view)).toBeNull(); +}); + +test("a step that joins the list later does not move somebody off theirs", () => { + const view = draw({ initialStepId: "two" }); + + view.rerender( + , + ); + + expect(view.getByRole("heading", { name: "Two" })).not.toBeNull(); + expect(progress(view).getAttribute("aria-valuetext")).toBe("Step 2 of 4"); +}); + +/** Three steps where the second is optional, the way Connect your apps is. */ +const OPTIONAL: OnboardingStep[] = [ + plain("one", "One"), + { + ...plain("apps", "Connect your apps"), + secondary: { label: "Not now" }, + footnote: "You can connect more apps later.", + }, + plain("three", "Three"), +]; + +test("an optional step says Continue, with Not now beside it, the same size", () => { + const view = draw({ initialStepId: "apps", steps: OPTIONAL }); + + const [continueButton, notNow] = footerButtons(view); + expect(footerButtons(view)).toHaveLength(2); + expect(continueButton?.textContent).toBe("Continue"); + expect(notNow?.textContent).toBe("Not now"); + // Under the primary on a phone, in the same column, and drawn the same height and shape. + expect(continueButton?.parentElement).toBe(notNow?.parentElement ?? null); + expect( + continueButton && + notNow && + continueButton.compareDocumentPosition(notNow) & + Node.DOCUMENT_POSITION_FOLLOWING, + ).toBeTruthy(); + for (const shape of ["h-[52px]", "w-full", "rounded-full"]) { + expect(continueButton?.className).toContain(shape); + expect(notNow?.className).toContain(shape); + } +}); + +test("the footnote sits just above the buttons", () => { + const view = draw({ initialStepId: "apps", steps: OPTIONAL }); + + const footnote = view.getByText("You can connect more apps later."); + expect(footnote.closest("footer")).not.toBeNull(); + expect( + footnote.compareDocumentPosition(primary(view)) & + Node.DOCUMENT_POSITION_FOLLOWING, + ).toBeTruthy(); +}); + +test("Continue and Not now both move on", async () => { + const continued = draw({ initialStepId: "apps", steps: OPTIONAL }); + fireEvent.click(continued.getByRole("button", { name: "Continue" })); + expect(await continued.findByRole("heading", { name: "Three" })).toBeTruthy(); + cleanup(); + + const skipped = draw({ initialStepId: "apps", steps: OPTIONAL }); + fireEvent.click(skipped.getByRole("button", { name: "Not now" })); + expect(await skipped.findByRole("heading", { name: "Three" })).toBeTruthy(); +}); + +test("a step's own action decides when to move on", async () => { + let handed: (() => void) | null = null; + const view = draw({ + initialStepId: "apps", + steps: [ + OPTIONAL[0] as OnboardingStep, + { + ...(OPTIONAL[1] as OnboardingStep), + primaryLabel: "Turn on notifications", + onPrimary: (next) => { + handed = next; + }, + }, + OPTIONAL[2] as OnboardingStep, + ], + }); + + fireEvent.click(view.getByRole("button", { name: "Turn on notifications" })); + // Nothing moved: the step holds the move until its own action is done. + expect(view.getByRole("heading", { name: "Connect your apps" })).toBeTruthy(); + act(() => (handed as unknown as () => void)()); + expect(await view.findByRole("heading", { name: "Three" })).toBeTruthy(); +}); + +test("a move handed out on one step does nothing once somebody has left it", async () => { + let handed: (() => void) | null = null; + const view = draw({ + initialStepId: "apps", + steps: [ + OPTIONAL[0] as OnboardingStep, + { + ...(OPTIONAL[1] as OnboardingStep), + onPrimary: (next) => { + handed = next; + }, + }, + OPTIONAL[2] as OnboardingStep, + ], + }); + + fireEvent.click(view.getByRole("button", { name: "Continue" })); + fireEvent.click(phoneBack(view) as HTMLButtonElement); + await view.findByRole("heading", { name: "One" }); + act(() => (handed as unknown as () => void)()); + await settleReactWork(); + expect(view.getByRole("heading", { name: "One" })).toBeTruthy(); +}); + +test("a done primary reads green with a check", () => { + const view = draw({ + initialStepId: "apps", + steps: [ + OPTIONAL[0] as OnboardingStep, + { + ...plain("apps", "Get notified"), + primaryLabel: "Notifications on", + primaryDone: true, + }, + OPTIONAL[2] as OnboardingStep, + ], + }); + + const [done] = footerButtons(view); + expect(footerButtons(view)).toHaveLength(1); + expect(done?.textContent).toBe("Notifications on"); + expect(done?.hasAttribute("data-done")).toBe(true); + expect(done?.className).toContain("bg-status-done-soft"); +}); + +test("an optional step keeps Not now's place when it goes, so the primary doesn't move", () => { + const view = draw({ + initialStepId: "apps", + steps: [ + OPTIONAL[0] as OnboardingStep, + { + ...plain("apps", "Get notified"), + primaryLabel: "Notifications on", + primaryDone: true, + optional: true, + }, + OPTIONAL[2] as OnboardingStep, + ], + }); + + expect(footerButtons(view)).toHaveLength(1); + const slot = footer(view).querySelector("[data-onboarding-secondary-slot]"); + expect(slot?.className).toContain("h-[52px]"); + expect(slot?.getAttribute("aria-hidden")).toBe("true"); +}); + +test("a waiting primary shows a spinner, says it is busy, and can't be pressed again", () => { + const view = draw({ + initialStepId: "apps", + steps: [ + OPTIONAL[0] as OnboardingStep, + { + ...plain("apps", "Get notified"), + primaryLabel: "Turn on notifications", + primaryPending: true, + secondary: { label: "Not now" }, + }, + OPTIONAL[2] as OnboardingStep, + ], + }); + + const waiting = primary(view); + expect(waiting.getAttribute("aria-busy")).toBe("true"); + expect(waiting.disabled).toBe(true); + expect(waiting.querySelector(".animate-spin")).not.toBeNull(); +}); + +test("a step whose own form moves on gets no primary, and keeps Back", () => { + const view = draw({ + initialStepId: "three", + steps: [ + ...STEPS.slice(0, 2), + { ...plain("three", "Three"), ownAction: true }, + ], + }); + + expect(footer(view).querySelector("[data-onboarding-primary]")).toBeNull(); + expect(phoneBack(view)).not.toBeNull(); + expect(desktopBack(view)).not.toBeNull(); +}); + +test("held by its caller, it asks to move and draws whatever step it is handed", async () => { + const asked: Array<[string, string]> = []; + const view = render( + asked.push([to, way])} + stepId="apps" + steps={OPTIONAL} + />, + ); + + fireEvent.click(view.getByRole("button", { name: "Continue" })); + fireEvent.click(phoneBack(view) as HTMLButtonElement); + expect(asked).toEqual([ + ["three", "forward"], + ["one", "back"], + ]); + // Nothing moved until the caller said so. + expect(view.getByRole("heading", { name: "Connect your apps" })).toBeTruthy(); + + view.rerender( + asked.push([to, way])} + stepId="one" + steps={OPTIONAL} + />, + ); + const heading = await view.findByRole("heading", { name: "One" }); + await waitFor(() => expect(document.activeElement).toBe(heading)); +}); + +/* + * THE DESKTOP'S STEP LIST. Every step by name, the one on screen marked as the current step, the + * ones before it ticked. A step already reached is a button back to it; one ahead is only its name. + */ + +test("on a desktop, the step list names every step and marks the one on screen", () => { + const view = draw({ initialStepId: "two" }); + + const items = within(stepList(view)).getAllByRole("listitem"); + expect(items.map((item) => item.textContent?.replace(/^\d/, ""))).toEqual([ + "One", + "Two", + "Three", + ]); + const current = stepList(view).querySelector('[aria-current="step"]'); + expect(current?.textContent).toContain("Two"); + // The list is the desktop's, the dashes the phone's. + expect(stepList(view).className).toContain("hidden lg:flex"); + expect(progress(view).parentElement?.className).toContain("lg:hidden"); +}); + +/* + * A SHORT SCREEN. Nine steps, a heading, a few lines and the buttons used to run past the bottom of + * a laptop's window on the longer steps, cutting Continue and Back in half. + * + * WHAT THIS CAN'T PROVE. happy-dom lays nothing out, so no test here can show that the buttons fit + * on any given screen. It pins the structure that makes them fit: on a desktop the frame is one + * screen tall, the step list's row is the only one that can shrink, the list is what scrolls, and + * the step on screen is scrolled into it. That this adds up to every step's buttons on screen at + * 1280 by 800 and 1280 by 700 was checked in a browser, step by step. + */ +test("on a desktop, a short screen scrolls the step list, never the step's words or its buttons", () => { + const scrolled: Array<{ row: Element; options: unknown }> = []; + const original = Element.prototype.scrollIntoView; + Element.prototype.scrollIntoView = function ( + this: Element, + options?: boolean | ScrollIntoViewOptions, + ) { + scrolled.push({ row: this, options }); + }; + try { + const view = draw({ initialStepId: "two" }); + + // One screen tall, and of the left's rows only the list's can shrink. + const frame = view.container.querySelector("[data-onboarding-frame]"); + expect(frame?.className).toContain("lg:h-svh"); + expect(frame?.className).toContain( + "lg:grid-rows-[minmax(10rem,auto)_auto_auto_1fr]", + ); + // The picture stretches to the screen rather than being set to its height, which would give + // that height to the empty last row before the list got its own. + const demo = view.container.querySelector("[data-onboarding-demo]"); + expect(demo?.className).toContain("lg:max-h-svh"); + expect(demo?.className).not.toContain("lg:h-svh"); + + // The list is what scrolls, inside a header and a nav that let it shrink. + const list = stepList(view).querySelector("ol"); + expect(list?.className).toContain("overflow-y-auto"); + expect(list?.className).toContain("min-h-0"); + expect(stepList(view).className).toContain("lg:min-h-0"); + expect(view.container.querySelector("header")?.className).toContain( + "lg:min-h-0", + ); + // The step's words and its buttons neither scroll nor shrink. + const intro = view.container.querySelector("[data-onboarding-intro]"); + for (const part of [intro, footer(view)]) { + expect(part?.className).not.toMatch(/overflow|min-h-0|max-h-/); + } + + // The step on screen is kept in the list's view, and the next one once Continue is pressed. + expect(scrolled.at(-1)?.row.textContent).toContain("Two"); + expect( + scrolled.at(-1)?.row.querySelector('[aria-current="step"]'), + ).not.toBeNull(); + expect(scrolled.at(-1)?.options).toEqual({ block: "nearest" }); + fireEvent.click(primary(view)); + expect(scrolled.at(-1)?.row.textContent).toContain("Three"); + } finally { + Element.prototype.scrollIntoView = original; + } +}); + +test("a step already reached can be picked from the list; one ahead cannot", async () => { + const view = draw({ initialStepId: "two" }); + + const list = within(stepList(view)); + expect(list.queryByRole("button", { name: "Three" })).toBeNull(); + expect(list.queryByRole("button", { name: "Two" })).toBeNull(); + + fireEvent.click(list.getByRole("button", { name: "One" })); + const heading = await view.findByRole("heading", { name: "One" }); + await waitFor(() => expect(document.activeElement).toBe(heading)); + + // Two has been seen now, so it is a way back; Three still is not. + expect( + within(stepList(view)).getByRole("button", { name: "Two" }), + ).toBeTruthy(); + expect( + within(stepList(view)).queryByRole("button", { name: "Three" }), + ).toBeNull(); + fireEvent.click(within(stepList(view)).getByRole("button", { name: "Two" })); + expect(await view.findByRole("heading", { name: "Two" })).toBeTruthy(); +}); + +test("held by its caller, a pick from the list is asked for like Back", () => { + const asked: Array<[string, string]> = []; + const view = render( + asked.push([to, way])} + stepId="three" + steps={STEPS} + />, + ); + + fireEvent.click(within(stepList(view)).getByRole("button", { name: "One" })); + expect(asked).toEqual([["one", "back"]]); +}); + +test("on a desktop, Back sits beside Continue, before it, and Back on a phone is hidden there", () => { + const view = draw({ initialStepId: "two" }); + + const back = desktopBack(view); + expect(back?.closest("footer")).not.toBeNull(); + expect(back?.textContent).toBe("Back"); + expect(back?.className).toContain("hidden"); + expect(back?.className).toContain("lg:inline-flex"); + expect(back?.className).toContain("lg:order-first"); + expect(phoneBack(view)?.parentElement?.className).toContain("lg:hidden"); + + fireEvent.click(back as HTMLButtonElement); + expect(view.getByRole("heading", { name: "One" })).toBeTruthy(); +}); + +/* + * THE DEMOS' CLOCK. A picture moves only while somebody can see it: on screen, in a shown tab. + */ + +/** An `IntersectionObserver` that reports only what the test tells it to. */ +class Sight { + static all: Sight[] = []; + readonly root = null; + readonly rootMargin = ""; + readonly thresholds = []; + targets: Element[] = []; + constructor(private readonly callback: IntersectionObserverCallback) { + Sight.all.push(this); + } + observe(target: Element) { + this.targets.push(target); + } + unobserve() {} + disconnect() { + this.targets = []; + } + takeRecords() { + return []; + } + static show(target: Element, isIntersecting: boolean) { + for (const sight of Sight.all) { + if (!sight.targets.includes(target)) continue; + sight.callback( + [{ target, isIntersecting } as IntersectionObserverEntry], + sight as unknown as IntersectionObserver, + ); + } + } +} + +function drawStage() { + return render( + + {({ beat, playing }) => ( +

    + Beat {beat} {playing ? "playing" : "still"} +

    + )} +     , + ); +} + +const stageOf = (view: ReturnType) => + view.container.querySelector("[data-demo]") as HTMLElement; + +test("a demo is one picture, read by its label, with nothing inside to reach", () => { + const view = drawStage(); + + const picture = view.getByRole("img", { name: "A test picture" }); + expect(picture.firstElementChild?.hasAttribute("inert")).toBe(true); +}); + +test("a demo plays its beats while on screen, and holds still once scrolled away", async () => { + const original = globalThis.IntersectionObserver; + globalThis.IntersectionObserver = + Sight as unknown as typeof IntersectionObserver; + try { + const view = drawStage(); + const stage = stageOf(view); + expect(stage.hasAttribute("data-playing")).toBe(true); + await waitFor(() => + expect(stage.getAttribute("data-demo-beat")).not.toBe("0"), + ); + + act(() => Sight.show(stage, false)); + expect(stage.hasAttribute("data-playing")).toBe(false); + expect(view.getByText(/still/)).toBeTruthy(); + const held = stage.getAttribute("data-demo-beat"); + await new Promise((resolve) => setTimeout(resolve, 80)); + expect(stage.getAttribute("data-demo-beat")).toBe(held); + + act(() => Sight.show(stage, true)); + expect(stage.hasAttribute("data-playing")).toBe(true); + } finally { + globalThis.IntersectionObserver = original; + Sight.all = []; + } +}); + +test("a demo in a hidden tab holds still until the tab is shown", async () => { + const view = drawStage(); + const stage = stageOf(view); + const hidden = Object.getOwnPropertyDescriptor(Document.prototype, "hidden"); + try { + Object.defineProperty(document, "hidden", { + configurable: true, + get: () => true, + }); + act(() => { + document.dispatchEvent(new Event("visibilitychange")); + }); + expect(stage.hasAttribute("data-playing")).toBe(false); + const held = stage.getAttribute("data-demo-beat"); + await new Promise((resolve) => setTimeout(resolve, 80)); + expect(stage.getAttribute("data-demo-beat")).toBe(held); + } finally { + Reflect.deleteProperty(document, "hidden"); + if (hidden) Object.defineProperty(Document.prototype, "hidden", hidden); + } + act(() => { + document.dispatchEvent(new Event("visibilitychange")); + }); + expect(stage.hasAttribute("data-playing")).toBe(true); +}); diff --git a/app/tests/onboarding-route.test.tsx b/app/tests/onboarding-route.test.tsx new file mode 100644 index 0000000000..3099ae9a52 --- /dev/null +++ b/app/tests/onboarding-route.test.tsx @@ -0,0 +1,653 @@ +import { + afterAll, + afterEach, + beforeAll, + beforeEach, + expect, + test, +} from "bun:test"; +import { GlobalRegistrator } from "@happy-dom/global-registrator"; +import { QueryClientProvider } from "@tanstack/react-query"; +import { + createMemoryHistory, + createRootRouteWithContext, + createRoute, + createRouter, + Outlet, + RouterProvider, +} from "@tanstack/react-router"; +import { + act, + cleanup, + fireEvent, + render, + waitFor, + within, +} from "@testing-library/react"; +import userEvent from "@testing-library/user-event"; +import { refreshNotificationPermission } from "@/lib/notifications/permission"; +import { queryClient } from "@/query-client"; +import type { RouterContext } from "@/router-context"; +import { Route as OnboardingRoute } from "@/routes/_authed/onboarding"; +import { settleReactWork } from "./settle-react-work"; + +/** + * Where somebody is in onboarding lives in the address: `?step=`. A reload stays on the step, + * browser Back walks back one, a consent comes back to the step it left from, and every move is + * saved, so arriving without a step opens the one they were on. The name step is the way out: Get + * started saves the name, completes onboarding, plays the orb's finish and opens the app. + */ + +beforeAll(() => GlobalRegistrator.register({ url: "http://localhost:3010" })); +afterAll(async () => { + await settleReactWork(); + GlobalRegistrator.unregister(); +}); + +const BOT = { + id: "general", + name: "General Assistant", + title: "Everyday work", + roleDescription: "", + avatarSeed: "general", + visibility: "private", + endpoint: null, + builtIn: true, + hasAuth: false, + mine: false, +}; + +const CHECKLIST = { + audience: "admin", + rows: [ + { + id: "model", + scope: "deployment", + title: "AI model", + required: true, + status: "needs_setup", + ready: false, + detail: "Without a model key, OpenBot can't answer.", + fix: { + kind: "env", + names: ["OPENAI_API_KEY"], + docsUrl: "https://github.com/CopilotKit/openbot#quick-start", + page: { label: "Credentials", path: "/admin/credentials" }, + }, + }, + ], +}; + +const originalFetch = globalThis.fetch; +let saved: number; +let saves: number[]; +let completed: boolean; +let named: string | null; +let localEvaluation: boolean; +/** Whether `/api/capabilities` answers, and how many times it was asked. */ +let capabilitiesFail: boolean; +let capabilitiesAsked: number; +const defaults = queryClient.getDefaultOptions(); + +beforeEach(() => { + // Another file's screens may have cached the signed-in person in the app's one query client. + queryClient.clear(); + queryClient.setDefaultOptions({ + ...defaults, + queries: { ...defaults.queries, retry: false }, + }); + saved = 0; + saves = []; + completed = false; + named = null; + localEvaluation = false; + capabilitiesFail = false; + capabilitiesAsked = 0; + (globalThis as { Notification?: unknown }).Notification = { + permission: "default", + requestPermission: async () => "default", + }; + refreshNotificationPermission(); + globalThis.fetch = Object.assign( + async (input: RequestInfo | URL, init?: RequestInit) => { + const path = String(input); + const method = init?.method ?? "GET"; + if (path === "/api/me") + return Response.json({ + user: { + id: "person-1", + email: "maya@example.com", + name: named ?? "maya@example.com", + needsName: named === null, + role: "admin", + onboarding: { + step: saved, + completedAt: completed ? "2026-10-23T00:00:00Z" : null, + }, + }, + }); + if (path === "/api/me/profile" && method === "PATCH") { + named = (JSON.parse(String(init?.body)) as { name: string }).name; + return Response.json({ profile: { name: named, needsName: false } }); + } + if (path === "/api/me/onboarding" && method === "POST") { + const body = JSON.parse(String(init?.body)) as { + step?: number; + completed?: boolean; + }; + if (body.completed) { + completed = true; + return Response.json({ + onboarding: { step: saved, completedAt: "2026-10-23T00:00:00Z" }, + }); + } + if (body.step === undefined) throw new Error("Missing onboarding step"); + saves.push(body.step); + saved = body.step; + return Response.json({ + onboarding: { step: saved, completedAt: null }, + }); + } + if (path === "/api/capabilities") { + capabilitiesAsked += 1; + return capabilitiesFail + ? Response.json({ error: "down" }, { status: 503 }) + : Response.json({ localEvaluation }); + } + if (path.startsWith("/api/agents")) + return Response.json({ agents: [BOT] }); + if (path === "/api/plugins/for/general/apps") + return Response.json({ + apps: [ + { + key: "notion", + displayName: "Notion", + logo: null, + account: "personal", + connect: "oauth", + state: "not_connected", + }, + ], + }); + if (path === "/api/setup") return Response.json({ checklist: CHECKLIST }); + return Response.json({}); + }, + { preconnect: originalFetch.preconnect }, + ); +}); +afterEach(() => { + cleanup(); + globalThis.fetch = originalFetch; + queryClient.clear(); + queryClient.setDefaultOptions(defaults); + window.sessionStorage.clear(); +}); + +function open(path: string, given?: ReturnType) { + const rootRoute = createRootRouteWithContext()({ + component: Outlet, + }); + const authedRoute = createRoute({ + id: "/_authed", + getParentRoute: () => rootRoute, + component: Outlet, + }); + const homeRoute = createRoute({ + path: "/", + getParentRoute: () => authedRoute, + component: () =>

    Home

    , + }); + const credentialsRoute = createRoute({ + path: "/admin/credentials", + getParentRoute: () => authedRoute, + component: () =>

    Credentials page

    , + }); + const wired = ( + OnboardingRoute as unknown as { + update: (options: unknown) => typeof OnboardingRoute; + } + ).update({ + id: "/onboarding", + path: "/onboarding", + getParentRoute: () => authedRoute, + }); + const history = given ?? createMemoryHistory({ initialEntries: [path] }); + const router = createRouter({ + routeTree: rootRoute.addChildren([ + authedRoute.addChildren([homeRoute, credentialsRoute, wired]), + ]), + history, + context: { queryClient }, + }); + const view = render( + + + , + ); + return Object.assign(view, { router, history }); +} + +const heading = (view: ReturnType, name: string) => + view.findByRole("heading", { name }); +const searchOf = (view: ReturnType) => + view.router.state.location.search as { + step?: string; + connected?: string; + open?: string; + }; +/** The phone's Back, at the top left. The desktop's is the same button beside Continue. */ +const back = (view: ReturnType) => { + const found = view.container.querySelector( + '[data-onboarding-back="phone"]', + ); + if (!found) throw new Error("no Back"); + return found; +}; +/** Continue, once the deployment's answer has let it be pressed. */ +const continueButton = async (view: ReturnType) => { + const button = await view.findByRole("button", { name: "Continue" }); + await waitFor(() => expect(button.hasAttribute("disabled")).toBe(false)); + return button; +}; + +test("arriving without a step writes the first one into the address, in place", async () => { + const view = open("/onboarding"); + + await heading(view, "Welcome to OpenBot"); + await waitFor(() => expect(searchOf(view).step).toBe("welcome")); + // In place: nothing behind it to go Back to inside onboarding. + expect(view.history.length).toBe(1); +}); + +test("Continue adds a step to the address and to history, and saves it", async () => { + const view = open("/onboarding?step=welcome"); + await heading(view, "Welcome to OpenBot"); + + fireEvent.click(await continueButton(view)); + + await heading(view, "Each Bot has its own browser and files"); + expect(searchOf(view).step).toBe("computer"); + expect(view.history.length).toBe(2); + await waitFor(() => expect(saves).toContain(2)); +}); + +test("the steps run welcome, computer, team, approval, apps, notify, setup, name", async () => { + const view = open("/onboarding?step=welcome"); + await heading(view, "Welcome to OpenBot"); + + const list = view.getByRole("navigation", { name: "Onboarding steps" }); + await waitFor(() => + expect( + within(list) + .getAllByRole("listitem") + .map((item) => item.textContent?.replace(/^\d/, "")), + ).toEqual([ + "Welcome", + "A browser and files of its own", + "Your team of Bots", + "You decide what it asks", + "Connect apps", + "Get notified", + "Check your setup", + "Your name", + ]), + ); + for (const next of [ + "Each Bot has its own browser and files", + "Pick a Bot or make your own", + "You decide what it asks about", + ]) { + fireEvent.click(await continueButton(view)); + await heading(view, next); + } + fireEvent.click(await continueButton(view)); + await heading(view, "Connect your apps"); +}); + +test("Continue and Back start the new step's picture at the top of its pane", async () => { + const view = open("/onboarding?step=computer"); + await heading(view, "Each Bot has its own browser and files"); + const pane = () => + view.container.querySelector("[data-onboarding-demo]") as HTMLElement; + pane().scrollTop = 120; + + fireEvent.click(await continueButton(view)); + + const team = await heading(view, "Pick a Bot or make your own"); + await waitFor(() => expect(document.activeElement).toBe(team)); + expect(pane().scrollTop).toBe(0); + + pane().scrollTop = 120; + fireEvent.click(back(view)); + const computer = await heading( + view, + "Each Bot has its own browser and files", + ); + await waitFor(() => expect(document.activeElement).toBe(computer)); + expect(pane().scrollTop).toBe(0); +}); + +test("browser Back goes to the step before, not out of onboarding", async () => { + const view = open("/onboarding?step=welcome"); + await heading(view, "Welcome to OpenBot"); + fireEvent.click(await continueButton(view)); + await heading(view, "Each Bot has its own browser and files"); + + await act(async () => view.history.back()); + + expect(await heading(view, "Welcome to OpenBot")).toBeTruthy(); + expect(searchOf(view).step).toBe("welcome"); +}); + +test("Back after Continue returns through history, so Forward still works", async () => { + const view = open("/onboarding?step=welcome"); + await heading(view, "Welcome to OpenBot"); + fireEvent.click(await continueButton(view)); + await heading(view, "Each Bot has its own browser and files"); + + fireEvent.click(back(view)); + + expect(await heading(view, "Welcome to OpenBot")).toBeTruthy(); + expect(view.history.canGoBack()).toBe(false); + await act(async () => view.history.forward()); + expect( + await heading(view, "Each Bot has its own browser and files"), + ).toBeTruthy(); +}); + +test("Back on a step opened by reload replaces it with the step before", async () => { + const view = open("/onboarding?step=team"); + await heading(view, "Pick a Bot or make your own"); + + fireEvent.click(back(view)); + + expect( + await heading(view, "Each Bot has its own browser and files"), + ).toBeTruthy(); + expect(searchOf(view).step).toBe("computer"); + expect(view.history.length).toBe(1); +}); + +test("on a desktop, a step already passed can be picked from the list", async () => { + const view = open("/onboarding?step=approval"); + await heading(view, "You decide what it asks about"); + + const list = view.getByRole("navigation", { name: "Onboarding steps" }); + expect(within(list).queryByRole("button", { name: /Connect apps/ })).toBe( + null, + ); + fireEvent.click(within(list).getByRole("button", { name: /Welcome/ })); + + expect(await heading(view, "Welcome to OpenBot")).toBeTruthy(); + expect(searchOf(view).step).toBe("welcome"); + // It has seen the approval step, so the list leads back to it. + fireEvent.click( + within( + view.getByRole("navigation", { name: "Onboarding steps" }), + ).getByRole("button", { name: /You decide what it asks/ }), + ); + expect(await heading(view, "You decide what it asks about")).toBeTruthy(); +}); + +test("a reload stays on the step in the address", async () => { + const view = open("/onboarding?step=apps"); + + expect(await heading(view, "Connect your apps")).toBeTruthy(); + expect(searchOf(view).step).toBe("apps"); +}); + +test("arriving with no step opens the one the server saved", async () => { + saved = 7; + const view = open("/onboarding"); + + expect(await heading(view, "Get notified")).toBeTruthy(); + await waitFor(() => expect(searchOf(view).step).toBe("notify")); +}); + +test("a failed consent says so on its step, and only `connected` leaves the address", async () => { + const view = open("/onboarding?step=apps&connected=failed"); + + expect(await heading(view, "Connect your apps")).toBeTruthy(); + expect((await view.findByRole("alert")).textContent).toBe( + "Couldn't connect that app. Try again.", + ); + await waitFor(() => expect(searchOf(view)).toEqual({ step: "apps" })); +}); + +test("a step that isn't one is ignored", async () => { + const view = open("/onboarding?step=nonsense"); + + expect(await heading(view, "Welcome to OpenBot")).toBeTruthy(); + await waitFor(() => expect(searchOf(view).step).toBe("welcome")); +}); + +test("after a reload, Back still returns through history, not a second entry", async () => { + const first = open("/onboarding?step=welcome"); + await heading(first, "Welcome to OpenBot"); + fireEvent.click(await continueButton(first)); + await heading(first, "Each Bot has its own browser and files"); + const history = first.history; + first.unmount(); + queryClient.clear(); + + // The same history, a new page: what a reload keeps. + const reloaded = open("/onboarding?step=computer", history); + await heading(reloaded, "Each Bot has its own browser and files"); + fireEvent.click(back(reloaded)); + + expect(await heading(reloaded, "Welcome to OpenBot")).toBeTruthy(); + expect(history.length).toBe(2); + expect(history.canGoBack()).toBe(false); +}); + +test("back from a consent, browser Back goes to the step before, not to the vendor", async () => { + window.sessionStorage.setItem( + "openbot-onboarding-consent", + JSON.stringify({ + step: "apps", + entry: "the-entry-that-left", + app: { serverId: "notion", brokered: false }, + }), + ); + const view = open("/onboarding?step=apps&connected=failed"); + + expect(await heading(view, "Connect your apps")).toBeTruthy(); + await waitFor(() => expect(view.history.length).toBe(2)); + expect(searchOf(view)).toEqual({ step: "apps" }); + // The app it left for is named, once its row has arrived. + await waitFor(async () => + expect((await view.findByRole("alert")).textContent).toBe( + "Couldn't connect Notion. Try again.", + ), + ); + expect( + window.sessionStorage.getItem("openbot-onboarding-consent"), + ).toBeNull(); + + await act(async () => view.history.back()); + expect(await heading(view, "You decide what it asks about")).toBeTruthy(); +}); + +test("the failed line goes once the person moves on, and doesn't come back with Back", async () => { + const view = open("/onboarding?step=apps&connected=failed"); + await view.findByRole("alert"); + + fireEvent.click(view.getByRole("button", { name: "Not now" })); + await heading(view, "Get notified"); + fireEvent.click(back(view)); + + expect(await heading(view, "Connect your apps")).toBeTruthy(); + await settleReactWork(); + expect(view.queryByRole("alert")).toBeNull(); +}); + +test("Get started on the name step saves the name, finishes, plays the orb's finish, then opens the app", async () => { + const view = open("/onboarding?step=name"); + await heading(view, "Let's get to it."); + // The name step's own form carries the way out; the frame draws no Continue. + expect(view.queryByRole("button", { name: "Continue" })).toBeNull(); + // Under the orb: on a phone the picture comes first. + expect( + view.container.querySelector("[data-onboarding-demo]")?.className, + ).toContain("row-start-2"); + + await userEvent.setup({ document }).type(view.getByLabelText("Name"), "Maya"); + fireEvent.click(view.getByRole("button", { name: "Get started" })); + + await waitFor(() => + expect( + view.container.querySelector( + '[data-onboarding-finish] [data-state="finished"]', + ), + ).not.toBeNull(), + ); + expect(named).toBe("Maya"); + expect(completed).toBe(true); + expect(view.queryByText("Home")).toBeNull(); + expect( + view.getByRole("button", { name: "Saving…" }).hasAttribute("disabled"), + ).toBe(true); + await waitFor(() => expect(view.getByText("Home")).toBeTruthy(), { + timeout: 3000, + }); +}); + +test("a checklist link goes through the name step, says what opens, and opens it after", async () => { + const view = open("/onboarding?step=setup"); + await heading(view, "Check your setup"); + + fireEvent.click(await view.findByRole("link", { name: "Credentials" })); + + await heading(view, "Let's get to it."); + expect(searchOf(view)).toEqual({ step: "name", open: "/admin/credentials" }); + expect( + await view.findByText("Credentials opens once you get started."), + ).toBeTruthy(); + expect(completed).toBe(false); + + await userEvent.setup({ document }).type(view.getByLabelText("Name"), "Maya"); + fireEvent.click(view.getByRole("button", { name: "Get started" })); + await waitFor(() => expect(view.getByText("Credentials page")).toBeTruthy(), { + timeout: 3000, + }); + expect(completed).toBe(true); +}); + +test("a page to open that isn't one of this app's is dropped", async () => { + const view = open("/onboarding?step=name&open=https%3A%2F%2Fevil.test"); + + await heading(view, "Let's get to it."); + expect(searchOf(view).open).toBeUndefined(); +}); + +test("Skip intro reaches Connect your apps, survives reload, and Back returns to the step it skipped from", async () => { + localEvaluation = true; + const first = open("/onboarding?step=welcome"); + await heading(first, "Welcome to OpenBot"); + await waitFor(() => + expect(first.getByRole("progressbar").getAttribute("aria-valuemax")).toBe( + "9", + ), + ); + fireEvent.click(first.getByRole("button", { name: "Skip intro" })); + await heading(first, "Connect your apps"); + await waitFor(() => expect(saves).toContain(6)); + expect(completed).toBe(false); + const history = first.history; + first.unmount(); + queryClient.clear(); + + const view = open("/onboarding?step=apps", history); + await heading(view, "Connect your apps"); + fireEvent.click(back(view)); + await heading(view, "Welcome to OpenBot"); + await act(async () => history.forward()); + await heading(view, "Connect your apps"); + fireEvent.click(view.getByRole("button", { name: "Not now" })); + await heading(view, "Get notified"); + fireEvent.click(view.getByRole("button", { name: "Not now" })); + await heading(view, "Check your setup"); + expect(completed).toBe(false); +}); + +test("somebody trying OpenBot alone gets the self-hosting step, with Talk to an engineer", async () => { + localEvaluation = true; + const view = open("/onboarding?step=approval"); + await heading(view, "You decide what it asks about"); + + fireEvent.click(await continueButton(view)); + + expect(await heading(view, "It runs on your machine")).toBeTruthy(); + expect(searchOf(view).step).toBe("selfhost"); + const engineer = view.getByRole("link", { name: "Talk to an engineer" }); + expect(engineer.getAttribute("target")).toBe("_blank"); + fireEvent.click(await continueButton(view)); + expect(await heading(view, "Connect your apps")).toBeTruthy(); +}); + +test("an organization's people never see the self-hosting step", async () => { + const view = open("/onboarding?step=approval"); + await heading(view, "You decide what it asks about"); + + fireEvent.click(await continueButton(view)); + + expect(await heading(view, "Connect your apps")).toBeTruthy(); + expect(view.queryByText("Running it yourself")).toBeNull(); +}); + +test("the self-hosting step in the address waits for the answer, rather than flashing step 1", async () => { + localEvaluation = true; + const view = open("/onboarding?step=selfhost"); + + expect(await heading(view, "It runs on your machine")).toBeTruthy(); + expect( + view.queryByRole("heading", { name: "Welcome to OpenBot" }), + ).toBeNull(); +}); + +test("until the deployment answers, the intro's Continue waits; a failure says so and offers Try again", async () => { + capabilitiesFail = true; + const view = open("/onboarding?step=welcome"); + await heading(view, "Welcome to OpenBot"); + + expect( + ( + await view.findByText("Couldn't load your setup. Try again to continue.") + ).getAttribute("role"), + ).toBe("alert"); + expect( + view.getByRole("button", { name: "Continue" }).hasAttribute("disabled"), + ).toBe(true); + + capabilitiesFail = false; + const asked = capabilitiesAsked; + fireEvent.click(view.getByRole("button", { name: "Try again" })); + await waitFor(() => expect(capabilitiesAsked).toBeGreaterThan(asked)); + fireEvent.click(await continueButton(view)); + expect( + await heading(view, "Each Bot has its own browser and files"), + ).toBeTruthy(); + expect( + view.queryByText("Couldn't load your setup. Try again to continue."), + ).toBeNull(); +}); + +test("a consent after Skip intro returns Back to the skipped-from step", async () => { + window.sessionStorage.setItem( + "openbot-onboarding-consent", + JSON.stringify({ + step: "apps", + from: "welcome", + entry: "the-entry-that-left", + }), + ); + const view = open("/onboarding?step=apps&connected=failed"); + await heading(view, "Connect your apps"); + await waitFor(() => + expect(view.router.state.location.state).toMatchObject({ + onboardingFrom: "welcome", + }), + ); + fireEvent.click(back(view)); + await heading(view, "Welcome to OpenBot"); +}); diff --git a/app/tests/onboarding-steps.test.tsx b/app/tests/onboarding-steps.test.tsx new file mode 100644 index 0000000000..a77757590d --- /dev/null +++ b/app/tests/onboarding-steps.test.tsx @@ -0,0 +1,569 @@ +import { + afterAll, + afterEach, + beforeAll, + beforeEach, + expect, + test, +} from "bun:test"; +import { GlobalRegistrator } from "@happy-dom/global-registrator"; +import { QueryClientProvider } from "@tanstack/react-query"; +import { + act, + cleanup, + fireEvent, + render, + waitFor, + within, +} from "@testing-library/react"; +import type { ReactNode } from "react"; +import { + type ConsentApp, + ConnectAppsStep, +} from "@/components/onboarding/connect-apps-step"; +import { + GRANTED_BEAT_MS, + notificationPreview, + useNotifyStep, +} from "@/components/onboarding/notify-step"; +import { StepIntro } from "@/components/onboarding/onboarding-frame"; +import { OnboardingWizard } from "@/components/onboarding/onboarding-wizard"; +import { agentKeys } from "@/lib/agents/queries"; +import { authKeys } from "@/lib/auth/queries"; +import type { AppAccess } from "@/lib/app-access/queries"; +import { refreshNotificationPermission } from "@/lib/notifications/permission"; +import { queryClient } from "@/query-client"; +import { settleReactWork } from "./settle-react-work"; + +/** + * Onboarding's two optional steps. Connect your apps lists the apps the default Bot uses as this + * person, connects the ones with a sign-in page from here, and comes back to this step. Get + * notified asks the browser, shows the answer, and moves on. + */ + +beforeAll(() => GlobalRegistrator.register({ url: "http://localhost:3010" })); +afterAll(async () => { + await settleReactWork(); + GlobalRegistrator.unregister(); +}); + +const BOT = { + id: "general", + name: "General Assistant", + title: "Everyday work", + roleDescription: "", + avatarSeed: "general", + visibility: "public", + endpoint: null, + builtIn: true, + hasAuth: false, + mine: false, +}; + +/** One app as `GET /api/plugins/for/:agentId/apps` answers it, not connected and connectable. */ +const app = (over: Partial): AppAccess => ({ + key: "notion", + displayName: "Notion", + logo: null, + provider: "catalogue", + account: "personal", + connect: "oauth", + authScheme: null, + accountMode: null, + state: "not_connected", + unavailable: null, + accountHolder: "you", + asksFirst: false, + granted: true, + ownerGrants: "admin", + givenByOwner: false, + needsConnect: false, + connectedAt: null, + requestKind: null, + request: null, + ...over, +}); + +const originalFetch = globalThis.fetch; +let apps: AppAccess[] | "error"; +let connectAnswer: Record; +let confirmed: boolean; +let requests: { path: string; method: string }[]; + +let permission: NotificationPermission; +let prompted: NotificationPermission; +let asked: number; +const originalNotification = (globalThis as { Notification?: unknown }) + .Notification; + +/** The app's one client, without its retries, so a failed read is an answer at once. */ +const defaults = queryClient.getDefaultOptions(); + +beforeEach(() => { + queryClient.clear(); + queryClient.setDefaultOptions({ + ...defaults, + queries: { ...defaults.queries, retry: false }, + }); + requests = []; + apps = [app({})]; + connectAnswer = { + authorizationUrl: "https://accounts.example/consent?state=s", + }; + confirmed = true; + permission = "default"; + prompted = "granted"; + asked = 0; + (globalThis as { Notification?: unknown }).Notification = { + get permission() { + return permission; + }, + requestPermission: async () => { + asked += 1; + permission = prompted; + return prompted; + }, + }; + refreshNotificationPermission(); + globalThis.fetch = Object.assign( + async (input: RequestInfo | URL, init?: RequestInit) => { + const path = String(input); + const method = init?.method ?? "GET"; + requests.push({ path, method }); + if (path === "/api/plugins/for/general/apps") { + if (apps === "error") + return Response.json({ error: "nope" }, { status: 503 }); + return Response.json({ apps }); + } + if (path.endsWith("/connection/confirm")) + return Response.json({ connected: confirmed }); + if (path.includes("/connect?returnTo=")) + return Response.json(connectAnswer); + if (path.startsWith("/api/agents")) + return Response.json({ agents: [BOT] }); + return Response.json({}); + }, + { preconnect: originalFetch.preconnect }, + ); +}); +afterEach(() => { + cleanup(); + queryClient.clear(); + queryClient.setDefaultOptions(defaults); + globalThis.fetch = originalFetch; + (globalThis as { Notification?: unknown }).Notification = + originalNotification; +}); + +function withQueries(children: ReactNode) { + queryClient.setQueryData(agentKeys.list(false), [BOT]); + return ( + {children} + ); +} + +function drawConnect( + props: { + failed?: boolean; + returned?: ConsentApp | null; + role?: "admin" | "member"; + } = {}, +) { + const left: Array<[string, ConsentApp]> = []; + const opened: Array<{ label: string; path: string }> = []; + if (props.role) + queryClient.setQueryData(authKeys.currentUser(), { + id: "u1", + email: "jo@example.com", + name: "Jo", + role: props.role, + }); + const view = render( + withQueries( + left.push([url, left_])} + openPage={(target) => opened.push(target)} + returned={props.returned ?? null} + />, + ), + ); + return Object.assign(view, { left, opened }); +} + +/** One app's row, once the list has arrived. */ +const rowOf = async (view: ReturnType, key: string) => + within( + await waitFor(() => { + const row = view.container.querySelector( + `[data-app-row="${key}"]`, + ); + if (!row) throw new Error(`no ${key} row`); + return row; + }), + ); + +test("Connect your apps lists the apps the default Bot uses as you, and no team accounts", async () => { + apps = [ + app({}), + app({ key: "linear", displayName: "Linear", provider: "composio" }), + app({ key: "team-drive", displayName: "Team Drive", account: "shared" }), + ]; + const view = drawConnect(); + + expect(await view.findByRole("list", { name: "Apps" })).toBeTruthy(); + const rows = Array.from( + view.container.querySelectorAll("[data-app-row]"), + ).map((row) => row.getAttribute("data-app-row")); + expect(rows).toEqual(["notion", "linear"]); + expect(requests).toContainEqual({ + path: "/api/plugins/for/general/apps", + method: "GET", + }); +}); + +test("Connect starts the real connection, coming back to this step", async () => { + const view = drawConnect(); + + fireEvent.click(await view.findByRole("button", { name: "Connect Notion" })); + + await waitFor(() => + expect(view.left).toEqual([ + [ + "https://accounts.example/consent?state=s", + { serverId: "notion", brokered: false }, + ], + ]), + ); + expect(requests).toContainEqual({ + path: "/api/plugins/servers/notion/connect?returnTo=onboarding", + method: "POST", + }); +}); + +test("a brokered app leaves saying so, so its return is asked of the vendor", async () => { + apps = [ + app({ + key: "composio-linear", + displayName: "Linear", + provider: "composio", + connect: "consent", + }), + ]; + const view = drawConnect(); + + fireEvent.click(await view.findByRole("button", { name: "Connect Linear" })); + + await waitFor(() => + expect(view.left[0]?.[1]).toEqual({ + serverId: "composio-linear", + brokered: true, + }), + ); +}); + +test("connected reads Connected, with nothing to press", async () => { + apps = [app({ state: "ready" })]; + const view = drawConnect(); + + expect( + await (await rowOf(view, "notion")).findByText("Connected"), + ).toBeTruthy(); + expect((await rowOf(view, "notion")).queryByRole("button")).toBeNull(); +}); + +test("a refused account offers Reconnect", async () => { + apps = [app({ state: "reconnect" })]; + const view = drawConnect(); + + expect( + await view.findByRole("button", { name: "Reconnect Notion" }), + ).toBeTruthy(); + expect( + (await rowOf(view, "notion")).getByText("Needs reconnecting."), + ).toBeTruthy(); +}); + +test("an app whose key somebody types is connected later, in Connections", async () => { + apps = [app({ key: "stripe", displayName: "Stripe", connect: "key" })]; + const view = drawConnect(); + + expect( + await (await rowOf(view, "stripe")).findByText( + "Connect it later in Connections.", + ), + ).toBeTruthy(); + expect((await rowOf(view, "stripe")).queryByRole("button")).toBeNull(); +}); + +test("an app the Bot was never given says an admin can give it, with nothing to press", async () => { + apps = [app({ state: "not_granted" })]; + const view = drawConnect(); + + expect( + await (await rowOf(view, "notion")).findByText( + "General Assistant can't use it yet. An admin can give it.", + ), + ).toBeTruthy(); + expect((await rowOf(view, "notion")).queryByRole("button")).toBeNull(); +}); + +test("an account the vendor already held is recorded on the press, with nowhere to go", async () => { + apps = [app({ provider: "composio", connect: "consent" })]; + connectAnswer = { connected: true }; + const view = drawConnect(); + + fireEvent.click(await view.findByRole("button", { name: "Connect Notion" })); + apps = [app({ provider: "composio", connect: "consent", state: "ready" })]; + + expect( + await (await rowOf(view, "notion")).findByText("Connected"), + ).toBeTruthy(); + expect(view.left).toEqual([]); +}); + +test("back from a brokered consent, the vendor is asked once, and the row reads its answer", async () => { + apps = [app({ provider: "composio", connect: "consent", state: "ready" })]; + const view = drawConnect({ + returned: { serverId: "notion", brokered: true }, + }); + + expect( + await (await rowOf(view, "notion")).findByText("Connected"), + ).toBeTruthy(); + await settleReactWork(); + expect( + requests.filter((request) => request.path.endsWith("/connection/confirm")), + ).toEqual([ + { path: "/api/plugins/servers/notion/connection/confirm", method: "POST" }, + ]); + expect(view.queryByRole("alert")).toBeNull(); +}); + +test("a brokered consent the vendor says did not finish is said on the step", async () => { + apps = [app({ provider: "composio", connect: "consent" })]; + confirmed = false; + const view = drawConnect({ + returned: { serverId: "notion", brokered: true }, + }); + + expect((await view.findByRole("alert")).textContent).toBe( + "Couldn't connect Notion. Try again.", + ); +}); + +test("an OAuth return is not asked of a vendor: the callback already recorded it", async () => { + apps = [app({ state: "ready" })]; + const view = drawConnect({ + returned: { serverId: "notion", brokered: false }, + }); + + await (await rowOf(view, "notion")).findByText("Connected"); + await settleReactWork(); + expect( + requests.some((request) => request.path.endsWith("/connection/confirm")), + ).toBe(false); +}); + +test("a consent that came back failed is said on the step", async () => { + const view = drawConnect({ + failed: true, + returned: { serverId: "notion", brokered: false }, + }); + + await view.findByRole("list", { name: "Apps" }); + expect((await view.findByRole("alert")).textContent).toBe( + "Couldn't connect Notion. Try again.", + ); +}); + +test("a list that could not be read says so, and never claims Connected", async () => { + apps = "error"; + const view = drawConnect(); + + expect( + await view.findByText( + "Couldn't load your apps. Continue, and connect them later in Connections.", + ), + ).toBeTruthy(); + expect(view.queryByText("Connected")).toBeNull(); +}); + +test("no apps to connect says how they arrive", async () => { + apps = []; + const view = drawConnect({ role: "member" }); + + expect(await view.findByText("No apps to connect yet")).toBeTruthy(); + expect( + view.getByText( + "When an admin adds apps, you can connect them in Connections.", + ), + ).toBeTruthy(); + expect(view.queryByRole("button", { name: "Add apps" })).toBeNull(); +}); + +test("an admin with no apps to connect is offered the page where apps are added", async () => { + apps = []; + const view = drawConnect({ role: "admin" }); + + fireEvent.click(await view.findByRole("button", { name: "Add apps" })); + expect(view.opened).toEqual([{ label: "Plugins", path: "/admin/plugins" }]); +}); + +/** Get notified between two plain steps, in the real frame. */ +function NotifyOnboarding() { + const notify = useNotifyStep("General Assistant"); + return ( + , + }, + notify, + { + id: "setup", + title: "Check your setup", + intro: () => , + }, + ]} + /> + ); +} + +test("Get notified previews the notification the app really sends", () => { + const view = render(); + + expect(view.getByRole("heading", { name: "Get notified" })).toBeTruthy(); + const preview = notificationPreview("General Assistant"); + expect(preview).toEqual({ + title: "General Assistant needs you", + body: "1 question, 1 approval", + }); + expect( + view.getByRole("img", { + name: `A notification from OpenBot: ${preview.title}. ${preview.body}.`, + }), + ).toBeTruthy(); + expect(view.getByText(preview.title)).toBeTruthy(); + expect( + view.getByText( + "Notifications show while OpenBot is open in a browser tab.", + ), + ).toBeTruthy(); +}); + +test("Turn on notifications asks the browser, reads Notifications on, then moves on", async () => { + const view = render(); + + fireEvent.click(view.getByRole("button", { name: "Turn on notifications" })); + + const on = await view.findByRole("button", { name: "Notifications on" }); + expect(on.hasAttribute("data-done")).toBe(true); + expect(asked).toBe(1); + expect(view.queryByRole("button", { name: "Not now" })).toBeNull(); + await act(async () => { + await new Promise((resolve) => setTimeout(resolve, GRANTED_BEAT_MS + 50)); + }); + expect( + await view.findByRole("heading", { name: "Check your setup" }), + ).toBeTruthy(); +}); + +test("while the browser's prompt is up, the button waits, and can't be pressed twice", async () => { + let answer: (value: NotificationPermission) => void = () => {}; + (globalThis as { Notification?: unknown }).Notification = { + get permission() { + return permission; + }, + requestPermission: () => + new Promise((resolve) => { + asked += 1; + answer = (value) => { + permission = value; + resolve(value); + }; + }), + }; + refreshNotificationPermission(); + const view = render(); + + fireEvent.click(view.getByRole("button", { name: "Turn on notifications" })); + const waiting = await waitFor(() => { + const button = view.container.querySelector( + "[data-onboarding-primary][data-pending]", + ); + if (!button) throw new Error("not waiting yet"); + return button; + }); + expect(waiting.disabled).toBe(true); + fireEvent.click(waiting); + expect(asked).toBe(1); + + await act(async () => answer("granted")); + expect( + await view.findByRole("button", { name: "Notifications on" }), + ).toBeTruthy(); +}); + +test("a prompt closed without an answer says where to turn them on, and offers Continue", async () => { + prompted = "default"; + const view = render(); + + fireEvent.click(view.getByRole("button", { name: "Turn on notifications" })); + + const why = await view.findByText( + "You can turn them on later from a Bot's page.", + ); + // Said just above the button, in the frame's footer, where a small screen can't scroll it away. + expect(why.closest("footer")).not.toBeNull(); + expect(view.queryByRole("button", { name: "Not now" })).toBeNull(); + // Not now's place is kept, so Continue stays where Turn on notifications was. + expect( + view.container.querySelector("footer [data-onboarding-secondary-slot]"), + ).not.toBeNull(); + fireEvent.click(view.getByRole("button", { name: "Continue" })); + expect( + await view.findByRole("heading", { name: "Check your setup" }), + ).toBeTruthy(); +}); + +test("Not now moves on and asks the browser nothing", async () => { + const view = render(); + + fireEvent.click(view.getByRole("button", { name: "Not now" })); + + expect( + await view.findByRole("heading", { name: "Check your setup" }), + ).toBeTruthy(); + expect(asked).toBe(0); +}); + +test("a no from the browser says where to allow them, and the button becomes Continue", async () => { + prompted = "denied"; + const view = render(); + + fireEvent.click(view.getByRole("button", { name: "Turn on notifications" })); + + expect( + await view.findByText( + "Notifications are blocked for this site. Allow them in your browser's site settings to turn them on.", + ), + ).toBeTruthy(); + fireEvent.click(view.getByRole("button", { name: "Continue" })); + expect( + await view.findByRole("heading", { name: "Check your setup" }), + ).toBeTruthy(); +}); + +test("a browser with no notifications says so, and offers Continue", () => { + (globalThis as { Notification?: unknown }).Notification = undefined; + refreshNotificationPermission(); + const view = render(); + + expect(view.getByText("This browser can't show notifications.")).toBeTruthy(); + expect(view.getByRole("button", { name: "Continue" })).toBeTruthy(); + expect(view.queryByRole("button", { name: "Not now" })).toBeNull(); +}); diff --git a/app/tests/profile.test.tsx b/app/tests/profile.test.tsx index 421a25c289..1e36399a9b 100644 --- a/app/tests/profile.test.tsx +++ b/app/tests/profile.test.tsx @@ -4,6 +4,7 @@ import { QueryClient, QueryClientProvider } from "@tanstack/react-query"; import { cleanup, fireEvent, render, waitFor } from "@testing-library/react"; import userEvent from "@testing-library/user-event"; import { + OnboardingNameOrb, OnboardingNameStep, ProfileNameForm, } from "@/components/settings/profile"; @@ -47,6 +48,19 @@ test("onboarding name step starts empty for a placeholder profile", () => { ).toBe(true); }); +test("the name step's orb is OpenBot's violet, listening, and plays its finish when onboarding ends", () => { + const view = render(); + const orb = () => + view.container.querySelector("[data-palette]") as HTMLElement; + expect(orb().getAttribute("data-palette")).toBe("violet"); + expect(orb().getAttribute("data-state")).toBe("listening"); + expect(view.container.querySelector("[data-onboarding-finish]")).toBeNull(); + + view.rerender(); + expect(orb().getAttribute("data-state")).toBe("finished"); + expect(orb().closest("[data-onboarding-finish]")).not.toBeNull(); +}); + test("onboarding name step preserves a saved name when revisited", () => { const view = render( diff --git a/app/tests/settings-appearance.test.tsx b/app/tests/settings-appearance.test.tsx new file mode 100644 index 0000000000..e822beed02 --- /dev/null +++ b/app/tests/settings-appearance.test.tsx @@ -0,0 +1,102 @@ +import { + afterAll, + afterEach, + beforeAll, + beforeEach, + expect, + test, +} from "bun:test"; +import { GlobalRegistrator } from "@happy-dom/global-registrator"; +import type { UserPreferences } from "@openbot/shared/user-preferences"; +import { QueryClient, QueryClientProvider } from "@tanstack/react-query"; +import { + createMemoryHistory, + createRootRoute, + createRouter, + RouterProvider, +} from "@tanstack/react-router"; +import { cleanup, render } from "@testing-library/react"; +import userEvent from "@testing-library/user-event"; +import { ThemeProvider } from "@/components/theme-provider"; +import { authKeys } from "@/lib/auth/queries"; +import { userPreferencesQueryOptions } from "@/lib/settings/message-list"; +import { settingsKeys } from "@/lib/settings/queries"; +import { THEME_STORAGE_KEY } from "@/lib/theme"; +import { Route as SettingsRoute } from "@/routes/_authed/settings/index"; + +beforeAll(() => GlobalRegistrator.register()); +beforeEach(() => { + window.localStorage.clear(); + document.documentElement.classList.remove("dark"); +}); +afterEach(() => cleanup()); +afterAll(() => GlobalRegistrator.unregister()); + +async function renderSettings() { + const queryClient = new QueryClient({ + defaultOptions: { + queries: { retry: false, staleTime: Number.POSITIVE_INFINITY }, + }, + }); + queryClient.setQueryData(authKeys.currentUser(), { + id: "alice", + email: "alice@example.com", + name: "Alice", + role: "user", + onboarding: null, + }); + const preferences: UserPreferences = { + messageListEmphasis: "thread", + selfHostBannerDismissed: true, + narration: null, + skin: "openbot", + }; + queryClient.setQueryData( + userPreferencesQueryOptions("alice").queryKey, + preferences, + ); + queryClient.setQueryData(settingsKeys.instructions(), ""); + const rootRoute = createRootRoute({ + component: SettingsRoute.options.component, + }); + const router = createRouter({ + routeTree: rootRoute, + history: createMemoryHistory({ initialEntries: ["/"] }), + }); + const view = render( + + + + + , + ); + await view.findByText("Appearance"); + return view; +} + +test("Appearance offers System, Light and Dark where the dark switch was", async () => { + const view = await renderSettings(); + expect(view.queryByRole("switch", { name: "Dark theme" })).toBeNull(); + const appearance = view.getByRole("combobox", { name: "Appearance" }); + expect(appearance.textContent).toContain("System"); + + const user = userEvent.setup({ document }); + await user.click(appearance); + const options = (await view.findAllByRole("option")).map( + (option) => option.textContent, + ); + expect(options).toEqual(["System", "Light", "Dark"]); +}); + +test("choosing Dark draws it and remembers it", async () => { + const view = await renderSettings(); + const user = userEvent.setup({ document }); + await user.click(view.getByRole("combobox", { name: "Appearance" })); + await user.click(await view.findByRole("option", { name: "Dark" })); + + expect(document.documentElement.classList.contains("dark")).toBe(true); + expect(window.localStorage.getItem(THEME_STORAGE_KEY)).toBe("dark"); + expect( + view.getByRole("combobox", { name: "Appearance" }).textContent, + ).toContain("Dark"); +}); diff --git a/app/tests/setup-checklist.test.tsx b/app/tests/setup-checklist.test.tsx new file mode 100644 index 0000000000..af9b1e0727 --- /dev/null +++ b/app/tests/setup-checklist.test.tsx @@ -0,0 +1,342 @@ +import { + afterAll, + afterEach, + beforeAll, + beforeEach, + expect, + test, +} from "bun:test"; +import { GlobalRegistrator } from "@happy-dom/global-registrator"; +import { QueryClientProvider } from "@tanstack/react-query"; +import { + cleanup, + fireEvent, + render, + waitFor, + within, +} from "@testing-library/react"; +import { StepIntro } from "@/components/onboarding/onboarding-frame"; +import { OnboardingWizard } from "@/components/onboarding/onboarding-wizard"; +import { SetupChecklist } from "@/components/onboarding/setup-checklist"; +import type { + SetupChecklist as Checklist, + SetupRow, +} from "@/lib/setup/queries"; +import { queryClient } from "@/query-client"; +import { settleReactWork } from "./settle-react-work"; + +/** + * The setup checklist onboarding's "Check your setup" step draws. It draws what the server decided, + * row by row, and lets somebody check again and open the page that closes a gap. + */ + +beforeAll(() => GlobalRegistrator.register({ url: "http://localhost:3010" })); +afterAll(async () => { + await settleReactWork(); + GlobalRegistrator.unregister(); +}); + +const originalFetch = globalThis.fetch; +let answer: () => Response; +/** What `POST /api/setup/check` does before the read it causes; nothing by default. */ +let onCheck: (() => void) | null; +let requests: { path: string; method: string }[]; +const defaults = queryClient.getDefaultOptions(); + +beforeEach(() => { + queryClient.clear(); + queryClient.setDefaultOptions({ + ...defaults, + queries: { ...defaults.queries, retry: false }, + }); + requests = []; + onCheck = null; + globalThis.fetch = Object.assign( + async (input: RequestInfo | URL, init?: RequestInit) => { + const path = String(input); + const method = init?.method ?? "GET"; + requests.push({ path, method }); + if (path === "/api/setup/check") onCheck?.(); + return answer(); + }, + { preconnect: originalFetch.preconnect }, + ); +}); +afterEach(() => { + cleanup(); + queryClient.clear(); + queryClient.setDefaultOptions(defaults); + globalThis.fetch = originalFetch; +}); + +const ROW = (row: Partial & Pick): SetupRow => ({ + scope: "deployment", + title: row.id, + required: false, + status: row.ready === false ? "needs_setup" : "ready", + ready: true, + detail: "Works.", + fix: null, + ...row, +}); + +/** What an admin is sent: Intelligence ready, the model key missing, no computer, Slack unset. */ +const ADMIN: Checklist = { + audience: "admin", + rows: [ + ROW({ + id: "intelligence", + title: "CopilotKit Intelligence", + required: true, + detail: "Keeps your conversations and what OpenBot remembers.", + }), + ROW({ + id: "model", + title: "AI model", + required: true, + ready: false, + detail: "Without a model key, OpenBot can't answer.", + fix: { + kind: "env", + names: ["OPENAI_API_KEY"], + docsUrl: "https://github.com/CopilotKit/openbot#quick-start", + page: { label: "Credentials", path: "/admin/credentials" }, + }, + }), + ROW({ + id: "computer", + title: "Bot computer", + ready: false, + detail: "Without it, Bots can't browse the web or work with files.", + fix: { + kind: "env", + names: ["AGENT_COMPUTER_URL", "COMPUTER_TOKEN"], + docsUrl: + "https://github.com/CopilotKit/openbot/blob/main/docs/configuration.md#computer-and-supervisor", + }, + }), + ROW({ id: "apps", title: "More apps" }), + ROW({ + id: "chat", + title: "Slack and Teams", + ready: false, + detail: "Without it, people can talk to OpenBot only here.", + fix: { + kind: "env", + names: ["OPENTAG_SHARED_SECRET", "OPENTAG_URL"], + docsUrl: + "https://github.com/CopilotKit/openbot/blob/main/docs/configuration.md", + }, + }), + ], +}; + +/** What a member is sent for the same deployment: no fix, and no names. */ +const MEMBER: Checklist = { + audience: "member", + rows: ADMIN.rows.map((row) => + row.ready ? row : { ...row, detail: "Your admin sets this up.", fix: null }, + ), +}; + +function serving(checklist: Checklist) { + answer = () => Response.json({ checklist }); +} + +function draw() { + const opened: Array<{ label: string; path: string }> = []; + const view = render( + + opened.push(target)} /> + , + ); + return Object.assign(view, { opened }); +} + +const rowOf = (view: ReturnType, id: string) => { + const row = view.container.querySelector( + `[data-setup-row="${id}"]`, + ); + if (!row) throw new Error(`no ${id} row`); + return within(row); +}; + +test("Ready and Needs setup rows come from the server's answer", async () => { + serving(ADMIN); + const view = draw(); + + await view.findByRole("list", { name: "Setup" }); + expect( + Array.from(view.container.querySelectorAll("[data-setup-row]")).map((row) => + row.getAttribute("data-setup-row"), + ), + ).toEqual(["intelligence", "model", "computer", "apps", "chat"]); + expect(rowOf(view, "intelligence").getByText("Ready")).not.toBeNull(); + expect(rowOf(view, "model").getByText("Needs setup")).not.toBeNull(); + expect(rowOf(view, "model").getByText("Required")).not.toBeNull(); + expect( + rowOf(view, "model").getByText( + "Without a model key, OpenBot can't answer.", + ), + ).not.toBeNull(); + expect( + view.getByText("3 things need setup, and 1 is required."), + ).not.toBeNull(); +}); + +test("a required gap is red, an optional one amber, and Ready green", async () => { + serving(ADMIN); + const view = draw(); + await view.findByRole("list", { name: "Setup" }); + + expect(rowOf(view, "model").getByText("Needs setup").className).toContain( + "text-status-problem", + ); + expect(rowOf(view, "chat").getByText("Needs setup").className).toContain( + "text-status-needs", + ); + expect(rowOf(view, "apps").getByText("Ready").className).toContain( + "text-status-done", + ); +}); + +test("an admin is shown the variable to set, the page, and the guide", async () => { + serving(ADMIN); + const view = draw(); + await view.findByRole("list", { name: "Setup" }); + + const model = rowOf(view, "model"); + expect(model.getByText("OPENAI_API_KEY")).not.toBeNull(); + const guide = model.getByRole("link", { name: "Setup guide" }); + expect(guide.getAttribute("href")).toBe( + "https://github.com/CopilotKit/openbot#quick-start", + ); + expect(guide.getAttribute("target")).toBe("_blank"); + + fireEvent.click(model.getByRole("link", { name: "Credentials" })); + expect(view.opened).toEqual([ + { label: "Credentials", path: "/admin/credentials" }, + ]); + + expect(rowOf(view, "chat").getByText("OPENTAG_SHARED_SECRET")).not.toBeNull(); +}); + +test("a member sees their admin sets it up, and no variable names", async () => { + serving(MEMBER); + const view = draw(); + await view.findByRole("list", { name: "Setup" }); + + const model = rowOf(view, "model"); + expect(model.getByText("Your admin sets this up.")).not.toBeNull(); + expect(model.getByText("Needs setup")).not.toBeNull(); + expect(view.container.textContent).not.toContain("OPENAI_API_KEY"); + expect(view.container.textContent).not.toContain("OPENTAG"); + expect(view.queryByRole("link", { name: "Setup guide" })).toBeNull(); + expect(view.queryByRole("link")).toBeNull(); +}); + +test("Check again asks the server to check again, and draws its answer", async () => { + serving(ADMIN); + const view = draw(); + await view.findByRole("list", { name: "Setup" }); + + onCheck = () => { + serving({ + audience: "admin", + rows: ADMIN.rows.map((row) => ({ + ...row, + status: "ready", + ready: true, + fix: null, + })), + }); + }; + fireEvent.click(view.getByRole("button", { name: "Check again" })); + + expect(await view.findByText("Everything is set up.")).not.toBeNull(); + expect(requests).toContainEqual({ path: "/api/setup/check", method: "POST" }); +}); + +test("a key nobody has checked reads Not checked, never Ready", async () => { + serving({ + audience: "admin", + rows: ADMIN.rows.map((row) => + row.id === "model" + ? { + ...row, + status: "not_checked", + ready: false, + detail: "Key set, not checked yet. Check again to ask OpenAI.", + fix: null, + } + : { ...row, status: "ready", ready: true, fix: null }, + ), + }); + const view = draw(); + await view.findByRole("list", { name: "Setup" }); + + const model = rowOf(view, "model"); + expect(model.getByText("Not checked")).not.toBeNull(); + expect(model.queryByText("Ready")).toBeNull(); + expect(view.getByText("1 thing isn't checked yet.")).not.toBeNull(); +}); + +test("optional gaps alone say OpenBot works without them", async () => { + serving({ + audience: "admin", + rows: ADMIN.rows.map((row) => + row.required ? { ...row, status: "ready", ready: true, fix: null } : row, + ), + }); + const view = draw(); + + expect( + await view.findByText("2 things need setup. OpenBot works without them."), + ).not.toBeNull(); +}); + +test("a check that fails says so, and Check again stays", async () => { + answer = () => Response.json({ error: "nope" }, { status: 503 }); + const view = draw(); + + expect( + await view.findByText( + "Couldn't check your setup. Check again, or carry on and look later.", + ), + ).not.toBeNull(); + expect(view.getByRole("button", { name: "Check again" })).not.toBeNull(); +}); + +test("moving on works with a required gap still open", async () => { + serving(ADMIN); + const view = render( + + , + demo: () => {}} />, + }, + { + id: "name", + title: "Your name", + intro: () => , + }, + ]} + /> + , + ); + + await view.findByText("3 things need setup, and 1 is required."); + const next = view.getByRole("button", { name: "Continue" }); + expect((next as HTMLButtonElement).disabled).toBe(false); + fireEvent.click(next); + await waitFor(() => + expect( + view.getByRole("heading", { name: "Let's get to it." }), + ).toBeTruthy(), + ); +}); diff --git a/app/tests/sidebar-row-actions.test.tsx b/app/tests/sidebar-row-actions.test.tsx new file mode 100644 index 0000000000..91bf7fd534 --- /dev/null +++ b/app/tests/sidebar-row-actions.test.tsx @@ -0,0 +1,253 @@ +import { + afterAll, + afterEach, + beforeAll, + beforeEach, + expect, + test, +} from "bun:test"; +import { GlobalRegistrator } from "@happy-dom/global-registrator"; +import { QueryClient, QueryClientProvider } from "@tanstack/react-query"; +import { + createMemoryHistory, + createRootRoute, + createRouter, + RouterProvider, +} from "@tanstack/react-router"; +import { cleanup, render, waitFor, within } from "@testing-library/react"; +import userEvent from "@testing-library/user-event"; +import { AppSidebar } from "@/components/app-sidebar/app-sidebar"; +import { SidebarProvider } from "@/components/ui/sidebar"; +import { agentKeys } from "@/lib/agents/queries"; +import { authKeys } from "@/lib/auth/queries"; +import { botLifecycleKeys } from "@/lib/bot-lifecycle/queries"; +import { type ChannelSummary, channelKeys } from "@/lib/channels/queries"; +import { userPreferencesQueryOptions } from "@/lib/settings/message-list"; +import { settleReactWork } from "./settle-react-work"; + +/* + * Pin and Delete on a sidebar row. They used to be on right-click alone, which nobody finds without + * being told and no keyboard reaches. Each row now has a "…" button beside it that opens the same + * two: it shows when the pointer is over the row or focus is in it, and it comes straight after the + * row in tab order. + */ + +class OfflineWebSocket extends EventTarget { + static readonly CONNECTING = 0; + static readonly OPEN = 1; + static readonly CLOSING = 2; + static readonly CLOSED = 3; + readyState = 0; + onclose = null; + onerror = null; + onmessage = null; + onopen = null; + close() {} + send() {} +} + +const originalFetch = globalThis.fetch; +const originalWebSocket = globalThis.WebSocket; +const clients: QueryClient[] = []; + +beforeAll(() => { + GlobalRegistrator.register({ url: "http://localhost/" }); + globalThis.WebSocket = OfflineWebSocket as unknown as typeof WebSocket; +}); +afterEach(() => { + cleanup(); + for (const client of clients) client.clear(); + clients.length = 0; + globalThis.fetch = originalFetch; +}); +afterAll(async () => { + await settleReactWork(); + globalThis.WebSocket = originalWebSocket; + GlobalRegistrator.unregister(); +}); + +function conversation( + id: string, + summary: string, + pinned = false, +): ChannelSummary { + return { + id, + name: "Expenses", + agentIds: ["expenses"], + threadId: `thread-${id}`, + active: true, + lastMessageAt: null, + summary, + lastMessage: null, + lastMessageAgentId: null, + createdAt: "2026-10-01T00:00:00.000Z", + pinned, + lastReadAt: null, + }; +} + +const requests: string[] = []; +let served: ChannelSummary[] = []; +beforeEach(() => { + requests.length = 0; + globalThis.fetch = Object.assign( + async (input: RequestInfo | URL, init?: RequestInit) => { + const method = init?.method ?? "GET"; + const url = String(input); + requests.push(`${method} ${url}`); + if (method !== "GET") return new Response(null, { status: 204 }); + if (url === "/api/channels") + return Response.json({ channels: served, nextCursor: null }); + return Response.json({ error: "not served" }, { status: 404 }); + }, + { preconnect: originalFetch.preconnect }, + ); +}); + +function renderSidebar(channels: ChannelSummary[]) { + served = channels; + const queryClient = new QueryClient({ + defaultOptions: { queries: { retry: false, staleTime: Infinity } }, + }); + clients.push(queryClient); + queryClient.setQueryData(authKeys.currentUser(), { + id: "maya", + name: "Maya Chen", + email: "maya@example.com", + role: "user", + onboarding: null, + }); + queryClient.setQueryData(userPreferencesQueryOptions("maya").queryKey, { + messageListEmphasis: "thread", + selfHostBannerDismissed: true, + narration: null, + skin: "openbot", + }); + queryClient.setQueryData(agentKeys.list(false), []); + queryClient.setQueryDefaults(botLifecycleKeys.attention, { + staleTime: Number.POSITIVE_INFINITY, + refetchInterval: false, + }); + queryClient.setQueryData(botLifecycleKeys.attention, []); + queryClient.setQueryData(channelKeys.list(), { + pages: [{ channels, nextCursor: null }], + pageParams: [""], + }); + const router = createRouter({ + routeTree: createRootRoute({ + component: () => ( + + + + ), + }), + history: createMemoryHistory({ initialEntries: ["/"] }), + }); + const view = render( + + + , + ); + return { view, user: userEvent.setup({ document }) }; +} + +/** + * What has focus, by its label, its address or its words. A string because a failed `toBe` between + * two elements prints both of them whole, which takes bun seconds on every retry inside `waitFor`. + */ +function focused() { + const element = document.activeElement; + return ( + element?.getAttribute("aria-label") ?? + element?.getAttribute("href") ?? + element?.textContent ?? + null + ); +} + +/** A conversation's row in the roster: the link to it, and the "…" button that belongs to it. */ +async function row(view: ReturnType, summary: string) { + const link = await view.findByRole("link", { name: new RegExp(summary) }); + const item = link.closest('[data-sidebar="menu-item"]'); + if (!item) throw new Error(`The row for "${summary}" is not a menu item`); + return { + link, + more: within(item).getByRole("button", { name: `Options for ${summary}` }), + }; +} + +test("each row's … button opens Pin and Delete, and pinning asks for it", async () => { + const { view, user } = renderSidebar([ + conversation("c1", "Trip to Lisbon"), + conversation("c2", "Quarterly receipts"), + ]); + const { more } = await row(view, "Trip to Lisbon"); + + await user.click(more); + const menu = await view.findByRole("menu"); + expect( + within(menu) + .getAllByRole("menuitem") + .map((item) => item.textContent), + ).toEqual(["Pin conversation", "Delete conversation…"]); + + await user.click(within(menu).getByRole("menuitem", { name: /^Pin/ })); + await waitFor(() => expect(requests).toContain("PUT /api/channels/c1/pin")); + // Only the row it was opened on. + expect(requests).not.toContain("PUT /api/channels/c2/pin"); +}); + +test("a pinned conversation's … menu offers Unpin", async () => { + const { view, user } = renderSidebar([ + conversation("c1", "Trip to Lisbon", true), + ]); + const { more } = await row(view, "Trip to Lisbon"); + + await user.click(more); + + expect( + await view.findByRole("menuitem", { name: /Unpin conversation/ }), + ).toBeTruthy(); +}); + +test("the … button comes right after its row in tab order, and the keyboard can pin from it", async () => { + const { view, user } = renderSidebar([ + conversation("c1", "Trip to Lisbon"), + conversation("c2", "Quarterly receipts"), + ]); + const first = await row(view, "Trip to Lisbon"); + const second = await row(view, "Quarterly receipts"); + + first.link.focus(); + expect(focused()).toBe("/channel/c1"); + await user.tab(); + expect(focused()).toBe(first.more.getAttribute("aria-label")); + await user.tab(); + expect(focused()).toBe(second.link.getAttribute("href")); + + await user.tab({ shift: true }); + await user.keyboard("{Enter}"); + await view.findByRole("menuitem", { name: /^Pin/ }); + await waitFor(() => expect(focused()).toBe("Pin conversation")); + await user.keyboard("{Enter}"); + + await waitFor(() => expect(requests).toContain("PUT /api/channels/c1/pin")); +}); + +test("Delete from the … menu is confirmed before anything is deleted", async () => { + const { view, user } = renderSidebar([conversation("c1", "Trip to Lisbon")]); + const { more } = await row(view, "Trip to Lisbon"); + + await user.click(more); + await user.click( + await view.findByRole("menuitem", { name: /Delete conversation/ }), + ); + + const dialog = await view.findByRole("dialog"); + expect(dialog.textContent).toContain("Expenses"); + expect(requests.some((request) => request.startsWith("DELETE"))).toBe(false); + + await user.click(within(dialog).getByRole("button", { name: "Delete" })); + await waitFor(() => expect(requests).toContain("DELETE /api/channels/c1")); +}); diff --git a/app/tests/sidebar-search-hotkey.test.tsx b/app/tests/sidebar-search-hotkey.test.tsx new file mode 100644 index 0000000000..194793938b --- /dev/null +++ b/app/tests/sidebar-search-hotkey.test.tsx @@ -0,0 +1,236 @@ +import { + afterAll, + afterEach, + beforeAll, + beforeEach, + expect, + test, +} from "bun:test"; +import { GlobalRegistrator } from "@happy-dom/global-registrator"; +import { QueryClient, QueryClientProvider } from "@tanstack/react-query"; +import { + createMemoryHistory, + createRootRoute, + createRouter, + RouterProvider, +} from "@tanstack/react-router"; +import { act, cleanup, render, waitFor } from "@testing-library/react"; +import { ThemeProvider } from "@/components/theme-provider"; +import { AppSidebar } from "@/components/app-sidebar/app-sidebar"; +import { SidebarProvider, useSidebar } from "@/components/ui/sidebar"; +import { agentKeys } from "@/lib/agents/queries"; +import { authKeys } from "@/lib/auth/queries"; +import { botLifecycleKeys } from "@/lib/bot-lifecycle/queries"; +import { channelKeys } from "@/lib/channels/queries"; +import { formatHotkey, getHotkey, HOTKEYS } from "@/lib/hotkeys/hotkeys"; +import { userPreferencesQueryOptions } from "@/lib/settings/message-list"; +import { settingsKeys } from "@/lib/settings/queries"; +import { Route as SettingsRoute } from "@/routes/_authed/settings/index"; +import { settleReactWork } from "./settle-react-work"; + +/* + * ⌘K, or Ctrl+K off a Mac, finds a conversation from anywhere: it opens the sidebar if it was + * collapsed, or the sheet on a narrow window where the sidebar is one, and puts the cursor in its + * search box. It is a modified key, so it works while somebody is typing in a field, too. + */ + +class OfflineWebSocket extends EventTarget { + static readonly CONNECTING = 0; + static readonly OPEN = 1; + static readonly CLOSING = 2; + static readonly CLOSED = 3; + readyState = 0; + onclose = null; + onerror = null; + onmessage = null; + onopen = null; + close() {} + send() {} +} + +/** The window's size, which is what decides whether the sidebar is a sheet (under 768px). */ +function setViewport(width: number, height: number) { + ( + window as unknown as { + happyDOM: { setViewport(size: { width: number; height: number }): void }; + } + ).happyDOM.setViewport({ width, height }); +} + +const originalFetch = globalThis.fetch; +const originalWebSocket = globalThis.WebSocket; +const clients: QueryClient[] = []; + +beforeAll(() => { + GlobalRegistrator.register({ url: "http://localhost/" }); + globalThis.WebSocket = OfflineWebSocket as unknown as typeof WebSocket; +}); +beforeEach(() => { + globalThis.fetch = Object.assign( + async () => Response.json({ error: "not served" }, { status: 404 }), + { preconnect: originalFetch.preconnect }, + ); +}); +afterEach(() => { + cleanup(); + for (const client of clients) client.clear(); + clients.length = 0; + globalThis.fetch = originalFetch; + setViewport(1024, 768); +}); +afterAll(async () => { + await settleReactWork(); + globalThis.WebSocket = originalWebSocket; + GlobalRegistrator.unregister(); +}); + +function signedIn() { + const queryClient = new QueryClient({ + defaultOptions: { queries: { retry: false, staleTime: Infinity } }, + }); + clients.push(queryClient); + queryClient.setQueryData(authKeys.currentUser(), { + id: "maya", + name: "Maya Chen", + email: "maya@example.com", + role: "user", + onboarding: null, + }); + queryClient.setQueryData(userPreferencesQueryOptions("maya").queryKey, { + messageListEmphasis: "thread", + selfHostBannerDismissed: true, + narration: null, + skin: "openbot", + }); + queryClient.setQueryData(agentKeys.list(false), []); + queryClient.setQueryDefaults(botLifecycleKeys.attention, { + staleTime: Number.POSITIVE_INFINITY, + refetchInterval: false, + }); + queryClient.setQueryData(botLifecycleKeys.attention, []); + queryClient.setQueryData(channelKeys.list(), { + pages: [{ channels: [], nextCursor: null }], + pageParams: [""], + }); + queryClient.setQueryData(settingsKeys.instructions(), ""); + return queryClient; +} + +function SidebarState() { + const { isMobile, open, openMobile } = useSidebar(); + return {(isMobile ? openMobile : open) ? "open" : "closed"}; +} + +/** The sidebar, collapsed, beside a field somebody could be typing in. */ +async function renderCollapsed() { + const queryClient = signedIn(); + const router = createRouter({ + routeTree: createRootRoute({ + component: () => ( + + + +