Repository navigation
Say what a Bot is doing, and in plain words what it asks to do - #916
Draft
jerelvelarde wants to merge 47 commits into
Draft
jerelvelarde wants to merge 47 commits into
jerelvelarde wants to merge 47 commits into
Conversation
The last two places still drawing boring-avatars' gradient circles now draw the orb in the colour the Bot's id picks, so a Bot looks the same in the sidebar, the conversation header and its profile. Nothing imports boring-avatars any more, so it leaves the app's dependencies.
…ll when asked A Bot's orb is a block, so a group conversation's 12px faces no longer sit on the header's text baseline and drop half out of their circles. It resizes its canvas when the screen's pixel density changes, so dragging a window to another screen or zooming no longer leaves it soft, and it follows the reduced-motion setting live rather than only when it first draws. The changelog and the stylesheet now describe what this step draws, not what later steps will.
The skins import the orb's state type from agent-orb now that ai-core is gone, OpenBot's mark keeps its agent-orb class for them, and their tests check the orb's colour rather than boring-avatars' SVG. The Appearance test's preferences carry the new skin field.
AgentOrb read "100%" as 100px, so with no Bot yet the header drew a 100px orb in a 44 or 52px slot. With a Bot, the classes that used to rescale boring-avatars' SVG squeezed the avatar's circle to 44px below 900px wide while the orb inside stayed 52px, clipped and off centre. The frame now picks 44 or 52px from the window's width and draws the face at that size, the selectors that matched only the old SVG are gone here and from OpenDots' Bot rows, and AgentOrb takes its size only in px and refuses anything else.
Every orb on screen redrew on every animation frame, uncapped, so a sidebar of conversations kept a few dozen canvases repainting at the screen's full rate while the app sat idle: a small orb's colour still drifted, so no two frames were ever the same. A small orb's colour now holds still, an orb skips any frame that would draw what its canvas already shows in the same theme and at the same density, and a small orb draws at most about thirty frames a second. A face at rest with the pointer still now draws only to blink.
…violet A Bot's face was one of nine colours and nothing else, so a team of four had about an even chance of two identical faces. In the Bots OpenBot ships with, three were the same sky blue, and release notes wore violet, OpenBot's own mark. A Bot is now drawn from its seed: one of the eight colours that aren't violet, with its colour masses turned, its hue nudged and its eyes spaced by the same hash, so two Bots in one colour still look apart. BotOrb takes the seed rather than a colour and draws OpenBot's violet when it has none. The OpenDots avatar test checked one colour in nine, so it passed or failed on the hash; it now checks that the seed reaches the orb.
GET /api/setup says, row by row, whether Intelligence, the model, the Bot computer, more apps and Slack and Teams work here, from their own answers rather than from a variable being set. POST /api/setup/check asks them again. An admin, or somebody running OpenBot alone, is told the page or the variable that closes a gap; everybody else is told only that it works or that their admin sets it up, and their response carries no variable name.
Onboarding is the only page somebody who has not finished it may open, so a consent started on its Connect your apps step returns to that step, worked or not, for OAuth and brokered apps alike. A consent cancelled at the vendor now returns to the screen it started on too, instead of the Connections list. The returnTo name is still narrowed to a closed set before it is sealed or used.
A step-by-step onboarding frame that works on a desktop screen too. One DOM, laid out by CSS: from lg up, the steps as a list (the ones reached can be picked) with the step's heading, sentence and Continue/Back on the left, and its picture filling the right; below lg, a dash for each step, a round Back and the buttons pinned to the bottom. Focus moves to each new heading. A demo's clock runs only while it is on screen in a shown tab, and holds still with reduced motion.
Welcome types a request under OpenBot's listening orb, a computer of its own shows the Bot's desktop with its pointer on an errand, the team shows three Bots' orbs taking turns at work, and an approval card waits for Allow once before it sends. Somebody running OpenBot alone also sees it on a laptop, with the agents they have, and for their organization. Each is a picture, read by its label, with nothing inside to click.
Connect your apps lists the apps the person's default Bot uses as them and connects the ones with a sign-in page right there, coming back to the step; a brokered consent is asked of the vendor on return, so the row turns Connected only when it is. Get notified previews the notification OpenBot really sends and asks the browser, sharing its answer with a Bot's page. Check your setup draws the server's checklist with a status pill per row, and Check again.
First run is now welcome, a browser and files of its own, the team of Bots, asking before it acts, running it yourself (only for somebody trying OpenBot alone), connect apps, get notified, check your setup, and the name step. Where somebody is lives in the address and is saved on every move, so a reload, browser Back and a consent's return keep their place. The name step stays last, under the orb: Get started saves the name, completes onboarding and plays the orb's finish, and a checklist link carries its page through it, since nothing else opens until onboarding is done.
…ws now The root's drop guard named the onboarding poster's composer, which is now a drawing inside onboarding's picture frame. The changelog says the computer check also runs once at start-up.
Found running the steps against a real install. Connect apps told an admin to wait for an admin; it now offers Plugins, carried through the name step like the checklist's links. The checklist's "Required" tag ran into each title, and the notification preview cut "General Assistant needs you" off at the Bot's name.
…s written out Setting OPENAI_BASE_URL to https://api.openai.com/v1 counted as a custom endpoint, so the check was skipped and the setup checklist said the model was not checked. The provider's own host is now checked as if no address were set.
…the box The step promised that a Bot asks before it sends, buys or deletes anything. Ask before making changes starts off, so out of the box a Bot asks only where a rule says to, and paying or changing account security is handed to the person. The step now says that, and where to turn asking on.
With AGENT_COMPUTER_URL set and no computer running, the usual state and always true on a first run, the row told an admin Bots can't browse the web and to set the variables and restart, for something already set. Not checked now says it is set up and none is running to check, and opens Computers.
Intelligence, model and computer answers were asked at boot and then served for as long as the server ran. An Intelligence slower to start than OpenBot read as a red required gap until somebody pressed Check again, and a key refused days after boot still read Ready. A read now asks again, in the background and under the shared 10-second limit, when an answer failed or is five minutes old, so the next read has it. Reads are answered from the last answers that are all in, so none waits on a slow fleet.
The changelog said the intro was replaced by another product's steps, and two comments cited it, which reads as a skin being added. They now say what OpenBot does. The changelog also said a cancelled connect returns to where it started from anywhere; only Connect your apps changed, so it says that.
At 1280 by 800 the self-hosting step and the name step ran past the bottom of the window, cutting Continue and Back in half; at 700 tall, four steps did. The left pane stacked the nine-step list, the step's words and its buttons, and the page was left to scroll. On a desktop the frame is now one screen tall and the step list's row is the only one that can shrink, so the step's words and its buttons always keep their full size. When the screen is too short for all of it, the list scrolls within itself, fades at the edge it scrolls under, and keeps the step on screen in view. Its rows and the space under it are a little tighter, so every step still fits whole at 1280 by 800. Only a window too short for the words and buttons themselves scrolls the page. The picture pane now stretches to the screen instead of being set to its height: a set height spanning every row gave that height to the empty last row before the list got any. It drops the phone's margin on a desktop, which would stop it short of the bottom on the name step.
Rows, destinations and the header's controls lift to the card's fill and shadow under the pointer and while open, with a visible focus ring. Row text is on type-sub and type-caption, and the unread dot is the brand colour. OpenBot's orb sits beside the product name, and the + button's tooltip names its shortcut. A busy conversation's orb works, and a still dot in the working colour says so as well: an orb under 40px only blinks, and reduced motion stills it. ChannelAvatar takes an optional state; its props are otherwise unchanged. Each conversation row has a "…" button (SidebarMenuAction showOnHover) with the Pin and Delete that were on right-click alone. It shows under the pointer and with focus in the row, and comes straight after the row in tab order. NavigationItem gains an optional badge, a count the sidebar draws beside the entry, set by useNavigationItems, which the sidebar now reads. Primary entries draw in list order, so a new destination and its count are added in lib/navigation.ts alone. ⌘K or Ctrl+K opens the sidebar, or its sheet on a narrow window, and focuses its search box. Settings lists it from the hotkey registry.
A Bot's reply is drawn one bubble per thought (split-bubbles.ts), each
bubble memoised so a streamed reply parses only the part still arriving.
A bubble holding a code block or a table takes the column's width from md
up; the rest keep an 82% cap. The person's messages sit in a tinted bubble,
and Thinking is three dots in the Bot's bubble, still with reduced motion.
New --bubble, --bubble-user and --radius-bubble tokens carry the colours
and the shape, and their words clear 4.5:1 in both themes.
Tool lines draw a hook, an outcome mark and a Tabler icon, and only a line
that worked carries the done mark. A call that was refused, failed or
declined is named for what it would have done, a declined one reads "You
declined" (toolResultDeclined) and is not counted as an issue, a group
where nothing worked never claims it did, and a step held on its approval
card reads "Waiting for you". Mail and calendar calls through any app are
named for what happened to the person's own ("Searched your email"), by
their words rather than a table of one Gmail plugin's tools.
The Bot's bubble stays the ghost variant, which the skins find a Bot's
words by, and OpenDots takes the bubble off again to keep its plain prose.
A Bot's page names its tools by what they do ("Send your email").
A conversation with one Bot shows that Bot's orb in its header, at 40px,
with a line under the conversation's name. The orb works while the Bot has
the turn, listens while the person types, and lands an answer (not a
failure or a Stop). The line says "Working · Searching the web" while a
step runs, "Needs you" while an approval waits in that conversation, and
nothing at rest. The conversation list's busy flag counts as working, so a
run the server resumes after Allow once shows as "Working"; the flag of
the tab's own turn does not outlast the turn. Groups keep their own header.
The state lives in components/channels/conversation-presence.tsx, a store
the conversation route provides and ConversationView writes. Each reader
subscribes to the fields it uses, so a keystroke reaches the orb and not
the line, and a new step reaches the line and not the orb. Reports carry
the conversation they came from, so the route, which stays mounted from
one conversation to the next, never shows the last one's turn.
The step comes from lib/channels/step.ts (stepFor, stepPhrase) in the
running words the step's own tool line uses. An app's call it cannot name
by what it does reads "Using <app>"; a question for the person and a
drawn interface are passed over.
The compact composer is a pill: 62px on a soft shadow around 44px
controls, the editor at the bubble's 17px, and the focus ring kept.
Its border stays, clear, so the skins still colour it. It takes a
placeholder ("Message Ana" in a conversation with one Bot, "Ask anything"
elsewhere) and an onTextChange. Every data-slot hook is where it was.
The pane that shows a Bot's computer beside its conversation is 440px
wide. Its header carries the Bot's orb, "<Bot>'s computer" ("Computer"
until the name arrives), Take control, and a dot while the conversation
needs the person. The title keeps its data-slot and holds the title alone;
Take control sits in a new `actions` slot beside the close button, and the
screen under it says who has control without a second button. The screen
is badged with the Bot's name, not the conversation's.
Under the screen, a step log lists the conversation's browser steps, read
from its messages (a gated click runs on the server and never reaches a
browser handler), among the session's command log, oldest first and
newest last. Each step is named in its line's words and says how it went:
done in green, waiting for you in amber, didn't work or blocked in red.
The conversation presence store carries the browser steps, stamped when
the tab first saw each one.
`DetailPanel` reports whether it would overlay. The pane opens by itself
only when it docks beside the conversation; on a narrow screen the
header's computer button lights up instead. The password box names the
site the secret goes to.
OpenBot ran the composer column flush, which the floating pill made obvious whenever the column was no wider than the composer, such as with the Bot's computer docked beside the chat. The column now insets it by 12px, as the OpenDuet skin already did; the two skins' comments say so.
A browser step folded into a tool line read its outcome from the browser's JSON alone, with declined and refused fixed at false. A click the person declined is answered by the server in words, with no JSON, so it read as done; one a boundary refused read "Didn't work". The computer pane's step log already told these apart. Its reading moves to `browserOutcome` beside `browserStepDetails`, and the step log and the transcript's step rows both read through it. In the transcript, a declined step reads "Declined", muted and not an issue; a refused one reads "Blocked" with the boundary's reason. A step cut off by Stop reads "Not completed", one a starting computer turned away reads "Not done yet", and the note that a step went to its approval card is no result.
A declined, blocked or cut-off browser step read in the past tense, as in "You declined: Clicked on page". It now reads "You declined: Click on page", in the conversation and in the computer's step log, where a step waiting on its approval card is named the same way.
…at you decided
The approval card read "Waiting for your approval: GMAIL SEND EMAIL on gmail"
with Allow once, Always allow here and Deny, then vanished once answered. It now
says the action in the words its step uses ("Send your email"), the app, the
place it happens, who it goes to and what it says, with the whole request under
Details. Decline and Allow once answer it; Always allow here sits quietly under
them because it saves a rule for this Bot, action and place. A hand-off keeps
"I did it myself" and "Don't do it". Once answered it folds to a pill saying how
it went (Allowed, Sent, Done, Declined, Didn't work) and a line saying who
decided and when.
The same card is in the conversation, on the Bot's page and in the list of what
needs you, where an answer stays folded until the page is left. A server call
alone in its turn now draws its card under its line, which says it waits for
you; before, that request showed nowhere in the conversation. A run of steps
held on a card says the same.
The decision writes the server's answer into the inbox before reading it
again, so a card stays folded when that read fails instead of offering its
buttons for a second, refused press. A decline the server finished with the
decline as its error still reads Declined, never Didn't work.
…akes Under the composer of a conversation with one Bot or several, a small muted caption reads "OpenBot is AI and can make mistakes.", in the product name this OpenBot is configured with. It sits inside the composer column, so the skins' hooks and insets are unchanged.
A step line for typing into a field or reading a page said "Used the browser" while the computer's step log beside it said "Filled in field" and "Read page". Up to two kinds of browser step are now named in the log's words; past that the line keeps one phrase.
…reads Yes Typing into a search box asked "Type" with a field reading "Submit: true", under a step line that said "Fill in field". The card now takes its title from the same names the conversation and the computer's step log use, now kept in one file, and a yes-or-no argument reads Yes or No, with "submit" worded as "Submit after typing".
The label column was sized for To, Cc and Subject, so "Submit after typing" broke over three lines. It now takes the width its label needs, from that size up to under half the card.
…rase "Fill in field" and "Click on page" read like log lines inside a sentence such as "Allow it to fill in field, or decline". The step that would happen is now "Fill in a field", "Click on the page", "Open a page" and so on, in the chat, on the card and in the computer's log. What a step did keeps the log's shorter form.
… what is agreed to The rule Always allow here saves covers this Bot, this action and this place (every command in the folder, every email from the app), but the card led with one call's details and named no place for apps or the Bot's own computer. A line under the button now says how far it reaches, and the button is described by it. The card's face drops what it has no room for (a second body, objects, fields past four). When it does, Details opens by itself.
…t vouch for It named the host of the last frame drawn, but the value goes to the element the Bot pointed at, and that frame can lag behind or show another page.
… already waiting waits A decline was only recognised when the server's sentence stood alone. A Bot trying the action again gets it after "Refused.", or inside a computer tool's JSON, so the line read "Blocked: Send your email" in red, counted as an issue and showed the words written for the Bot. And the answer to a request that was already waiting was read as a decline, so the line said "You declined" while the first card still had no answer. Both are now read in every shape they arrive in. A repeat of a waiting request waits with it while the conversation has something waiting, and reads as not done once it doesn't.
… tool calls it has TanStack Query polls once for every component reading a query, and fetches only merge while one is in flight. Every tool call row read the inbox twice, mounted at different moments as a reply streamed, so thirty calls asked for it sixty times every ten seconds. The rows and cards now read the cache without a poll of their own. The conversation polls it once, and so does the Bot page, whose cards used to keep themselves fresh.
A turn that stopped for an approval counted as answered. The browser sends a gated click to its card and stops the run with no failure and no Stop, and a call the server holds ends its turn the same way, so the orb played its landing while the line said "Needs you". A turn that ends on a call still waiting for the person no longer counts as answered, and an approval waiting in the conversation holds off any landing.
…itself The check read declined[1]?.output under toBeUndefined, which also passes when the step is missing; the assertion-shape guard refuses that shape.
The changelog still said the box names the site the value goes to. Since the password box stopped naming a host, which it could not vouch for, it says the value goes straight to the page and the Bot never sees it; the changelog now says the same.
This was referenced Oct 31, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Depends on #913 and #915. Its own change is the commits after those: compare.
The problem
When a Bot asks before acting, its card reads "Waiting for your approval:", then the tool's id with its underscores turned to spaces, "on" and a scope, over Allow once, Always allow here and Deny. Always allow here saves a rule for this Bot, this action and this place, whatever the arguments: every email from that app, every command in that folder. The card shows one call and says nothing of that reach. Once answered it vanishes, and a call the server holds alone in its turn draws no card in the conversation at all.
Reading back is no easier: a tool line can't tell a decline from a result, so a declined step reads as done.
The approach
The card says what will happen, and how far a yes reaches. It names the action in the words its step uses ("Send your email"), the app, where it happens, who it goes to and what it says, with the whole request under Details, which opens by itself when the face leaves something out. Decline and Allow once answer it. Always allow here sits quietly under them, over a line that describes the button: this Bot won't ask again before doing this there, "whatever the details". A hand-off asks "I did it myself" or "Don't do it". Answered, it folds to how it went (Allowed, Sent, Done, Declined, Didn't work) and who decided when. The same card is in the conversation, on the Bot's page and in what needs you, and a lone server call now draws its card under its line.
A conversation polls its approvals once. Every tool row read the inbox twice, each with its own poll, so thirty calls asked sixty times every ten seconds. Rows and cards now read the cache, and the conversation polls it.
A step says how it went. Only a line that worked carries the done mark; one refused, failed or declined is named for what it would have done. A decline reads "You declined" and is not an issue, and so does a call tried again after a decline, which arrives after "Refused." or inside a computer's JSON. A repeat of a request already waiting waits with it rather than reading as declined.
The conversation shows the Bot. A reply is one bubble per thought, with lists, code and tables kept whole. With one Bot, the header carries its orb, which works while the Bot has the turn, listens while the person types, and lands an answer, though not a failure, a Stop or an approval hold. Under the name, "Working · Searching the web" says which step it is on and "Needs you" says an approval waits there. Under the composer, every conversation says OpenBot is AI and can make mistakes.
The Bot's computer sits beside it. The 440px pane carries the Bot's orb, Take control, and a step log under the screen that says how each browser step went. It opens by itself only when it docks beside the conversation; on a narrow screen the computer button lights up instead. The password box says it goes to "the page", not a site's name, because the frame drawn can lag behind the page the Bot points at.
The sidebar gives each conversation a "…" button with the Pin and Delete that were on right-click alone, and ⌘K opens its search.
What is not covered
toolCallIdalone.Verification
typecheck, lint, format:check and the assertion-shape guard are clean. The full app suite is 2416 pass, 0 fail, against 2270 on #915 and 2129 on upstream
main. New files, all inapp/tests/:approval-view.test.ts: the action in words, who decided, how it ended.lone-step-approval.test.tsx: a lone server call's card; a declined send is never named sent.approval-inbox-poll.test.tsx: thirty tool calls, one inbox request a poll.tool-line.test.tsx,chat-step.test.tsx,step-log.test.tsx: done only when it worked; a decline repeated reads declined; a waiting repeat waits.conversation-presence.test.tsx: Working and the step, Needs you, no landing on a Stop or an approval hold.split-bubbles.test.ts,transcript-bubbles.test.tsx: one bubble per thought, code and tables whole.computer-pane.test.tsx: opens beside, never over; the page, not a site.channel-avatar.test.tsx,sidebar-row-actions.test.tsx,sidebar-search-hotkey.test.tsx,ai-notice.test.tsx: the busy orb, the "…" menu, ⌘K, the AI line.Merge notes
This and #914 both change
app/src/components/channels/avatar.tsxandapp/tests/user-preferences.test.tsx. Whichever merges second combines the two avatars and updatesapp/tests/channel-avatar.test.tsxand the pane-title test inapp/tests/computer-pane.test.tsxto hold the Bots list. The combined change is small and already written for whichever lands second.