Repository navigation
Enhance vision mode with improved input handling and error checks - #3269
Merged
Merged
Conversation
Act runs whose model takes images now see the tab as screenshots and act with coordinates instead of the accessibility tree (Settings > Advanced > Vision mode, on by default; the tree path stays for text-only models and when the setting is off). - Shared modules (byte-identical in both builds): vision-mode, vision-budget (per-provider sizes that no server rescales, 28/32 px grids, WebP/JPEG, token estimates), vision-diff (cell diff, regions, patch/keyframe/none planning), vision-session (image ids f/p/z, image->CSS maps, staleness), vision-tools (tool surface, prompt, labels, click_at translation), vision-encoder (canvas resample/encode), vision-runtime (keyframes, observation after batches, native tools, pruning). - click_at is rewritten into a canonical coordinate click before the batch pipeline, so every gate and hard guard still applies; Chrome dispatches trusted CDP pointer events at the exact point, Firefox a synthetic sequence from the content script. - Captures account for CDP clip coordinates being document coordinates and for output = clip x scale x DPR (retina, browser zoom); the real capture is resampled to the planned size so mapping stays exact. - Only the current frame, its patches and zooms stay in the request. - Legacy auto-screenshots: clip now starts at the scroll offset (was blank on scrolled pages) and divides the DPR out (was 2x on retina). - Settings UI, locales (23), config export, trace runtime config, docs. - Tests: unit, agent run loop (both builds), Chromium DPR 1/2/3 pipeline, installed MV3 e2e at DPR 1/2 and browser zoom. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WKLAisk5u8VVdGkG8ZwtUf
…detail - Vision mode is on by default for vision-capable models, and the MV3 quality fixtures run under model names WebBrain recognizes as such (minimax-m3, gpt-5.4-mini) while scripting accessibility-tree answers. They now seed visionMode: 'off' with their other settings; docs say so. - Vision frames always ask for detail 'high': OpenAI's low detail would shrink a frame to 512 px server-side and every returned point would be off by that factor. The vision screenshot size setting governs size. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WKLAisk5u8VVdGkG8ZwtUf
…fety Fixes from an independent review of the vision-mode branch. - Chrome captures under browser zoom: CDP clips are in device-independent pixels (CSS px x zoom), so clip origin and size are scaled by cssVisualViewport.zoom. At 125% the frame showed only 80% of the viewport and points were off by the zoom. The legacy auto-screenshot clip gets the same factor. Firefox frames always pass the visual-viewport rect (no classic scrollbars in the image). - Dispatch checks run on what is under the pointer (vision-dom-probe.js, isolated world in Chrome, content-script world in Firefox): repeated clicks on controls, drags that would click a control or start/end in an opaque frame, and wheels over a focused number field are refused; drag_at is blocked on protected messaging pages like iframe_click. - A refused click_at stops the rest of its batch. - Each run starts on its own frame: the previous run's screen is retired, a failed first capture also retires it for the model, ids/epochs continue across sessions and pruning keeps the last frame by position (a model switch or worker restart showed an old frame before). - Vision tool, capture and encoding errors become tool results or notes instead of ending the run; the batch stability check fails closed. - Malformed image ids are refused; normalized zoom fallback size; top-left corner nudge; hover no longer focuses selects; GLM-4.xV and InternVL use 0..1000 points; nested runs restore the outer vision snapshot; sessions are dropped with their tab. - Tests: refusals, batch interrupt, error paths, multi-run epochs, the real DOM probe in Chromium, and pixel-exact marker checks in the MV3 run at DPR 1/2, zoom 125%/150% and scrolled pages. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WKLAisk5u8VVdGkG8ZwtUf
…le stub - The dispatch probe treats an element drawn with a pointer cursor as a control (framework buttons are often a bare <div> with a listener), for repeated clicks and for drags inside it, unless the drag starts on a slider, scrollbar, resizer, draggable item or grab handle. - drag_at also runs the click submit probe at its start point and refuses a drag that starts on a submit, send, delete or payment control. - The Chrome probe remakes a cached isolated world once if it did not survive (debugger re-attach on the same document). - test/runtime-lifecycle.mjs stubs the new visionModeReady promise that handleMessage awaits (8 chat-start tests threw a ReferenceError). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WKLAisk5u8VVdGkG8ZwtUf
…hurn, faster steps From a Mastodon trace of the vision branch: - No accessibility-tree text reaches a vision run's model: the pre-decision capture still binds keyboard input but is not added to the conversation, feedback messages carry events and the URL only, and applying feedback skips the tree read. - Page churn no longer throws decisions away. click_at, hover_at and drag_at are vision_point calls and scroll_at a scroll call; the pixels around the point are re-checked before dispatch, and the page monitor rebases vision operations over passive or unattributed churn while the element at the point is unchanged. A person's input still voids them. - type_text / press_keys right after a click_at of the same response are checked by focus at the clicked point instead of being blocked by the page's reaction to that focus. - scroll_at dispatches under churn, and the monitor counts the wheel events each fenced notch owns, so a late notch is no longer taken for a person's scroll (which fenced the next actions). - A set-aside decision is remade on the current screen; before a decision an unchanged screen (micro changes aside) is not sent again. - Latency: the settle wait ends on micro changes (caret, clock), the Chrome click reuses the file-chooser settle as its new-tab quiet window (late tabs are still rerouted before the next screen update), and the first frame waits for a cookie-banner decline in progress. The MV3 fixture's four steps took about 7.5 s instead of 15 s. - Prompts: the universal cookie/paywall/PDF guidance uses vision tools, site notes say which element tools a vision run lacks, and a site workflow that needs the element inventory switches the run to the tree path (system prompt rebuilt when the vision state flips). Tests: vision unit and agent suites (micro changes, settle, churn before a decision, site notes, ledger fallback, new-tab watch), a real-monitor wheel test, and the MV3 fixture now types in the same response as its click under a 250 ms ticker, scrolls, follows a target=_blank link and asserts one screen update per step with no tree text. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WKLAisk5u8VVdGkG8ZwtUf
Review fixes for #522: - Patches follow the provider's image grid like frames and zooms do. planFrameUpdate grows each patch to multiples of profile.align (at least two cells, centred on the change, inside the frame) and merges patches that touch after that, so a Qwen2/2.5-VL or UI-TARS server (28 px grid) receives a patch unchanged and the 1:1 map recorded by registerPatches holds for the points the model reads from it. - detectVisionProfile compares the base URL's host with api.openai.com instead of looking for the name anywhere in the URL (CodeQL: incomplete URL substring sanitization). Tests: grid-aligned patch planning (covering the change, merged when touching, unchanged without a grid), a Qwen2.5-VL agent run whose patch is 28 px aligned and whose patch-local point maps 1:1 to the frame, and host-based OpenAI detection. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WKLAisk5u8VVdGkG8ZwtUf
Review fix for #522: the Firefox vision click went straight to an unguarded synthetic click, so a file input, its label or an upload button that clicks one could open a native chooser that screenshots cannot show or dismiss. - content.js visionInput runs the click inside clickWithoutNativeFilePicker, returning the blocked-chooser result at once or the guard id for settlement (both content scripts, which stay identical). - The Firefox agent settles that guard after a vision click, as it does for an ordinary click, so a chooser opened a moment later is caught. - A blocked chooser is reported in vision terms in both builds (visionFilePickerBlockedResult): upload_file is not offered in vision mode, so the result no longer points the model at it. Tests: the real content-script click blocks a file input, a label for one and a button that clicks one, catches a deferred chooser at settlement and settles an ordinary control clean; the Firefox agent settles the guard and strips its id; the blocked result never names upload_file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WKLAisk5u8VVdGkG8ZwtUf
…ates Review fix for #522: a drag that ends inside a same-origin frame moved in frame-local coordinates but released at top-level ones, so a slider or canvas committing its value on pointerup/mouseup could jump. Every move and the release now carry the coordinates of the document under the point (a point with nothing under it keeps the last frame's offset); both content scripts keep the identical vision input code. Tests: a drag across a slider in a bordered same-origin frame moves and releases at the same frame-local point; the vision input block is asserted identical in both builds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WKLAisk5u8VVdGkG8ZwtUf
A drag after an earlier action of its batch was re-checked against the live screen at its start only; its end went through the session's recorded patches, so a drop target the earlier action changed still got the drag. Both ends are now resolved first and checked against one fresh capture (_visionResolvePointsForTool), and the refusal names the end that moved. In Chrome, the update after a click consumed the late-tab watch even when no tab had appeared yet, dropping the rest of its four seconds. A watch that has seen nothing now stays armed until its deadline; a tab that opens while the update's screen settles is rerouted and the screen read again, and one that opens while the model decides is rerouted at the next update. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WKLAisk5u8VVdGkG8ZwtUf
Synthetic pointer events never move a native <input type=range> thumb or select text, so in Firefox a drag_at on a slider or across text sent its events and reported success while nothing changed. The drag's press now focuses what it lands on and starts what a person's press would: - on a range input the value under the pointer is set through the native setter (the input's own step sanitization snaps it), with an input event per change and change on release; - on selectable text or in a text field the text from the press to the pointer is selected, after a cancelable selectstart, never on links, images, draggable items, other controls or user-select: none. Both follow the pointer in the element's own document, past a frame's edge included, and the result reports rangeValue or selectedChars. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WKLAisk5u8VVdGkG8ZwtUf
The Firefox drag emulation checked el.disabled, which is false for a range input or text field inside a disabled <fieldset>; such a slider was moved and its input/change events fired. Both helpers now test :disabled, which covers the fieldset too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WKLAisk5u8VVdGkG8ZwtUf
A person's click on a disabled button does nothing, and el.click() (the ordinary content click path) honours that, but events the vision input dispatches reach the control's listeners all the same: a disabled submit with a JavaScript handler could act, even while an earlier submission was pending. A click or drag whose press lands on a disabled control, disabled by its fieldset or inside one included, is now refused before anything is sent (visionInputRefused: disabled_control). A link inside a disabled fieldset is not a disabled control and still clicks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WKLAisk5u8VVdGkG8ZwtUf
…the caret A double click in an input or textarea called el.select(), taking the whole field, so the type_text that usually follows replaced all of it instead of the word the model pointed at. The clicked caret position (caretPositionFromPoint, or caretRangeFromPoint) now decides: a double click selects the word, run of spaces or punctuation mark under the pointer (Intl.Segmenter word boundaries; the right half of a word's last letter still counts as the word), a triple click a textarea's line or the whole input. Page text uses the same word boundaries instead of Selection.modify, which took the previous word when the caret sat at a word's start. A single click in a text field or in editable text now also puts the caret under the pointer, as a person's press does; focus() alone left it wherever it was. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WKLAisk5u8VVdGkG8ZwtUf
…e next Each synthetic hover, click or drag appeared on its target with over and enter events only; nothing ever left the previous one, so a hover menu or tooltip that closes on mouseleave (or on a mouseout whose relatedTarget is outside it, as React's onMouseLeave reads it) stayed open after the model pointed elsewhere. The content script now keeps the element the vision pointer was last over and moves off it as a mouse does: out on it and leave on the ancestors the pointer exits, over on the new target and enter on the ancestors it enters (shared ancestors get neither), pointer events before mouse events, relatedTarget set within a document. Drag moves cross elements the same way with the button held, and a wheel moves the pointer to its point first, as the Chrome path's mouseMoved does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WKLAisk5u8VVdGkG8ZwtUf
The synthetic wheel scrolled the first scroll container under the point with room, else the document, with a programmatic scrollBy, which ignores what decides a real wheel: an inner pane already at its end never passed the wheel to the page, a pane with overscroll-behavior: contain did, a frame that cannot scroll never scrolled the page around it, and a page locked with overflow: hidden behind a modal scrolled anyway. _visionWheelTarget now routes the delta as the browser does: the nearest scroller that can still move that way, an exhausted one passing it on unless its overscroll-behavior keeps it, then the viewport unless its overflow locks it, then the page around a same-origin frame; nothing scrolls when nothing would. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WKLAisk5u8VVdGkG8ZwtUf
…item
Synthetic pointer and mouse events never start the browser's drag and
drop, so in Firefox a drag_at on an HTML5 draggable item (a sortable
list, a kanban card) never reached its dragstart, dragover or drop
handlers while the tool reported success, and drag_drop, which does
dispatch DragEvents, is not offered in vision mode.
A drag whose press lands on a draggable="true" item now runs the
sequence as the browser does (_visionHtmlDrag): dragstart on the item
with a DataTransfer the page fills (a cancelled dragstart starts no drag
and the pointer drag goes on as mouse events), then per step drag on the
item, dragenter on a newly entered element followed by dragleave on the
one left, and dragover on the current one, whose cancelling makes it a
drop target (for the drop effect the page leaves, where the engine lets
a constructed DataTransfer carry one); on release drop there, or
dragleave when nothing accepted it, then dragend. A pointercancel marks
the drag's start and no pointer or mouse events follow, as for a
person's drag. The result reports html5Drag {started, dropped,
dropHandled}.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WKLAisk5u8VVdGkG8ZwtUf
A zoom taken after a page-changing action in the same response captured the updated screen but was registered with the session's sequence from before the update; the update after the batch then registered its patches with a newer sequence, so a point read from that fresh zoom was refused as stale_area although the zoom already showed the change. Such a zoom now takes a frame-scale capture beside it, and the update that sends it compares that capture with its own frame (_visionRefreshZooms): the zoom is made as current as the update (markZoomCurrent) and stays stale only where the screen changed after its capture. Both browser copies share the change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WKLAisk5u8VVdGkG8ZwtUf
A delete the user asked for was refused in vision mode. The deletion guard binds a click to the control and payload it checked only through a ref or selector and refused every coordinate click (click_required), so no click_at could ever delete. An audit of every guard for the same assumption (refs, selectors, tree reads, DOM text snapshots) found more: - Deletions and commits after a correction bind a vision click to the control the submit probe read at its point; the vision dispatch reads the point again right before input goes out and sends nothing when the authorized control moved away. - Loop detection keys vision actions on the screen they acted on (VisionSession.screenVersion): the same point, key or wheel after a visible change is progress, on an unchanged screen it still stops. Vision clicks and arrow keys get no DOM-snapshot noProgress verdict, a native dropdown is never a failed click, and loop messages name click_at, zoom and the screen. - The delivery checkpoint counts take_screenshot, scroll_at and wait; a visible change ends the streak. Page focus churn keeps look, zoom and wait responses. - Every vision batch ends with one screen update, including batches that leave early (a refused done); done's unlabeled device-scale screenshot is not taken; zoom and take_screenshot are observations; on-screen status texts are recorded while a submission waits for proof; the opening frame is plan evidence; refusals name vision tools; gmail_count_results stays in the tool list. - Whole-thread reads, WhatsApp Web and Telegram keep the tree path. - A pending CAPTCHA gate gets WebBrain's own root tree read before the next action. - The submit probe reads points inside open shadow roots, and plain content at a vision point is a non-submit target (frames, canvases and application regions stay opaque). - Firefox: a recipient-bound vision click refuses right, middle and repeated clicks; the toolbar check keeps vision clicks on the vision dispatch in both builds. Tests: test/vision-mode-guards.mjs (both builds), MV3 deletion (named and vague) and repeated-click fixtures, the probe in Chromium. The cookie-banner wiring test now counts the first-frame settles the branch added earlier. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WKLAisk5u8VVdGkG8ZwtUf
…ion press A slider, canvas or press-and-hold control that calls setPointerCapture on its pointerdown and reads the moves and the release from the capturing element lost the drag under the synthetic Firefox input: the vision pointer is no active browser pointer, so Firefox throws for its id (the handler stops there) and nothing routes its events, while drag_at still reported success. A control that drops the implicit capture first (react-aria's press) never completed a click either. While a vision click or drag runs, the capture methods of the press's window now answer for the vision pointer's id (through wrappedJSObject and exportFunction in a Firefox content script; other ids still reach the page's own methods, which are put back when the sequence ends). Each later pointer event and its compatibility mouse event go to the capturing element, in its document's coordinates, with gotpointercapture before them and lostpointercapture right after the release or cancel; an element removed from the document loses capture as in the browser. Without a capture request nothing changes. The browser test drives a thumb whose handlers are on itself, a track that captures its thumb and a react-aria-style press, against capture methods that throw for the vision pointer's id as Firefox's do, both from the page world and as from a Firefox content script. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WKLAisk5u8VVdGkG8ZwtUf
A same-origin frame drawn at a CSS scale (a transform or zoom on itself or an ancestor) took Firefox vision input and the vision DOM probe at the wrong place: the point was only shifted by the frame's position, so with transform: scale(0.5) a control at the frame's (100,100) was looked for at (50,50), and the click went to whatever was there and reported success. The Firefox input (content.js _visionFrameMap) and the shared DOM probe now map a point into a frame past its border and padding and undo the scale it is drawn at, by comparing the frame's drawn box with the size of its own viewport; a drag maps every point, its range, selection and captured-pointer coordinates included, through the same chain. A frame drawn rotated, skewed, flipped, on a motion path or in 3D, which no scale maps, is refused for input (frame_unmappable) and read by the probe as a frame it cannot inspect. The Chrome click submit probe's child-frame mapping likewise takes the frame's border, padding and drawn scale into account. Browser tests click and drag in a scaled frame and in a frame inside a scaled ancestor, refuse a rotated frame, and read the same points with the DOM probe (both builds) and the Chrome click submit probe. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WKLAisk5u8VVdGkG8ZwtUf
- Firefox hover reveals CSS :hover. Synthetic events run a page's hover listeners but match nothing :hover, so a menu or tooltip only CSS revealed stayed hidden while hover_at reported success. While the vision pointer rests on an element, it and its ancestors (through shadow roots and out of same-origin frames) carry an attribute, and each :hover rule the content script can read is copied, keyed to it, into a style sheet the document or shadow root adopts (@media, @supports, @layer, @container, @import media and nesting kept; :host(:hover) included). The pointer moving on moves it; a person's own pointer moving ends it. A page with no readable :hover rule is left untouched. - hover_at and drag_at bind the page monitor to their point, as a translated click does: _visionTranslateToolCalls resolves the CSS point (a drag's start; its end stays under the pixel check) and page feedback prepares the operation with it, so unrelated churn before dispatch no longer voids them. - A Firefox click or drag on a frame the content script cannot enter (another site's, a plugin, one still loading) is refused (opaque_frame) instead of stopping at the frame element and reporting success. - Points read from a frame the window was resized or zoomed since are refused before a batch runs (viewport_changed, with a full frame next), which catches a resize page feedback had not reported yet; the rest of the batch is skipped with them. Tests: the :hover emulation (a CSS-only menu, a rule in @media, a shadow root's :host(:hover), a person's pointer ending it, no residue on a page without :hover rules), the monitor preparation points of hover_at, drag_at and scroll_at in the run loop, an opaque frame refused, and a resize while the model decides, reported and not. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WKLAisk5u8VVdGkG8ZwtUf
…ng until captured - drag_at is a consequential completion action, like the drag_drop it replaces: a drag that moves a slider, reorders an item or draws on a canvas now opens verification debt, so done in the same batch, or without a look at the result, is refused as unverified. - take_screenshot no longer counts as an observation. It only asks for the frame the screen update after its batch captures, and that update records itself as auto_screenshot once it exists; when the capture after an action failed, take_screenshot plus done in one batch no longer passes as verified. A zoom, captured when it runs, still counts. - The :hover emulation reads a style sheet again only when its rule count changes, so a large site's rules are read once. Run-loop tests: take_screenshot and done after an update that could not be captured (refused, then accepted once the frame was captured), and a drag finished in its own batch (refused); the classification in both builds; a :hover rule the page adds later is picked up. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WKLAisk5u8VVdGkG8ZwtUf
…ascade The :hover emulation copied whole rules into an adopted style sheet, so a :hover selector listed with others (`.menu:hover, .save`) carried the others along and, appended after the page's sheets, overrode later rules: Save was hidden whatever the pointer was over, and a later rule of equal specificity lost to the copy. Now nothing is copied. While the vision pointer rests on an element, every :hover in the selectors of the readable style rules also accepts the hover attribute, in place: `.menu:hover .submenu, .save` reads `.menu:is(:hover, [data-webbrain-hover]) .submenu, .save`. Each rule keeps its place in the cascade, its specificity (:is takes its argument's), its conditions and the other selectors of its list, and `:not(:hover)` stays its negation. The page's selectors are put back when the emulation ends (a person's own pointer moving), and a sheet is read again only when its rule count changed. Also: on a protected Chrome Web Store page, hover_at, drag_at, scroll_at and zoom stop with the protected-page result, as click and hover do (take_screenshot sends nothing to the page). The browser test adds the mixed selector list, a later rule of equal specificity still winning over the hover, and the page's selectors, shadow roots' included, restored. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WKLAisk5u8VVdGkG8ZwtUf
Synthetic pointer and mouse events perform no slider activation, so a click_at on a volume or playback slider's track reported success and left the value where it was; only drag_at set the value. An uncancelled primary press of a vision click on an enabled <input type=range> now sets the value under the point the way the drag does (_visionRangeDrag: the step snapping, an `input` event), with `change` on release, and the result reports the rangeValue. A press the page cancels, and a right or middle click, move nothing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WKLAisk5u8VVdGkG8ZwtUf
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WKLAisk5u8VVdGkG8ZwtUf
test/firefox-vision-e2e.mjs installs the Firefox build into an isolated headless Firefox over WebDriver BiDi and sends vision_input to the real content script, Xray world, wrappedJSObject and exportFunction included. It covers CSS :hover and its restore on a person's own pointer move, pointer capture, a press that drops the implicit capture, a range track click, word and drag text selection, a wheel, a click in a scaled frame and the refusal of another origin's frame. It uses the window's first tab, since Playwright's Firefox cannot open one over BiDi. npm run test:firefox-vision:e2e runs it (FIREFOX_BINARY if needed). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WKLAisk5u8VVdGkG8ZwtUf
Vision mode applies only to Act runs whose model accepts images, and some runs keep the accessibility tree whatever the setting, so the default option is Auto rather than On, like Lean mode's beside it. The setting is stored as 'auto'. A stored 'on' (or true), and an exported configuration that carries it, reads as 'auto', and a stored 'on' exports as 'auto'. Every locale shows the Auto label it already uses for Lean mode. Traces record 'auto'. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WKLAisk5u8VVdGkG8ZwtUf
With Firefox trusted automation on, a vision click_at (one left click) and hover_at now go out as trusted input at the point through the local companion, instead of the content script's synthetic sequence, when the companion reports point input. - The companion's connect reports capabilities (point_input). Its new click_at and hover_at actions take a whole-pixel CSS point of the run's top document. They recheck that the element the extension marked is still the one at the point, and the click runs under the companion's usual native-action marker. - content.js visionInput runs every check of its synthetic sequence, then marks the element document.elementFromPoint gives (the one the page monitor prepared for the point), returns the rounded point and, for a click, arms the file-chooser guard. A point over an embedded frame or inside an open shadow root stays synthetic, because there the monitor binds the host but sees the events come from inside it. - bidi/client.js visionInput prepares and performs the input, fencing it like the synthetic sequence (the run already prepared the vision point). It returns null, so the agent sends input synthetically, when the run has no companion, the companion has no point input, or the input stays synthetic. A disconnected run never falls back. - A double, triple, right or middle click, a drag and a wheel stay synthetic. In Firefox a BiDi wheel scrolls only the window's selected tab: in headless Firefox the wheel event reached a background tab but nothing scrolled. A run's tab can be in the background. Tests: companion and client units; the content preparation in Chromium; the page monitor's attribution of the trusted click and hover (and why an open shadow root stays synthetic); and the Firefox e2e, which now also sends a trusted click_at and hover_at through the production session. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WKLAisk5u8VVdGkG8ZwtUf
The LM Studio plugin's URL guard matched IPv6 literals with string prefixes and a dotted-quad regex for IPv4-mapped addresses. The WHATWG URL parser rewrites [::ffff:127.0.0.1] to [::ffff:7f00:1], so every IPv4-mapped loopback or private address passed the guard, and the DNS layer skipped literal IPs entirely, so fetch_url / research_url reached local services with allowPrivate off. 0.0.0.0 and :: also passed. Parse literals to bytes and match CIDR prefixes. IPv6 addresses that embed an IPv4 address (IPv4-mapped, IPv4-compatible, SIIT, NAT64 64:ff9b::/96, 6to4, Teredo server and client) get the IPv4 rules for the embedded address. Add 0.0.0.0/8, multicast and reserved IPv4, ::, fec0::/10 and ff00::/8, and make 100.64.0.0/10 a real /10. The DNS layer now classifies literal IPs itself instead of returning early. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With a Cloud Bridge token configured, the Chrome and Firefox bridges sent the token, browserId and installationId in the first hello and then took connection_approved from whoever answered on the loopback port. A process that won the port could collect a reusable token and run cloud_* actions. The hello now carries a fresh 32-byte nonce instead. The controller must answer auth_challenge with HMAC-SHA256(token, label, "controller", both nonces); the extension verifies it with WebCrypto and only then sends its own proof (role "extension", both nonces, browserId, installationId) with the IDs. The token never crosses the socket. Until the controller has proved itself, connection_pending/approved are ignored; a rejection is still honoured. A controller that fails gets nothing, the socket closes and the bridge retries with a backoff that open no longer resets. The old bearer hello stays available only behind the deprecated storage flag webbrainCloudBridgeLegacyBearerAuth (exactly true), passed through cloud-runs.js and watched by background.js. Tokenless mode, used by the MCP server, LM Studio and the Cloud sidecar, is unchanged. The example approval server implements the handshake (--legacy-bearer keeps the old hello), and the protocol doc gives the exact transcript, a test vector and what a backend has to implement. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ddresses fetch_url, read_page_source and research_url validated only the URL text, so a public hostname whose DNS answer was loopback, RFC1918, unique-local or link-local (cloud metadata) passed and its response reached the model. Both builds now read the server address the browser actually connected to from webRequest (details.ip) for the tool's own fetch, and for every top-level response of research_url's hidden tab, and discard the response when that address is blocked. Resolved addresses use the same tiers as IP literals, including the local-network opt-in. Firefox also gains the "dns" permission and refuses a hostname before sending when any answer is blocked; it skips the response check for proxied requests. Docs describe the check and its limits (Chrome cannot stop the request itself, proxies, no-address fallback), and the local-network setting text in every locale no longer says DNS is unchecked. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…context The main agent put the tab URL and document.title into a bracketed [Current page context] note in the same user-role message as the task, after only removing framing characters. They now sit inside the _wrapUntrusted boundary the planner and tiny Ask already use. The fields keep their one-line sanitizing, so the note still strips as one unit and the progress page scope still reads its URL. The default-on "Always allow API mutations" setting also waived the network permission prompt for model-initiated POST/PUT/PATCH/DELETE through fetch_url/research_url. A new install now stores apiMutationSitePrompt: true, and there the setting still allows those requests but each host goes through the normal network permission prompt. /allow-api and a Cloud run's own grant still waive it. Existing installs are not migrated and keep the earlier behavior. Cloud runs that ask for API mutations add their run-scoped grant whenever no grant waives the prompt. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A coordinate click, type_text without a selector and press_keys were authorized against the top-level page host, while Chrome's CDP input delivers them to whichever document is under the point or has focus, including an embedded cross-origin payment, identity or wallet frame. Chrome now resolves that receiving document before the permission gate (cdp-client.js resolveInputFrame: hit test or focus, descending into process-isolated frames and <object>/<embed>, with focus, scroll and origin read in an isolated world) and charges its origin's host. An unidentifiable or opaque-origin frame fails closed, and after a permission card the receiver is checked again so a page cannot move the input into another document while the card is open. Firefox's generic input stays in the top document: its synthetic events cannot enter a cross-origin frame, and trusted companion input on a frame element (or a companion press over one) is now refused, except the ungated Tab, Escape and hover. Both gates honour the trusted _inputFrameUrl only for these tools and always overwrite it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ing others
list_downloads returned the browser's recent download history with full
local paths and source URLs, and read_downloaded_file read any download id
it was given and sent its bytes to the model's provider.
Each tab's conversation now keeps the ids of its own downloads: the ids
WebBrain's download tools return, plus every download that starts while a
WebBrain run is active on the tab (downloads.onCreated). The set is saved
with the conversation, restored only into the same conversation, and
cleared with it.
list_downloads looks up only those ids and names files by basename.
read_downloaded_file reads them as before; any other completed download is
read only after the user approves that exact file on a new WebBrain card
("Read this file once"), which never times out. Unattended runs cannot ask
and read nothing. Chrome and Firefox are changed alike (Firefox asks only
for the captured data: payloads it can read).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The dependabot bump of zod to 4.6.6 (#508) left two zods in the plugin: our tool schemas were built with zod 4, while @lmstudio/sdk's tool() wraps them in its own bundled zod 3 z.object(). zod 3 calls _parse() on each field, which zod 4 schemas do not have, so every tool call failed with "keyValidator._parse is not a function". The SDK's zod-to-json-schema also saw no zod 3 fields and advertised every tool to the model with an empty parameter list. Pin zod to ^3.25.76, the major @lmstudio/sdk requires (^3.22.4, still the case in 2.1.0). npm now dedupes to one zod shared by the plugin and the SDK. No source changes. Add test/toolParameters.test.mjs, which registers the tools through main() and checks the SDK's checkParameters, implementation and JSON schema paths. It fails on main and passes with this pin. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The bridge refused every memory run after the 256th started in one process
("Restart the MCP server"). That suits a short-lived MCP process but breaks a
host that keeps one bridge alive for the app's lifetime, such as webbrain-me:
a frequently scheduled memory bot fails until the app restarts.
Keep the same 256-scope memory bound, but count retained scopes instead of
lifetime starts. A run's redaction scope is never evicted before the extension
reports it terminal, so failed and late runs keep full protection. Finished
scopes are recycled oldest-finished first when room is needed, and their
callback credentials leave the global redaction set only when no retained run
still uses them. When every slot holds an unfinished run, the bridge rereads
the extension's run list once (it hydrates first, so it is authoritative) to
catch runs that finished unpolled or whose start was refused, and refuses with
a 409 only if all 256 are still live. Memory starts without a runId get a
bridge-minted one so every scope can be tracked.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
No workflow ran `lmstudio-plugin`'s `npm test`, so dependency bumps that break the plugin (like the zod 4 bump in #508) merged green. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Pre-hiding added autoconsent's <style id="autoconsent-prehide"> when an
Act run started and removed it 2 s later, after the turn was released.
The page monitor counts any stylesheet it does not own as page churn, so
that removal refused the agent's next action ("action context unverified",
"browser changed during action preparation"). It broke native-chat,
native-chat-presubmit, bulk-resume, mastodon-publication,
mastodon-delete-confirmation and the WhatsApp contact matrix harnesses.
The runtime wrapper now creates the pre-hide and rules style elements
itself and registers them in window.__wbPageMonitorDecorations before
they reach the page, and undoPrehide no longer inserts and removes a
style when nothing was pre-hidden. A page's own element with the same id
is not claimed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…Settings While WebBrain works on a tab, the page edges now show one of five effects, chosen in Settings → General → Agent activity effect: - Neural circuit (default): traces grow in from the edges and signals run along them toward the element WebBrain acts on - Aurora: a soft band that drifts around the frame - Scan: corner brackets and a slow scanning beam, with lock-on brackets around each target - Gears: meshing cog trains turning in the corners - Classic glow: the previous pulsing border Each effect is a click-through canvas (pointer-events: none) registered as a page-monitor decoration, hidden for screenshots like the old glow, and held still under prefers-reduced-motion. The content script reads agentIndicatorStyle from storage.local and follows storage.onChanged, so a new choice applies to a running task at once. The setting is exported and validated on import, and all 23 locales carry the new strings. The agent cursor keeps one smaller size (36×42), glides between targets, presses with a ripple only on clicks, and between actions stays where it last acted, fading in and out until the task ends. It moves with CSS transitions and the Web Animations API, so it never rewrites its style attribute per frame. Tests: a page-monitor-ui case (Chromium and Firefox) cycles every style and checks that none of it is reported as page feedback and that real clicks still reach the page; a run.js case covers the General tab select, saving, the live listener, and config export/import. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EsDq6CXeqKxZ1FNZay6B3R
…d motion The canvas effects and the cursor already check prefers-reduced-motion in script. The classic glow and the target ring are CSS animations, so the injected stylesheet now turns them off under the same media query (the ring keeps a static shadow and only fades). The page-monitor-ui case checks both, plus a still circuit frame, with reducedMotion emulated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EsDq6CXeqKxZ1FNZay6B3R
Classify LM Studio fetch targets by IP bytes so IPv4-mapped IPv6 cannot reach loopback
Bumps [dnspython](https://github.com/rthalley/dnspython) from 2.8.0 to 2.9.0. - [Release notes](https://github.com/rthalley/dnspython/releases) - [Changelog](https://github.com/rthalley/dnspython/blob/main/doc/whatsnew.rst) - [Commits](rthalley/dnspython@v2.8.0...v2.9.0) --- updated-dependencies: - dependency-name: dnspython dependency-version: 2.9.0 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com>
…auth Authenticate the bridge controller before sending any credential
…ivate Block network-tool responses from hostnames that resolve to private addresses
Quote the page URL and title as untrusted data and keep site prompts for default-on API writes
Charge coordinate clicks and focused typing to the frame that receives them
mcp-server: bound concurrent bot-memory scopes, not lifetime runs
Run the LM Studio plugin tests in CI
…ations Cookie banners: register autoconsent's styles as page decorations
Agent indicator: Neural circuit page effect by default, four more in Settings
…s/sources/deepseek-v4-flash/dnspython-2.9.0 Bump dnspython from 2.8.0 to 2.9.0 in /docs/vision-models/sources/deepseek-v4-flash
Limit download tools to this conversation's files and ask before reading others
Pin LM Studio plugin zod to the SDK's major so tool arguments parse
|
@esokullu is attempting to deploy a commit to the esokullu's projects Team on Vercel. A member of the Team first needs to authorize it. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.