[earakely-scale]

Adds the computer driver SDK to agentenv-protocol: a driver is a ComputerDriverEnvironment subclass that implements screen operations (capture, click, type, …), and the base class serves the computer contract from them. Linear: FD-4380 (M0.0, the SDK prerequisites) and FD-4348 (M0.2, the base class) of the CUA Env epic. Nothing under src/agent_env/ changes; core gains one test.

SDK changes

• Structured errors. AgentEnvError(code, message, retryable, applied, hint, status). Raised in an extension it answers with its status and those fields in the error body; raised in a tool (mcp's FastMCP and fastmcp) it answers isError with "<code>: <message>" as text and the fields as structured content. Other extension failures stay extension_failed / 500, now applied: "unknown". invoke_extension raises ExtensionCallError, an httpx.HTTPStatusError carrying the parsed fields, so callers that catch HTTPStatusError (DeployedEnv.invoke) keep working.
• Several methods per extension. @extension(uri, op=...): handlers sharing a URI merge into one card entry, each method with its own verb and endpoint (/agentenv/ext/<extension>/<op> by default). extension_params(uri) adds extension-level params; declared_extensions() advertises declaration-only ones. Without op, routes and cards are unchanged.
• @tool metadata. annotations, structured_output and meta on both FastMCPs; a tool returning Annotated[CallToolResult, Model] is sent as is and listed with Model's schema.
• Choosing what is served. served_tools(tools), served_extensions(extensions) and served_operations(operations) choose at mount which tools, extensions (and methods of one) and data-plane methods an env serves; returning something with no handler is refused. update_tool(name, description=...) changes a served tool's description after the mount, on the card and in tools/list, on both FastMCPs; mounted_card() is the card the mounted application serves, and the card route serves it as it stands.
• Tool access without a gateway. tool_access = True filters tools/list and refuses tools/call per caller role, and serves disable-tool/v1 / enable-tool/v1 at the gateway's paths and bodies, so modify_env_tool_access works on an env with no gateway. The gateway keeps its own code (its image installs no protocol); tst/unit/env/gateway/tool_access_parity_test.py runs the same rule cases, 5,000 random step sequences included, against both.

agentenv_protocol.computer

• Operations (@display, @capture, @click, @drag, @scroll, @type_text, @press; the pointer and keyboard groups; @set_display; @run_in_session and @write_files with Linux defaults), checked when the class is defined, chosen per platform by predicates (os, distribution, version, arch, display).
• The seven computer_* tools and the group tools, with structured results, output schemas, a one-line summary as text, and the errors table's codes and applied.
• Frames (f<n>, native crops that map back exactly, last 64 kept, stale on a display change, each caller's latest), coordinate spaces (frame and norm, refused rather than clamped, the default written into tool descriptions), the input queue (one call at a time, input_busy, others_since_frame).
• Framework extensions: computer-ready (real checks; tools refuse device_unavailable until they pass once), computer-display, computer-spaces, computer-keyboard, computer-exec (idempotent by request_id, output capped, timed out; a background app's open pipes are drained rather than waited on).
• Data plane: data/add into the session user's home from inline bytes, https URLs and read grants, atomic with path checks; data/get of the settings' data_paths, inline or under a write_namespace grant.
• Layout. The driver class (driver.py) holds the features and hooks. Its state lives in one ComputerRuntime (self._computer: operations, frames, queue, readiness, held keys, command runs). card.py decides what is served, through the hooks above. results.py has each tool's result model, which gives both its output schema and its result. checks.py has the definition-time checks. The mixins (tools.py, extensions.py, dataplane.py) declare what they read from the driver.
• x and y are optional on click and scroll in every schema, with a description saying when they are needed: a computer without computer-pointer/v1 refuses a call that leaves them out (invalid_argument).
• An example driver, packages/agentenv-protocol/examples/computer_x11/: the X11 driver the end-to-end runs used (mss capture, XTEST input through python-xlib, both groups, an XTEST ready check, reconnect on a dropped X connection), an image that runs it on Xvfb beside a Tk practice app, and a README. test_computer_x11_example.py runs it against fake mss and Xlib modules. M0.3's reference driver and computer_driver env type come later.
• Settings from AGENTENV_COMPUTER_SETTINGS. A new computer extra installs mcp, uvicorn and Pillow (uv.lock adds Pillow, from PyPI). agentenv_protocol.computer is not imported by import agentenv_protocol.

Design choices, as built

The 13 choices in the design doc stand. Where the build adds to or departs from them:

1. Shared tool-access rules: the gateway keeps its own code, one rule-case table tests both.
2. Routes for several methods: /agentenv/ext/<extension>/<op> with op, today's otherwise.
3. Error transport: the body carries the meaning; ExtensionCallError subclasses httpx.HTTPStatusError.
4. Role names: rules keyed by default and *; no role means default.
5. Frames: f<n>, last 64, latest per caller identity, which is default for every caller until FD-4386; caller_identity(context) is the seam.
6. Operation signatures as designed, called positionally.
7. Pointer group: four operations; the base class supplies computer_pointer_list (the default pointer only, no leasing).
8. Card features as class attributes, plus max_key_repeat, max_scroll_amount and layouts, which the tools needed to bound repeat, amount and the keyboard layout.
9. Default limits as designed, all in ComputerSettings.
10. Settings from one JSON env var, AGENTENV_COMPUTER_SETTINGS.
11. The computer extra is mcp, uvicorn, Pillow; mss and python-xlib go to the reference driver (M0.3).
12. Readiness gating: tools refuse until the ready checks pass once; a tool call runs the checks itself if nothing has yet, so a driver nobody polls doesn't refuse forever.
13. data/add: inline bytes, https URLs and read grants (ReadObject under the data-objects/v1 marker's read); no data/reset.

Left out, since nothing uses them yet: @list_windows (no default in this PR), the pointer and window tool parameters (leasing and windows come later), and a shipped fake screen (it lives in packages/agentenv-protocol/tests/computer_fake.py).

agentenv_protocol.computer is a normal public API with no experimental marker; a break after merge gets a !. plugin-api reads only agent_env, so it doesn't cover the protocol.

Testing

• make unit-test (the protocol's tests and core's unit tier) and make lint pass; the protocol's tests also pass under Python 3.10.
• Coverage with pytest-cov, run for this PR (not added to the dev extra): every module under computer/ 100%, errors.py and tool_access.py 100%. Every changed line in agent_env_environment.py and client.py is covered except the base _describe_registered_tool stub, which mirrors _register_tool's: an app without MCP tools refuses update_tool before it gets there.
• tst/unit/env/gateway/process_scope_socket_test.py::test_an_armed_time_trigger_keeps_firing_after_a_client_leaves failed once under full parallel load and passes alone; it waits 0.3 s for a virtual-time mark and predates this branch.
• End to end on a real X server (EC2), through agent-env and the gateway, with a real agent, and chaos testing: contract 58/58, gateway 9/9, Claude Code agent 4/4 runs, chaos 29/29. Details in feat(protocol): add the computer driver SDK #336 (comment).
• Rerun at ba1a5f5, after Greptile's findings on the example driver and the runtime (a cancelled timed hold, a click at the pointer queued behind a drag, input during a screenshot) were fixed: its own image passes the contract 58/58. With its driver code in the bundle image: chaos 29/29, gateway 9/9 and Claude Code 2/2 (Sonnet 5.5 and Opus 5.5). A real-X probe of the keyboard fixes: computer_key with A and ! types A!; a Shift held by computer_keyboard_down stays down through typing and a Shift-scroll (ABc); 59 distinct CJK characters, more than Xvfb's 19 spare keycodes, plus 10 repeats come out exact.
• Rerun at 91a7ccf, after the runtime and card refactor, on local Docker (arm64): contract 58/58, chaos 29/29, gateway 9/9, and Claude Code 2/2 (Sonnet 5.5 and Opus 5.5, all six checks each, 9 and 8 computer_* calls).

Docs

The protocol README gains the SDK changes and a "Computer drivers" section. The website should get a "Computer drivers" page covering the same: what a driver is, the operations table, what the base class serves, errors and settings.

🤖 Generated with Claude Code

Confidence Score: 5/5

The latest changes appear safe to merge.

Summary

Adds the computer driver SDK, structured errors, serving hooks, and tool-access support.

• Computer drivers provide screen operations that agents can call through shared tools.
• Computer drivers expose display, keyboard, and session-command settings through framework endpoints.
• Computer drivers can transfer files to and from the session user's home.
• Extension callers get named methods and structured details when an operation fails.
• Environments can choose their served tools and update tool details after mounting.
• Environments can filter and restrict tools by caller role without a gateway.
• An X11 example driver connects the computer SDK to a virtual desktop.

_{Reviews (12) · Last reviewed commit: "fix(protocol): find every keycode a comp..." · Reviewed by Greptile}

[socket-security]

No dependency changes detected. Learn more about Socket for GitHub.

👍 No dependency changes detected in pull request

[earakely-scale]

End-to-end and chaos results at 1c71b35

The only later commit, 659293f, realigns lines in tools.py.

The SDK was exercised as a driver author would use it. An X11 driver built on ComputerDriverEnvironment (mss capture, XTEST input through python-xlib, about 250 lines) runs in an Ubuntu 24.04 image with Xvfb and a Tk fixture app, the Fixture Desk. The fixture records everything that reaches it, so every effect is read back from the app's own state, not inferred from the driver's answers. The protocol came from a wheel built from this branch. The driver and harness are e2e scaffolding and aren't in this PR; M0.3's reference driver replaces them.

Contract on a real X server (EC2 c6i.4xlarge, agent-environments-admin): 58/58

• Every tool, against the fixture:
  • Unicode typing through the keymap (Hello, Wörld! 東京), chords and repeats;
  • clicks in frame and norm space, right and double clicks;
  • native crops whose points map back exactly;
  • scroll then select, a drag through a waypoint, the pointer group, Shift held across typed text;
  • screenshot after acting, and wait.
• Refusals leave the fixture unchanged: out of bounds, an unknown frame, a bad key with its hint, control characters, a count over the limit.
• 16 concurrent clicks all land.
• The extensions: spaces (descriptions rewritten), keyboard, display, exec (session environment, timeout, 1 MiB cap, a background GUI app, idempotent request_id, missing binary), data/add picked up by the app, data/get of the data paths, and per-role tool access with no gateway.

Through agent-env and the existing gateway (agent-env run of a bundle; deploy_env, modify_env_tool_access, env_outcome_verifier): 9/9

• The gateway proxies image blocks, structured content and error fields.
• The driver's extensions and data/add go through its /svc/desktop/... REST proxy.
• The modify_env_tool_access step hides a tool.

A real agent (deploy_agent claude-code, prompt_agent, env_outcome_verifier): 4/4 runs passed

• The task: type a name, tick Subscribe, scroll to and select Item 37 in a 100-item list, click a grid cell, and Save. Six checks are read from the fixture's state, including that the name arrived as real key events.
• Claude Code used only computer_* tools, 8–9 calls a run, with screenshot: true on its inputs to check each step without a separate screenshot call:
  • three runs on anthropic/claude-sonnet-5-5, 43–51 s each;
  • one run on anthropic/claude-opus-5-5, 47 s.

Chaos (EC2): 29/29

• X server killed under load: calls answer device_unavailable, none hangs (slowest 0.09 s). The first full success comes 0.57 s after the kill, computer-ready flips to not ready and back, and the driver stays up.
• Driver container restarted: old frame ids are refused stale_frame, and it serves again from f1.
• Flood: 154 mixed calls over three sessions in 3.1 s with no errors and all 64 clicks landed. 20 concurrent typed words never interleave. With a 0.3 s queue wait, 7 of 8 one-second drags are refused input_busy (retryable, applied no), and none is half delivered.
• Fuzz: 1,000 random malformed tool calls and garbage REST bodies. No transport failure, every error carries a known code, and the driver is still ready.
• Network with netem 150 ms ± 50 ms delay and 3% loss: all calls succeed, and a 2 MB response is capped correctly.
• Data plane: 30 concurrent data/add writes to one path leave exactly one payload whole, with no temp files. A 25 MiB inline file takes 0.8 s. ../, absolute and symlink escapes are refused.
• Exec:
  • a timed-out process tree is killed;
  • 50 concurrent commands run;
  • 10 concurrent duplicates of one request_id run once;
  • 100 MB of output is capped in 0.1 s;
  • stdin is closed.
• Memory: driver RSS goes from 67 MB to 80 MB after 500 frames, and stays there after 700.
• Fixture killed mid-session: input is still delivered.

Fixed on this branch because of these runs

• 441ac12: a driver's own _button helper silently replaced the base class's, so every click failed. Base-class internals are now _computer_*, and a driver that redefines one is refused when it is defined.
• 9aee10c: frames held raw pixels, 300 MB after 500 frames at 1280×800 and over 400 MB at the bench's 1920×1080. They now keep their PNG (80 MB).

For M0.3, not this PR

• The bundle loader finds an env's name only from @environment_card(name=), not @computer_driver(name=), so the e2e env needed environment_name in env.toml. The computer_driver env type should read it.
• A driver running as PID 1 doesn't reap the orphans a timed-out command leaves (they're killed, but stay zombies). The reference image should run the driver under an init or as a session service.