feat: add optional pi agent SDK support

CHANGELOG.md

@@ -7,6 +7,21 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Added
+
+- Optional pi Agent SDK adapter for isolated LiteLLM runs, Docker execution, structured judging, cancellation, and turn limits.
+- Provider-specific runner option types with backwards-compatible Claude defaults.
+
+### Changed
+
+- Apply the `on-success` Docker cleanup policy uniformly: adapters keep the container unless the run completed with a successful result.
+- Take provider option defaults from `adapter.defaultOptions` only; the Claude adapter owns the isolated `settingSources: []` default.
+
+### Fixed
+
+- Scrub configured credential values of at least eight characters from pi messages, errors, telemetry, and judge inputs, including nested error causes.
+- Keep scrubbed DOMExceptions as real DOMExceptions so telemetry span finalization cannot throw on abort errors.
+
 ## [0.3.1]
 
 ### Fixed

README.md

@@ -1,6 +1,6 @@
 # @metamask/agent-runner
 
-Reusable TypeScript runner for `@anthropic-ai/claude-agent-sdk` with optional Langfuse/OpenTelemetry lifecycle support.
+Reusable TypeScript runner for the Claude Agent SDK and optional pi Agent SDK, with Langfuse/OpenTelemetry lifecycle support and Docker execution.
 
 This package wraps the Claude Agent SDK `query()` behind a provider adapter, normalizes the streamed message types into a discriminated union, collects result metadata, and exposes `flush()` / `shutdown()` so short-lived CI and eval processes do not lose telemetry spans.
 
@@ -79,7 +79,84 @@ const result = await runner.runAgent({
 console.log(result.sessionId, result.totalCostUsd, result.durationMs);
 ```
 
-By default the runner passes `settingSources: []` to the Claude SDK for isolated settings. Callers can override that in `defaultOptions` or per-run `options` when they intentionally want SDK settings loaded from other sources.
+By default the Claude adapter passes `settingSources: []` to the SDK for isolated settings. Callers can override that in `defaultOptions` or per-run `options` when they intentionally want SDK settings loaded from other sources.
+
+## Pi usage
+
+Pi is an optional peer dependency. Claude-only installations keep their existing Node requirements and do not need pi installed. Pi execution requires Node.js 22.19.0 or newer and the supported SDK version:
+
+```sh
+npm install @metamask/agent-runner @earendil-works/pi-coding-agent@0.83.0
+export LITELLM_BASE_URL=http://localhost:4000
+export LITELLM_API_KEY=your-key
+```
+
+The endpoint can include `/v1`; the adapter does not append it twice. Pi uses the OpenAI Chat Completions protocol through LiteLLM. Native pi providers and stored OAuth credentials are not loaded.
+
+```ts
+import { createAgentRunner, createPiAdapter } from '@metamask/agent-runner';
+
+const runner = createAgentRunner({
+  adapter: createPiAdapter(),
+  defaultOptions: {
+    model: 'your-litellm-model-alias',
+    tools: ['read', 'bash', 'edit', 'write'],
+    maxTurns: 10,
+  },
+  sandbox: { type: 'docker', image: 'node:22-bookworm' },
+});
+
+const result = await runner.runAgent({
+  prompt: 'Run the tests and fix failures.',
+  signal: AbortSignal.timeout(120_000),
+});
+
+const verdict = await runner.judge(result, {
+  rubric: 'Evaluate correctness.',
+  scoreFields: [{ name: 'correctness', min: 0, max: 10 }],
+});
+```
+
+The adapter determines the runner's option and prompt types. Existing Claude calls remain unchanged. Pi accepts string prompts and these package-owned options:
+
+| Option          | Behavior                                                                                                                              |
+| --------------- | ------------------------------------------------------------------------------------------------------------------------------------- |
+| `model`         | Required LiteLLM model alias. Judging inherits it from runner defaults.                                                               |
+| `cwd`           | Absolute working directory, defaulting to `process.cwd()`. Docker maps it into the mounted workspace.                                 |
+| `systemPrompt`  | Replacement system prompt.                                                                                                            |
+| `tools`         | Exact built-in allowlist: `read`, `bash`, `edit`, `write`, `grep`, `find`, `ls`. Defaults to the first four; `[]` disables all tools. |
+| `maxTurns`      | Positive integer limiting model turns, including their tool execution. Unlimited for task runs when omitted.                          |
+| `contextWindow` | Declared model context capacity, default 128000 tokens.                                                                               |
+| `maxTokens`     | Maximum output tokens per generation, default 8192. This is not a whole-run limit.                                                    |
+| `reasoning`     | Whether the model supports reasoning, default `false`.                                                                                |
+| `input`         | Declared model modalities, default `['text']`. This adapter currently accepts text prompts only.                                      |
+| `cost`          | Optional USD-per-million-token prices: `input`, `output`, `cacheRead`, `cacheWrite`.                                                  |
+
+Unknown options, command-scoped tool selectors, and Claude permission options are rejected before execution. They are never silently ignored. Direct pi tools execute with the host process's permissions; use Docker when isolation is required.
+
+Every pi run uses a fresh in-memory session. Settings, extensions, skills, context files, and prompt templates are not discovered. Automatic retries and compaction are disabled. Persistence, resume, custom tools, and interactive steering are not supported.
+
+Pi emits the same normalized messages and telemetry as Claude. Reaching `maxTurns` produces `result.success: false`; cancellation or a throwing `onMessage` callback returns a partial result and stops active work. Cost is omitted when pricing for consumed tokens is unknown. Explicit zero prices report zero.
+
+`runAgent({ signal })` and `judge(..., { signal })` cancel only pi runs. The Claude adapter does not observe `signal` yet, so passing one there is a silent no-op.
+
+### Pi sandbox and judge behavior
+
+Docker installs pi 0.83.0 and copies the shared pi execution modules into the container. `bridge.sdkVersion`, if supplied, must match. With `bridge.install: false`, the runner performs no version check, so make sure the bridge directory already contains a compatible install. The Node version check applies inside the container, so a Node 20 host can still launch a supported pi container.
+
+Pi's default forwarded environment is `LITELLM_BASE_URL`, `LITELLM_API_KEY`, `HTTP_PROXY`, `HTTPS_PROXY`, and `NO_PROXY`. An explicit `forwardEnv` list or `false` overrides it. Credentials are passed through the existing Docker environment-file mechanism, not serialized into query options. The existing Docker security caveats still apply: enabled tools can read credentials inside the container.
+
+Pi judging uses only a terminating `submit_judgment` tool and rejects caller tool customization. It inherits model settings, but not task tools or task turn limits. Its default limit is five turns. It uses the runner's sandbox; `judge(..., context, { sandbox: false, signal })` can override sandboxing and cancel the judge. A sandboxed judge starts its own container rather than reusing the task container.
+
+Claude judging and the legacy Claude fallback for custom adapters without `runStructured()` remain unchanged. To judge a transcript with another provider, use a separate runner. Custom adapters can implement `runStructured()`, `getStructuredDefaults()`, and `getRunMetadata()`. Adapter `defaultOptions` are merged under per-run options; adapters that supply none change nothing.
+
+To run the local-protocol Docker smoke test after building:
+
+```sh
+RUN_DOCKER_TESTS=1 yarn vitest run src/sandbox/docker/pi-integration.test.ts --coverage.enabled=false
+```
+
+It requires a working Docker daemon and `host.docker.internal` connectivity, but no external model credentials.
 
 ## Telemetry usage
 
@@ -277,7 +354,7 @@ The run loop catches all errors and returns them in the result rather than throw
 
 ## Docker sandbox
 
-The Claude adapter can execute agent runs inside a Docker container instead of
+The Claude and pi adapters can execute agent runs inside a Docker container instead of
 the host process. This isolates filesystem writes, environment variables, and
 spawned subprocesses (including the Claude Agent SDK's own tools) from the
 host. The sandbox is opt-in: when no `sandbox` is configured the adapter runs
@@ -347,7 +424,7 @@ const result = await runner.runAgent({
 | `shmSize`          | `string`                              | Size of `/dev/shm` (e.g. `512m`, `2g`).                                                                                                                  |
 | `unsafeDockerArgs` | `string[]`                            | Extra raw arguments forwarded to `docker run`. Not validated.                                                                                            |
 | `setupCommands`    | `string[]`                            | Shell commands executed inside the container before the agent starts. Useful for installing extra dependencies or seeding state.                         |
-| `cleanup`          | `'always' \| 'on-success' \| 'never'` | When to remove the container. Defaults to `'always'`. `'on-success'` keeps the container on failure for inspection.                                      |
+| `cleanup`          | `'always' \| 'on-success' \| 'never'` | When to remove the container. Defaults to `'always'`. `'on-success'` keeps the container unless the run completed with a successful result.              |
 | `bridge`           | `DockerSandboxBridgeConfig`           | Bridge runtime options: `install`, `nodeCommand`, `npmCommand`, `sdkVersion`. Defaults install the host's installed Claude Agent SDK version on the fly. |
 
 `SandboxConfig` is a discriminated union on `type`; today only `'docker'`

eslint.config.mjs

@@ -50,7 +50,13 @@ const config = createConfig([
   // sandbox modules all run host-side and need the nodejs eslint extends so
   // they can call into the platform without per-line disables.
   {
-    files: ['src/adapters/claude-adapter.ts', 'src/sandbox/**/*.ts'],
+    files: [
+      'src/adapters/*-adapter.ts',
+      'src/pi-runtime.ts',
+      'src/credential-redactor.ts',
+      'src/judge/executor.ts',
+      'src/sandbox/**/*.ts',
+    ],
     ignores: ['**/*.test.ts'],
     extends: nodejs,
     rules: {

package.json

@@ -62,6 +62,7 @@
   },
   "devDependencies": {
     "@arethetypeswrong/cli": "^0.15.3",
+    "@earendil-works/pi-coding-agent": "0.83.0",
     "@lavamoat/allow-scripts": "^3.0.4",
     "@lavamoat/preinstall-always-fail": "^2.0.0",
     "@metamask/auto-changelog": "^6.1.0",
@@ -91,6 +92,14 @@
     "vite": "^8.0.8",
     "vitest": "^4.1.4"
   },
+  "peerDependencies": {
+    "@earendil-works/pi-coding-agent": "0.83.0"
+  },
+  "peerDependenciesMeta": {
+    "@earendil-works/pi-coding-agent": {
+      "optional": true
+    }
+  },
   "resolutions": {
     "fflate": "0.8.2"
   },
@@ -100,6 +109,8 @@
   "packageManager": "yarn@4.16.0+sha256.ba05224324578801b9cc98170d64aa50b9a36733b440fb0942306da3fbbdc7d1",
   "lavamoat": {
     "allowScripts": {
+      "@earendil-works/pi-coding-agent>@earendil-works/pi-ai>@google/genai": false,
+      "@earendil-works/pi-coding-agent>@earendil-works/pi-ai>@google/genai>protobufjs": false,
       "@lavamoat/preinstall-always-fail": false,
       "@langfuse/otel>@opentelemetry/exporter-trace-otlp-http>@opentelemetry/otlp-transformer>protobufjs": false,
       "@opentelemetry/sdk-node>@opentelemetry/exporter-logs-otlp-grpc>@grpc/grpc-js>@grpc/proto-loader>protobufjs": false,

src/adapters/claude-adapter.test.ts

@@ -679,6 +679,22 @@ describe('createClaudeAdapter', () => {
       expect(adapter.capabilities).toStrictEqual({ sandboxes: ['docker'] });
     });
 
+    it('defaults to isolated settingSources', () => {
+      const adapter = createClaudeAdapter();
+      expect(adapter.defaultOptions).toStrictEqual({ settingSources: [] });
+    });
+
+    it('describes runs for telemetry metadata', () => {
+      const adapter = createClaudeAdapter();
+      expect(adapter.getRunMetadata?.({})).toStrictEqual({
+        model: 'unknown',
+        maxTurns: 0,
+      });
+      expect(
+        adapter.getRunMetadata?.({ model: 'claude-x', maxTurns: 3 }),
+      ).toStrictEqual({ model: 'claude-x', maxTurns: 3 });
+    });
+
     it('runs in-process and bypasses the sandbox machinery when no sandbox is set', async () => {
       claudeMocks.query.mockReturnValueOnce(yieldMessages([]));
 
@@ -855,7 +871,9 @@ describe('createClaudeAdapter', () => {
     it('closes the container on bridge success when cleanup is `on-success`', async () => {
       const handle = makeHandle();
       sandboxMocks.createDockerSandbox.mockResolvedValueOnce(handle);
-      sandboxMocks.runDockerClaudeBridge.mockReturnValueOnce(makeBridge([]));
+      sandboxMocks.runDockerClaudeBridge.mockReturnValueOnce(
+        makeBridge([{ type: 'result', subtype: 'success' }]),
+      );
 
       const adapter = createClaudeAdapter();
       await drain(
@@ -870,6 +888,26 @@ describe('createClaudeAdapter', () => {
       expect(handle.unregisterCleanup).not.toHaveBeenCalled();
     });
 
+    it('keeps the container when the result failed under cleanup `on-success`', async () => {
+      const handle = makeHandle();
+      sandboxMocks.createDockerSandbox.mockResolvedValueOnce(handle);
+      sandboxMocks.runDockerClaudeBridge.mockReturnValueOnce(
+        makeBridge([{ type: 'result', subtype: 'error_max_turns' }]),
+      );
+
+      const adapter = createClaudeAdapter();
+      await drain(
+        adapter.run({
+          prompt: 'sandboxed',
+          options: {},
+          sandbox: { ...baseSandbox, cleanup: 'on-success' },
+        }),
+      );
+
+      expect(handle.close).not.toHaveBeenCalled();
+      expect(handle.unregisterCleanup).toHaveBeenCalledTimes(1);
+    });
+
     it('re-throws close errors when the bridge run succeeds but cleanup fails', async () => {
       const handle = makeHandle();
       handle.close.mockRejectedValueOnce(new Error('close boom'));

src/adapters/claude-adapter.ts

@@ -35,7 +35,20 @@ import { translateClaudeSdkMessages } from './claude-message-translator.js';
 export function createClaudeAdapter(): ProviderAdapter {
   return {
     name: 'claude',
+    // Isolated settings by default so the SDK does not read user or project
+    // configuration files unless a caller opts in via run options.
+    defaultOptions: { settingSources: [] },
     capabilities: { sandboxes: ['docker'] },
+    /**
+     * Describes the Claude run for telemetry.
+     *
+     * @param options - Claude options.
+     * @returns Model and turn limit.
+     */
+    getRunMetadata: (options) => ({
+      model: options.model ?? 'unknown',
+      maxTurns: options.maxTurns ?? 0,
+    }),
     /**
      * Runs the Claude query and yields translated agent messages.
      *
@@ -114,6 +127,7 @@ async function* runWithSandbox(
   let bridgeError: unknown;
   let bridgeFailed = false;
   let bridgeCompleted = false;
+  let resultSucceeded = false;
   let closeError: unknown;
   let closeFailed = false;
   try {
@@ -124,7 +138,12 @@ async function* runWithSandbox(
       request: { prompt: prepared.prompt, options: prepared.options },
     });
 
-    yield* translateClaudeSdkMessages(bridgeMessages);
+    for await (const message of translateClaudeSdkMessages(bridgeMessages)) {
+      if (message.type === 'result') {
+        resultSucceeded = message.success;
+      }
+      yield message;
+    }
     bridgeCompleted = true;
   } catch (cause) {
     bridgeError = cause;
@@ -133,7 +152,7 @@ async function* runWithSandbox(
     const consumerAborted = !bridgeCompleted && !bridgeFailed;
     if (
       consumerAborted ||
-      shouldCloseContainer(normalized.cleanup, bridgeCompleted)
+      shouldCloseContainer(normalized.cleanup, bridgeCompleted, resultSucceeded)
     ) {
       try {
         await handle.close();
@@ -157,7 +176,7 @@ async function* runWithSandbox(
 /**
  * Decides whether the sandbox container should be removed at the end
  * of a run based on the cleanup policy and whether the bridge run
- * completed naturally.
+ * completed naturally with a successful result.
  *
  * This function ONLY governs the non-abort cases. When the consumer
  * aborts iteration early (`break`, `iterator.return()`, or an
@@ -167,19 +186,21 @@ async function* runWithSandbox(
  * running.
  *
  * @param cleanup - Cleanup policy from the normalized sandbox config.
- * @param succeeded - Whether the bridge completed naturally without
+ * @param completed - Whether the bridge completed naturally without
  *   throwing and without being abandoned mid-stream.
+ * @param succeeded - Whether the run produced a successful result.
  * @returns Whether to call `handle.close()`.
  */
 function shouldCloseContainer(
   cleanup: 'always' | 'on-success' | 'never',
+  completed: boolean,
   succeeded: boolean,
 ): boolean {
   switch (cleanup) {
     case 'always':
       return true;
     case 'on-success':
-      return succeeded;
+      return completed && succeeded;
     case 'never':
       return false;
     default: