feat(mothership): answer a long workflow tool run still running, and stop detached runs on Stop by waleedlatif1 · Pull Request #9507 · simstudioai/sim

waleedlatif1 · 2026-10-31T08:00:35Z

Summary

A run_workflow, run_workflow_until_block, run_from_block or run_block call held the chat turn until its run settled, up to the execution limit. A synchronous workflows run from the CLI can already detach after --wait-timeout (#9464), and the worker waits on it in 240 s slices (simstudioai/mothership#872). This PR does the same for the workflow tools, and fixes Stop for both kinds of detached run.

Workflow tools answer a long run still running. Sim now declares the workflow-tool-wait-timeout client capability. It honors the waitTimeoutMs that the worker names on a workflow tool frame. The contract is synced from the mothership protocol.ts.

Server-run call: the handler gives the run its execution id before it starts. Once the wait passes and the run's log exists, the call answers { status: "running", executionId, workflowId }. A run still in preflight or startup is answered once it becomes readable, or with however it settles. The run goes on in-process. The tool execution keeps holding it (retainSandboxExecution), so the call's lease and Stop still reach it until it settles.
Browser-run call: once the wait passes and a browser holds the call's execution claim, the call is settled as background. This is the same detach a browser sends on pagehide. The browser's run goes on, still bound to the call.
Status field: a background workflow completion now also carries status: "running" next to its ids. The worker waits on that shape.
No change for other frames: a frame without waitTimeoutMs, or with a malformed value, holds the call as before.

Fix: Stop now stops a detached run. Once a sliced workflows run detaches, no call's abort signal reaches it. Stop delivery cancelled only browser runs bound to open calls. So a Stop in the middle of a slice ended the chat but left the draft run going until it finished.

Stop now also cancels the runs that this chat run's calls started and left running:

a workflows run that the CLI answered with exit 4 and its run record;
a workflow tool that answered still running.

A run the chat only waited on (workflows runs wait) is excluded, because something else started it. The cancel is the same in-process abort plus durable cancel flag that a browser run gets.

Companion: simstudioai/mothership#881

Type of Change

Testing

Unit tests:

mutations.test.ts:
- once the wait passes, answer still running by the run's id, and keep holding the run;
- answer running only once the run's log exists;
- a run that never starts answers with how it settles;
- a run that settles inside the wait answers normally;
- a bound server-run id is reused;
- with no wait, the call holds.
handlers.test.ts: the frame's wait reaches only workflow tools, and a malformed wait is ignored.
repository.test.ts: which calls count as having started a detached run. This includes a run behind global flags. A wait on another run never counts.
payload.test.ts: the capability is declared.
client.test.ts and the confirm-route test: the background shape now includes status.

Integration tests, against real Postgres and Redis (workflow-client-settlement.integration.ts, bun run test:integration workflow-client-settlement, 15/15):

A browser-claimed run_workflow past a 500 ms wait goes through the real claim, settlement, published confirmation and waiter. It is answered background with { workflowId, executionId, status: running }. Its row stays delivered, claimed and unsettled, so Stop still finds it.
getDetachedWorkflowExecutions over stored tool rows returns the CLI and tool runs that detached, and leaves out a wait on another run and a settled tool.

Commands:

vitest run lib/mothership app/api/copilot lib/organizations lib/workflows/application: 274 files, 3649 passed.
tsc --noEmit (apps/sim): clean.
explicit-any ratchet: clean.
application-graph and monorepo boundaries: clean.
unused-exports: baseline shrunk by one.

Live, through the real controller (saved-run-read.live.test.ts; real runCopilotLifecycle, sim_cli handler, embedded CLI, v2 execute and runs routes, executor; scripted model; worker = the companion branch). The draft is Start, then Wait 300 s, then Wait 200 s, then a Function returning {invoice: 42, settled: "after-waits"}.

Case	What happened
declared (CLI)	`workflows run --wait-timeout 240` (exit 4, 240.1 s), then `runs wait` (exit 4, 240.05 s), then `runs wait` (exit 0, 29.1 s), then `runs get --include-output` (0.02 s). Turn 509.7 s. The model saw `completed` with the output.
tool (`run_workflow`)	The call answered still running after 240.05 s, with `executionId`. Then `runs wait` (exit 4, 240.0 s), then `runs wait` (exit 0, 29.1 s), then the read-back (0.03 s). Turn 509.6 s. The model saw `status: completed` and `output.result = {invoice: 42, settled: "after-waits"}`. The run's trigger is `copilot`. Its call's lease held until the run ended (500.4 s).
declared-stop, before the fix	The Stop button's request (`/api/copilot/chat/abort`) arrived 6 s into the first `runs wait` slice. The turn ended about 2 s later, but the run was still `running` a minute after.
declared-stop, with the fix	Same Stop. The run was `cancelled` 26 ms after the Stop, and the turn ended at 246.7 s.

Reviewers, please focus on two places:

answerWithinWait in mutations.ts: what holds the run after the early answer, and its abort signal.
detachedWorkflowExecutionId in repository.ts: which calls count as having started a run.

Known limit: when a turn's provenance registry is already incomplete, results are withheld, and the still-running answer is withheld with them. Only the execution id survives, in effect, so the worker cannot recognize it and the model sees a withheld result naming the execution. A server re-run after a takeover does not keep the frame's wait and holds the call as before.

Checklist

Code follows project style guidelines
Self-reviewed my changes
Tests added/updated and passing
No new warnings introduced
I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

vercel · 2026-10-31T08:00:41Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
docs	Ready	Preview	Oct 31, 2026 9:18am UTC

github-actions · 2026-10-31T08:01:53Z

⚠️ Cross-repo companion check

One or more companion PRs aren't merged into staging yet. Merging this without them will leave copilot and sim out of sync — merge them in lockstep.

❌ simstudioai/mothership#881 — OPEN, not merged (targets staging) — feat(worker): wait on a workflow tool's run in slices on a Sim that answers it s

cubic-dev-ai

Review completed against the latest diff

_{Reply with feedback, questions, or to request a fix.

Turn on auto-fix | Re-trigger cubic}

greptile-apps · 2026-10-31T08:04:37Z

Confidence Score: 5/5

[Medium impact] The PR appears safe to merge; no new actionable issues were found.

Summary

Workflow tools can answer long runs with their execution IDs while those runs continue. Stop also finds workflow runs left running by earlier calls.

Workflow tool calls can answer while their runs continue.
Stop cancels detached workflow runs started by the chat.

The previous threads were rechecked. The helper changes and replacement tests address their concerns; no new findings were accepted.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Workflow tool starts] --> B{Run settles within wait?}
  B -->|Yes| C[Return result]
  B -->|No| D[Wait for readable run log]
  D --> E[Return running with execution ID]
  E --> F[Worker waits on the run]
  E --> G[Run keeps its execution ownership]
  H[User presses Stop] --> I[Find open and detached runs]
  I --> J[Abort locally and record cancellation]

_{Reviews (13) · Last reviewed commit: "fix(mothership): detach only a browser r..." · Reviewed by Greptile}

waleedlatif1 · 2026-10-31T08:20:51Z

@greptile

waleedlatif1 · 2026-10-31T08:20:52Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-10-31T08:20:57Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 24 files

_{Reply with feedback, questions, or to request a fix.

Turn on auto-fix | Re-trigger cubic}

waleedlatif1 · 2026-10-31T08:27:25Z

@greptile

waleedlatif1 · 2026-10-31T08:27:26Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-10-31T08:27:31Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

waleedlatif1 · 2026-10-31T08:27:51Z

@greptile

waleedlatif1 · 2026-10-31T08:27:53Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-10-31T08:27:57Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 24 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Turn on auto-fix | Re-trigger cubic}

waleedlatif1 · 2026-10-31T08:31:20Z

@greptile

waleedlatif1 · 2026-10-31T08:31:21Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-10-31T08:31:25Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 24 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Turn on auto-fix | Re-trigger cubic}

waleedlatif1 · 2026-10-31T09:06:48Z

@greptile

waleedlatif1 · 2026-10-31T09:06:49Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-10-31T09:06:55Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 24 files

_{You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.

Turn on auto-fix | Re-trigger cubic}

… worker's wait A run_workflow* call held the turn until its run settled, up to the execution limit, while a synchronous workflows run can already detach after --wait-timeout (#9464). Sim now declares workflow-tool-wait-timeout and honors the waitTimeoutMs a workflow tool frame names (synced from the mothership contract): - Server-run call: the handler gives the run its execution id up front and, once the wait passes, answers {status: running, executionId, workflowId}. The run goes on in-process; the tool execution keeps holding it, so its lease and a Stop through the call's abort signal still reach it. - Browser-run call: once the wait passes and a browser holds the call's execution claim, the call is settled background, the detach a browser sends on pagehide. The browser's run goes on, still bound to the call, so a Stop cancels it. - A background workflow completion now carries status: running beside its ids, the shape the worker waits on. A frame without waitTimeoutMs, or one with a malformed value, holds the call as before.

…ast a call A synchronous workflows run the worker waits on in slices detaches after its first --wait-timeout (#9464, mothership#872), and from then on no call's abort signal reaches it. Stop delivery cancelled only browser workflow runs bound to open calls, so a Stop mid-slice ended the chat while the draft run went on to completion. Driven through the real controller and the Stop route (live case in the next commit), the run was still running a minute after the Stop. Stop delivery now also cancels the runs this chat run's calls started and left running: a workflows run the CLI answered still running (exit 4 and its run record), and a workflow tool answered still running. A run the chat only waited on (workflows runs wait) is not one, since something else started it. The cancel is the same in-process abort and durable cancel flag a browser run gets; a detached run registers its in-process aborter. Autopsy: the slice tests on both sides covered waits that settle, lapse or lose an answer, and none stopped the chat mid-slice; the Stop tests seeded only open browser-run calls. stop-delivery.test.ts and the live declared-stop case now cover it.

…slice through the real controller Two live cases beside the sliced draft run (opt-in, SIM_HELPERS_SMOKE and MSHIP_WORKER_ROOT): - tool: run_workflow on a Sim declaring workflow-tool-wait-timeout answers its 500 s run still running after 240 s, the worker waits on it as run-wait slices and reads it back, and the model sees the settled run's output. - declared-stop: the Stop button's request while the worker waits on a detached draft run ends the turn and cancels the run. The harness now marks the turn's tool execution and billing as the chat route does, registers and wires the turn's controller as lifecycle/start.ts does, gives the tool case a real AES key to seal its provenance, and keeps logged errors' messages in the evidence.

… without a JSON round trip

…un is dispatched Review of the wait: - A run still in authorization or admission when the wait passes is not answered running under an id nothing has used yet; the call waits until the run is handed to the executor (onDispatch) or settles. - The browser-run detacher sleeps interruptibly and stops on either outcome of its waiter, so no derived promise rejects unhandled. - The live case takes its AES key from generateRandomHex. - The mock-only Stop delivery test is dropped: the live declared-stop case drives the real Stop route, storage and cancellation, and fails without the fix (the run stays running); detachedWorkflowExecutionId keeps its unit test.

…real Postgres and Redis The browser-run detach and the detached-run read now run through the real claim, settlement, published confirmation, waiter and stored tool rows (workflow-client-settlement.integration.ts), replacing unit tests that asserted only on mocks.

…nterruptible sleep

…un is readable A run past the call's wait was answered running as soon as it was handed to the executor, before the executor wrote its run log, so an immediate workflows runs wait could find no run. The call now answers running only once the run's log exists, re-reading it each second until then or until the run settles; the dispatch hook is gone.

…annot be read

… exists, and never discard a run that settled meanwhile

…ailed detach

waleedlatif1 · 2026-10-31T09:13:23Z

@greptile

waleedlatif1 · 2026-10-31T09:13:24Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-10-31T09:13:29Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 24 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.

Turn on auto-fix | Re-trigger cubic}

github-actions Bot added the requires-mothership-merge Has a companion PR on the mothership/copilot side — merge in lockstep label Oct 31, 2026

waleedlatif1 force-pushed the feat/run-workflow-slices branch from b1614a3 to 2be49a8 Compare October 31, 2026 08:03

cubic-dev-ai Bot reviewed Oct 31, 2026

View reviewed changes

Comment thread apps/sim/lib/mothership/request/tools/workflow-client-fallback.ts Resolved

Comment thread apps/sim/lib/mothership/request/tools/workflow-client-fallback.ts Resolved

Comment thread apps/sim/lib/mothership/tools/handlers/workflow/mutations.ts Resolved

greptile-apps Bot reviewed Oct 31, 2026

View reviewed changes

Comment thread apps/sim/lib/mothership/agent-cli/saved-run-read.live.test.ts Resolved

Comment thread apps/sim/lib/mothership/request/application/stop-delivery.test.ts Resolved

vercel Bot deployed to Preview October 31, 2026 08:10

View deployment

vercel Bot temporarily deployed to Preview October 31, 2026 08:12 Inactive

vercel Bot temporarily deployed to Preview October 31, 2026 08:20 Inactive

greptile-apps Bot reviewed Oct 31, 2026

View reviewed changes

Comment thread apps/sim/lib/mothership/request/tools/workflow-client-fallback.test.ts Resolved

cubic-dev-ai Bot reviewed Oct 31, 2026

View reviewed changes

Comment thread apps/sim/lib/mothership/request/tools/workflow-client-fallback.ts Resolved

vercel Bot temporarily deployed to Preview October 31, 2026 08:27 Inactive

cubic-dev-ai Bot reviewed Oct 31, 2026

View reviewed changes

greptile-apps Bot reviewed Oct 31, 2026

View reviewed changes

Comment thread apps/sim/lib/mothership/tools/handlers/workflow/mutations.ts Resolved

waleedlatif1 force-pushed the feat/run-workflow-slices branch from ab242a1 to 2b23efe Compare October 31, 2026 08:31

cubic-dev-ai Bot reviewed Oct 31, 2026

View reviewed changes

vercel Bot temporarily deployed to Preview October 31, 2026 08:32 Inactive

vercel Bot deployed to Preview October 31, 2026 09:05

View deployment

vercel Bot temporarily deployed to Preview October 31, 2026 09:06 Inactive

cubic-dev-ai Bot reviewed Oct 31, 2026

View reviewed changes

Comment thread apps/sim/lib/mothership/request/tools/workflow-client-fallback.ts Resolved

Comment thread apps/sim/lib/mothership/request/tools/workflow-client-fallback.ts Resolved

waleedlatif1 added 14 commits October 31, 2026 02:13

test(mothership): type the stopped run fixture and keep logged errors…

7543bf3

… without a JSON round trip

refactor(mothership): wait on a workflow tool's run with the shared i…

04886f7

…nterruptible sleep

fix(mothership): find a detached workflows run behind global CLI flags

3cc2d8d

fix(mothership): keep waiting on a workflow tool's run when its log c…

b0e7295

…annot be read

fix(mothership): answer a browser run still running only once its log…

8c1169f

… exists, and never discard a run that settled meanwhile

fix(mothership): retry a failed call read in the browser-run detacher

41e4d9a

fix(mothership): keep watching a browser-run call that is still pending

20a3a28

fix(mothership): detach only a browser run still going, and retry a f…

08c08f0

…ailed detach

waleedlatif1 force-pushed the feat/run-workflow-slices branch from 3e10a62 to 08c08f0 Compare October 31, 2026 09:13

cubic-dev-ai Bot reviewed Oct 31, 2026

View reviewed changes

vercel Bot deployed to Preview October 31, 2026 09:18

View deployment

waleedlatif1 merged commit 3e23d39 into staging Oct 31, 2026
130 of 132 checks passed

waleedlatif1 deleted the feat/run-workflow-slices branch October 31, 2026 17:39

Conversation

waleedlatif1 commented Oct 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Type of Change

Testing

Checklist

Uh oh!

vercel Bot commented Oct 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 31, 2026

⚠️ Cross-repo companion check

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

greptile-apps Bot commented Oct 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 5/5

Summary

Diagram

Uh oh!

Uh oh!

Uh oh!

waleedlatif1 commented Oct 31, 2026

Uh oh!

waleedlatif1 commented Oct 31, 2026

Uh oh!

cubic-dev-ai Bot commented Oct 31, 2026

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

waleedlatif1 commented Oct 31, 2026

Uh oh!

waleedlatif1 commented Oct 31, 2026

Uh oh!

cubic-dev-ai Bot commented Oct 31, 2026

Uh oh!

waleedlatif1 commented Oct 31, 2026

Uh oh!

waleedlatif1 commented Oct 31, 2026

Uh oh!

cubic-dev-ai Bot commented Oct 31, 2026

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

waleedlatif1 commented Oct 31, 2026

Uh oh!

waleedlatif1 commented Oct 31, 2026

Uh oh!

cubic-dev-ai Bot commented Oct 31, 2026

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

waleedlatif1 commented Oct 31, 2026

Uh oh!

waleedlatif1 commented Oct 31, 2026

Uh oh!

cubic-dev-ai Bot commented Oct 31, 2026

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

waleedlatif1 commented Oct 31, 2026

Uh oh!

waleedlatif1 commented Oct 31, 2026 •

edited

Loading

vercel Bot commented Oct 31, 2026 •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

greptile-apps Bot commented Oct 31, 2026 •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading