Skip to content

fix(onlion): a stalled request body cannot hold an invocation; slow reports name their stage - #139

Merged
wefio merged 3 commits into
mainfrom
fix/body-deadline-and-stage
Oct 30, 2026
Merged

wefio merged 3 commits into
mainfrom
fix/body-deadline-and-stage

Conversation

@wefio

@wefio wefio commented Oct 30, 2026

Copy link
Copy Markdown
Owner

Doc owner: docs/decisions/implemented/2026-10-30-body-deadline-and-stage.{md,zh-CN.md,terms.yaml} (added in this PR).

Split of the 19:34-19:50 window (12 kills at 120 s):

  • 7 rows, 601-667 MB — business logic, the embedding chain (already addressed by the per-invocation breaker and the fold deadline).
  • 5 rows, constant 51.3359 MB — memory never moved, i.e. the invocation was still waiting for its body. readJson awaited reader.read() with no deadline: a stalled upload held the entire budget and died at the ceiling with nothing to show.

Fixes: BODY_TIMEOUT_MS (30 s, NMG_BODY_TIMEOUT_MS) → 408 request_body_timeout with the stream cancelled; and the 60 s slow-request line now carries stage_isolate (the measured stage the isolate is inside), so the next kill names its stage instead of needing a code read.

Not verified

Not deployed. The 51 MB reading is consistent with the stalled-body mechanism but I have not seen a per-row log proving it for these exact five; the stage field is what will prove or disprove it next window. The Y group could also be an edge-side wait that never reaches the handler — in that case no nmg_request line appears and the fix is irrelevant rather than wrong.

wefio added 3 commits October 30, 2026 19:58
…to the ceiling

The 19:34-19:50 window split its twelve 120 s kills in two: seven at 601-667 MB
(business logic, the embedding chain) and five at a constant 51.3359 MB - a
number that never moved, which is what an invocation looks like when it is still
waiting for its request body. readJson awaited reader.read() with no deadline of
its own, so an upload that stopped halfway held the whole budget and was then
killed with nothing to show.

readJson now gives the body BODY_TIMEOUT_MS (30 s, NMG_BODY_TIMEOUT_MS to
change) and answers 408 request_body_timeout, cancelling the stream.

Also: the 60 s slow-request line reports `stage_isolate`, the measured stage the
isolate is currently inside. Every 120 s kill so far needed a code read to say
where it was; one string assignment per measured stage turns that into a field.
On this platform (fresh isolate per request) the reading is exact; where an
isolate is reused it is approximate, which the field name says.

Both are additive; 37 PASS.
…n production

The 2026-10-30 regression - callbacks.wasm deferred behind a factory, so
`new WebAssembly.Instance(fn, ...)` threw and every SQLite-backed /search
answered 503 in 75-150 ms - was invisible to `npm run verify`: Miniflare runs the
wrangler build, which never executes the inline-wasm plugin. Only the endpoint
found it.

`test-edgeone-package.mjs` builds the EdgeOne function and asserts the artefact,
not the source:
- callbacks.wasm is inlined as an eager WebAssembly.Module (the loader imports it
  as one);
- sqlite3.wasm is compiled on first use, so a request that never reads a
  partition does not decode or compile 1.5 MiB;
- every inline base64 payload still compiles to a real module with exports;
- the OPFS installers are gated off via globalThis.sqlite3ApiConfig - the only
  place the loader reads that setting; init()'s argument is silently ignored,
  which is exactly what the first attempt at that flag did;
- the built entry still exports onRequest.

Also tried and rejected: a Miniflare test for the stalled-body deadline. workerd
buffers the request body before invoking the handler, so the stream never reaches
readJson and the test hangs instead of exercising the deadline. The deadline is
verified by reading the code path and by one live behaviour (408 within the
window); a test that a no-op would pass is worse than no test, and one that hangs
the suite is worse still.

38 PASS (verify now includes the packaging test).
@wefio
wefio merged commit 0c6057d into main Oct 30, 2026
8 checks passed
@wefio
wefio deleted the fix/body-deadline-and-stage branch October 30, 2026 12:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant