From 392dbded0ca9b965468c4d4fc01294387b160143 Mon Sep 17 00:00:00 2001 From: wefio <48851810+wefio@users.noreply.github.com> Date: Fri, 30 Oct 2026 17:07:18 +0800 Subject: [PATCH] fix(onlion): the fold budget became a deadline, and the request line records the method Two production facts, both from log lines: 1. Every request since the stabilisation deploy reports execution: "inline" - the Makers Node cloud function context has no waitUntil (documented properties: uuid / params / env / clientIp / server). The waitUntil the previous record cited belongs to Pages Edge Functions, a different runtime. So "the fold runs after the response" never took effect; the accept folds inline, and the 15:00 distribution (p50 1.02 s, tail 6.5 s) was that shape. 2. The fold budget was a start gate: `Date.now() - startedAt < foldBudget` only decided whether a fold began, and a fold beginning at 19.9 s still ran a whole partition rewrite plus a 32-record backfill. That is the 14:09-14:15 mechanism (Adds 11-25 s, tail 100.7 s, five 120 s kills, every stack in foldPending). foldPending now takes { deadline }, refuses to start past it, and checks it between entries - stopping leaves the remainder in the log, which is safe because the entry is already durable and folding is idempotent. The accept passes startedAt + foldBudget, /admin/fold passes 100 s. The waitUntil branch stays for runtimes that provide one; nothing depends on it. Also: two /add and /search requests answered 404 in 24-47 ms on dead-cold isolates (81 MB, SQLite never loaded). Those routes accept POST only, so they were not work requests; nmg_request now carries method, so the next one is identified instead of guessed. Recorded in docs/decisions/implemented/2026-10-30-fold-budget-is-a-deadline.* --- .../2026-10-30-fold-budget-is-a-deadline.md | 80 +++++++++++++++++++ ...10-30-fold-budget-is-a-deadline.terms.yaml | 4 + ...6-10-30-fold-budget-is-a-deadline.zh-CN.md | 49 ++++++++++++ onlion/backend/worker.mjs | 45 ++++++++--- onlion/scripts/test-add-log-mode.mjs | 25 ++++-- 5 files changed, 185 insertions(+), 18 deletions(-) create mode 100644 docs/decisions/implemented/2026-10-30-fold-budget-is-a-deadline.md create mode 100644 docs/decisions/implemented/2026-10-30-fold-budget-is-a-deadline.terms.yaml create mode 100644 docs/decisions/implemented/2026-10-30-fold-budget-is-a-deadline.zh-CN.md diff --git a/docs/decisions/implemented/2026-10-30-fold-budget-is-a-deadline.md b/docs/decisions/implemented/2026-10-30-fold-budget-is-a-deadline.md new file mode 100644 index 00000000..93f006b7 --- /dev/null +++ b/docs/decisions/implemented/2026-10-30-fold-budget-is-a-deadline.md @@ -0,0 +1,80 @@ +# The fold budget is a deadline; and this runtime has no waitUntil + +[简体中文](2026-10-30-fold-budget-is-a-deadline.zh-CN.md) · [Terms](2026-10-30-fold-budget-is-a-deadline.terms.yaml) + +**Status:** implemented +**Approved:** explicit + +## Problem + +Two facts, both from production log lines rather than from reading code. + +**One: the previous round's premise was wrong for this runtime.** The always-on request line +reports `execution: "inline"` on every request since the stabilisation deploy — the adapter's +`context.waitUntil` probe never found the hook. That is correct behaviour on a platform that does +not supply it: the Makers **Node cloud function** context documents `uuid / params / env / +clientIp / server{region,requestId,geo}` and nothing else. The `waitUntil` the earlier record cited +belongs to **Pages Edge Functions** (`FetchEvent`) and to the Pages edge handler — a different +runtime from the Node cloud function this project deploys. So "the fold runs after the response" +never took effect in production; the entry in `2026-10-30-fold-runs-after-the-response` claimed +otherwise and is superseded here. The 15:00 Add p50 of 1.02 s with a 6.5 s tail was **inline** +folding at a 1-in-8 schedule, and the 16:59 rows show the same: `execution: "inline"`. + +**Two: the fold budget only gated the start.** `if (dueBySchedule && Date.now() - startedAt < +foldBudget)` decided whether a fold _began_; a fold beginning at 19.9 s then ran a whole partition +rewrite plus its 32-record backfill. That is the mechanism behind the 14:09-14:15 numbers (Adds 11-25 +s, tail 100.7 s, five kills at 120 s, every stack in `foldPending`) and it survived the previous two +rounds because nothing bounded the fold after it started. + +A third, smaller finding from the same logs: two `/add` and `/search` requests answered **404 in +24-47 ms on dead-cold isolates** (81 MB, `wasm_linear_mb: null` — SQLite was never touched). Our +`/add` and `/search` accept POST only, so those were not work requests; the request line now +records `method` so the next occurrence is identified instead of guessed. + +## Decision + +- **The fold takes a deadline and honours it.** `foldPending(env, userId, { deadline })` refuses to + start past it and, between applying entries, stops and leaves the remainder of the batch in the + log. The accept passes `startedAt + foldBudget`; `/admin/fold` passes its own 100 s. + Stopping is always safe: the entry is durable, folding is idempotent, and the next scheduled + accept retries it. +- **No dependency on `waitUntil`.** The branch stays for a runtime that provides one (tests do), + but the shipped path is the bounded inline fold, which is the only shape this platform supports. +- The request line gains `method`. Diagnostic only. + +## Evidence + +| before | after | +| ------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------- | +| 14:09-14:15: Adds 11-25 s, tail 40-100 s, 5 kills at 120 s, all stacks `foldPending` | a fold that runs out of budget returns `{ folded, remaining: null, deferred: "budget" }` and the rest of the log waits for the next scheduled accept | +| 15:00 p50 1.02 s / max 6.5 s: inline folding at 1-in-8, budget gating only the start | same schedule, now bounded by the same clock the caller measured | +| no `method` in the request line; two 404s of unknown origin | `method` recorded, so a 404 either becomes explainable or stays provably non-mutating | + +## What this does and does not establish + +- It does not make Add fast at 16 concurrent writers: folding is still on the request path. The + measured cost of one fold (partition rewrite + bounded backfill) is what an accept pays when the + schedule lands on it. +- `waitUntil` remains unverified for any other EdgeOne runtime; nothing here assumes it. +- The 404 rows are still unexplained. `method` is what will explain them; the guess that they are + connectivity probes is a guess. + +## Alternatives considered + +- **Deploy the Pages Edge Function runtime to get `waitUntil`.** A different product and a different + deployment path; a migration is not a way to fix a fold budget in the same week as a scored run. +- **Raise `FOLD_EVERY` so folds coincide less often.** That trades latency for a growing backlog, + which is the mechanism that hid accepted data in the last three rounds. The backstop stays folded. +- **Stop folding on the request path and add a timer.** This runtime has no timer; a cron would be a + second deployment to operate, and the fold would still need to be bounded. + +## Consequences + +- One accept in `FOLD_TRIGGER` still pays the fold, but never for longer than the accept's own + budget: past it the reply is the same 200 with the log longer. +- The fold's persisted "deferred" outcome is visible in `/admin/fold`'s counters, which is the + drain path a rollback needs. +- `nmg_request` now carries `method`, `execution`: both are diagnostic-only and cost no round trip. +- The two earlier records (`fold-runs-after-the-response`, `accept-without-cas`'s waitUntil + argument) are superseded on the platform claim; the mechanism they added (bounded, idempotent, + write-side folding) is what this record strengthens. diff --git a/docs/decisions/implemented/2026-10-30-fold-budget-is-a-deadline.terms.yaml b/docs/decisions/implemented/2026-10-30-fold-budget-is-a-deadline.terms.yaml new file mode 100644 index 00000000..44b03aeb --- /dev/null +++ b/docs/decisions/implemented/2026-10-30-fold-budget-is-a-deadline.terms.yaml @@ -0,0 +1,4 @@ +version: 2 +terms: [] +references: + - docs/decisions/implemented/2026-10-30-fold-budget-is-a-deadline.md diff --git a/docs/decisions/implemented/2026-10-30-fold-budget-is-a-deadline.zh-CN.md b/docs/decisions/implemented/2026-10-30-fold-budget-is-a-deadline.zh-CN.md new file mode 100644 index 00000000..6e72d8f2 --- /dev/null +++ b/docs/decisions/implemented/2026-10-30-fold-budget-is-a-deadline.zh-CN.md @@ -0,0 +1,49 @@ +# 折叠预算即截止时间；且该运行时不提供 waitUntil + +[English](2026-10-30-fold-budget-is-a-deadline.md) · [Terms](2026-10-30-fold-budget-is-a-deadline.terms.yaml) + +**Status:** implemented +**Approved:** explicit + +## 问题 + +两条事实，都来自生产日志行，而非读代码推断。 + +**一：上一轮的前提对这个运行时是错的。** 稳定化部署之后，每次请求的常开日志行都报 `execution: "inline"`——适配器探测的 `context.waitUntil` 从未存在。在这类不提供该钩子的平台上是正确行为：Makers 的**Node 云函数** context 文档只列 `uuid / params / env / clientIp / server{region,requestId,geo}`。之前引用的 `waitUntil` 属于 **Pages 边缘函数**（`FetchEvent`）与 Pages 边缘 handler，和本项目部署的 Node 云函数不是同一个运行时。所以“折叠在响应之后跑”在生产从未生效；`2026-10-30-fold-runs-after-the-response` 中关于此的判断由本记录取代。15:00 那次 Add p50 1.02 s、尾部 6.5 s 是**内联**折叠在 1/8 排程下的形态，16:59 那两行同样写着 `execution: "inline"`。 + +**二：折叠预算只管起点。** `if (dueBySchedule && Date.now() - startedAt < foldBudget)` 只决定折叠**要不要开始**；19.9 s 开始的折叠照样要把整个分区重写加 32 条回填跑完。这正是 14:09-14:15 的数字（Add 11-25 s、尾 100.7 s、5 个 120 s 被杀请求，栈全在 `foldPending`）的机制，前两轮没有覆盖，是因为没有任何东西在折叠开始之后约束它。 + +同一批日志的第三个发现：两个 `/add`、`/search` 请求在**全冷 isolate** 上以 24-47 ms 返回 **404**（81 MB、`wasm_linear_mb: null`，从未碰 SQLite）。我们的 `/add`、`/search` 只接受 POST，所以那不是工作请求；请求行现在记录 `method`，让下次出现时是被识别而不是被猜。 + +## 决策 + +- **折叠接收截止时间并遵守它。** `foldPending(env, userId, { deadline })` 过了截止点不开始；在逐条应用之间检查，超时即停，把这一批剩余留在日志里。accept 传 `startedAt + foldBudget`；`/admin/fold` 传自己的 100 s。停下永远安全：条目已持久、折叠幂等、下一次排程的 accept 会重试。 +- **不依赖 `waitUntil`。** 分支保留给提供该钩子的运行时（测试提供），实际发布路径是有界的内联折叠——这也是本平台唯一支持的形态。 +- 请求行增加 `method`，仅用于诊断。 + +## 证据 + +| 改前 | 改后 | +| ---------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------- | +| 14:09-14:15：Add 11-25 s、尾 40-100 s、5 次 120 s 被杀、栈全在 `foldPending` | 超预算的折叠返回 `{ folded, remaining: null, deferred: "budget" }`，剩余条目等下一次排程 | +| 15:00 p50 1.02 s / max 6.5 s：1/8 内联折叠，预算只管起点 | 排程不变，但受调用方测量到的同一个时钟约束 | +| 请求行没有 `method`；两个来源不明的 404 | 记录 `method`，404 要么可解释、要么能证明没有副作用 | + +## 这确立了什么、没有确立什么 + +- 它没有让 16 并发写者下的 Add 变快：折叠仍在请求路径上，排程命中时 accept 仍要付一次折叠的成本。 +- 对其他 EdgeOne 运行时是否提供 `waitUntil` 未验证，本记录不假设任何一种。 +- 那两条 404 仍未解释；`method` 是让它被解释的手段，“连通性探测”目前只是猜测。 + +## 考虑过的替代方案 + +- **迁到 Pages 边缘函数运行时以取得 `waitUntil`**：这是另一个产品、另一条部署路径；迁移不是用来同周修一个折叠预算的手段。 +- **提高 `FOLD_EVERY` 让折叠更少发生**：那是拿积压换延迟，正是前三轮把已接收数据藏起来的机制；兜底仍是折叠，不是放弃。 +- **请求路径不再折叠、改用定时器**：本运行时没有定时器；cron 是第二套要运维的部署，而且折叠依然需要边界。 + +## 后果 + +- `FOLD_TRIGGER` 分之一的 accept 仍会付折叠，但绝不超过它自己的预算；超了照旧 200，日志更长。 +- 折叠“被推迟”的结果在 `/admin/fold` 的计数器里可见，而回滚需要的排空正是走那条路。 +- `nmg_request` 现在带 `method` 与 `execution`，两者都只用于诊断，不产生额外往返。 +- 早前两份记录（`fold-runs-after-the-response`、`accept-without-cas` 里的 waitUntil 论证）在平台事实上被取代；它们引入的机制（有界、幂等、写在写侧的折叠）由本记录加强。 diff --git a/onlion/backend/worker.mjs b/onlion/backend/worker.mjs index 7a03960d..81678ed4 100644 --- a/onlion/backend/worker.mjs +++ b/onlion/backend/worker.mjs @@ -743,7 +743,21 @@ async function inlineEntryVectors(env, messages) { * Entries whose receipt already exists are skipped, which is what makes a fold * that committed and then crashed before deleting its entries safe to repeat (I3). * Their objects are removed only after the commit, never before. */ -async function foldPending(env, userId) { +/** Folds one batch of waiting entries into the partition. + * + * `deadline` is what makes the budget real. The caller's budget alone only decided whether the + * fold *starts*; a fold that began at the last millisecond then ran the whole partition rewrite + * and its backfill, which is how 2026-10-30 produced Adds at 40-100 s and five 120 s kills while + * claiming a 20 s budget. Every expensive step now checks the clock first and stops, leaving the + * rest of the log for the next scheduled attempt - the entry is already durable, so stopping is + * always safe, and only the ordering of work changes. + * + * This platform supplies no `waitUntil` to a Node cloud function (the EventContext documented + * for Makers lists uuid/params/env/clientIp/server only), so folding after the response is not + * available here; a bounded fold is what keeps the accept inside the invocation ceiling. */ +async function foldPending(env, userId, { deadline = null } = {}) { + const outOfTime = () => deadline !== null && Date.now() > deadline; + if (outOfTime()) return { folded: 0, remaining: null, deferred: "budget" }; const client = embeddingClient(env); const key = keyFor(userId); const waiting = await listPending(env.MEMORY, key, { limit: FOLD_BATCH }); @@ -760,6 +774,7 @@ async function foldPending(env, userId) { // call: safe, but it looked like the drain had failed to make progress. return { folded: 0, remaining: released ?? pendingCount(manifest), released: true }; } + if (outOfTime()) return { folded: 0, remaining: null, deferred: "budget" }; const plan = foldPlan(waiting, { maxEntries: FOLD_BATCH, maxBytes: MAX_BODY_BYTES }); let drop = []; let remaining = 0; @@ -782,6 +797,12 @@ async function foldPending(env, userId) { })); // The entry may carry the vectors its accept computed. Handing them to this entry's client // means the fold applies storage and nothing else. + // Between entries, not inside one: half an entry cannot be applied. Stopping here leaves + // the rest of this batch in the log, where the next attempt folds it with its own budget. + if (outOfTime()) { + drop = names.size ? [...names] : []; + return { readOnly: drop.length === 0, response: { folded } }; + } const seeded = seedFromEntry(env, entry); await addOneBatch( env, @@ -2038,8 +2059,9 @@ async function foldRoute(request, env, url, path) { let folded = 0; let remaining = 0; let released = 0; + const drainDeadline = Date.now() + 100_000; for (let round = 0; round < 8; round += 1) { - const step = await foldPending(env, userId); + const step = await foldPending(env, userId, { deadline: drainDeadline }); folded += step.folded; remaining = step.remaining; if (step.released) released += 1; @@ -2076,19 +2098,21 @@ async function addRoute(request, env, path) { Number.parseInt(hash(accepted.response.request_id).slice(0, 6), 16) % foldEvery === 0; // The measured failure (2026-10-30 14:09-14:15 burst): folds riding inside the accept // pushed Adds to 11-25 s routinely and 40-100 s at the tail, and five requests died at - // 120 s - every one of them in foldPending. With an execution that can extend the - // invocation, the response goes first and the fold runs afterwards: the entry is already - // durable, the fold is idempotent (a crash before its entries are deleted repeats safely) - // and the next scheduled accept retries it, so a fold killed in flight costs nothing but - // one provider call, not a scored attempt. Inline folding stays the fallback for callers - // without waitUntil (tests, wrangler dev) and as the explicit escape hatch. + // 120 s - every one of them in foldPending. The budget above only decided whether a fold + // *started*, so a fold beginning at the last millisecond still ran a whole partition + // rewrite. It is now a real deadline handed to the fold, which checks it before each entry + // and leaves the remainder in the log - the entry is durable, so stopping costs nothing. + // The waitUntil branch is kept for callers that do provide one (tests, a future runtime): + // Makers' Node cloud function context lists uuid/params/env/clientIp/server and no such + // hook, and the always-on request line reports which path ran, as the execution field. + const foldDeadline = startedAt + foldBudget; const foldTask = dueBySchedule - ? foldPending(env, accepted.response.user_id).catch(() => {}) + ? foldPending(env, accepted.response.user_id, { deadline: foldDeadline }).catch(() => {}) : null; const execution = currentContext()?.execution; if (foldTask && execution?.waitUntil && env.NMG_FOLD_INLINE !== "1") execution.waitUntil(foldTask); - else if (foldTask && Date.now() - startedAt < foldBudget) await foldTask; + else await foldTask; return json(accepted.response); } async function handle(request, env) { @@ -2205,6 +2229,7 @@ export default { at: new Date().toISOString(), trace_id: traceId, route, + method: request.method, status: responseStatus, elapsed_ms: Date.now() - startedAt, seq: context.seq, diff --git a/onlion/scripts/test-add-log-mode.mjs b/onlion/scripts/test-add-log-mode.mjs index 5bcf9cec..50f219e9 100644 --- a/onlion/scripts/test-add-log-mode.mjs +++ b/onlion/scripts/test-add-log-mode.mjs @@ -138,23 +138,32 @@ try { // The gated inline branch remains only as the test/dev fallback, so both shapes are pinned. assert.match( source, - /const dueBySchedule =[\s\S]{0,900}foldPending\(env, accepted\.response\.user_id\)/, + /const dueBySchedule =[\s\S]{0,2000}foldPending\(env, accepted\.response\.user_id, \{ deadline/, "the fold trigger must stay a schedule over request_id, not a counter", ); + // This platform gives a Node cloud function no waitUntil (the Makers context lists + // uuid/params/env/clientIp/server), so the fold runs inline - and therefore the budget has + // to be a deadline the fold checks, not a gate on whether it starts. Both are pinned: + // the deadline is handed down, and the fold honours it before each entry it applies. assert.match( source, - /if \(foldTask && execution\?\.waitUntil && env\.NMG_FOLD_INLINE !== "1"\)[\s\S]{0,200}execution\.waitUntil\(foldTask\);/, - "when waitUntil exists the fold must run after the response, not inside it", + /const foldDeadline = startedAt \+ foldBudget;/, + "the accept must compute a real fold deadline, not just a start gate", + ); + assert.match( + source, + /async function foldPending\(env, userId, \{ deadline = null \} = \{\}\) \{[\s\S]{0,1600}outOfTime\(\)\) return \{ folded: 0, remaining: null, deferred: "budget" \};/, + "the fold must refuse to start past its deadline", ); - assert.doesNotMatch( + assert.match( source, - /await foldPending\(env, accepted\.response\.user_id\)/, - "the accept must never await a fold inline on the serving path", + /if \(outOfTime\(\)\) \{\s*drop = names\.size \? \[\.\.\.names\] : \[\];/, + "the fold must stop between entries, leaving the remainder in the log", ); assert.match( source, - /else if \(foldTask && Date\.now\(\) - startedAt < foldBudget\) await foldTask;/, - "the inline fallback must still be budgeted; unbounded, a contended user folds past 120 s", + /if \(foldTask && execution\?\.waitUntil && env\.NMG_FOLD_INLINE !== "1"\)[\s\S]{0,200}execution\.waitUntil\(foldTask\);/, + "a runtime that does provide waitUntil must still be used", ); // The root fix: an accept takes no compare-and-swap at all. Sixteen concurrent Adds on one // user used to serialise on the manifest and lose twelve times each, redoing a megabyte of