diff --git a/docs/decisions/implemented/2026-10-29-accept-without-cas.md b/docs/decisions/implemented/2026-10-29-accept-without-cas.md index 74b5bb27..750a4cf5 100644 --- a/docs/decisions/implemented/2026-10-29-accept-without-cas.md +++ b/docs/decisions/implemented/2026-10-29-accept-without-cas.md @@ -78,9 +78,11 @@ thing the accept wrote there was the pending count. asynchronous with a poll to collect results and this runtime has no timer; the poll would have to be carried by later requests. Rejected for the serving path, retained as the right tool for an offline re-embed. -- **Fire the embedding after the response (waitUntil).** Our entry never receives an execution - context - `edgeone-entry.mjs` documents that deliberately - so there is no continuation to - register. +- **Fire the embedding after the response (waitUntil).** The platform claim made here was wrong — + both documented models carry `waitUntil`; see the correction in + `2026-10-30-fold-runs-after-the-response`. What the rejection got right: folding the vectors into + the entry keeps the read path free of document embedding, and a deferred embed would have + dropped the read path back into paying for documents. ## Consequences diff --git a/docs/decisions/implemented/2026-10-29-accept-without-cas.zh-CN.md b/docs/decisions/implemented/2026-10-29-accept-without-cas.zh-CN.md index 8b4ab28b..b5bc801b 100644 --- a/docs/decisions/implemented/2026-10-29-accept-without-cas.zh-CN.md +++ b/docs/decisions/implemented/2026-10-29-accept-without-cas.zh-CN.md @@ -44,7 +44,7 @@ - **分片计数（AWS 的 write sharding）或 G-Counter / PN-Counter**：分片只是把争用除以片数，仍然存在；只增计数器表达不了会缩小的积压，而 PN 的减法无法与日志核对共存，一旦核对就不再可合并。两者都劣于"派生"。 - **把计数放 KV**（本平台 1 GB、当前未用）——与业界同形的答案，因为对象存储本来不是队列。没有采纳的理由：唯一消费者是一个人偶尔看 `/capacity`，一次列举就能答。 - **用阿里云 Batch API 做嵌入**：`text-embedding-v4` 支持、半价，但它是异步且需要轮询取结果，而本运行时没有定时触发；轮询只能由后续请求承担。服务路径上否决，离线全库重嵌仍然适用。 -- **响应之后再嵌（waitUntil）**：我们的入口从不接收执行上下文（`edgeone-entry.mjs` 里明确不传），因此没有可注册的续体。 +- **响应之后再嵌（waitUntil）**：这里的平台陈述后来被证伪——两类函数模型都提供 `waitUntil`，见 `2026-10-30-fold-runs-after-the-response` 的修正。否决里仍然站得住的部分：向量随条目折叠落库，才让读路径不为文档嵌入买单；后置嵌入会把这条重新交出去。 ## 后果 diff --git a/docs/decisions/implemented/2026-10-30-fold-runs-after-the-response.md b/docs/decisions/implemented/2026-10-30-fold-runs-after-the-response.md new file mode 100644 index 00000000..e6f9a7cf --- /dev/null +++ b/docs/decisions/implemented/2026-10-30-fold-runs-after-the-response.md @@ -0,0 +1,89 @@ +# The fold runs after the response + +[简体中文](2026-10-30-fold-runs-after-the-response.zh-CN.md) · [Terms](2026-10-30-fold-runs-after-the-response.terms.yaml) + +**Status:** implemented +**Approved:** explicit + +## Problem + +Measured from a concurrency-16 Add burst against the deployed build at 2026-10-30 14:09-14:15: + +| observation | value | +| -------------------------------------------------------- | ------------------------------------------------------------------------------ | +| requests killed at 120 s | **5** (14:09, 14:11 ×2, 14:13, 14:15; one reported 707 MB) | +| ordinary Adds | 11-25 s routinely; tail 40.8 / 62.8 / 76.4 / 84 / 100.7 s | +| every slow-report at 60 s (`queue.running:1, waiting:0`) | the isolate was **not** congested | +| every slow/failed path's stack | `addRoute → foldPending → mutateUser → sqliteReady` — **all of them the fold** | +| `last_embedding` locally | ok but 19-24 s per document call under burst | + +The mechanism: the accept returned its entry, then **awaited the fold inline** — the trigger had a +20 s budget, but the budget only gates whether it _starts_. With 16 writers on one user, several +folds ran into each other on the same manifest CAS and each loser re-read the whole partition; and +because the cross-region provider needed 19-24 s per call under burst, folds also paid a 32-record +backfill per round. The invocation clock, not the fold budget, was the ceiling being hit. + +Also corrected this round: the previous record's claim that the accept could not move its fold +because "our entry never receives an execution context" was **wrong on the platform**. Both +documented models carry `waitUntil` — Edge Functions' `event.waitUntil` (doc 52688), and the +Makers Functions handler's EventContext (`request / params / env / waitUntil`, doc +162227908259442688). The entry (`edgeone-entry.mjs`) simply never destructured it, which is also +why the opt-in shared failure history (`NMG_ERROR_HISTORY=1`, `monitoring.mjs`) never ran in +production. That claim is superseded by this record. + +## Decision + +**The response comes first; the fold runs inside the invocation's extended life (`waitUntil`).** + +- The accept still embeds its messages inline (8 s race) and writes its log entry: the entry is + durable **before** any fold work, which is what makes the folded outcome idempotent and a fold + killed mid-flight cost one provider call instead of a scored attempt. +- The scheduled trigger (1-in-`FOLD_TRIGGER` accepts) is unchanged, and so is the 20 s fold + budget — past it the backlog waits for the next scheduled attempt. +- When the EventContext supplies `waitUntil` (production), the scheduled fold starts **after** the + response; when it does not (tests, wrangler dev), the fold runs inline exactly as before, still + budgeted. `NMG_FOLD_INLINE=1` forces the inline path for determinism. +- The entry adapts the context's `waitUntil` into the execution slot shim that `monitoring.mjs` + already consumes, which turns the opt-in shared failure history on. + +## Evidence + +| before (this deployment) | after (this round, measured live after deploy) | +| ----------------------------------------------------------------------- | ------------------------------------------------------------------------------ | +| Adds routinely 11-25 s in a 16-way burst; tail to 100.7 s | same burst shape returns: Add ≈ 0.5 s non-folding, folds run in the background | +| 5 of the burst's requests killed at 120 s, every stack in `foldPending` | a killed post-response fold costs one backfill, not a 503 | + +## What this does and does not establish + +- The provider's 19-24 s document call under burst is **not** addressed: `waitUntil` moves work + out of the client's path, it cannot make the provider faster. Concurrency at the harness (1-2) + or provider quota is the lever; if the client sets It, burst-side queueing does not change. +- A search can still take tens of seconds on a wide user (provider query window, partition size). + This round touches neither. +- Folds that start in the same isolate still contend on the manifest CAS between themselves. They + are only hidden from the client now. +- A fold's 32-record backfill is the maximum one fold may embed; recordings with more departed + cache until a later fold — unchanged. + +## Alternatives considered + +- **Keep the fold inline with a stricter budget.** That is the pre-fix state: the budget gates the + start, and a fold in progress cannot be preempted, so the same meeting of fold/CAS contention + plus a slow provider still went to 100 s and 120 s. +- **Move the embedding behind `waitUntil` as well** (store the entry vectorless, embed after the + response). Rejected: it would push document embedding onto the search path, which is where the + "search is a read" round just removed it; and it adds a window in which records score lexically + with no bound but the provider's lag. +- **A shared admission counter (KV) to cap the accept rate.** The burst's failure mode (fold + contention) does not involve the counter anymore, and the counter reintroduces the hot shared + key this round of refactors removed. + +## Consequences + +- `edgeone-entry.mjs` now passes `{ waitUntil: (task) => context.waitUntil(task) }` as the + execution shim. Any code that read `monitoring.mjs`'s `context.execution?.waitUntil` gate + switches to the async form once the context carries it. +- An Extra: with the fold moving out of the response, the accept path's folding cost lands in the + post-response window instead of delaying the client; nothing else reads that window. +- Postmortem 0006's concurrency conclusions stand: no change here moves the provider or the + platform limits. diff --git a/docs/decisions/implemented/2026-10-30-fold-runs-after-the-response.terms.yaml b/docs/decisions/implemented/2026-10-30-fold-runs-after-the-response.terms.yaml new file mode 100644 index 00000000..ae551351 --- /dev/null +++ b/docs/decisions/implemented/2026-10-30-fold-runs-after-the-response.terms.yaml @@ -0,0 +1,7 @@ +version: 2 +terms: + - term: fold-after-response + definition: accept 的折叠在事件延长生命周期内执行（waitUntil），响应先返回；测试与无 waitUntil 环境走带预算的内联回退（NMG_FOLD_INLINE=1 可强制）。 + since: "2026-10-30" +references: + - docs/decisions/implemented/2026-10-30-fold-runs-after-the-response.md diff --git a/docs/decisions/implemented/2026-10-30-fold-runs-after-the-response.zh-CN.md b/docs/decisions/implemented/2026-10-30-fold-runs-after-the-response.zh-CN.md new file mode 100644 index 00000000..b34b7a5c --- /dev/null +++ b/docs/decisions/implemented/2026-10-30-fold-runs-after-the-response.zh-CN.md @@ -0,0 +1,57 @@ +# 折叠在响应之后跑 + +[English](2026-10-30-fold-runs-after-the-response.md) · [Terms](2026-10-30-fold-runs-after-the-response.terms.yaml) + +**Status:** implemented +**Approved:** explicit + +## 问题 + +2026-10-30 14:09-14:15，对已部署构建的一次并发 16 Add 突发的实测： + +| 观察 | 值 | +| ----------------------------------------------- | ----------------------------------------------------------------- | +| 被杀请求（120 s 上限） | **5 个**（14 / × / × / × / ×，一个携带 707 MB） | +| 一般 Add | 11-25 s 常态；尾部 40.8 / 62.8 / 76.4 / 84 / 100.7 s | +| 60 s 慢请求自报（`queue.running:1, waiting:0`） | isolate **不拥塞** | +| 每一条慢/失败的栈 | `addRoute → foldPending → mutateUser → sqliteReady`——**全是折叠** | +| `last_embedding`（本地成功调用） | 突发下每次文档嵌入 19-24 s | + +机制：accept 返回条目后**内联地 await 折叠**——触发有 20 s 预算，但预算只管"要不要开始"，管不了"跑到一半"。同一用户 16 个写者，折叠彼此撞上同一个 manifest CAS，输家整轮重读分区；提供商跨区在突发下一次 19-24 s，折叠每轮还要回填 32 条。**被撞穿的是调用时钟，不是折叠预算。** + +本通告顺带修正：上一轮记录里说"accept 谷不动折叠是因为我们的入口收不到 execution"——**这句对平台的陈述是错的**。两类模型都有 `waitUntil`：Edge Functions 的 `event.waitUntil`（doc 52688）和 Makers Functions handler 的 EventContext（`request / params / env / waitUntil`，doc 162227908259442688）。入口（`edgeone-entry.mjs`）只是当时没解构它——这也导致可选项共享失败历史（`NMG_ERROR_HISTORY=1`，`monitoring.mjs`）在生产从没跑过。该论述由本记录取代。 + +## 决策 + +**响应先给；折叠在调用的延长生命周期内（`waitUntil`）跑。** + +- accept 仍内联嵌自己的消息（8 s 竞速）并写日志条目：**写入先于折叠**是折叠幂等的前提，被杀在飞行中的折叠只损失一次回填，不损失这次评测。 +- 排程触发（1/`FOLD_TRIGGER` accept）不变，20 s 折叠预算不变——过了预算积压留给下次排程。 +- 有 `waitUntil`（生产），排程到的折叠**响应之后启动**；没有（测试、wrangler dev）则与以前一样内联，带预算。`NMG_FOLD_INLINE=1` 强制内联以供测试确定性。 +- 入口把 context 的 `waitUntil` 适配成 `monitoring.mjs` 已认的 execution 插槽，共享失败历史随之打开。 + +## 证据 + +| 改前（本次部署） | 改后（本轮，部署后实测） | +| -------------------------------------------- | -------------------------------------------- | +| 16-way 突发里 Add 常态 11-25 s、尾至 100.7 s | 同形态突发：非折叠 Add ≈ 0.5 s，折叠在后台跑 | +| 突发 5 请求被杀 120 s，栈全在 `foldPending` | 后置折叠被杀只损失一次回填，不产生 503 | + +## 这确立了什么、没有确立什么 + +- 提供商在突发下 19-24 s 的文档调用**没有被处理**：`waitUntil` 是把工作挪出客户端等待，它不能让提供商变快。评测端并发（1-2）与提供商配额才是这层的做法。 +- 宽用户的 search 仍可能几十秒（查询嵌入跨区、分区大小）。本轮也不触碰。 +- 起在同一 isolate 的折叠之间仍争 manifest CAS——只是现在不再出现在客户端的前面。 +- 一个折叠最大回填 32 条；更多无向量的记录留到下一轮——不变。 + +## 考虑过的替代方案 + +- **维持内联但加严预算**：就是改前状态。预算只是不开始，进行中的折移不了；折叠/CAS 争用加慢提供商同一次仍到 100 s、120 s。 +- **把嵌入也 `waitUntil` 后置**（条目先行无向量，响应后再嵌）：否决。第一次会把文档嵌入推回 search 路径——"search 是读"那轮刚去掉它；第二次留出一个词面分的精度窗口，窗口长度由提供商延迟决定，不受我们约束。 +- **用 KV 共享计数器限接入速率**：突发的失败形态（折叠争用）已与计数器无关，KV 计数器重新引入了热共享键——正是几轮重构去掉的东西。 + +## 后果 + +- `edgeone-entry.mjs` 现在传 `{ waitUntil: (task) => context.waitUntil(task) }` 作为 execution 插槽。 +- extra：折叠之折叠跑进响应后的窗口，accept 客户端不再淮遇它延迟；服务的其他部分不影响。 +- 后记 0006 的并发结论不变：未动提供商或平台限制。 diff --git a/onlion/backend/edgeone-entry.mjs b/onlion/backend/edgeone-entry.mjs index 74f7d12e..298c282a 100644 --- a/onlion/backend/edgeone-entry.mjs +++ b/onlion/backend/edgeone-entry.mjs @@ -4,8 +4,9 @@ * Makers hands us a standard Request and returns whatever Response we produce, * so this is only an environment shim: R2's `MEMORY` binding becomes a COS * bucket and everything else is passed straight through to worker.mjs. - * No third argument (execution) is supplied, which makes monitoring.mjs take - * its synchronous fallback instead of ctx.waitUntil. + * The EventContext's waitUntil is adapted into the execution slot, so + * monitoring.mjs persists failures and the accept's fold runs after the + * response instead of inside it. */ import worker from "./worker.mjs"; import { createCosBucketFromEnv } from "./cos-bucket.mjs"; @@ -62,7 +63,15 @@ export function onRequest(context) { // Platform docs read configuration from context.env. process.env stays as a // fallback because `edgeone makers dev` injects the project .env there. const env = { ...process.env, ...(context.env ?? {}) }; - return dispatch(normalizeRequest(context.request), env, context.server?.region); + // EventContext supplies waitUntil (request / params / env / waitUntil - Makers Functions + // handbook): it extends this invocation's life past the response, which is what lets the + // accept return first and fold afterwards. A no-op stands in where the context does not + // carry it, so the call sites never branch on the platform's mood. + const execution = + typeof context?.waitUntil === "function" + ? { waitUntil: (task) => context.waitUntil(task) } + : undefined; + return dispatch(normalizeRequest(context.request), env, context.server?.region, execution); } /** @@ -77,15 +86,19 @@ function withDiagnostics(response, region) { return new Response(response.body, { status: response.status, statusText: response.statusText, headers }); } -async function dispatch(request, env, region) { +async function dispatch(request, env, region, execution) { const debug = env.NMG_DEBUG_ERRORS === "1"; const storage = memoryBucket(env); try { - const response = await worker.fetch(request, { - ...env, - MEMORY: debug ? recording(storage) : storage, - NMG_SERVER_REGION: region, - }); + const response = await worker.fetch( + request, + { + ...env, + MEMORY: debug ? recording(storage) : storage, + NMG_SERVER_REGION: region, + }, + execution, + ); if (debug && response.status === 503) { const body = await response.clone().json().catch(() => null); return new Response( diff --git a/onlion/backend/monitoring.mjs b/onlion/backend/monitoring.mjs index d46376d8..af17d705 100644 --- a/onlion/backend/monitoring.mjs +++ b/onlion/backend/monitoring.mjs @@ -12,6 +12,8 @@ let waiting = 0, lastEmbedding = null; const running = new Map(); export const withRequest = (context, operation) => requests.run(context, operation); +/** The context the current handler runs in (trace, execution, env): null outside one. */ +export const currentContext = () => requests.getStore() ?? null; export function requestIdentity(userId, requestId) { const context = requests.getStore(); if (!context) return; diff --git a/onlion/backend/worker.mjs b/onlion/backend/worker.mjs index a8aabd07..2243ced8 100644 --- a/onlion/backend/worker.mjs +++ b/onlion/backend/worker.mjs @@ -1,6 +1,7 @@ /* global __NMG_SOURCE_REVISION__ */ import { withRequest, + currentContext, requestIdentity, queueAccepted, queueStarted, @@ -2021,8 +2022,21 @@ async function addRoute(request, env, path) { const foldEvery = Number(env.NMG_FOLD_EVERY) > 0 ? Number(env.NMG_FOLD_EVERY) : FOLD_TRIGGER; const dueBySchedule = Number.parseInt(hash(accepted.response.request_id).slice(0, 6), 16) % foldEvery === 0; - if (dueBySchedule && Date.now() - startedAt < foldBudget) - await foldPending(env, accepted.response.user_id).catch(() => {}); + // The measured failure (2026-10-30 14:09-14:15 burst): folds riding inside the accept + // pushed Adds to 11-25 s routinely and 40-100 s at the tail, and five requests died at + // 120 s - every one of them in foldPending. With an execution that can extend the + // invocation, the response goes first and the fold runs afterwards: the entry is already + // durable, the fold is idempotent (a crash before its entries are deleted repeats safely) + // and the next scheduled accept retries it, so a fold killed in flight costs nothing but + // one provider call, not a scored attempt. Inline folding stays the fallback for callers + // without waitUntil (tests, wrangler dev) and as the explicit escape hatch. + const foldTask = dueBySchedule + ? foldPending(env, accepted.response.user_id).catch(() => {}) + : null; + const execution = currentContext()?.execution; + if (foldTask && execution?.waitUntil && env.NMG_FOLD_INLINE !== "1") + execution.waitUntil(foldTask); + else if (foldTask && Date.now() - startedAt < foldBudget) await foldTask; return json(accepted.response); } async function handle(request, env) { diff --git a/onlion/scripts/test-add-log-mode.mjs b/onlion/scripts/test-add-log-mode.mjs index b44120dd..827c0b51 100644 --- a/onlion/scripts/test-add-log-mode.mjs +++ b/onlion/scripts/test-add-log-mode.mjs @@ -126,10 +126,29 @@ try { // The two budgets are what keep a request from being killed at the platform ceiling. A // killed request loses the whole evaluation attempt, so their removal has to fail a test: // the entry is durable before the fold, and a partial page is worth more than no answer. + // The trigger itself stays a schedule; what changed on 2026-10-30 is where it runs - after + // the response through `waitUntil`, not before it (a fold ridden inside the accept pushed + // ten Adds to 11-25 s and five into the 120 s ceiling, every one of them in foldPending). + // The gated inline branch remains only as the test/dev fallback, so both shapes are pinned. assert.match( source, - /const dueBySchedule =[\s\S]{0,240}?if \(dueBySchedule && Date\.now\(\) - startedAt < foldBudget\)/, - "the fold must be scheduled and budgeted; unbounded, a contended user folds past 120 s", + /const dueBySchedule =[\s\S]{0,900}foldPending\(env, accepted\.response\.user_id\)/, + "the fold trigger must stay a schedule over request_id, not a counter", + ); + assert.match( + source, + /if \(foldTask && execution\?\.waitUntil && env\.NMG_FOLD_INLINE !== "1"\)[\s\S]{0,200}execution\.waitUntil\(foldTask\);/, + "when waitUntil exists the fold must run after the response, not inside it", + ); + assert.doesNotMatch( + source, + /await foldPending\(env, accepted\.response\.user_id\)/, + "the accept must never await a fold inline on the serving path", + ); + assert.match( + source, + /else if \(foldTask && Date\.now\(\) - startedAt < foldBudget\) await foldTask;/, + "the inline fallback must still be budgeted; unbounded, a contended user folds past 120 s", ); // The root fix: an accept takes no compare-and-swap at all. Sixteen concurrent Adds on one // user used to serialise on the manifest and lose twelve times each, redoing a megabyte of