Repository navigation
fix(onlion): an embedding outage costs latency, not the write - #135
Merged
Merged
Conversation
Tencent's analysis of 16:52-17:52 split 74 timeouts into three classes; class B (39 rows, 52.7%) is ours and it is a mechanism: 1. the accept's inline embed loses its 8 s race, so the entry is stored vectorless; 2. the fold backfills, syncRecordEmbeddings throws on the provider timeout; 3. the throw leaves rememberBatch -> addOneBatch -> mutateUser, which treats it as a lost compare-and-swap and re-runs the whole attempt - up to twelve times, each paying the same 12 s timeout. 12 x 12 s = 144 s, which is the 120 s ceiling with only the fold on the stack. The records were already durable; every retry re-embedded them for nothing. Fixes: - the backfill never fails its caller, except EmbeddingPreparation, which is control flow (runPart unwinds the SQLite session, embeds outside it, re-enters). Swallowing that one silently dropped records - the fold-must-stay-searchable test caught it. - monitoring gains a provider circuit breaker (NMG_EMBED_BREAKER_MS, default 30 s) on dependency_timeout / embedding_timeout / embedding_network_error / 429 / 5xx. While open an embedding call fails in microseconds as embedding_suppressed; records are stored vectorless and score lexically until a later fold backfills them. A 400 does not trip it: that is our bug. Class A (31 rows, 53 ms, 54.48 MB) and C (4 rows) never reached the handler, so the worker cannot fix them; reducing the 600-800 MB per-instance baseline is the separate next task, and the record says so. Recorded in docs/decisions/implemented/2026-10-30-embedding-breaker.*
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
docs/decisions/implemented/2026-10-30-embedding-breaker.{md,zh-CN.md,terms.yaml}syncRecordEmbeddingsthrows on the provider timeout, that throw leavesrememberBatch → addOneBatch → mutateUser, andmutateUserre-runs the whole attempt up to 12 times, each paying the same 12 s timeout (12 × 12 s = 144 s). The records were already durable.EmbeddingPreparation— that one is control flow (runPartunwinds the SQLite session, embeds outside it, re-enters); swallowing it silently dropped records and the fold-searchability test caught it.NMG_EMBED_BREAKER_MS, default 30 s) ondependency_timeout/embedding_timeout/embedding_network_error/ 429 / 5xx: while open, an embedding call fails in microseconds asembedding_suppressed; records are stored vectorless and score lexically until a later fold backfills them. A 400 does not trip it.Not verified
Not deployed yet (local verify 37 PASS). Class A (31 rows, 53 ms, 54.48 MB) and class C (4 rows) never reached the handler — no
nmg_requestline — so the worker cannot fix them; reducing the 600-800 MB per-instance baseline is a separate task, stated as such in the record. The breaker window (30 s) is a judgement, not a measurement.