Repository navigation
fix(onlion): a partition that cannot be read shortens the page, not the answer - #140
Merged
Merged
Conversation
…he answer The 20:18-20:20 window is a different fault from every earlier one: four /search answered 503 at stage=partition_download with code dependency_timeout in 29-31 s, while /add returned 200 on the same instance three seconds earlier. A 5xx on /search is a failed attempt for the harness; a shorter page still scores. /add already degrades - the read path did not. search now treats an unreadable partition as a missing one: partial=true, degraded=<error code>, and after two consecutive failures it stops paying the timeout per partition and answers with what it read. Anything that is not a timeout/reset/replaced-snapshot still propagates, because a bug must stay visible. Also fixed the field that sent the operator down the wrong path: failure records attached embedding_host/embedding_model whenever the environment had them, so a partition_download timeout appeared next to the embedding provider and looked like a shared dependency. They are attached only when the failing stage is an embedding stage. Test: fold a user, delete the partition object, search -> 200 with partial=true and degraded=snapshot_changed. Before this change the same call surfaced as a failure. 39 PASS.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Doc owner:
docs/decisions/implemented/2026-10-30-search-degrades-on-unreadable-partition.{md,zh-CN.md,terms.yaml}(added in a follow-up commit on this branch).Evidence (20:20:21 and 20:20:57, same instance, same user):
/search503 atstage=partition_download,dependency_timeout, 29-31 s,sqlite_loaded:false, while/addanswered 200 at 24 s three seconds earlier on the same instance. Not a platform kill — a business 5xx — and/searchwas the only path without a degrade route.Fix: an unreadable partition (timeout / reset / replaced snapshot) marks the page
partialwith adegradedcode; after two consecutive failures the loop stops paying the timeout per partition. Anything else still propagates.Also: failure records attached
embedding_host/embedding_modelwhenever they were configured, which is why apartition_downloadtimeout looked like an embedding fault. They are attached only for embedding stages now.Not verified
Not deployed. The test deletes a partition object to force the condition; the live cause of that 29-31 s download timeout (object store under a saturated egress while the embedding calls go cross-region) is not diagnosed by this change — it only stops one broken partition from becoming a failed attempt.