Repository navigation
perf(outbox): resume each retention batch where the previous run's batch ended - #9446
Merged
Merged
Conversation
…tch ended The completed-event prune deletes one oldest-first batch per type per run. Every batch started its scan of outbox_event_type_created_idx at the oldest row of its type, and the rows earlier batches deleted stay in that index until vacuum removes them, so while a large backlog drains each batch first walks everything deleted since the last vacuum. That walk grows with each run: on a read replica during a large drain it took about 0.6s before the first live row, and a batch can reach the 10s statement timeout. Each batch now resumes after the (created_at, id) of the last row the previous run's batch deleted, the same shape as the idempotency sweep's resume. Batches are ordered by (created_at, id) so ties on created_at split exactly, the cursor keeps PostgreSQL's microsecond text form, and the row comparison becomes a created_at >= bound on the index, with the full comparison as a filter. Resuming from a cursor, the same batch took about 20ms. The prune runs once per processor run, so the cursor has to outlive the run; it is kept in Redis as a hint, one MGET before the batches and one pipeline after. A row the pass went by before it completed, or while an overlapping run held its lock, is covered by the next pass: the cursor is cleared when a batch comes up short of the cutoff, and a pass older than an hour starts over from the oldest row. Without Redis, or with an unreadable cursor, a batch starts from the oldest row as before. Batch size, timeout, cadence and which rows are eligible are unchanged. Autopsy: the retention suite asserted which rows each run deleted, which a scan from the oldest row gets right, so nothing observed where a batch started. The new Redis cases complete a row behind the cursor between runs: resuming leaves it for the next pass and the old scan takes it at once, so the first case fails on the previous code. The others split one creation time across batches with pending rows mixed in, and restart a pass that is over an hour old or whose cursor is unreadable.
|
The latest updates on your projects. Learn more about Vercel for GitHub. |
Contributor
 [Medium impact] The PR appears safe to merge; the previous cursor issue is fixed. SummaryCompleted outbox cleanup now resumes each event type after the previous batch’s last
The previous finding is fully addressed. No new actionable issues were found. Reviews (2) · Last reviewed commit: "fix(outbox): start a pass from the oldes..." · Reviewed by Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
All reported issues were addressed across 2 files
Reply with feedback, questions, or to request a fix.
Turn on auto-fix | Re-trigger cubic
…names no real time A cursor whose JSON parsed but whose creation time was not a calendar time reached the batch's ::timestamp cast and failed the run, which stopped every later type's prune until the cursor expired. The cursor's creation time is now checked as PostgreSQL's ISO text with a calendar round trip, so such a cursor is dropped like an unreadable one and the pass starts from the oldest row.
Collaborator
Author
Collaborator
Author
|
@cubic-dev-ai review this PR |
Contributor
@waleedlatif1 I have started the AI code review. It will take a few minutes to complete. |
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
outbox_event_type_created_idxuntil vacuum, so while a backlog drains each batch first walks everything deleted since the last vacuum (about 0.6s before the first live row on a read replica; a batch can reach the 10s statement timeout)(created_at, id)of the last row the previous run's batch deleted, the same resume shape as the idempotency sweep. Batches order by(created_at, id)so ties split exactly; the cursor keeps PostgreSQL's microsecond text form. The row comparison plans as acreated_at >=index bound with the full comparison as a filter (plainEXPLAINchecked); resuming, the same batch took about 20msMGETbefore the batches, one pipeline after). The cursor clears when a batch comes up short of the cutoff, and a pass older than an hour starts over from the oldest row, so a row the pass went by before it completed, or while an overlapping run held its lock, is still pruned by the next passcompleted, retention window) are unchangedType of Change
Testing
lib/core/outbox/retention.integration.tsagainst real PostgreSQL and Redis: a row completed behind the cursor between runs is left for the next pass and then pruned (fails on the previous code, which takes it at once); one creation time split across batches with pending rows and non-prunable types mixed in deletes exactly the eligible rows; a pass over an hour old and an unreadable cursor both restart from the oldest row. Reverting the short-batch reset, the id tiebreak, or the pass age limit each turns its case redlib/core/outboxunit testsbun run lint,bun run type-check(apps/sim),bun run check:audits(61 audits)Checklist
test-auditauthoring gate)