Repository navigation
fix(state): a cancelled backup copy keeps the context's error - #1838
Merged
Merged
Conversation
When the deadline lands inside a batch, database/sql rolls the copy's transaction back and closes its statement on its own goroutine. The next row then fails with "sql: statement is closed", and the backup's reason lost the deadline (CI run 39759536215). The copy now puts the context's error in front once the context has ended. TestHistoryMaintenanceWaitsForCancelledRollback cancelled on a 50 ms timer. Under load the timer could fire in the backfill's read, before it takes the writer mutex, so the rollback the test held was a read's and the free mutex looked like an early release (run 40185497314). The test now cancels once the write is under way. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Review: the archive write has its own 1 s limit, so a test goroutine that stalls between seeing the write and cancelling gets DeadlineExceeded. Either context error shows the maintenance stopped on its context. The changeset now covers a cancel and state tables too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two
internal/statetests that failed CI on PRs that did not touch them. One shows a small real bug in backup error reporting. The other is a timing assumption in the test, not a data-safety bug.1. Backup copy loses the deadline (real bug, small)
Run 39759536215 (master):
Cause.
copyVerifiedTableinserts rows through a statement prepared on a transaction that uses the backup context. When the deadline lands inside a batch, database/sql rolls the transaction back on its own goroutine and closes its statements. The next row'sExecContextthen fails withsql: statement is closed, not the context's error. The reason still named the limit (the outer wrapper adds it fromcontext.Cause), buterrors.Is(err, context.DeadlineExceeded)was false. A user's cancel in the same spot came out as "statement is closed", with no "canceled".Fix. Once the context has ended, the copy puts the context's error in front of the database error:
copy history_hot: context deadline exceeded: sql: statement is closed. The final commit's error gets the same treatment and the table name. Patch changeset.Evidence.
TestBackupCopyCancelledInsideABatchKeepsTheContextErrorcancels after the first row, waits until database/sql has rolled back and returned the connection, then sends the second row. On master it fails every time withcopy t: sql: statement is closed, the CI message. It passes on this branch.2. Maintenance rollback test: timing, not an ordering bug
Run 40185497314 (PR #1807), race job:
Cause. The test cancels with a 50 ms timeout and holds the first rollback the driver sees.
backfillSeriesHoursSnapshotreads in a deferred transaction first and takeshistoryWriteMuonly before it writes. Under CI load the 50 ms can run out during those reads. The rollback the test then holds belongs to a read-only transaction that never took the mutex, so the free mutex looks like an early release.Why it is not a data-safety bug. When a write is under way, the deferred cleanup runs
tx.Rollback(), thenconn.Close(), thenhistoryWriteMu.Unlock(). A transaction begun from asql.Connholds that connection's close lock until database/sql's own rollback has returned, soconn.Close()waits for SQLite's rollback before the unlock. That is the property the test checks, and it holds. A read cancelled before the write holds no writer slot in SQLite (WAL, deferred transaction), so not holding the mutex there is correct.Fix (test only). The test now cancels when the driver reports that the write has started, instead of on a timer. It accepts either context error, because the archive write also has its own 1 s limit.
Evidence. On master, 60 ms of delay in the test driver's reads (a local probe, not committed) reproduces the CI message in the backfill case. The archive and completion cases instead fail with
maintenance ended before rollback. With the same probe this branch passes all three: the mutex stays held until the rollback finishes.Review
An independent review (Sonnet) checked the ordering claim against database/sql in Go 1.26.9 (CI) and 1.27.1. It found no bug: all three paths unlock after
conn.Close(), andconn.Close()waits for the rollback. It also mutated the code: droppingconn.Close()from the backfill cleanup, or unlocking before rollback and close, makes the changed test fail with the original message, so the test still guards the property. The second commit applies its two notes: accept either context error, and reword the changeset.It also found a separate, older risk, which this PR does not change.
history_plain.go:337,store_ts.go:396andhistory_migration.go:302holdhistoryWriteMuaround a transaction begun on the*sql.DB, not on asql.Conn. If that context is cancelled,tx.Rollback()can return before SQLite has rolled back, and the mutex is released early. SQLite's own writer lock still protects the data, so the next writer waits or gets BUSY; nothing is lost.store_ts.go:528already uses aConnfor this reason. I have not checked whether those paths run with a context that can be cancelled.Runs
-count=34of the four rollback and backup tests (204 runs each): all pass. Master also passes this local load; both flakes need CI's load, which is why the evidence above uses deterministic reproductions.internal/statepackage under the race detector: passes (270 s).go vet ./...passes.🤖 Generated with Claude Code