Repository navigation
fix(state): three history writers hold their mutex until rollback ends - #1845
Merged
Merged
Conversation
The control loop's history write, the SQLite history import and the plain archive import held historyWriteMu around a transaction begun on the *sql.DB. When that context ended, database/sql could report the Tx done before SQLite had rolled it back, and the mutex went to the next writer early. They now begin the transaction on a connection of their own and close it before unlocking, as the batch and maintenance paths already do. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Review note: the control loop writes through recordHistoryBatches, which used a Conn already. recordHistoryBatch serves RecordTick*, which only tests call, so the changeset no longer names the control loop. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Found by the review of #1838 on 25 October 2026 and left out of that PR.
Problem
Three history writers held
historyWriteMuaround a transaction begun on the*sql.DB:recordHistoryBatch(store_ts.go), behindRecordTick*, with a 30 s limit. Only tests callRecordTick*; the control loop writes throughrecordHistoryBatches, which already used a connection of its own;writeHourBuckets(history_plain.go), the import of plain archive files at start-up;mergeHistoricalSamples(history_migration.go), the import of SQLite history.When the context of such a transaction ends during a write, database/sql rolls it back on its own goroutine. The writer's
tx.Rollback()can then returnErrTxDonebefore SQLite has finished, and the writer unlocks. The next writer gets the mutex while SQLite still holds its write lock, so it waits or fails with BUSY. SQLite's own lock keeps the data safe; nothing is lost.Fix
A new
beginHistoryWritebegins the transaction on asql.Connand returns a function that rolls back and closes that connection. The three writers defer it after their unlock, so it runs first.Conn.Closewaits until database/sql's rollback has returned, so the mutex now stays held until SQLite has rolled back. The batch, archive and maintenance paths already work this way (store_ts.go,archive_write.go,history_series_hour.go); #1838 explains why.The connection count is unchanged: a transaction on the
*sql.DBalready held one pooled connection for its whole life.Evidence
TestHistoryMaintenanceWaitsForCancelledRollbackfrom #1838 becomesTestHistoryWriterWaitsForCancelledRollback, with three new cases:tick,cold-hoursandimport. Each holds SQLite's rollback in the test driver once the write has started, cancels, and checks that the writer has neither returned nor released the mutex. On master, 15 of 15 runs of the new cases (3 ×-count=5) fail:The writer returned before the driver even reached the rollback it holds, which is the early return in its plainest form. On this branch all six cases pass, also with
-race -count=3.go vet ./...passes.go test ./internal/... ./cmd/ftw/...passes.Not changed
The other
historyWriteMuusers either use a connection already or write without a context that can end (s.history.Exec,Begin()withcontext.Background()), so a cancel cannot release their mutex early.Review
An independent review (Sonnet) found no bug in the fix. It traced
Conn.Closein Go 1.26.0, 1.26.9 and 1.27.1:Conn.BeginTxtakes the connection's close lock and onlyTx.close, after the driver's rollback, gives it back, soClosewaits for the rollback. It checked the defer order on every return and panic path (endfirst, thenhistoryWriteMu, thenarchiveViewMu), that nothing inside the three transactions usess.historydirectly, and that the pool (4 open, 2 idle) needs no extra connection. modernc.org/sqlite implementsResetSessionandIsValid, so a cancelled transaction's connection goes back through the same check on either path. WithbeginHistoryWritechanged back tos.history.BeginTx, the three new cases failed 15 of 15 and the old three passed.Its one finding: the changeset and this description said the fix covers the control loop's write. It does not. The control loop writes through
recordHistoryBatches, andrecordHistoryBatchserves onlyRecordTick*, which only tests call. The second commit takes the control loop out of the changeset; the user-visible fix is in the two imports.RecordTick*is now test-only code with a comment that still calls it the control-loop tick; removing it would be a separate change across 14 test files.Patch changeset.
🤖 Generated with Claude Code