Skip to content

fix(state): three history writers hold their mutex until rollback ends - #1845

Merged
frahlg merged 2 commits into
masterfrom
fix/history-conn
Oct 25, 2026
Merged

frahlg merged 2 commits into
masterfrom
fix/history-conn

Conversation

@frahlg

@frahlg frahlg commented Oct 25, 2026 •

Copy link
Copy Markdown
Member

Found by the review of #1838 on 25 October 2026 and left out of that PR.

Problem

Three history writers held historyWriteMu around a transaction begun on the *sql.DB:

  • recordHistoryBatch (store_ts.go), behind RecordTick*, with a 30 s limit. Only tests call RecordTick*; the control loop writes through recordHistoryBatches, which already used a connection of its own;
  • writeHourBuckets (history_plain.go), the import of plain archive files at start-up;
  • mergeHistoricalSamples (history_migration.go), the import of SQLite history.

When the context of such a transaction ends during a write, database/sql rolls it back on its own goroutine. The writer's tx.Rollback() can then return ErrTxDone before SQLite has finished, and the writer unlocks. The next writer gets the mutex while SQLite still holds its write lock, so it waits or fails with BUSY. SQLite's own lock keeps the data safe; nothing is lost.

Fix

A new beginHistoryWrite begins the transaction on a sql.Conn and returns a function that rolls back and closes that connection. The three writers defer it after their unlock, so it runs first. Conn.Close waits until database/sql's rollback has returned, so the mutex now stays held until SQLite has rolled back. The batch, archive and maintenance paths already work this way (store_ts.go, archive_write.go, history_series_hour.go); #1838 explains why.

The connection count is unchanged: a transaction on the *sql.DB already held one pooled connection for its whole life.

Evidence

TestHistoryMaintenanceWaitsForCancelledRollback from #1838 becomes TestHistoryWriterWaitsForCancelledRollback, with three new cases: tick, cold-hours and import. Each holds SQLite's rollback in the test driver once the write has started, cancels, and checks that the writer has neither returned nor released the mutex. On master, 15 of 15 runs of the new cases (3 × -count=5) fail:

--- FAIL: TestHistoryWriterWaitsForCancelledRollback/tick (0.04s)
    history_rollback_test.go:255: the write ended before rollback: context canceled

The writer returned before the driver even reached the rollback it holds, which is the early return in its plainest form. On this branch all six cases pass, also with -race -count=3.

  • go vet ./... passes.
  • go test ./internal/... ./cmd/ftw/... passes.

Not changed

The other historyWriteMu users either use a connection already or write without a context that can end (s.history.Exec, Begin() with context.Background()), so a cancel cannot release their mutex early.

Review

An independent review (Sonnet) found no bug in the fix. It traced Conn.Close in Go 1.26.0, 1.26.9 and 1.27.1: Conn.BeginTx takes the connection's close lock and only Tx.close, after the driver's rollback, gives it back, so Close waits for the rollback. It checked the defer order on every return and panic path (end first, then historyWriteMu, then archiveViewMu), that nothing inside the three transactions uses s.history directly, and that the pool (4 open, 2 idle) needs no extra connection. modernc.org/sqlite implements ResetSession and IsValid, so a cancelled transaction's connection goes back through the same check on either path. With beginHistoryWrite changed back to s.history.BeginTx, the three new cases failed 15 of 15 and the old three passed.

Its one finding: the changeset and this description said the fix covers the control loop's write. It does not. The control loop writes through recordHistoryBatches, and recordHistoryBatch serves only RecordTick*, which only tests call. The second commit takes the control loop out of the changeset; the user-visible fix is in the two imports. RecordTick* is now test-only code with a comment that still calls it the control-loop tick; removing it would be a separate change across 14 test files.

Patch changeset.

🤖 Generated with Claude Code

The control loop's history write, the SQLite history import and the
plain archive import held historyWriteMu around a transaction begun on
the *sql.DB. When that context ended, database/sql could report the Tx
done before SQLite had rolled it back, and the mutex went to the next
writer early. They now begin the transaction on a connection of their
own and close it before unlocking, as the batch and maintenance paths
already do.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 25, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-25T13:20:26.460906Z de75dbd PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

Review note: the control loop writes through recordHistoryBatches,
which used a Conn already. recordHistoryBatch serves RecordTick*, which
only tests call, so the changeset no longer names the control loop.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@frahlg
frahlg merged commit 3606fd1 into master Oct 25, 2026
22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant