[frahlg]

Problem

Triage item C1, Core half. Nightly random rig run on master 83c2b24, 27 Oct 2026.

An inverter outage makes the Sungrow refuse every write. The registry cannot confirm driver_default_mode, so the driver returns ErrControlBlocked and Core excludes it (driver refused control … driver_default_mode returned false). When the device came back, Core:

• did not send the default again, because the defaulted latch was still set;
• kept the driver out of dispatch until driverRefusalRetryInterval (5 min) had passed.

During that time the inverter kept its last forced command, and the fuse guard could not use the battery. This happened in 8 of 10 seeds (207532 to 207539). The logs show driver exclusion retry window elapsed exactly 5 min after the refusal. The refusal and the first failed poll (modbus reconnect backoff active) come within about 1 s of each other, in either order: in 207533 the refusal came at 15:01:07.115 and the first failed poll at 07.381. Forced command still running after the device came back (house elapsed seconds):

Seed	Window	Command
207533	654–888	187 / 2668 W
207535	579–812	170 / 2632 W (charge)
207537	213–510	187 / 3854 W
207538	1034–1209	187 / 4674 W

In 207534 the battery discharged 5000 W from 5.6 % to 5.0 % while FTW was not dispatching it. In 207537 polls failed from 15:15:00 to 15:15:50, and the driver stayed excluded until 15:20:01.

Change

go/cmd/ftw/driver_failure_default.go:

• An exclusion counts as an outage if, when it was excluded, the driver's last read had failed or the watchdog had it offline. It also counts if, while it is excluded, the watchdog takes it offline or a read fails before any good read since the exclusion. The last case covers a command that hits the dead link before a poll does, in an outage shorter than the 60 s watchdog (207536).
• On the first update tick with a good reading (not offline, last poll succeeded), the tracker:
  • clears the defaulted entry and returns the driver, so driver_default_mode goes out on that tick;
  • takes the driver back into dispatch, as the retry window does, with one refusal allowed before it is out again.
• If the trial command is refused while reads fail again, the next return retries it at once. If the device reads fine and refuses, that is a real refusal and the 5 min backoff applies as before. A device that has read fine since it refused keeps the full window, even if a later read fails.
• If driver_default_mode fails again, the registry's own recovery timer (100 ms doubling to 30 s) keeps retrying it.
• Third and fourth commits: after an early trial, a failed read does not count as an outage for one retry interval (5 min) from the trial, or until the excluded action is accepted. This stops the flaky-link storm (a device that refuses with good reads would get a default and a trial after every failed read). Before the fourth commit the guard had no end: a driver that dispatch left idle after its return kept it, so its next short dead-link outage held the inverter for the full 5 min again (review of 8cf230f).

What this does not fix

Taking the driver back into dispatch only opens the gate. Nothing goes to the device unless dispatch computes a target for it. On a Sungrow the resent default does nothing yet: the driver reports the default as held without writing it once EMS_WRITE_ATTEMPTS is reached. So the old forced command runs until FTW's next target. Rig replay of 207535 on this branch: after the return, status.dispatch stayed [] while grid was about -1 W, and the inverter held 170/2630 W for 67 s (233 s on master). The rig's R8 check will still fail on that seed until the device-drivers half lands.

Rig replay of this branch before the second commit (seeds 207535 to 207537, driver pin 57984984a9bf), seconds the inverter ran a forced command while up and not dispatched:

Seed	master	first commit
207537	264 s	19 s
207535	233 s + 171 s	67 s + 0 s
207536, first two outages	30 s and 44 s	16 s and 16 s
207536, 29 s outage	270 s	about 270 s

The second commit targets the last row. The rig has not been rerun on it.

Tests

In go/cmd/ftw/driver_failure_default_test.go:

• TestDriverExcludedDuringOutageGetsDefaultWhenItReadsAgain: a short outage (reads failing, no watchdog) and a long one (watchdog offline). Three battery commands are refused while offline. The test expects a default request and re-admission on the first fresh tick, and no second default after that.
• TestDriverRefusingAfterOutageKeepsRetryWindowWhileOnline: a refusal during a second dropout is retried on the next return. A refusal while reads are fine keeps the 5 min window, even with one failed read in between.
• TestDriverExcludedJustBeforeShortOutageGetsDefaultWhenItReadsAgain: the command is refused while the last read was good, then reads fail for 30 s, under the watchdog. Covers both the ErrControlBlocked path and three plain refusals. Expects a default and re-admission on the first good read.
• TestDriverRefusingOnFlakyLinkGetsOneEarlyTrial (third commit): a device that refuses with good reads while every other read fails. Over 5 min it gets one early trial and two defaults. On the second commit it got 75 trials and 150 defaults.
• TestSecondDeadLinkOutageAfterReturnGetsDefault (fourth commit): outage, return, then a gap, then a 29 s dead-link outage (207536 shape). With dispatch idle for 6 min, or a battery command accepted in the gap, the driver gets its default and comes back on the first good read. With dispatch idle for only 1 min it still waits out the window; that case pins the trade-off below.
• TestAcceptedTrialEndsGuardWhileOtherActionStreakRemains (test-only commit): on a hybrid inverter a refused curtail keeps the refusal record alive after the battery trial is accepted. A 29 s dead-link outage one minute later still gets its default and comes back on the first good read. It checks that an accepted trial ends the guard; with the reset line removed it fails (got [], want [sungrow]) and no other test catches it.

The first two fail on master (got [], want [sungrow]); the third fails on the first commit the same way. The fourth fails on the second commit. The fifth fails on the third commit (got [], want [sungrow], 6 min case). All pass now. The reviewer's TestReviewSecondDeadLinkOutageAfterReturn passes its 6 min and accepted cases; its 1 min case expects a return and fails by design (see Risk). The reviewer's edge tests (no release while reads fail or offline, no default storm when a returned driver is still blocked, Modbus exception with good reads keeps the full window) also pass. make verify and go test -race ./cmd/ftw pass. No real devices were touched.

Risk

Low. The change only affects drivers excluded while their reads were failing or offline, or whose next read after the exclusion failed. For those, the default can go out one extra time, and they rejoin dispatch sooner. A device that keeps dropping out gets one early trial each time it comes back, at most one per 5 min. A second dead-link outage that starts within 5 min of an early trial, before the excluded action was accepted, still waits out the 5 min window, as on master; the owner should decide whether that is acceptable. A device that refuses with good reads and whose very next read happens to fail gets one early trial; if it refuses that trial with good reads, the 5 min window applies, even if later reads fail. Reads failing at the exclusion or the watchdog taking it offline still count as an outage after a trial.

Stack and merge order

Only open PR touching driver_failure_default.go. Independent of #1911 (C9); they can merge in either order. Branch is up to date with master (7840c33).

🤖 Generated with Claude Code

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-27T16:58:27.301308Z	f922e32	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.