Repository navigation
Guard nightly sync against disk exhaustion and blocked index writes - #26
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The October 28–29 nightly syncs spent over an hour harvesting before every repository's search synchronization failed. The shared production filesystem reached 98% usage, causing Elasticsearch's flood-stage read-only block even while cluster health remained green. Redis image-cache use remained about 232 MiB; inactive Wherehence development runs and build outputs accumulated on the same disk.
Add a read-only preflight before repository discovery/harvesting. CI streams the checked-out standard-library Python script to the SSH host, so it takes effect without rebuilding the application image. It requires 15% available disk on the host and Elasticsearch filesystems, checks the target index's write/read-only/metadata blocks, and fails closed if Elasticsearch cannot be inspected. It neither removes data nor disables disk protections.
Validation: seven focused regression tests pass, Ruff passes, and deployment invariants pass. A live preflight correctly reports the low disk reserve and
read_only_allow_deleteblock. Operational recovery archives 13 explicitly approved inactive Wherehence run trees to the existing private Storage Box, checks SHA-256 and a full downloaded archive comparison, then removes unchanged local copies. Live databases and application mounts are excluded. Full nightly recovery verification follows disk recovery.