Repository navigation
Verify cache seeds without gating deploy smoke jobs #435
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
prk-Jr
wants to merge
7
commits into
main
Choose a base branch
from
fix/verify-cache-seeds
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
7 commits
Select commit
Hold shift + click to select a range
e3bfd10
Verify cache seed publication without gating deploy smokes
prk-Jr 523c3e1
Require exact cache publication and confine seed cleanup
prk-Jr 3373552
Verify source cache publication before restore probes
prk-Jr fba5216
Merge branch 'main' into fix/verify-cache-seeds
prk-Jr d43ba8b
Merge branch 'main' into fix/verify-cache-seeds
aram356 6e5a134
Merge branch 'main' into fix/verify-cache-seeds
prk-Jr 2650767
Verify compiler cache seeds with remote read-back hits
prk-Jr File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
62 changes: 62 additions & 0 deletions
62
.github/actions/setup-rust-build-cache/tests/publication.sh
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,62 @@ | ||
| #!/usr/bin/env bash | ||
| # Mutate the workflow itself so contract checks cannot merely test a detached if. | ||
| set -euo pipefail | ||
| ACTION_DIR=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd) | ||
| WORK_DIR=$(mktemp -d "${TMPDIR:-/tmp}/edgezero-cache-publication-test.XXXXXX") | ||
| trap 'rm -rf -- "$WORK_DIR"' EXIT | ||
| # shellcheck source=common.sh | ||
| source "$ACTION_DIR/tests/common.sh" | ||
| workflow="$ACTION_DIR/../../workflows/deploy-action.yml" | ||
|
|
||
| reject_mutation() { | ||
| local name="$1" expression="$2" diagnostic="$3" status=0 | ||
| yq "$expression" "$workflow" >"$WORK_DIR/workflow.yml" | ||
| CACHE_WORKFLOW_FILE="$WORK_DIR/workflow.yml" bash "$ACTION_DIR/tests/fixture.sh" >"$WORK_DIR/log" 2>&1 || status=$? | ||
| [[ "$status" == 1 ]] || fail "workflow guard accepted mutation: $name (exit $status)" | ||
| grep -Fq "$diagnostic" "$WORK_DIR/log" || { | ||
| cat "$WORK_DIR/log" >&2 | ||
| fail "mutation failed for an unrelated reason: $name" | ||
| } | ||
| } | ||
|
|
||
| for kind in source target; do | ||
| publication=".jobs.\"cache-restore-seed\".steps[] | select(.name == \"Require exact $kind seed publication\")" | ||
| reject_mutation 'missing final gate' \ | ||
| ".jobs.\"cache-restore-seed\".steps |= map(select(.name != \"Require exact $kind seed publication\"))" \ | ||
| 'missing exact publication assertion' | ||
| reject_mutation 'hardcoded first hit' \ | ||
| "($publication | .env.CACHE_SEED_FIRST_HIT) = \"true\"" \ | ||
| 'publication assertion ignores first lookup output' | ||
| reject_mutation 'hardcoded second hit' \ | ||
| "($publication | .env.CACHE_SEED_SECOND_HIT) = \"true\"" \ | ||
| 'publication assertion ignores second lookup output' | ||
| reject_mutation 'skipped final gate' \ | ||
| "($publication | .if) = \"false\"" \ | ||
| 'publication assertion can be skipped' | ||
| reject_mutation 'ignored final failure' \ | ||
| "($publication | .\"continue-on-error\") = true" \ | ||
| 'publication assertion failure is ignored' | ||
| reject_mutation 'gate always succeeds' \ | ||
| "($publication | .run) = \"true\"" \ | ||
| 'publication assertion accepts unverified outputs' | ||
| reject_mutation 'gate always fails' \ | ||
| "($publication | .run) = \"false\"" \ | ||
| 'publication assertion rejects a verified exact hit' | ||
| for step in "$kind-save-1" "$kind-verify-1" "$kind-save-2" "$kind-verify-2"; do | ||
| for input in path key; do | ||
| reject_mutation "$step wrong $input" \ | ||
| "(.jobs.\"cache-restore-seed\".steps[] | select(.id == \"$step\") | .with.$input) = \"wrong\"" \ | ||
| "$step uses a different cache $input" | ||
| done | ||
| if [[ "$step" == *-2 ]]; then | ||
| reject_mutation "$step wrong retry condition" \ | ||
| "(.jobs.\"cache-restore-seed\".steps[] | select(.id == \"$step\") | .if) = \"false\"" \ | ||
| "$step retries without a first-attempt miss" | ||
| else | ||
| reject_mutation "$step skipped initial attempt" \ | ||
| "(.jobs.\"cache-restore-seed\".steps[] | select(.id == \"$step\") | .if) = \"false\"" \ | ||
| "$step skips the initial publication attempt" | ||
| fi | ||
| done | ||
| done | ||
| printf 'source and target publication workflow mutation contracts passed\n' | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
58 changes: 58 additions & 0 deletions
58
.github/actions/setup-rust-build-cache/tests/seed-compiler-cache.sh
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,58 @@ | ||
| #!/usr/bin/env bash | ||
| set -euo pipefail | ||
|
|
||
| # Called by deploy-action.yml; seed.sh covers publication and cleanup failures. | ||
| # Publish each target before downstream jobs test first-build compiler reuse. | ||
| fixture="${GITHUB_WORKSPACE:?}/cache-restore-fixture" | ||
| target_dir=$(realpath "${CARGO_TARGET_DIR:?}") | ||
|
prk-Jr marked this conversation as resolved.
Resolved
|
||
| expected="$(realpath "${RUNNER_TEMP:?}")/edgezero-rust-cache/cache-restore-smoke-v2/target" | ||
| [[ "$target_dir" == "$expected" ]] || { | ||
| printf '::error::compiler seed must use its confined fixture target directory\n' >&2 | ||
| exit 1 | ||
| } | ||
| cd "$fixture" | ||
| host=$(rustc -vV | sed -n 's/^host: //p') | ||
| [[ "$host" =~ ^[A-Za-z0-9_][A-Za-z0-9_.-]*$ ]] || { | ||
| printf '::error::rustc must report a safe host target component\n' >&2 | ||
| exit 1 | ||
| } | ||
| # Every build must reach the remote cache, including the read-back build. | ||
| build_fresh() { | ||
| debug_dir="$target_dir/$target/debug" | ||
| [[ ! -L "$target_dir/$target" && ! -L "$debug_dir" ]] || { | ||
| printf '::error::compiler seed cleanup must not follow target symlinks\n' >&2 | ||
| exit 1 | ||
| } | ||
| rm -rf -- "$debug_dir" | ||
| sccache --zero-stats | ||
| cargo build --locked --lib --target "$target" | ||
| stats=$(sccache --show-stats --stats-format json) | ||
| printf 'Compiler seed for %s (attempt %s): %s\n' "$target" "$attempt" "$stats" | ||
| } | ||
|
|
||
| compiler_hit() { | ||
| jq -e '(.stats.cache_hits.counts.Rust // 0) > 0 and (.stats.cache_errors.counts.Rust // 0) == 0 and (.stats.cache_read_errors // 0) == 0 and (.stats.cache_write_errors // 0) == 0' <<<"$stats" >/dev/null | ||
| } | ||
|
|
||
| for target in "$host" wasm32-wasip1; do | ||
| for attempt in 1 2 3; do | ||
| build_fresh | ||
| if compiler_hit; then | ||
| break | ||
| fi | ||
| # The pinned backend can report a successful write despite failed upload | ||
| # finalization. Only a fresh build's remote hit verifies publication. | ||
| if jq -e '(.stats.cache_writes // 0) > 0 and (.stats.cache_errors.counts.Rust // 0) == 0 and (.stats.cache_read_errors // 0) == 0 and (.stats.cache_write_errors // 0) == 0' <<<"$stats" >/dev/null; then | ||
| build_fresh | ||
| if compiler_hit; then | ||
| break | ||
| fi | ||
| fi | ||
| if [[ "$attempt" == 3 ]]; then | ||
| printf '::error::failed to verify compiler cache publication for %s after three attempts\n' "$target" >&2 | ||
| exit 1 | ||
| fi | ||
| printf '::warning::compiler cache seed failed; retrying after backoff\n' | ||
| sleep "$((attempt * 60))" | ||
| done | ||
| done | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.