A CLI orchestrator for CIFAR-10 speedrun benchmarks. Runs published training scripts (airbench, hiverge, hlb) and your own model under a fair, cached, reproducible protocol on a single GPU.
Status: v0 in development. The full design spec lives in
docs/DESIGN.mdand is the canonical source of truth for the project.
The CIFAR-10 speedrun is an active competitive benchmark — train to 94% test accuracy in the shortest wallclock time on one GPU. Each record-holder open-sources their training script, so the leaderboard is a chain of GitHub repos. scriptbench takes that chain plus your own architecture, runs them all under a fair interleaved schedule, and produces structured results you can compare.
It is an orchestrator, not a training framework. Every script runs in a fresh subprocess with its own recipe (optimizer, schedule, augmentation). scriptbench captures stdout, samples GPU telemetry, parses results, and caches by fingerprint so unchanged code is never re-run.
- Python 3.11+
- uv — install once with
curl -LsSf https://astral.sh/uv/install.sh | sh(or see uv's install docs for pipx/Homebrew/Windows variants). uv manages the project venv, lockfile, and per-adapter environments. - NVIDIA GPU + CUDA drivers — required to actually run benchmarks. Pure-logic tests (
pytest -m 'not gpu') and all quality gates run on any machine.
# Clone upstream model repos
./scripts/setup_adapters.sh
# Install scriptbench (with torch + torchvision for adapter scripts)
uv sync --extra gpu
# Run a single airbench training and cache the result
uv run scriptbench run --models airbench --runs 1
# Run airbench, hiverge, hlb each 3 times under a fair schedule
uv run scriptbench run --models airbench,hiverge,hlb --runs 3
# Add your own model (your scripts live in models/, see models/README.md)
uv run scriptbench add --name mymodel --script models/my_model.py \
--kind speedrun --target-acc 0.94 \
--accuracy-regex "test_acc=([0-9.]+)" --time-regex "wall=([0-9.]+)s"
# Compare your model against airbench (uses the last 3 cached runs of each)
uv run scriptbench run --models mymodel --runs 3
uv run scriptbench compare --baseline airbench --challenger mymodel \
--metric time_to_target_seconds --runs 3
# Inspect cached runs / cache size / drop stale entries
uv run scriptbench list --runs
uv run scriptbench cache stats
uv run scriptbench cache clear --older-than 7dEvery command produces colored human output in a TTY and newline-delimited JSON otherwise (or with --output json). See docs/DESIGN.md §11 for the full CLI surface.
uv sync # install runtime + dev deps
uv run pre-commit install # installs both pre-commit and pre-push hooksPre-commit runs ruff format and ruff check (fast). Pre-push runs the full gate: pyright --strict, vulture, deptry, pylint (duplicate-code), and pytest -m 'not gpu' with the 90% coverage threshold. See docs/CODE-RULES.md §Enforcement.
Layered, with strict downward dependencies. See docs/DESIGN.md §5 for the full breakdown.
CLI (typer)
→ Orchestrator (schedule, retry)
→ Cache (fingerprint, lookup)
→ Runner (subprocess, GPU sample, parse)
→ Adapter (per-script logic)
→ Storage (SQLite + filesystem)
Subprocess isolation, always. Foreign training code is never imported into our process — eliminates dependency conflicts and GPU state leaks between models.
Your training scripts live in models/ — that directory is
checked into version control. Cloned upstream repos
(adapters/_repos/) are gitignored. See models/README.md
for the full registration walkthrough; the short version is "drop a
script in, run scriptbench add, then scriptbench run."
docs/DESIGN.md— full design spec; canonical source for everything belowdocs/TECH-STACK.md— language, libraries, toolingdocs/CODE-RULES.md— engineering standardsdocs/AGENT-RULES.md— how AI agents collaborate on this codebasedocs/papers/— relevant ML papers and notes (populated during v2; see DESIGN §13)docs/adapter-guides/— per-adapter reference (populated as adapters land)docs/notes/iteration-log.md— running log of experiments and decisionsmodels/README.md— how to register and run your own training scripts
MIT. Free to use, modify, and redistribute; copyright stays with LOME-AI LLC. See LICENSE.