scriptbench — Design Document

Status: Design spec, ready for implementation Audience: Implementing developer (human or AI agent). This document is the developer's only context. Everything required to build scriptbench is here. Last updated: 2026-05-05

1. What scriptbench is

scriptbench is a CLI tool that orchestrates CIFAR-10 training scripts from multiple competing repositories, runs them under a fair scheduling protocol, and produces structured benchmark results. Its primary purpose is to let a user (typically an ML researcher or AI coding agent) test their own model architecture against published state-of-the-art speedrun scripts on identical hardware, with reproducible and cacheable results.

It is an orchestrator, not a training framework. It does not train models itself. It runs other people's training scripts as subprocesses, captures their output, and aggregates results. Each model brings its own training recipe (optimizer, schedule, augmentation) — scriptbench does not normalize these. The point is to compare each architecture as its authors designed it, not under a common loop.

Why this exists

The CIFAR-10 speedrun is an active competitive benchmark — train a model to 94% test accuracy in the shortest possible wallclock time on a single GPU. The current public record (as of late 2025) is ~1.98 seconds on an A100 (hiverge/cifar10-speedrun), preceded by ~2.59s (KellerJordan/cifar10-airbench), preceded by ~6.3s (tysam-code/hlb-CIFAR10). Each record-holder open-sources their full script, so "the leaderboard is the code." There is no central submission portal; the chain of GitHub repos is the leaderboard.

Researchers iterating on new architectures want to measure themselves against these baselines on their own hardware. Doing this manually is tedious and error-prone — different scripts have different output formats, different timing conventions, different stopping criteria. scriptbench standardizes the orchestration layer so iteration is fast and comparisons are honest.

Primary user

The intended user is an AI coding agent driving experiments on behalf of a human researcher, or the researcher directly via CLI. Output is optimized for both: every command produces verbose JSON for agents and a colored human-readable view for terminals.

2. Goals and non-goals

Goals

Run multiple CIFAR-10 training scripts in a fair, reproducible, cached manner
Support adding new scripts (including the user's own model) with minimal boilerplate
Produce structured, queryable, machine-readable results with rich metadata
Cache results aggressively so unchanged code is never re-run
Be agent-friendly: JSON output, deterministic CLI surface, clear error messages
Remain a small, hackable codebase — the developer (or an agent) should be able to read the whole thing in an hour

Non-goals

Not a training framework. Does not implement optimizers, schedulers, or model layers.
Not a hosted service. Local-only. No multi-user, no auth, no remote execution.
Not a hyperparameter sweeper. Each script runs as the author wrote it.
Not a model zoo with one shared training loop. Each script honors its own recipe.
Not focused on apples-to-apples training method normalization. We compare systems (model + recipe + optimizer + augmentation), not isolated architectures.
Not a generic ML benchmarking framework. CIFAR-10 first; other tasks possibly later but out of scope for v1.

3. Design principles

These principles drive every decision below. When in doubt, defer to them.

Iteration speed over completeness. A working tool with three adapters next week beats a perfect tool with fifteen adapters next year. Cache aggressively, ship incrementally.
Subprocess isolation, always. Every script runs in a fresh subprocess. We never import foreign training code into our own process. This eliminates dependency conflicts, GPU state leakage between models, and cascading failures.
Honor the original recipe. Adapters do not modify training scripts. They invoke them as published, parse their output, and report what happened. If a script was tuned for an A100, that's how it runs on a 5090.
Both audiences, every output. Every command produces equivalent human and machine-readable views from the same underlying data. Never log information to one that doesn't appear in the other.
Reproducibility receipts. Every run records its full environment (GPU, drivers, library versions, script hash, seed). Six months later, you can answer "what produced this number?"
Fingerprint-based cache, not time-based. A run is cached by what it depends on, not when it happened. Edit the script → cache misses. Don't edit → instant return forever.
Fail loud, fail typed. Errors raise typed exceptions with context. Never silently degrade.
No cooldowns, no warmups in the hot loop. User explicitly traded thermal/cold-start variance for iteration speed. The interleaved schedule absorbs most of this.

4. Tech stack — final decisions

Component	Choice	Why
Language	Python 3.11+	Ecosystem proximity to ML scripts, agent fluency
Package manager	`uv`	Fast, deterministic, makes per-adapter venvs trivial
CLI framework	`typer`	Typed function-signature CLI, zero boilerplate
Schemas / validation	`pydantic` v2	Runtime validation + JSON serialization from types
Human terminal output	`rich`	Tables, progress bars, syntax-highlighted JSON
GPU monitoring	`nvidia-ml-py`	Canonical NVML binding, low overhead
Subprocess management	stdlib `subprocess`	Adequate; no async needed for sequential runs
Database	stdlib `sqlite3`	Zero-config, single-file, fast for our query volume
Test framework	`pytest`	Standard
Type checker	`pyright` (strict)	Stronger than mypy, faster, agent-friendly errors
Linter / formatter	`ruff`	Fast, single tool, replaces black + flake8 + isort
Profiling integration	NVIDIA Nsight Systems via `nsys` CLI	No code coupling, captures everything

Rejected alternatives

TypeScript / bun: Considered seriously. Stronger type system, native JSON. Rejected because every script we orchestrate is Python; ecosystem proximity wins. Note: this is genuinely close — if future scriptbench grows beyond pure orchestration, the choice was Python; if it stays pure orchestration, TypeScript would have been equally defensible.
Rust / Go: Single-binary distribution is nice, but no Python interop is a real cost for ML tooling. Slower iteration loop hurts a tool whose entire point is fast iteration.
Bash + jq: Fine for a 50-line prototype; falls apart on retry logic, structured errors, schedule generation.

5. Architecture overview

scriptbench is a layered architecture with strict downward dependencies:

┌───────────────────────────────────────────┐
│ CLI layer (typer commands)                │  cli.py
├───────────────────────────────────────────┤
│ Orchestrator (schedule, run loop, retry)  │  orchestration/
├───────────────────────────────────────────┤
│ Cache layer (fingerprint, lookup, store)  │  cache/
├───────────────────────────────────────────┤
│ Runner (subprocess, GPU sampling, parse)  │  orchestration/runner.py
├───────────────────────────────────────────┤
│ Adapter (per-script logic)                │  adapters/
├───────────────────────────────────────────┤
│ Storage (SQLite + filesystem)             │  cache/store.py
└───────────────────────────────────────────┘

Each layer depends only on layers below it. The CLI never touches subprocesses. The runner never knows about caching. The cache never knows what an adapter is — it just sees opaque fingerprints and result blobs. This makes every layer independently testable.

Module structure (full directory tree)

scriptbench/
  pyproject.toml             # uv-managed, declares deps and entrypoint
  README.md                  # quick start
  
  scriptbench/
    __init__.py
    __main__.py              # python -m scriptbench → cli.app()
    cli.py                   # typer entrypoint, command definitions
    config.py                # paths, env-var overrides, defaults
    schemas.py               # all pydantic models (RunResult, etc.)
    errors.py                # typed exception hierarchy
    
    adapters/
      __init__.py
      base.py                # Adapter Protocol
      _registry.py           # discover and load adapters
      airbench.py            # KellerJordan/cifar10-airbench
      hiverge.py             # hiverge/cifar10-speedrun
      hlb.py                 # tysam-code/hlb-CIFAR10
      user_script.py         # generic adapter for "add your own"
    
    orchestration/
      __init__.py
      schedule.py            # interleaved schedule generator (pure)
      runner.py              # one-run execution: subprocess + sampling + parse
      loop.py                # outer loop: schedule → cache check → runner → store
      gpu_sampler.py         # NVML sampling thread
      env_capture.py         # environment fingerprinting
    
    cache/
      __init__.py
      fingerprint.py         # script hashing, dep walking, key generation
      store.py               # SQLite + filesystem read/write
      policy.py              # invalidation rules
    
    output/
      __init__.py
      human.py               # rich-based formatters
      json_out.py            # newline-delimited JSON for agents
      summary.py             # comparison, leaderboards, top-2-of-3 aggregation
    
    docs_index.py            # serves docs/ directory as searchable index
  
  docs/                      # human + agent reference material (see §13)
    papers/
    notes/
    adapter-guides/
  
  tests/
    test_schedule.py
    test_fingerprint.py
    test_cache.py
    test_runner.py
    test_adapters/
      test_airbench.py
      ...
    fixtures/
      sample_outputs/        # captured stdout from each script for parser tests
  
  scripts/                   # NOT to be confused with model scripts; dev tooling
    setup_adapters.sh        # clones each upstream repo into adapters/_repos/

6. Data model (schemas)

All defined in schemas.py using pydantic v2. These are the canonical shapes; serialize as JSON, store in SQLite as JSON columns plus indexed scalars.

`RunContext`

Everything an adapter needs to run.

class RunContext(BaseModel):
    adapter_name: str
    seed: int
    extra_args: list[str] = []         # adapter-specific CLI args
    profile: bool = False              # enable Nsight profiling
    timeout_seconds: int = 600         # kill if exceeds
    working_dir: Path
    env_overrides: dict[str, str] = {}

`RunResult`

The canonical output of one run. Discriminated union by kind.

class GpuStats(BaseModel):
    peak_memory_mb: int
    avg_utilization_pct: float
    avg_power_w: float
    temp_start_c: int
    temp_end_c: int
    samples_path: Path | None          # jsonl of per-second samples

class EpochRecord(BaseModel):
    epoch: int
    loss: float | None
    accuracy: float | None
    time_s: float

class EnvCapture(BaseModel):
    python_version: str
    torch_version: str
    cuda_version: str
    cudnn_version: str
    gpu_model: str
    gpu_uuid: str
    driver_version: str
    hostname: str
    timestamp_utc: datetime

class RunResultBase(BaseModel):
    schema_version: int = 1
    run_id: str                        # UUID
    adapter_name: str
    adapter_version: str
    fingerprint: str                   # cache key
    seed: int
    
    wallclock_seconds: float           # measured by orchestrator
    script_reported_seconds: float | None  # measured by script itself, if reported
    
    exit_code: int
    success: bool
    error_message: str | None = None
    
    epochs: list[EpochRecord] = []
    gpu: GpuStats
    env: EnvCapture
    
    stdout_path: Path
    stderr_path: Path
    profile_path: Path | None = None
    
    cached: bool = False               # True if returned from cache
    cache_hit_count: int = 0           # set by cache layer

class SpeedrunResult(RunResultBase):
    kind: Literal["speedrun"] = "speedrun"
    target_accuracy: float             # e.g. 0.94
    final_accuracy: float
    time_to_target_seconds: float | None  # null if target not reached

class AccuracyResult(RunResultBase):
    kind: Literal["accuracy"] = "accuracy"
    peak_accuracy: float
    final_accuracy: float
    epoch_budget: int

RunResult = Annotated[
    SpeedrunResult | AccuracyResult,
    Field(discriminator="kind"),
]

`ModelConfig`

Registered adapter configuration.

class ModelConfig(BaseModel):
    name: str                          # unique
    adapter_class: str                 # dotted path
    kind: Literal["speedrun", "accuracy"]
    description: str
    paper_url: str | None = None
    repo_url: str | None = None
    docs_path: Path | None = None      # within docs/

`ScheduleEntry`

class ScheduleEntry(BaseModel):
    position: int                      # 0-indexed in schedule
    adapter_name: str
    seed: int
    run_index: int                     # 0..(num_runs-1) for this model

class Schedule(BaseModel):
    entries: list[ScheduleEntry]
    num_models: int
    runs_per_model: int
    interleaved: bool

`AggregatedScore`

class AggregatedScore(BaseModel):
    adapter_name: str
    runs: list[RunResult]
    score: float                       # the official number (top-2 mean)
    score_metric: str                  # "time_to_94" or "peak_accuracy"
    all_run_values: list[float]
    discarded_run_index: int           # which run was the worst
    std_dev: float

7. Adapter protocol

Adapters are the extension point. Adding a new script means writing one file.

# adapters/base.py

from typing import Protocol, runtime_checkable
from ..schemas import RunContext, RunResult

@runtime_checkable
class Adapter(Protocol):
    name: str
    version: str                       # bump to invalidate cache
    kind: Literal["speedrun", "accuracy"]
    
    def fingerprint_inputs(self, ctx: RunContext) -> dict:
        """All inputs that affect the result. Used for cache key.
        
        Must include: adapter version, script file hash, dependency hashes,
        seed, and any adapter-specific config. Must NOT include: timestamps,
        observed metrics, run IDs.
        """
    
    def build_command(self, ctx: RunContext) -> list[str]:
        """Subprocess argv to invoke the script."""
    
    def parse_output(
        self,
        stdout: str,
        stderr: str,
        exit_code: int,
        wallclock_s: float,
        gpu_stats: GpuStats,
        env: EnvCapture,
    ) -> RunResult:
        """Convert raw process output into a typed RunResult."""
    
    def expected_runtime_seconds(self) -> float:
        """Best estimate of how long one run takes. Used for progress bars
        and timeout calibration. May be wildly wrong; that's fine."""
    
    def repo_path(self) -> Path:
        """Where this script lives on disk."""

Adapters are pure orchestration — they do not import the training script's modules. They only read its files (for hashing) and parse its stdout.

Initial adapters to implement (in order)

airbench.py — wraps cifar10-airbench/airbench94_muon.py. Parses Keller Jordan's stdout format.
hiverge.py — wraps cifar10-speedrun/cifar10_speedrun.py. Similar parsing.
hlb.py — wraps hlb-CIFAR10/main.py. Tysam's custom progress format.
user_script.py — generic adapter taking a script path + a config (regex for accuracy line, regex for time line, kind="speedrun" or "accuracy"). This is what the user uses for their own model.

Each adapter ships with fixture files in tests/fixtures/sample_outputs/<adapter>/ containing real captured stdout/stderr from successful runs. Parser tests run against these fixtures so we don't need a GPU to run unit tests.

8. Scheduling

Algorithm

Generate a schedule of (adapter, seed, run_index) triples that interleaves runs across models so each model gets early/middle/late slots in the session. This averages out thermal drift, system noise, and GPU warm-up effects across all models — the user explicitly chose to skip cooldowns/warmups in favor of iteration speed, so the schedule is doing the heavy lifting on fairness.

For N models × K runs, the canonical fair schedule uses K random permutations of [0..N), concatenated (a randomized complete block design). Each block is a full permutation; concatenating K blocks gives K runs of each model with the within-block order varied.

# orchestration/schedule.py

import random
from ..schemas import Schedule, ScheduleEntry

def make_interleaved_schedule(
    adapter_names: list[str],
    runs_per_model: int,
    seeds: list[int] | None = None,
    rng_seed: int = 0,
) -> Schedule:
    rng = random.Random(rng_seed)
    if seeds is None:
        seeds = list(range(runs_per_model))
    assert len(seeds) == runs_per_model
    
    entries: list[ScheduleEntry] = []
    for run_index in range(runs_per_model):
        block = list(range(len(adapter_names)))
        rng.shuffle(block)
        for slot, model_idx in enumerate(block):
            entries.append(ScheduleEntry(
                position=len(entries),
                adapter_name=adapter_names[model_idx],
                seed=seeds[run_index],
                run_index=run_index,
            ))
    
    return Schedule(
        entries=entries,
        num_models=len(adapter_names),
        runs_per_model=runs_per_model,
        interleaved=True,
    )

Scoring: top-2-of-3 average

# output/summary.py

def aggregate_top_k_of_n(results: list[RunResult], keep_top: int = 2) -> AggregatedScore:
    """For speedrun: lower time is better. For accuracy: higher acc is better.
    Sort, keep the best `keep_top`, take the mean."""
    ...

For speedrun results, the sortable metric is time_to_target_seconds; lower is better. For accuracy results, it's peak_accuracy; higher is better. The discarded run is logged but not used in the score.

Same seed across models per run-index

Run index 0 uses seed 0 for every model. Run index 1 uses seed 1 for every model. This isolates model differences from data-ordering differences. Within a single model, seeds 0/1/2 give three independent samples of its variance.

Pre-flight CUDA context init

Before the schedule starts, the orchestrator does:

import torch
torch.zeros(1, device="cuda")
torch.cuda.synchronize()

This eats the worst of the cold-start CUDA init before any timed run, without being a real warmup. Cheap insurance; user explicitly approved this.

9. Runner

The runner is responsible for one run from invocation to typed result.

# orchestration/runner.py

class Runner:
    def __init__(self, gpu_sampler: GpuSampler, env_capturer: EnvCapturer):
        ...
    
    def run_one(self, adapter: Adapter, ctx: RunContext) -> RunResult:
        # 1. Snapshot environment
        env = self.env_capturer.capture()
        
        # 2. Start GPU sampler thread
        self.gpu_sampler.start()
        
        # 3. Build command and invoke subprocess
        cmd = adapter.build_command(ctx)
        start = time.monotonic()
        proc = subprocess.Popen(cmd, stdout=PIPE, stderr=PIPE, cwd=adapter.repo_path())
        
        # 4. Wait with timeout
        try:
            stdout_b, stderr_b = proc.communicate(timeout=ctx.timeout_seconds)
        except subprocess.TimeoutExpired:
            proc.kill()
            raise AdapterTimeoutError(...)
        
        wallclock = time.monotonic() - start
        
        # 5. Stop sampler, get GPU stats
        gpu_stats = self.gpu_sampler.stop()
        
        # 6. Persist stdout/stderr to logs/ before parsing
        stdout_path = self._persist(stdout_b, ...)
        stderr_path = self._persist(stderr_b, ...)
        
        # 7. Parse via adapter
        result = adapter.parse_output(
            stdout=stdout_b.decode(),
            stderr=stderr_b.decode(),
            exit_code=proc.returncode,
            wallclock_s=wallclock,
            gpu_stats=gpu_stats,
            env=env,
        )
        
        return result

GPU sampling thread

# orchestration/gpu_sampler.py

class GpuSampler:
    """Samples NVML once per second on a background thread.
    Aggregates into GpuStats at stop()."""
    
    def __init__(self, sample_interval_s: float = 1.0):
        ...
    
    def start(self) -> None: ...
    
    def stop(self) -> GpuStats:
        # Returns peak memory, avg utilization, avg power, temp delta
        # Also writes per-sample jsonl to gpu_samples_path
        ...

Sample fields per tick: timestamp, gpu_util_pct, mem_used_mb, power_w, temp_c, sm_clock_mhz. These get written to a .jsonl file alongside the run's other artifacts. The aggregate (peak, mean) goes into GpuStats for the result.

Profiling mode

When ctx.profile == True, the runner wraps the command:

nsys profile --stats=true --output=<path>.nsys-rep <original command>

This produces a binary trace file plus a stats summary printed to stdout. The trace path goes into RunResult.profile_path. Profiling adds 5–30% overhead, so timing is no longer comparable to leaderboard. Use only when iterating on a single model.

The stats summary lines (top kernels by time) get parsed into a separate optional field on the result, top_kernels: list[KernelTiming], so agents can see hot spots without parsing the binary trace.

10. Cache layer

The cache is the feature most responsible for iteration speed. Get it right.

Fingerprint composition

def compute_fingerprint(adapter: Adapter, ctx: RunContext) -> str:
    components = {
        "adapter_name": adapter.name,
        "adapter_version": adapter.version,
        "script_hash": hash_file(adapter.script_path()),
        "deps_hash": hash_local_imports(adapter.script_path()),
        "seed": ctx.seed,
        "extra_args": ctx.extra_args,
        "profile": ctx.profile,
        "gpu_uuid": current_gpu_uuid(),
        "torch_version": torch.__version__,
        "cuda_version": cuda_version(),
        "schema_version": RunResult.SCHEMA_VERSION,
    }
    # Plus anything adapter.fingerprint_inputs() returns
    components.update(adapter.fingerprint_inputs(ctx))
    
    canonical = json.dumps(components, sort_keys=True)
    return hashlib.sha256(canonical.encode()).hexdigest()

Local import walking

# cache/fingerprint.py

def hash_local_imports(script_path: Path) -> str:
    """Parse the script with ast, find imports of local modules,
    recursively hash those files. Returns combined hash."""
    
    visited: set[Path] = set()
    
    def walk(path: Path) -> None:
        if path in visited:
            return
        visited.add(path)
        tree = ast.parse(path.read_text())
        for node in ast.walk(tree):
            if isinstance(node, (ast.Import, ast.ImportFrom)):
                for name in extract_module_names(node):
                    local = resolve_local_module(name, path.parent)
                    if local is not None:
                        walk(local)
    
    walk(script_path)
    
    h = hashlib.sha256()
    for p in sorted(visited):
        h.update(p.name.encode())
        h.update(p.read_bytes())
    return h.hexdigest()

Stdlib and site-packages imports are excluded — we trust torch.version to capture relevant changes there. Only files inside the adapter's repo directory count as "local."

Storage layout

.scriptbench/                 # default cache root, configurable
  cache.db                    # SQLite metadata
  runs/
    <YYYY-MM-DD>/
      <run_id>/
        result.json
        stdout.log
        stderr.log
        gpu_samples.jsonl
        profile.nsys-rep      # only if --profile
  scripts/
    <script_hash>.py          # snapshot of the exact script that ran
  config.toml                 # user settings

SQLite schema

CREATE TABLE runs (
    run_id          TEXT PRIMARY KEY,
    fingerprint     TEXT NOT NULL,
    adapter_name    TEXT NOT NULL,
    adapter_version TEXT NOT NULL,
    seed            INTEGER NOT NULL,
    success         INTEGER NOT NULL,
    score           REAL,                  -- main metric (time_to_94 or peak_acc)
    timestamp_utc   TEXT NOT NULL,
    result_path     TEXT NOT NULL,         -- relative to cache root
    schema_version  INTEGER NOT NULL
);

CREATE INDEX idx_runs_fingerprint ON runs(fingerprint);
CREATE INDEX idx_runs_adapter ON runs(adapter_name);
CREATE INDEX idx_runs_timestamp ON runs(timestamp_utc);

CREATE TABLE annotations (
    run_id          TEXT NOT NULL,
    key             TEXT NOT NULL,
    value           TEXT NOT NULL,
    PRIMARY KEY (run_id, key),
    FOREIGN KEY (run_id) REFERENCES runs(run_id)
);

runs is append-only and immutable. annotations is the mutable side table for user notes ("flagged as anomalous", "this is the v2 architecture", etc.).

Cache lookup flow

# cache/store.py

class Cache:
    def lookup(self, fingerprint: str) -> RunResult | None:
        row = self.db.execute(
            "SELECT result_path FROM runs WHERE fingerprint = ? AND success = 1 ORDER BY timestamp_utc DESC LIMIT 1",
            (fingerprint,),
        ).fetchone()
        if row is None:
            return None
        return self._load_result(row["result_path"])
    
    def store(self, result: RunResult) -> None:
        # Atomic write: tmp file, then rename
        ...
        self.db.execute("INSERT INTO runs VALUES (...)", ...)
        self.db.commit()
    
    def clear(
        self,
        adapter_name: str | None = None,
        older_than: timedelta | None = None,
        fingerprint: str | None = None,
    ) -> int:
        """Returns count of entries removed."""
        ...

Cache invalidation rules

The cache is automatic-by-fingerprint; the user only intervenes for unusual situations.

Default: cache hit → return immediately, log cached: true in output.
--no-cache: always run; do still store the new result.
--refresh-cache: invalidate matching fingerprint, then run, then store.
bench cache clear: remove all entries.
bench cache clear --model X: remove only entries for adapter X.
bench cache clear --older-than 7d: remove old entries.
Schema bump: when RunResult.SCHEMA_VERSION changes, all entries with old schema are treated as misses (but not deleted, in case user wants to inspect).
Failed runs are not cached by default. Add --cache-failures to override.

11. CLI surface

All commands defined in cli.py using typer. Every command supports --output {human,json} (default human in TTY, json otherwise) and --verbose.

Commands

scriptbench run
    --models <names>            comma-separated; or "all"
    --runs <n>                  default 3
    --interleave / --no-interleave   default true
    --seeds <ints>              comma-separated; default 0,1,2,...
    --no-cache                  always run, don't read cache
    --refresh-cache             invalidate then run
    --profile                   enable Nsight Systems
    --timeout <seconds>         default 600
    --output {human,json}       default depends on TTY
    --output-file <path>        also write results here
    --rng-seed <int>            schedule generator seed; default 0

scriptbench list                List registered adapters
    --runs                      Or list past runs instead
    --filter <expr>             SQL-ish filter for past runs

scriptbench compare
    --baseline <name>
    --challenger <name>
    --metric {time_to_94, peak_accuracy, ...}
    --runs <n>                  use last N runs of each
    --output {human,json}

scriptbench cache stats         Show cache size, hit count, oldest entry
scriptbench cache clear         Remove all cache
scriptbench cache clear --model <name>
scriptbench cache clear --older-than <duration>
scriptbench cache invalidate --fingerprint <hash>

scriptbench add                 Register a user script as an adapter
    --name <id>
    --script <path>
    --kind {speedrun, accuracy}
    --target-acc <float>        for speedrun
    --epoch-budget <int>        for accuracy
    --accuracy-regex <pattern>  parse final accuracy from stdout
    --time-regex <pattern>      parse script-reported time

scriptbench remove --name <id>  Unregister an adapter

scriptbench docs                Open the docs/ directory in the browser
scriptbench docs list           List all registered docs
scriptbench docs show <topic>   Print a doc to stdout

scriptbench env                 Print captured environment (for bug reports)
scriptbench version

Output modes

Human mode: rich tables, live-updating progress bars during runs, syntax-highlighted JSON for the final summary, colored deltas in compare.

JSON mode: newline-delimited JSON. One object per event:

{"event": "schedule_generated", "schedule": {...}}
{"event": "run_started", "position": 3, "adapter": "airbench", "seed": 1}
{"event": "run_completed", "result": {...}} — full RunResult
{"event": "run_cached", "result": {...}} — cache hit
{"event": "session_summary", "scores": [...], "schedule": {...}}

Agents stream-parse this. Critically, run_completed events arrive incrementally during long sessions, so the agent can see partial progress without waiting for the whole session.

Exit codes

0: success, all runs completed (cached or not)
1: at least one run failed but session continued
2: configuration error (bad args, missing adapter)
3: cache corruption detected
4: fatal environment error (no GPU, NVML unavailable)
130: user interrupt (Ctrl+C)

12. Environment capture

Every result records the full environment so a number can be reproduced months later.

# orchestration/env_capture.py

def capture_env() -> EnvCapture:
    return EnvCapture(
        python_version=platform.python_version(),
        torch_version=_torch_version(),
        cuda_version=_cuda_runtime_version(),
        cudnn_version=_cudnn_version(),
        gpu_model=_nvml_gpu_name(),
        gpu_uuid=_nvml_gpu_uuid(),
        driver_version=_nvml_driver_version(),
        hostname=socket.gethostname(),
        timestamp_utc=datetime.now(timezone.utc),
    )

Implementation notes:

Use nvidia-ml-py for GPU info — do not shell out to nvidia-smi for capture (slow, parsing-prone)
Torch version captured by importing torch in a subprocess; the orchestrator's torch may not match the adapter's torch if per-adapter venvs are used
GPU UUID is included in the fingerprint — same model on two different physical 5090s will have separate caches. This is intentional; thermal/binning differences can affect speedrun timings.

13. Documentation space (`docs/`)

The user explicitly wants a place for both humans and AI agents to learn about the field. The docs/ directory is that place. It is part of the repository (not generated, not gitignored) and ships with scriptbench.

Structure

docs/
  README.md                    # entry point, what's here, how to use
  
  papers/
    README.md                  # bibliography with one-line summaries
    speedrun/
      jordan-2024-cifar10-3.29s.pdf       # arXiv 2404.00498
      jordan-2024-cifar10-3.29s.notes.md
      page-2018-how-to-train-resnet.md     # link + notes (not a paper, blog series)
      hiverge-2025-blog.md                 # link + notes
    optimizers/
      muon-2024.pdf
      muon-2024.notes.md
      lookahead-2019.pdf
      adamw-2017.pdf
    architectures/
      resnet-2015.pdf
      preact-resnet-2016.pdf
      densenet-2017.pdf
      vit-2020.pdf
    augmentation/
      mixup-2017.pdf
      cutmix-2019.pdf
      autoaugment-2018.pdf
    background/
      cifar10-krizhevsky-2009.pdf
      bag-of-tricks-2018.pdf
  
  notes/                       # original notes by users/agents
    iteration-log.md           # running log of experiments
    failure-modes.md           # things that broke and why
    gpu-quirks/
      rtx-5090.md              # 5090-specific gotchas
      a100.md
  
  adapter-guides/              # per-adapter reference
    airbench.md
    hiverge.md
    hlb.md
    user_script.md             # how to write your own adapter
  
  reading-order.md             # suggested learning path for newcomers

Content guidelines

Papers: store the actual PDF wherever copyright permits; otherwise a .md file with the citation, abstract, and link. Each paper gets a sibling .notes.md with: one-paragraph plain-English summary, key result, why it matters for scriptbench, related work in our docs.
Notes: free-form. Iteration log entries are timestamped and signed by the agent or user.
Adapter guides: each one explains what the script does, what its outputs mean, what its training recipe is (optimizer, schedule, augmentation), what the published numbers are, and what's known to break.

Initial papers to include (developer should fetch and add these)

The developer should populate docs/papers/ with the following on initial setup. PDFs should be downloaded from arXiv where available; otherwise create a .md file with the citation and link.

Speedrun-specific:

Jordan, K. (2024). "94% on CIFAR-10 in 3.29 Seconds on a Single GPU." arXiv:2404.00498. Required — describes the airbench method.
Page, D. (2018). "How to Train Your ResNet" (8-part blog series). The original sub-30-second-CIFAR-10 work that started the speedrun lineage. Link: https://myrtle.ai/learn/how-to-train-your-resnet/
Hiverge (2025). CIFAR-10 speedrun blog post. Link: https://www.hiverge.ai/blog/cifar-speedrun. Describes the AI-evolved 1.98s record.

Optimizers:

Jordan, K. (2024). "Muon: An optimizer for hidden layers in neural networks." Notes/blog. The optimizer behind both airbench94_muon and hiverge.
Zhang et al. (2019). "Lookahead Optimizer: k steps forward, 1 step back." arXiv:1907.08610. Used in pre-Muon airbench.
Loshchilov & Hutter (2017). "Decoupled Weight Decay Regularization." arXiv:1711.05101. AdamW.

Architectures (for context on the broader space):

He et al. (2015). "Deep Residual Learning for Image Recognition." arXiv:1512.03385. ResNet.
He et al. (2016). "Identity Mappings in Deep Residual Networks." arXiv:1603.05027. Pre-activation ResNet.
Huang et al. (2017). "Densely Connected Convolutional Networks." arXiv:1608.06993. DenseNet.
Dosovitskiy et al. (2020). "An Image is Worth 16x16 Words." arXiv:2010.11929. ViT.

Augmentation & training tricks:

Zhang et al. (2017). "mixup: Beyond Empirical Risk Minimization." arXiv:1710.09412.
He et al. (2018). "Bag of Tricks for Image Classification." arXiv:1812.01187. Critical reading for understanding why training recipes matter.

Background:

Krizhevsky, A. (2009). "Learning Multiple Layers of Features from Tiny Images." The original CIFAR-10 paper.

The docs/papers/README.md file should be a real bibliography — one line per paper with link, year, and a one-sentence "why it matters." Make it scannable.

`scriptbench docs` command

The CLI exposes the docs directory:

scriptbench docs              # open docs/ in default browser/editor
scriptbench docs list         # list all docs with titles
scriptbench docs show <topic> # print a markdown file to stdout

This is intentionally simple. We don't ship a docs server. The point is to make the directory discoverable to agents that don't know to look there otherwise.

14. Error handling

Typed exception hierarchy in errors.py:

class ScriptbenchError(Exception): ...

class AdapterError(ScriptbenchError): ...
class AdapterNotFoundError(AdapterError): ...
class AdapterParseError(AdapterError): ...
class AdapterTimeoutError(AdapterError): ...
class AdapterScriptFailedError(AdapterError): ...

class CacheError(ScriptbenchError): ...
class CacheCorruptError(CacheError): ...
class CacheSchemaMismatchError(CacheError): ...

class EnvironmentError(ScriptbenchError): ...
class NoGpuError(EnvironmentError): ...
class NvmlUnavailableError(EnvironmentError): ...

class ConfigurationError(ScriptbenchError): ...

CLI commands catch these at the top level and translate to:

A human-readable error message (rich-formatted, colored)
A structured JSON error event in JSON output mode
The appropriate exit code

Stack traces are suppressed by default but printed with --verbose or SCRIPTBENCH_DEBUG=1.

Run-level failures

If a single run fails (script crashes, parser fails, OOM), the orchestrator:

Logs the failure
Continues with the rest of the schedule
Marks the result as success: false in the output
Counts the failed run toward retry quota (default: 1 retry per run)

If after retries a model has fewer than keep_top successful runs, its score is null and a warning is emitted.

15. Testing strategy

tests/
  test_schedule.py       # property: every model appears K times, every position covered
  test_fingerprint.py    # same inputs → same hash; different inputs → different hash
  test_cache.py          # round-trip, invalidation, atomic write under failure
  test_runner.py         # uses subprocess fixtures; verifies parsing + GPU sampling shape
  test_adapters/
    test_airbench.py     # parser correctness against captured stdout fixtures
    test_hiverge.py
    test_hlb.py
    test_user_script.py
  test_cli.py            # smoke test of every command with --help and basic invocations
  fixtures/
    sample_outputs/
      airbench/successful_run_001.stdout.txt
      airbench/successful_run_002.stdout.txt
      airbench/oom_failure_001.stdout.txt
      airbench/oom_failure_001.stderr.txt
      hiverge/...
      hlb/...

GPU-required tests are marked @pytest.mark.gpu and skipped by default. The pre-push gate runs the non-GPU suite. A separate pytest -m gpu invocation runs end-to-end on a real machine.

Property-based testing (via hypothesis) for schedule.py and fingerprint.py — these are pure functions with strong invariants and benefit from generative testing.

16. Implementation phases

v0 — minimum viable (target: one weekend)

cli.py with run, list, cache clear
schemas.py with RunResult (just SpeedrunResult, no AccuracyResult yet)
adapters/base.py + adapters/airbench.py
orchestration/runner.py (subprocess + basic GPU sampling)
orchestration/schedule.py
cache/ with file-based fingerprint→JSON lookup, no SQLite yet
output/human.py and output/json_out.py
Basic tests on schedule and fingerprint
README

This works end to end with airbench. ~600-800 lines.

v1 — usable (target: second weekend)

Add hiverge and hlb adapters
Add user_script.py adapter
SQLite metadata layer
compare command
add and remove commands
AccuracyResult variant
Full cache subcommand suite
Profiling mode

v2 — polished (third weekend onward)

docs/ directory populated with papers and notes
docs command
Per-adapter venv support via uv
Annotations table
More tests
Reading-order doc

Beyond v2: optional features like multi-GPU, custom metrics, hyperparameter sweeps within an adapter.

17. Conventions and style

Type hints everywhere. pyright --strict must pass. No Any without a comment explaining why.
No print() in library code. All output goes through output/. Library raises exceptions; CLI presents them.
Pydantic for all data crossing module boundaries. Internal helpers can use plain dataclasses if they don't need validation.
Path objects, not strings. pathlib.Path everywhere a path is involved.
No global state except for: the typer app, the logger, and the cache root path resolved at startup.
Atomic writes everywhere. Anything that writes to a file users care about uses tmp-then-rename.
Module docstrings explain what the module does and what depends on it. Function docstrings explain what the function does in one sentence; longer explanations go in module docs or docs/notes/.
Logging via logging stdlib module, not print. Default level WARNING. --verbose raises to INFO. SCRIPTBENCH_DEBUG=1 raises to DEBUG.

18. Things explicitly not in scope (and why)

Concurrent runs across multiple GPUs. Not needed for v1; would complicate scheduling and cache. Defer.
Distributed training adapters. Speedruns are single-GPU by design. If we ever added a multi-GPU benchmark, it'd be a new adapter kind.
Web UI / dashboard. User explicitly chose CLI-only. JSON output makes it trivial to build a dashboard later if anyone wants to.
Automated regression detection. "Alert me if my model's score drops" is a real use case but layered on top — agents can do this trivially by comparing two compare outputs.
Cloud GPU rental integration. Out of scope. Run scriptbench on whatever GPU you have.
Auto-discovery of upstream repo updates. Adapters pin to specific commits via the snapshot mechanism. Updating an adapter is a deliberate action, not a background task.

19. Setup notes for the developer

First steps

uv init scriptbench && cd scriptbench
uv add typer pydantic rich nvidia-ml-py
uv add --dev pytest pyright ruff hypothesis
Set up the directory structure from §5
Implement schemas.py, schedule.py, fingerprint.py first — these are pure and testable without a GPU
Then runner.py, then airbench adapter
Last: CLI wiring

Cloning upstream model repos

Adapters expect upstream repos to live at known paths. A setup script in scripts/setup_adapters.sh:

#!/usr/bin/env bash
mkdir -p adapters/_repos
cd adapters/_repos
git clone https://github.com/KellerJordan/cifar10-airbench.git
git clone https://github.com/hiverge/cifar10-speedrun.git
git clone https://github.com/tysam-code/hlb-CIFAR10.git

The adapter's repo_path() returns these paths. Pin to specific commits in the adapter's version field — bump the version when intentionally updating.

Per-adapter environments (recommended for v1)

Different upstream repos may pin different torch versions. Use uv to create per-adapter venvs:

uv venv adapters/_repos/cifar10-airbench/.venv
uv pip install -r adapters/_repos/cifar10-airbench/requirements.txt --python adapters/_repos/cifar10-airbench/.venv/bin/python

The adapter's build_command() then prefixes with the venv's Python:

def build_command(self, ctx):
    venv_python = self.repo_path() / ".venv/bin/python"
    return [str(venv_python), "airbench94_muon.py"]

This isolates dependency conflicts cleanly.

GPU prerequisites

NVIDIA GPU with CUDA-capable driver
CUDA Toolkit (whatever the upstream repos require, typically 12.x)
Nsight Systems CLI (nsys) installed if profiling mode will be used. apt install nvidia-nsight-systems or download from NVIDIA.

Smoke test

After v0 is implemented:

uv run scriptbench run --models airbench --runs 1

Expected: one airbench run, ~2-3s training time on a 5090, JSON output with success: true, cache populated, second invocation returns cached result instantly.

20. Glossary

Adapter: a Python module that knows how to invoke one training script and parse its output. Implements the Adapter protocol.
Fingerprint: SHA-256 hash of all inputs that affect a run's result. The cache key.
Speedrun: a benchmark whose metric is "wallclock time to reach target accuracy." Lower is better.
Accuracy benchmark: a benchmark whose metric is "peak accuracy within fixed time/epoch budget." Higher is better.
Run: one invocation of one script with one seed. Produces one RunResult.
Session: one invocation of scriptbench run. Contains many runs.
Schedule: the ordering of runs within a session. Interleaved by default for fairness.
Top-2-of-3: scoring rule. For each model, take three runs, drop the worst, average the remaining two.
Cache hit / miss: the cache is keyed by fingerprint. Hit means we found a previous result with the same fingerprint and return it without running.

21. Open questions for future iterations

These are deferred decisions, not blockers for v1:

Should compare support more than two models at once? (Probably yes; defer until needed.)
Should we support custom metrics beyond time-to-target and peak-accuracy? (Maybe; would require generalizing scoring.)
Should the cache support remote storage (S3, etc.) for sharing across machines? (Out of scope unless requested.)
Should there be a "reproducibility check" command that takes an old run and re-runs it to see if the score still matches? (Useful, but easy to script externally.)
Should adapter discovery be plugin-based via entry points, or hardcoded? (Start hardcoded; switch to entry points if/when third parties want to ship adapters.)

End of document

The implementing developer should be able to build v0 from this document alone. If anything is ambiguous, default to the simpler interpretation and note the assumption in docs/notes/iteration-log.md. When in doubt, refer back to the design principles in §3.

FilesExpand file tree

DESIGN.md

Latest commit

History