Skip to content

feat(perf): compare nightly iOS runs and run the suite every night - #3886

Merged
thymikee merged 1 commit into
perf/ios-command-suitefrom
perf/nightly-comparison
Oct 22, 2026
Merged

feat(perf): compare nightly iOS runs and run the suite every night#3886
thymikee merged 1 commit into
perf/ios-command-suitefrom
perf/nightly-comparison

Conversation

@thymikee

@thymikee thymikee commented Oct 21, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Summary

Un-parks perf-nightly (#1781 A3): iOS runs nightly at 02:40 UTC; dispatch takes rounds, warmup, scenarios, baseline size and branch. Each run uploads perf-ios-summary (90 days) and perf-ios-raw (30 days), compares with earlier runs, and writes the verdict to the job summary with warnings for regressions and new failures.

pnpm perf:compare [--to <ref>] [--from <ref>|<a>..<b>] does the same locally (run id, date, latest, or a local output). A step is flagged only when its median moves more than 20%, 3× the night-to-night scaled MAD, 2× its own cv/√n, and 50 ms against the median of up to 7 earlier full-suite nights; fewer than 3 nights give no verdict. 15 files. Android stays report-only behind a dispatch input.

Validation

  • Dispatches on this branch: 39664219771 and 39671171208 found an iOS 26.2 "Open in" prompt after trigger-app-event and a late route change after answering it (both fixed in layer 2); 39679264668 at the stack head: success, 101/105 clean, comparison against the two earlier runs in the job summary (100 fixed, 3 failing, 2 insufficient-baseline).
  • pnpm perf:compare --to 39679264668 --branch perf/nightly-comparison downloads and reproduces that verdict locally.
  • vitest scripts/perf and repo guards: pass. Gate receipt: pnpm check:affected --run --head 60cf336129 (stack head, includes this layer): passed, exactCommit: true; FAIL-OPEN full set, every local check passed; native, device, and coverage lanes are GitHub-authoritative..

Risk: the first nights have no verdicts until 3 nights exist.

🤖 Generated with Claude Code

@github-actions

github-actions Bot commented Oct 21, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Size Report

Metric Base Current Diff
Installed (including dependencies) 5.52 MB 5.52 MB +83 B
Package (unpacked) 5.52 MB 5.52 MB +83 B
Package (download) 1.67 MB 1.67 MB +19 B

Startup median (7 runs, lower is better):

Scenario Base Current Diff
CLI --version 28.6 ms 25.5 ms -3.1 ms
CLI --help 52.4 ms 52.1 ms -0.4 ms

@github-actions

github-actions Bot commented Oct 21, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor
PR Preview Action v1.8.1
Preview removed because the pull request was closed.
2026-10-22 02:53 UTC

@thymikee @claude
feat(perf): compare nightly iOS runs and run the suite every night
60cf336 
pnpm perf:compare judges one run against earlier runs: by default the seven
nights before it, or a chosen run, date, range, or local output. A step's
baseline is the median of its nightly medians. A change is flagged only when
it is larger than 20%, three times the night-to-night scaled MAD, twice the
run's own cv/sqrt(n), and 50 ms; new failures and fixed steps are flagged
too.

perf-nightly runs the iOS suite at 02:40 UTC again (#1781 A3). Each run uploads
perf-ios-summary (90 days) and perf-ios-raw (30 days), and writes the
comparison to the job summary with warnings for regressions and new
failures. Android stays report-only behind a dispatch input.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@thymikee
thymikee force-pushed the perf/nightly-comparison branch from f55d37e to 60cf336 Compare October 21, 2026 21:29
@thymikee
thymikee force-pushed the perf/ios-command-suite branch from 6454e27 to b0b469b Compare October 21, 2026 21:29
@thymikee

thymikee commented Oct 21, 2026

Copy link
Copy Markdown
Member Author

The code in PR #3886 at 60cf336 is ready. I found no blocking problems in the nightly compare and the scheduled suite run.

Not blocking, and you can take or leave these: coversTarget in https://github.com/callstack/agent-device/blob/60cf336/scripts/perf/nightly-artifacts.ts#L107 admits any completed run with the same platform and a matching scenario set, so a manual dispatch with rounds=1 or warmup=0, or a second dispatch on the same day, counts as a baseline night (the rule could also require the same config.rounds and config.warmup, and optionally prefer event=schedule); the README at https://github.com/callstack/agent-device/blob/60cf336/scripts/perf/README.md#L85 says fewer than three nights gives insufficient-baseline, but https://github.com/callstack/agent-device/blob/60cf336/scripts/perf/compare.ts#L275-L279 emits new-failures and fixed from a single night, so the docs or the gate order should agree; and nothing tests the baseline selection in main() at https://github.com/callstack/agent-device/blob/60cf336/scripts/perf/compare-cli.ts#L148 (the exclude-run, the coversTarget filter, the startedAt cutoff for a local --to), so moving it into a pure function in nightly-artifacts.ts would make it testable.

The checks at this head do not give a result yet. The 13 checks that are not passing are all cancelled and one is still running, and none failed. The cancellations look like superseded runs on this stacked branch. The diff touches scripts/perf and test/repo-guards/npm-package-scripts.test.ts only, so the lint, typecheck, coverage and Repo Guards lanes are the ones that matter.

I did not open dispatch runs 39664219771, 39671171208 or 39679264668, and I did not reproduce the job summary or the local perf:compare output. I also did not run vitest scripts/perf, the repo guards or the fallow gates, so I could not check whether summary-fixture.ts trips the production-vs-test export check.

Please re-run the cancelled checks at 60cf336 before merge. It would also help to settle the baseline admission rule and the failure-verdict docs before the first scheduled night.

@thymikee thymikee added the ready-for-human Valid work that needs human implementation, judgment, or maintainer merge label Oct 21, 2026
@thymikee
thymikee merged commit 00d3d0c into perf/ios-command-suite Oct 22, 2026
24 of 37 checks passed
@thymikee
thymikee deleted the perf/nightly-comparison branch October 22, 2026 02:52
thymikee added a commit that referenced this pull request Oct 22, 2026
@thymikee @claude
feat(perf): compare nightly iOS runs and run the suite every night (#…
34ec510 
…3886)

pnpm perf:compare judges one run against earlier runs: by default the seven
nights before it, or a chosen run, date, range, or local output. A step's
baseline is the median of its nightly medians. A change is flagged only when
it is larger than 20%, three times the night-to-night scaled MAD, twice the
run's own cv/sqrt(n), and 50 ms; new failures and fixed steps are flagged
too.

perf-nightly runs the iOS suite at 02:40 UTC again (#1781 A3). Each run uploads
perf-ios-summary (90 days) and perf-ios-raw (30 days), and writes the
comparison to the job summary with warnings for regressions and new
failures. Android stays report-only behind a dispatch input.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@thymikee thymikee mentioned this pull request Oct 22, 2026
Merged
thymikee added a commit that referenced this pull request Oct 22, 2026
@thymikee @claude
feat(perf): time every iOS command against the fixture app (#3885)
d217300 
* feat(perf): time every iOS command against the fixture app

pnpm perf --platform ios now drives the test app through seven scenarios that
time each catalog command the iOS simulator serves, including the heavy feed
and a raw capture with the XCTest tree backend pinned. The suite records a
reason for every command it does not time, and a unit test keeps that
partition complete.

Each sample keeps the client daemon round trip and a phase profile from its
--debug request log: runner commands, host process spawns, snapshot sources,
retries, stabilization, and fallbacks. A run writes summary.json, report.md,
and the raw daemon and session logs. The iOS Settings tour is removed; the
Android tour stays as it was.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(perf): compare nightly iOS runs and run the suite every night (#3886)

pnpm perf:compare judges one run against earlier runs: by default the seven
nights before it, or a chosen run, date, range, or local output. A step's
baseline is the median of its nightly medians. A change is flagged only when
it is larger than 20%, three times the night-to-night scaled MAD, twice the
run's own cv/sqrt(n), and 50 ms; new failures and fixed steps are flagged
too.

perf-nightly runs the iOS suite at 02:40 UTC again (#1781 A3). Each run uploads
perf-ios-summary (90 days) and perf-ios-raw (30 days), and writes the
comparison to the job summary with warnings for regressions and new
failures. Android stays report-only behind a dispatch input.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* fix(perf): judge nights, samples, and failures soundly; keep logs on interrupt

- Rolling baselines take one scheduled run per UTC night (the newest
  usable one) and page through workflow runs until enough nights are
  found. Explicit --to/--from references still accept any trigger.
- A run measures at least three rounds. A timing verdict needs three
  successful samples tonight and on each counted baseline night.
- Failures get their own verdict. more-failures flags tonight failing
  more than every baseline night, in count and share; a faster median
  with worse failures is not an improvement.
- The interrupt handler and the normal finish share one finalizer:
  save logs, stop session and daemon within a deadline, save again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(perf): pin the tree backend on a capture it serves, and share host readers

- The forced-tree step ran a raw capture. The runner honors a backend pin only on the
  regular plan, so the raw plan recovered through private AX and the expectation passed
  a degraded row. The step is now a regular pinned capture on the home screen, and its
  expectation requires the pinned backend with the runner's `requested-backend` code.
- Perf metadata reads the revision, toolchain, and simulator through the readers of
  the iOS snapshot benchmark, moved to scripts/lib/bench-host.ts.
- host-kit and the daemon client export the phase names that the profiler reads.
- summary-fixture.ts becomes summary.fixtures.ts; the nightly header matches 3 rounds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(perf): keep the daemon request phase literal

Only scripts read it, so a root export has no production consumer and fails the
fallow audit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(perf): stop the log stream right after it starts

#3892 fixed the iOS simulator logs stop hang, so the untimed wait before
logs-stop is no longer needed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
thymikee added a commit that referenced this pull request Oct 22, 2026
@thymikee @claude
feat(perf): time every Android command against the fixture app (#3902)
f552d87 
* feat(test-app): add a Bluesky-depth heavy feed screen

A JavaScript-only route that reproduces a heavy React Native home feed: 30
nested containers above a virtualized list of composite posts with image,
link, quote, and quote-with-media embeds. Its native accessibility tree is
deeper than 65 levels, so XCTest tree capture fails with
kAXErrorIllegalArgument and recovers through private AX, as on Bluesky
1.136. Settings links to it with open-heavy-feed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(perf): time every iOS command against the fixture app

pnpm perf --platform ios now drives the test app through seven scenarios that
time each catalog command the iOS simulator serves, including the heavy feed
and a raw capture with the XCTest tree backend pinned. The suite records a
reason for every command it does not time, and a unit test keeps that
partition complete.

Each sample keeps the client daemon round trip and a phase profile from its
--debug request log: runner commands, host process spawns, snapshot sources,
retries, stabilization, and fallbacks. A run writes summary.json, report.md,
and the raw daemon and session logs. The iOS Settings tour is removed; the
Android tour stays as it was.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(perf): compare nightly iOS runs and run the suite every night (#3886)

pnpm perf:compare judges one run against earlier runs: by default the seven
nights before it, or a chosen run, date, range, or local output. A step's
baseline is the median of its nightly medians. A change is flagged only when
it is larger than 20%, three times the night-to-night scaled MAD, twice the
run's own cv/sqrt(n), and 50 ms; new failures and fixed steps are flagged
too.

perf-nightly runs the iOS suite at 02:40 UTC again (#1781 A3). Each run uploads
perf-ios-summary (90 days) and perf-ios-raw (30 days), and writes the
comparison to the job summary with warnings for regressions and new
failures. Android stays report-only behind a dispatch input.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* fix(perf): judge nights, samples, and failures soundly; keep logs on interrupt

- Rolling baselines take one scheduled run per UTC night (the newest
  usable one) and page through workflow runs until enough nights are
  found. Explicit --to/--from references still accept any trigger.
- A run measures at least three rounds. A timing verdict needs three
  successful samples tonight and on each counted baseline night.
- Failures get their own verdict. more-failures flags tonight failing
  more than every baseline night, in count and share; a faster median
  with worse failures is not an improvement.
- The interrupt handler and the normal finish share one finalizer:
  save logs, stop session and daemon within a deadline, save again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(perf): time every Android command against the fixture app

Run iOS and Android from one shared suite: platform deltas are data on
each step, and one harness owns rounds, sessions, and daemon restarts.
The heavy-feed scenario times the Android helper's one-shot
instrumentation capture by restarting the daemon with the helper
session off. Request profiles name the helper transport that answered,
adb spawns, and helper gestures. The nightly runs both platforms and
compares each against its own nights; the Settings tour is removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(perf): keep Android clipboard and restore failures out of timings

The API 35 and 36 emulator images ship no shell clipboard service, so
clipboard is an Android exclusion. A daemon restart that reopens the
session clears the stuck flag, and one that cannot reopen it skips the
rest of the scenario as restore_failed. The one-shot step warms the
restarted daemon's helper install check untimed. latest and date
references resolve to the newest run with the platform's artifact.
The tour-only --n alias is gone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(perf): time every iOS command against the fixture app

pnpm perf --platform ios now drives the test app through seven scenarios that
time each catalog command the iOS simulator serves, including the heavy feed
and a raw capture with the XCTest tree backend pinned. The suite records a
reason for every command it does not time, and a unit test keeps that
partition complete.

Each sample keeps the client daemon round trip and a phase profile from its
--debug request log: runner commands, host process spawns, snapshot sources,
retries, stabilization, and fallbacks. A run writes summary.json, report.md,
and the raw daemon and session logs. The iOS Settings tour is removed; the
Android tour stays as it was.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(perf): compare nightly iOS runs and run the suite every night (#3886)

pnpm perf:compare judges one run against earlier runs: by default the seven
nights before it, or a chosen run, date, range, or local output. A step's
baseline is the median of its nightly medians. A change is flagged only when
it is larger than 20%, three times the night-to-night scaled MAD, twice the
run's own cv/sqrt(n), and 50 ms; new failures and fixed steps are flagged
too.

perf-nightly runs the iOS suite at 02:40 UTC again (#1781 A3). Each run uploads
perf-ios-summary (90 days) and perf-ios-raw (30 days), and writes the
comparison to the job summary with warnings for regressions and new
failures. Android stays report-only behind a dispatch input.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* fix(perf): judge nights, samples, and failures soundly; keep logs on interrupt

- Rolling baselines take one scheduled run per UTC night (the newest
  usable one) and page through workflow runs until enough nights are
  found. Explicit --to/--from references still accept any trigger.
- A run measures at least three rounds. A timing verdict needs three
  successful samples tonight and on each counted baseline night.
- Failures get their own verdict. more-failures flags tonight failing
  more than every baseline night, in count and share; a faster median
  with worse failures is not an improvement.
- The interrupt handler and the normal finish share one finalizer:
  save logs, stop session and daemon within a deadline, save again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(perf): pin the tree backend on a capture it serves, and share host readers

- The forced-tree step ran a raw capture. The runner honors a backend pin only on the
  regular plan, so the raw plan recovered through private AX and the expectation passed
  a degraded row. The step is now a regular pinned capture on the home screen, and its
  expectation requires the pinned backend with the runner's `requested-backend` code.
- Perf metadata reads the revision, toolchain, and simulator through the readers of
  the iOS snapshot benchmark, moved to scripts/lib/bench-host.ts.
- host-kit and the daemon client export the phase names that the profiler reads.
- summary-fixture.ts becomes summary.fixtures.ts; the nightly header matches 3 rounds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(perf): read Android phase names from their emitters, and document the pinned capture

- platform-android exports the helper capture and gesture phase names through
  its mechanics entry; the profiler and its test import them.
- The README's snapshot-path section and parity table name the iOS pinned
  capture on home-reads and the reason code it must carry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(perf): name the failed restore when restarting the daemon, and fix the local compare example

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(perf): keep the daemon request phase literal

Only scripts read it, so a root export has no production consumer and fails the
fallow audit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(perf): stop the log stream right after it starts

#3892 fixed the iOS simulator logs stop hang, so the untimed wait before
logs-stop is no longer needed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(android): record the answering helper transport on its capture timer

The snapshot helper's capture timers now name the transport they drove, and the
session timer records whether the session answered. profile.ts reads those
fields per event instead of inferring a silent session from event order, and no
longer re-spells the producer names platform-android owns.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

ready-for-human Valid work that needs human implementation, judgment, or maintainer merge

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@thymikee