Repository navigation
feat(perf): time every iOS command against the fixture app - #3885
Conversation
Local proof run (raw report.md)Source report.mdagent-device perf — ios
Times are milliseconds. See scripts/perf/README.md for each metric. The second table of each lifecycle
home-reads
catalog
form
automation
heavy-feed
session
Backends and fallbacks
Failures
CoverageMeasured commands (43): action-button, alert, app-switcher, apps, appstate, artifacts, back, batch, capabilities, click, clipboard, close, devices, diff, doctor, events, fill, find, focus, gesture, get, home, is, keyboard, logs, longpress, network, open, orientation, perf, prepare, press, react-native, record, screenshot, scroll, settings, snapshot, swipe, trace, trigger-app-event, type, wait.
|
Size Report
Startup median (7 runs, lower is better):
|
6454e27 to
b0b469b
Compare
|
I found no blocking defect in b0b469b, and the 14 checks on this PR pass. The diff touches only scripts/perf, the parked perf-nightly.yml and vitest.config.ts, so no production package code changes. I did not open the local proof report or the artifacts of CI run 39679264668, so the 102/105 and 101/105 counts rest on your description. I also did not run vitest scripts/perf or the forced-backend capture, so I have not confirmed that the daemon honors the snapshotPreferredBackend pin through the SDK transport. Please attach that capture, showing the sdk-forced-tree step served by the tree backend, before merge. Not blocking, and you can take or leave these: (1) metadata.ts re-implements readTarget, readGitRevision and readToolchain from scripts/ios-snapshot-benchmark/host.ts, with no dirty flag, no 'available' filter and a silent undefined for an unknown UDID. One module should record iOS bench host and target identity, so move those readers to scripts/lib and build RunMetadata from them. (2) profile.ts and profile.test.ts copy phase names and the diag prefix as string literals from their emitters, and the fallback labels use data.error message text. Each phase the profiler reads should be declared once by its emitter and imported in both files, with fallbacks keyed on failure codes. (3) In ios-suite.ts, step() takes Could you keep the step table and profile.ts, but delete metadata.ts and build RunMetadata from the host.ts readers moved to scripts/lib? Could the Android tour run on the same SuiteScenario runner, so harness.ts, scenario.ts and platform-profiles.ts collapse into one table with a single round, session and daemon-stuck loop? Could forced-backend.ts wrap the SDK's own default transport to add the pin, instead of rebuilding daemon lookup from src internals, or could that one step wait until the pin has a supported surface? That would remove roughly 250 to 350 production lines and the second harness. It needs exported diagnostic phase-name constants from platform-apple and host-kit first, plus the shared scripts/lib home for bench host and target identity. Before merge, the forced-backend capture above must be attached. Please also reuse the host.ts readers, and either share the round loop with the Android harness or explain why there are two harnesses. |
|
|
Addressed the four findings in 558da5c (follow-up commit, no force-push). Each finding below lists the fix, the test, and the repro on 00d3d0c. 1. Dispatch runs can replace independent baseline nights (P2)Fix:
On 00d3d0c: 2. Timing verdicts accept fewer than three successful measurements (P2)Fix:
Tests:
On 00d3d0c: the repro gives 3. Worsening failures can be reported as an improvement (P2)Fix: Each row now has two separate verdicts:
Tests:
On 00d3d0c: the repro gives 4. Interrupted runs lose diagnostic logs (P2)Fix:
It runs once.
On 00d3d0c: the interrupt handler ran only Mutation checks (each fails the named test)Each of these mutations fails its test:
Docs
New head: 558da5c
No device run: the suite measures the same steps. Only teardown ordering and deadlines changed. Co-Authored-By: Claude Opus 5.5 noreply@anthropic.com |
|
The Mutation Harness Canary failures at 558da5c don't come from this PR. The lane is advisory and never gates ( |
|
The earlier review of b0b469b asked for one capture of the sdk-forced-tree step served by the tree backend. At 558da5c that capture is still missing, so the code needs no change yet, but the PR needs this evidence before merge. scripts/perf/forced-backend.ts is unchanged in the delta, and the author comments do not supply a run. The nightly compare treats sdk-forced-tree like any other step. If the daemon ignores the snapshotPreferredBackend pin, a degraded-backend row gets judged as a timing baseline, and nobody can tell it ran on the wrong backend. Please attach one run, local or CI, at this head. It should include the sdk-forced-tree sample's request log line showing the pinned backend, and a summary.json and report.md where that sample's profile reports snapshot backend 'tree', not degraded. Not blocking, and you can take or leave these: summary-fixture.ts is imported only by the three test files but counts as production code (and may show up as an unused export), so a test-helper name or a tests/fixtures path would fit better; and the header in perf-nightly.yml cites n=5 medians while the nightly now runs 3 rounds, so it could say that one failed sample withholds that step's timing verdict. CI shows two red areas that look unrelated to this PR. Lint fails on the line cap in packages/platform-apple/src/snapshot-route.test.ts:1003, which this PR does not touch. The Mutants kernel-errors jobs fail in the Stryker dry run on the host-kit exec-trace test, and this PR does not touch host-kit. I did not run vitest scripts/perf or your mutation checks, and I did not open the CI artifacts, so the 102/105 and 101/105 counts rest on the PR body. The compare step has not run against real scheduled baselines yet, so its paging and selection against the live GitHub API are unverified. Before merge, please attach the sdk-forced-tree capture and answer the open simplicity question: does the perf lane need both host readers and two harnesses? |
pnpm perf --platform ios now drives the test app through seven scenarios that time each catalog command the iOS simulator serves, including the heavy feed and a raw capture with the XCTest tree backend pinned. The suite records a reason for every command it does not time, and a unit test keeps that partition complete. Each sample keeps the client daemon round trip and a phase profile from its --debug request log: runner commands, host process spawns, snapshot sources, retries, stabilization, and fallbacks. A run writes summary.json, report.md, and the raw daemon and session logs. The iOS Settings tour is removed; the Android tour stays as it was. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…3886) pnpm perf:compare judges one run against earlier runs: by default the seven nights before it, or a chosen run, date, range, or local output. A step's baseline is the median of its nightly medians. A change is flagged only when it is larger than 20%, three times the night-to-night scaled MAD, twice the run's own cv/sqrt(n), and 50 ms; new failures and fixed steps are flagged too. perf-nightly runs the iOS suite at 02:40 UTC again (#1781 A3). Each run uploads perf-ios-summary (90 days) and perf-ios-raw (30 days), and writes the comparison to the job summary with warnings for regressions and new failures. Android stays report-only behind a dispatch input. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…interrupt - Rolling baselines take one scheduled run per UTC night (the newest usable one) and page through workflow runs until enough nights are found. Explicit --to/--from references still accept any trigger. - A run measures at least three rounds. A timing verdict needs three successful samples tonight and on each counted baseline night. - Failures get their own verdict. more-failures flags tonight failing more than every baseline night, in count and share; a faster median with worse failures is not an improvement. - The interrupt handler and the normal finish share one finalizer: save logs, stop session and daemon within a deadline, save again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
558da5c to
fed2a4c
Compare
…t readers - The forced-tree step ran a raw capture. The runner honors a backend pin only on the regular plan, so the raw plan recovered through private AX and the expectation passed a degraded row. The step is now a regular pinned capture on the home screen, and its expectation requires the pinned backend with the runner's `requested-backend` code. - Perf metadata reads the revision, toolchain, and simulator through the readers of the iOS snapshot benchmark, moved to scripts/lib/bench-host.ts. - host-kit and the daemon client export the phase names that the profiler reads. - summary-fixture.ts becomes summary.fixtures.ts; the nightly header matches 3 rounds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
I found no code problem at fed2a4c that must change before merge, and the 23 checks are green. The change touches only scripts/perf, the nightly workflow, package.json, vitest.config.ts and one repo-guard test, with no package code changes. I did not run vitest or any gate myself, I did not open the CI artifacts (the 101/105 count rests on the PR body), and the pin evidence is the local report at 7a801ca, not a run at fed2a4c. Before merge, please update the PR body to cover the folded-in compare layer, the nightly schedule and the 16 exclusions, and answer the question below. Not blocking, and you can take or leave each: (1) scripts/perf/ios-suite.ts:50 lets the pinned row pass when One question on design: |
#3885 was rebased onto main after #3884 merged. This merge takes its new head with the Android layer's two commits replayed on it. Suite conflicts resolve to the renamed suite.ts and suite-harness.ts: the pinned-tree capture moves to home-reads (iOS only) and the Android one-shot step keeps its transport check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Update: this PR is now rebased on main after #3884 merged. The maintainer did the rebase, and its content matches 558da5c. The new head is 8958314, one commit on top of fed2a4c. This comment covers each open point. 1. sdk-forced-tree capture (blocking): the row was degraded, and the step is now fixedWhat the old step measured. I ran the heavy-feed scenario at fed2a4c, before any change, on my own iPhone 17 Pro simulator (iOS 27.0, Xcode 27.1). All three Why. The daemon does honor the pin. It skips the host AX bridge, and the request log shows
The tree backend cannot serve the heavy feed at all. A tree-pinned row on that screen can only be degraded or failed. Fix (8958314).
Evidence at 8958314. Command:
Request log The daemon request log does not record request flags, so no line there spells out Every other failed sample in that run is one of the three known heavy-feed gaps ( 2. Two host readers, two harnesses (blocking question)
3. Phase-name constantshost-kit now exports The three platform-apple phases ( 4. Non-blocking notes
Validation at 8958314
report.md (home-reads, heavy-feed at 8958314)agent-device perf — ios
Times are milliseconds. See scripts/perf/README.md for each metric. The second table of each home-reads
heavy-feed
Backends and fallbacks
Failures
CoverageMeasured commands (19): artifacts, back, batch, diff, events, find, gesture, get, is, logs, longpress, perf, press, react-native, screenshot, scroll, snapshot, swipe, wait.
Co-Authored-By: Claude Opus 5.5 noreply@anthropic.com |
Only scripts read it, so a root export has no production consumer and fails the fallow audit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Gate update: the first
Co-Authored-By: Claude Opus 5.5 noreply@anthropic.com |
Brings in #3892, which fixes the iOS simulator logs stop hang. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * origin/main: perf(ios): read each resolved text input through one snapshot (#3882) fix(gate): take the host validation lock for heavy gates (#3900) fix(logs): keep an early iOS simulator logs stop from hanging the session (#3892) fix(apple): clean the runner derived path the runner uses (#3897) test(provider-scenarios): open deadline scenarios without the test IME (#3903) test(host-kit): stub os.homedir for the exec-trace late-close test (#3898)
#3892 fixed the iOS simulator logs stop hang, so the untimed wait before logs-stop is no longer needed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Head is now aed5aa7:
Device check on my own iPhone 17 Pro simulator (iOS 27.0) at aed5aa7, with
Co-Authored-By: Claude Opus 5.5 noreply@anthropic.com |
|
The earlier notes from fed2a4c are now addressed, and I found no new problems in the code at aed5aa7. The checks are green: 23 checks, none failing. There are no conflicts. I did not run vitest, check:affected, check:production-exports or any device run. The 3/3 run at aed5aa7 and the tree/healthy/requested-backend x3 run at 8958314 rest on your comments. I did not open the CI job logs. I did not review #3902, so I could not confirm that it merges ios-harness.ts and the Android harness into one suite-harness.ts. One question: if the nightly checkout already holds untracked files when collectMetadata runs, would |
thymikee
added
the
ready-for-human
* origin/main: perf(ios): confirm fill focus from the tapped input before the keyboard (#3906) feat(perf): time every iOS command against the fixture app (#3885) fix(ios): stop the XCTest-channel penalty from re-arming itself (#3896) test(android): pin that a child vanishing mid-read reads as gone (#3905) ci: skip device smoke lanes when a PR changes no lane input (#3904) fix(ios): verify deep-tree back on the sole screen (#3894) fix(ios): synthesize long presses and the gesture viewport on deep trees (#3893) perf(ios): read each resolved text input through one snapshot (#3882) fix(gate): take the host validation lock for heavy gates (#3900) fix(logs): keep an early iOS simulator logs stop from hanging the session (#3892) fix(apple): clean the runner derived path the runner uses (#3897) test(provider-scenarios): open deadline scenarios without the test IME (#3903) test(host-kit): stub os.homedir for the exec-trace late-close test (#3898) feat(test-app): add a Bluesky-depth heavy feed screen (#3884)
1 participant
Summary
pnpm perf --platform ios --udid <udid>times every catalog command that the iOS simulator serves, against the fixture app. The suite has 105 steps in 7 scenarios: lifecycle, home reads, catalog, form, automation, heavy feed, and session. The report lists 16 excluded commands with a reason, andios-suite.test.tskeeps measured + excluded equal to the catalog.Each sample records wall time, the client
daemon_requesttime, and a profile from its--debugrequest log. The output issummary.json,report.md, and the raw logs. The nightly workflow andperf:comparecome from #3886, which merged into this branch.home-reads/snapshot-forced-treepins the XCTest tree through the SDK. Its expectation requirestreewith the runner'srequested-backendcode, so a degraded capture fails instead of becoming a baseline.longpress,swipe, andbackfail today (XCTestkAXErrorIllegalArgument). They stay as measured steps so that a fix shows asfixed.scripts/lib/bench-host.ts. host-kit and the daemon client export the phase names that the profiler reads.46 files. This PR is rebased on main after #3884, and main is merged in for #3892.
Validation
home-reads,heavy-feedlocally (3 rounds + 1 warmup, iPhone 17 Pro, iOS 27.0).snapshot-forced-treewas served bytree/healthy/requested-backend×3. The only failures were the 3 heavy-feed gaps. The evidence is in the PR comments.--scenarios automationran clean (1 warmup + 3 rounds).logs-start→logs-stoppassed 3/3 with nodaemon_stuck, now that the fix(logs): keep an early iOS simulator logs stop from hanging the session #3892 workaround is removed.vitest scripts/perf scripts/lib scripts/ios-snapshot-benchmark: passed.pnpm check:affected --run --head 8958314a16: passed at aed5aa7 (exactCommit: true, FAIL-OPEN full set: 36 passed, 25 GitHub-authoritative skips).🤖 Generated with Claude Code