diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index c0f9fa98..dc3e28f1 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -194,7 +194,7 @@ Test-only and never published or bundled. Each adapter runs one scenario (open ` Each scenario runs twice, with and without BiDi. The fake answers the BiDi commands the adapters send, including the navigation, context, element lookup and script calls WebdriverIO routes over BiDi once a session offers it, and a registered preload puts the collector into every later document from birth, as in Chrome. A difference an adapter is allowed to show lives in `src/allowlist.ts` with a mandatory reason, and an entry that stops matching fails the run, so a closed gap forces its own removal. -Every run, passing or failing, dumps its frames, ground truth, violations and scenario output to `packages/conformance/out/-.json` and rebuilds `out/index.html`, a self-contained page laying each run out as a timeline of documents, command rows and DOM anchors with every violation beside the event it concerns. CI uploads `out/` as the `conformance-report` artifact when the `build` job fails. +Every run, passing or failing, dumps its frames, ground truth, violations and scenario output to `packages/conformance/out/-bidi-.json` and rebuilds `out/index.html`, a self-contained page laying each run out as a timeline of documents, command rows and DOM anchors with every violation beside the event it concerns. CI uploads `out/` as the `conformance-report` artifact when the `build` job fails. Beside the rules, the adapters are compared with each other: each run is reduced to what a dashboard user sees (steps, outcomes, the page each step replays, snapshots, run controls) and any difference between adapters in the same mode fails unless a declared gap explains it. The rules catch every adapter being wrong in the same way; the comparison catches a difference no rule anticipated. Adapters are registered once in `src/adapters.ts`. diff --git a/docs/ADAPTERS.md b/docs/ADAPTERS.md index 7c1889c7..fb039bc9 100644 --- a/docs/ADAPTERS.md +++ b/docs/ADAPTERS.md @@ -200,9 +200,19 @@ CONFORMANCE_LOGIN_URL=/login CONFORMANCE_BROKEN_URL=/broken ``` -3. Returns the recorded frames, the fake's ground truth (which documents were born and when, which navigations failed), the exit code and the output. The scenario times out after 60 s. +3. Returns the recorded frames, the fake's ground truth (which documents were born and when, which navigations failed, which requests each document made, and what the fake served over BiDi), the exit code and the output. The scenario times out after 60 s. -The recording backend accepts only `/worker` upgrades, serves the real collector bundle at `/api/collector` as the real backend does, and stubs `/api/element-scripts`. The fake WebDriver models two pages (`PAGES`), answers `/url`, `/title`, `/element(s)`, element `value`/`click`/`clear`/`text`, `/execute/*`, `/screenshot`, `/window`, `/window/handles`, `/window/rect`, and returns `null` for anything else. It returns a BiDi `webSocketUrl` only when the scenario runs with `bidi: true` **and** the new-session request asked for `webSocketUrl: true`; registering a preload (`script.addPreloadScript`) then gives every later document a collector anchored at birth (`src/fake-bidi.ts`). Over that socket the fake answers what a client routes through BiDi once it has it (`browsingContext.navigate`, `browsingContext.getTree`, `browsingContext.locateNodes`, `script.callFunction`), from the same two-page model, so WebdriverIO drives the same pages and the same user actions are stamped whichever protocol carried them; everything else is acknowledged. +The recording backend accepts only `/worker` upgrades, serves the real collector bundle at `/api/collector` as the real backend does, and stubs `/api/element-scripts`. The fake WebDriver models two pages (`PAGES`), answers `/url`, `/title`, `/element(s)`, element `value`/`click`/`clear`/`text`, `/execute/*`, `/screenshot`, `/window`, `/window/handles`, `/window/rect`, and returns `null` for anything else. Each page loads its document plus the two `SUBRESOURCES`; the loads are the ground truth's `requests`, and a drain of the collector returns them once as Resource Timing rows (no status, no headers, `source: 'resource-timing'`), as `packages/script` reports them. + +Every scenario runs in three BiDi modes (`BIDI_MODES` in `src/constants.ts`), each the shape of a real driver: + +| Mode | The fake serves | The real driver it stands for | +|---|---|---| +| `live` | a `webSocketUrl`, every command answered, and `network.beforeRequestSent` / `network.responseCompleted` emitted for each load to the clients that subscribed | Chrome | +| `silent` | a `webSocketUrl` and every command answered, subscriptions accepted, no event ever emitted | Appium 3.7's BiDi shim on XCUITest | +| `off` | no `webSocketUrl` at all | Safari | + +A `webSocketUrl` is returned only when the mode offers one **and** the new-session request asked for `webSocketUrl: true`; registering a preload (`script.addPreloadScript`) then gives every later document a collector anchored at birth (`src/fake-bidi.ts`). Over that socket the fake answers what a client routes through BiDi once it has it (`browsingContext.navigate`, `browsingContext.getTree`, `browsingContext.locateNodes`, `script.callFunction`), from the same two-page model, so WebdriverIO drives the same pages and the same user actions are stamped whichever protocol carried them; everything else is acknowledged. The fake recognises the collector, not your adapter's way of using it, so inject and drain however your language makes natural: @@ -225,13 +235,13 @@ Every scenario performs these steps, in this order, with nothing else in between 7. Navigate to `CONFORMANCE_BROKEN_URL`, which fails with HTTP 500; catch and ignore the error. 8. Quit the session and disable the adapter. Exit 0. -The browser therefore creates exactly three documents. +The browser therefore creates exactly three documents and makes nine requests: each document plus `/app.js` and `/app.css`. ### Adding your language 1. Add your id to `AdapterId` in `src/types.ts`. 2. Add `scenarios/_scenario.` performing the steps above, or, for an adapter that only runs under its framework's CLI, a folder with a config and a spec (`scenarios/nightwatch/` and `scenarios/wdio/` are the models: the config reads the fake's origin from `CONFORMANCE_WEBDRIVER` and points the runner at it, with `start_process: false` for Nightwatch and `hostname`/`port` for WebdriverIO). Read the three variables by name (the Python scenario repeats them, since it cannot import `SCENARIO_ENV`). -3. Add one entry to `ADAPTERS` in `src/adapters.ts`: `ready()` (your toolchain and build are present) and `prepare()` (the command, args and env that launch your scenario, plus an optional `cleanup`, and a `cwd` for a CLI that resolves its runner and plugins from the working directory, as WebdriverIO does). `tests/conformance.test.ts` then runs your adapter in both modes, holds it to every rule, and compares it with every other adapter. Expect the first run to be red: classify each line as a suite assumption to fix, a difference to declare with a reason, or an adapter bug to fix, and merge only when it is green. `ready()` may only skip a local run; in CI a missing prerequisite fails. +3. Add one entry to `ADAPTERS` in `src/adapters.ts`: `ready()` (your toolchain and build are present) and `prepare()` (the command, args and env that launch your scenario, plus an optional `cleanup`, and a `cwd` for a CLI that resolves its runner and plugins from the working directory, as WebdriverIO does). `tests/conformance.test.ts` then runs your adapter in all three BiDi modes, holds it to every rule, and compares it with every other adapter. Expect the first run to be red: classify each line as a suite assumption to fix, a difference to declare with a reason, or an adapter bug to fix, and merge only when it is green. `ready()` may only skip a local run; in CI a missing prerequisite fails. 4. Make sure CI installs your toolchain before `pnpm test` in `.github/workflows/ci.yml` (Python's install step is there). Run it locally with `pnpm test packages/conformance`. Vitest only collects `packages/**/tests/**/*.test.ts`. @@ -253,20 +263,23 @@ Defined in `src/checker.ts`, names in `RULES` (`src/constants.ts`): | `replace-targets-a-sent-row` | Every `replaceCommand.oldTimestamp` names a row already sent. | | `one-row-per-browser-action` | The number of `navigate`, `click` and `fill` rows equals what the fake browser received, after applying `replaceCommand` the way the dashboard does (shared `replaceCommand`: by `id`, then the last row with the old timestamp). | | `run-controls-are-served` | The dashboard offers every control the runner's defaults allow unless `metadata.options.runCapabilities` turns it off (shared `resolveRunCapabilities`, the dashboard's own rule), so a stream that advertises any control, by declaration or by silence, serves it: `launchCommand` or `rerunCommand` in the metadata (the backend's shell path, falling back from either to the other), `configFile` there, or a `config` frame (its framework path, which reruns a WDIO or Nightwatch config). With none, the backend falls back to the WebdriverIO binary, so the control runs the wrong tool. | +| `every-browser-request-listed` | For every url the browser loaded, the `networkRequests` rows (after folding re-sends under the same `id`, as the dashboard does) number at least the loads the browser made. Fewer means the page's Resource Timing was withheld although nothing else served the request. | +| `one-row-per-browser-request` | For every url the browser loaded, the rows number at most the loads the browser made. More means two sources listed one load, or a re-sent row carried a new id. The two rules are separate so a declared gap can cover the first without excusing the second. | +| `request-row-carries-what-the-browser-served` | With BiDi `live` (a socket handed to the session, not merely offered), every row carries a status and response headers. Otherwise every row either carries those, as the page's fetch/XHR patch supplies them, or is marked `source: 'resource-timing'`: a row with neither passes as a driver's and misleads the detail card. | ### Cross-adapter agreement -The rules check each adapter against what the fake browser did. A second check needs no rules: within each mode, every adapter's stream is reduced to what a dashboard user sees (`src/normalize.ts`: each step's shared action name, target, typed text, outcome and the page it replays, the pages snapshotted, the run controls offered; timestamps, ids, ports and raw command names dropped) and compared with the others (`src/agreement.ts`). Any difference fails as one of `adapters-agree-on-actions`, `-on-errors`, `-on-pages`, `-on-snapshots` or `-on-run-controls`, unless a declared gap explains it. A gap explains a difference only where its holder's own run shows the gap on that same step or document and the other adapter's run is clean there, so one adapter's limitation cannot hide a second bug in the other. This is what catches a difference no rule anticipated; the rules catch every adapter being wrong in the same way. +The rules check each adapter against what the fake browser did. A second check needs no rules: within each mode, every adapter's stream is reduced to what a dashboard user sees (`src/normalize.ts`: each step's shared action name, target, typed text, outcome and the page it replays, the pages snapshotted, the run controls offered, the requests listed; timestamps, ids, ports and raw command names dropped) and compared with the others (`src/agreement.ts`). Any difference fails as one of `adapters-agree-on-actions`, `-on-errors`, `-on-pages`, `-on-snapshots`, `-on-run-controls` or `-on-requests`, unless a declared gap explains it. A gap explains a difference only where its holder's own run shows the gap on that same step or document and the other adapter's run is clean there, so one adapter's limitation cannot hide a second bug in the other. This is what catches a difference no rule anticipated; the rules catch every adapter being wrong in the same way. Navigation rows are recognised through `mapCommandToAction` (`method: 'navigate'`), with the url taken from a bare string arg or a W3C `{url}` arg, so name your navigation command accordingly. ### Reading a failure -A failing test prints the `unexplained` violations, each `{rule, message, subject?}`, for example `get(http://127.0.0.1:53211/login) at 1760000000412 replays no document; the browser was on http://127.0.0.1:53211/login` under `row-replays-its-page`. Each run also writes `packages/conformance/out/-.json` (frames, ground truth, violations, output) and rebuilds `packages/conformance/out/index.html`, a report with each run's documents, anchors and rows on one timeline and the violations placed on the event they concern (`src/report.ts`, `src/report-html.ts`, `src/timeline.ts`). The report opens with "Do the adapters agree?": per mode, a column per adapter and a row per step, differing rows highlighted. A failing run opens with a "Where to look" block: for each unexplained rule, what it means, its usual cause, and which of the responsibilities in section 2 it tests, linked to that section (`src/rule-guide.ts`). It names no adapter's files: the section says what the responsibility requires, and your adapter's own structure says where it lives. The directory is gitignored; CI uploads it as the `conformance-report` artifact when the job fails. A failing `exitCode` assertion includes the scenario's full output. +A failing test prints the `unexplained` violations, each `{rule, message, subject?}`, for example `get(http://127.0.0.1:53211/login) at 1760000000412 replays no document; the browser was on http://127.0.0.1:53211/login` under `row-replays-its-page`. Each run also writes `packages/conformance/out/-bidi-.json` (frames, ground truth, violations, output) and rebuilds `packages/conformance/out/index.html`, a report with each run's documents, anchors and rows on one timeline and the violations placed on the event they concern (`src/report.ts`, `src/report-html.ts`, `src/timeline.ts`). The report opens with "Do the adapters agree?": per mode, a column per adapter and a row per step, differing rows highlighted. A failing run opens with a "Where to look" block: for each unexplained rule, what it means, its usual cause, and which of the responsibilities in section 2 it tests, linked to that section (`src/rule-guide.ts`). It names no adapter's files: the section says what the responsibility requires, and your adapter's own structure says where it lives. The directory is gitignored; CI uploads it as the `conformance-report` artifact when the job fails. A failing `exitCode` assertion includes the scenario's full output. ### Allowlist entries -`src/allowlist.ts` `KNOWN_GAPS` lists differences an adapter is allowed to show: `{adapter, bidi, rules, reason, timingDependent?, lacksRunControls?}`. An entry is legitimate only for a documented limitation that cannot be fixed in the adapter (for example, the `