Skip to content

feat(accessibility): wait for a report to finish processing before reading its issues - #499

Closed
Aaryan430 wants to merge 5 commits into
browserstack:mainfrom
Aaryan430:feat/a11y-wait-for-processing
Closed

Aaryan430 wants to merge 5 commits into
browserstack:mainfrom
Aaryan430:feat/a11y-wait-for-processing

Conversation

@Aaryan430

Copy link
Copy Markdown
Collaborator

Stacked on #496 (which is stacked on #494). GitHub can't stack a PR on a fork branch, so the earlier commits here are theirs. Review only the last commit, 1643f4a. It does not depend on #498.

An agent that fetched an SDK build's accessibility issues right after the test run reported 14 issues as final ("✅ All issues retrieved"). The build completed minutes later with 27. After the tests end, a build stays finalizing, which on an advanced scan is the dashboard's "advanced issue processing", and the export serves whatever exists so far.

Recorded on a fresh build, polling every 5 s:

Time Build status (list_builds) Issues the API returns
tests running running 0
tests end finalizing 14
~3 min later completed 27

Change

fetchAccessibilityIssues reads the report's status before reading its issues:

Source Where the status comes from
Website Scanner run GET …/scans/{id}/scan_runs/{run}/status
Workflow Analyzer report GET …/reports/{id} → summary.status
Automated Tests build its project's list_builds row (the build overview has no status), found by id in the newest builds
  • While the report is processing (queued / running / processing / finalizing), a first read waits: it polls every 10 s for up to 5 min, the same budget startAccessibilityScan uses, and sends progress notifications. Then it reads the issues and says it waited.
  • If processing outlasts the wait, the answer starts with a warning that these are only the issues found so far, the total will grow, and to fetch again. It ends with "All issues found so far retrieved; the report is still processing" instead of "✅ All issues retrieved".
  • A later page doesn't wait again but still warns. A status that can't be read is not waited on (no behaviour change).

Verification

Ran a fresh SDK test (accessibility + advanced on) and called this branch's build over MCP the moment the run ended:

  +1s progress: The report is still finalizing; waiting for it to complete (1s of up to 5m 00s).
  ...
+133s progress: The report is still finalizing; waiting for it to complete (2m 13s of up to 5m 00s).
+148s The report was still processing (status: finalizing); waited 2m 24s for it to complete before reading its issues.
      Retrieved 7 accessibility issues (Total: 27)

Without this change the same call returns 14. New tests cover each source's status read (including a build found on the second page of its project's list, and an unreadable status), no wait when complete, waiting through finalizing (fake timers), the warning when processing outlasts the wait, no wait on later pages, and progress notifications. npm run build (lint, format, 1336 tests, tsc) passes.

API follow-up (optional): the Automated Tests build overview (GET /api/automated-tests/v1/builds/{id}) has no status, so the MCP reads it from the project's build list. get_build_summary already holds the TestHub object list_builds takes it from, so adding it there is one line and would make this a single call.

🤖 Generated with Claude Code

Aaryan430 and others added 5 commits October 19, 2026 15:00
The issues export has a row for every occurrence. In a grouped report each
repeated occurrence carries a "Duplicate Issue of" link to its main issue, and
the dashboard leaves those rows out of its issue count. startAccessibilityScan
and fetchAccessibilityIssues counted every row, so a scan the dashboard shows
as 79 issues was reported as "We found 122 issues".

The parser now returns main issues only, folds each duplicate into the issue
its link points at (duplicate_count), and returns how many duplicates it left
out; both tools state that number next to the total. A duplicate whose main
issue cannot be identified is counted only there. Reports without the
"Duplicate Issue of" column are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The scan and issue tools return one text block per message. A client that
joins the blocks with no separator (Claude Code does) runs the report link
into the next block's first word: "...&Key-Pair-Id=KWDR32L2KWU1BWe found 122
issues". The key pair id is opaque, so an agent copying the link takes the
"W" with it and the download fails with 403 InvalidKey. Each block now ends
with a newline.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…low Analyzer reports

fetchAccessibilityIssues read only Website Scanner runs, so an agent that ran
an SDK test with accessibility on, or was handed a Workflow Analyzer report,
had no way to read its issues. The API already exports both the same way as a
scan run: start a CSV export, poll the same route with the task id, download
the link. Only the route and its parameter names differ.

The tool now takes exactly one source: scanId with scanRunId, buildId (the
Accessibility build id, thBuildId in the report link), or workflowReportId.
The fetcher builds each source's export URLs; the parser, duplicate folding
and paging are unchanged. A build the API does not find (the Automate build
id is one) gets a message naming the id to pass instead, and the next-page
hint repeats the source's own arguments.

Rows now carry affected_page, since builds and Workflow Analyzer reports span
several pages, and source_test when the export has a "Source test" column
(Automated Tests builds).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…V link

An agent handed a dashboard report link had to find the ids itself, and
fetchAccessibilityIssues returned parsed issues but not the CSV. Both
now go through the tool.

reportUrl takes a Website Scanner or Automated Tests report link. An
Automated Tests link carries its build id. A Website Scanner link carries
one run's thBuildId, which only the scan list can map to ids: each scan
there carries its latest run's link, so the newest scans are walked
(15 pages) for a match. A link to an older run is reported as such, with
the scan's id, rather than guessed at. The list is walked at its default
page size: at limit=50 or 100 its cursor skips scans.

Every answer now includes the CSV's download link, as startAccessibilityScan
already does. When a link was resolved the answer says to what, and the
next-page hint uses the ids, so later pages skip the lookup.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ading its issues

An agent that fetched an SDK build's issues right after the test run got 14
issues, called final ("All issues retrieved"); the build completed minutes
later with 27. After the tests end a build stays "finalizing" (advanced
issue processing, for an advanced scan) and the export serves whatever
exists so far.

fetchAccessibilityIssues now reads the report's status first: a scan run's
status endpoint, a Workflow Analyzer report's summary, and for a build its
project's build list, since the build overview carries no status. While it
is queued, running, processing or finalizing, a first read waits, polling
every 10 s for up to 5 min with progress notifications, then reads the
issues and says it waited. If processing outlasts the wait, the answer
says the issues are only those found so far and to fetch again, instead of
"All issues retrieved". A later page does not wait again, and a status
that cannot be read is not waited on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Oct 21, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration
  • Configuration used: Central YAML (base), Organization UI (inherited), Workspace UI (inherited)
  • Review profile: ASSERTIVE
  • Plan: Enterprise
  • Run ID: 9ddebfbb-74e9-4218-8a27-b29cb0d7d1a5

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@Aaryan430

Copy link
Copy Markdown
Collaborator Author

Superseded by #503, the same change raised from a branch in this repo instead of a fork, so the stacked PRs show only their own changes.

@Aaryan430 Aaryan430 closed this Oct 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant