Skip to content

ci(e2e): wait for stable doc screenshots, strip exclude-from-example-doc lines - #4429

Merged
mfazekas merged 1 commit into
mainfrom
fix/stable-doc-screenshots
Oct 29, 2026
Merged

mfazekas merged 1 commit into
mainfrom
fix/stable-doc-screenshots

Conversation

@mfazekas

Copy link
Copy Markdown
Contributor

Description

Nightly Percy doc screenshots differ from run to run (label placement, tile timing, a wall-clock timestamp). docScreenshots.e2e.js now saves a screenshot only once two consecutive ones, 500 ms apart, match within 10 pixels, leaving out the status bar. The cap is 10 s, and on timeout it keeps the last one and logs the example. The comparison runs in a separate node process (e2e/changedPixels.js, ~0.15 s), like cropTop.js. UserLocationUpdates gets excludeFromPercy: true for its Timestamp: line, and its doc site screenshot is still taken.

scripts/example-docs.ts now drops lines ending with // exclude-from-example-doc (until now they leaked into the docs) and blocks between /* exclude-from-example-doc:start */ and /* exclude-from-example-doc:end */. Both markers are documented in CONTRIBUTING.md and AGENTS.md.

Checklist

  • I've read CONTRIBUTING.md
  • I updated the doc/other generated code with running yarn generate in the root folder
  • I have tested the new feature on /example app.
    • In V11 mode/ios
    • In New Architecture mode/ios
    • In V11 mode/android
    • In New Architecture mode/android
  • I added/updated a sample - if a new feature was implemented (/example)

Screenshot OR Video

Local runs used ios.release.ci with RNMBX_E2E_MAPS_IDLE=1 on an iPhone 17 simulator, and the Percy copies were compared with pixelmatch (threshold 0.1).

  1. Two normal runs of the flaky subset (MapUnMount, SourceLayerVisibility, GradientLine, ShowAndHideLayer, StyleJson, ChoroplethLayerByZoomLevel, ThirdPartyVectorSource, QuerySourceFeatures, UserLocationUpdates) plus ShowClick and GeoJSONSource were identical, except UserLocationUpdates (4286 px, the timestamp). The flakiness didn't show up locally, and every example settled at the first comparison.
  2. With the idle wait removed to mimic a slow runner, the first screenshot of MapUnMount differed by 43264 px (1.47%) between runs. The settled one was identical, and matched the normal run too.
  3. Full run: 74/74 passed in 1083 s, and the stable capture took 120 s of that. 72 examples settled after 2 screenshots (~1.5 s), ImageOverlay after 6, and CustomNativeUserLocation hit the 10 s cap.

The generated example pages for ViewportGestures, UserLocationPadding and UserLocationUpdates no longer show the ExampleMetadata import, and no other page changes.

…doc lines

Doc screenshots are now taken once two consecutive screenshots match (status bar
left out), with a 10 s cap. UserLocationUpdates is excluded from Percy as it shows
a wall-clock timestamp. example-docs.ts drops lines ending with
// exclude-from-example-doc and blocks between exclude-from-example-doc:start/end.
@mfazekas
mfazekas marked this pull request as ready for review October 29, 2026 16:24
@mfazekas
mfazekas merged commit 47a736a into main Oct 29, 2026
10 checks passed
@mfazekas
mfazekas deleted the fix/stable-doc-screenshots branch October 29, 2026 16:24

This branch was successfully deployed

1 active deployment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant