Skip to content

feat: report actual component removals in the Markdown fidelity checker - #4316

Merged
khadni merged 10 commits into
mainfrom
feat/component-coverage-report
Oct 20, 2026
Merged

khadni merged 10 commits into
mainfrom
feat/component-coverage-report

Conversation

@khadni

@khadni khadni commented Oct 14, 2026 •

Copy link
Copy Markdown
Collaborator

Some JSX components disappear from agent-facing exports without a clear signal. This PR records actual component removals during conversion and reports them through the existing Markdown fidelity checker, which now also verifies the final TXT output and has fewer false positives.

Changes

  • transformMarkdown accepts an optional onDropComponent observer, passed through buildMarkdownArtifact and renderLlmsPageMarkdown. Events are delivered only after a successful transformation, and the observer cannot change the output.
  • unsupported now means a JSX subtree was actually removed. Gaps in static source expectations are reported as unverifiable.
  • The checker verifies Markdown and final per-page TXT output. Identical findings are deduplicated.
  • New or corrected expectations: DocCards/DocCard, MethodTabs, inline CopyText, em/i, a, img (compared by URL and alt text), and tab slots written on one line. FeedPage is unverifiable (its export is a placeholder).
  • The Source line of latest CCIP pages is expected at their published URL (/ccip/..., not /ccip/v2/...).
  • One reviewed exception: PolicyInstanceBuilder (interactive tool; equivalent information exists elsewhere).
  • Focused runs (--path) list detailed findings. CI warnings and job summaries list only new findings; CI stays non-blocking.

Baseline and commands

The baseline is fully regenerated: 21,129 stale identities → 1,195 known findings. It records known debt, not approved losses.

  • --update-baseline: regenerates the baseline from a complete run, excluding exceptions; refuses if a page check fails.
  • --omit-present: omits successful evidence from the JSON report (CI artifact ~127 MB → ~0.9 MB).
  • --publish: writes the public component inventory; a failed run keeps the previous file.

Reproduce the baseline: npm run check:markdown-fidelity -- --update-baseline --omit-present

Public report and CI

/reports/component-coverage.json uses schemaVersion 2. Refresh it weekly with npm run report:markdown-fidelity from up-to-date main, in a dedicated PR containing only that file. The Vercel build is unchanged and no TXT bundle is rewritten. The CI filter also runs validation on conversion dependency updates (MDX, remark, unified).

Validation

  • Strict full-corpus check: 1,215 routes, zero new findings.
  • missing 3,073 → 480, unverifiable 1,213 → 598 compared with the checker on main. The 117 remaining unsupported are real removals.

Follow-ups

  • Remaining removals: data lists (FeedList, StreamsNetworkAddressesTable), landing-page navigation (ToolsUtilitiesGrid, LayoutHero, TabGrid), ReportSchemaTabs, ChangelogSnippet, the interactive Remix tutorial steps (CCIP), and interactive tools to record as exceptions.
  • Checker limitations: raw HTML tables are unverifiable.

@vercel

vercel Bot commented Oct 14, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
documentation Ready Ready Preview Oct 20, 2026 7:27pm UTC

Request Review

@khadni khadni changed the title Report unhandled components in LLM bundles feat: report actual component removals in the Markdown fidelity checker Oct 19, 2026
@khadni
khadni merged commit 801de5f into main Oct 20, 2026
27 checks passed
@khadni
khadni deleted the feat/component-coverage-report branch October 20, 2026 19:37

This branch was successfully deployed

1 active deployment
Preview — bdaa5acd Deployed Oct 20, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants