Repository navigation
fix(extraction): collapse a damaged C #if group to its first branch before parsing - #1736
danusha2345 wants to merge 2 commits into
Conversation
e1165f8 to
9ce9e30
Compare
|
Second commit, jq lost Two more damage shapes (betaflight, the 7 phantom |
|
Merged current |
|
Merged current |
5cb0e8a to
291ad0e
Compare
695943f to
009912e
Compare
922640b to
d047047
Compare
… before parsing
tree-sitter-c keeps every branch of a `#if … #elif … #else … #endif` group
and parses each as a run of block items, so a group is only harmless when
each branch is complete on its own. Three shapes in real C are not: a branch
that begins with `else` (`} #ifdef X else if (…) { … } #endif`), a branch
that ends with a bare `if (…)` header whose body follows the `#endif` (the
ST HAL's per-device flash-latency tables), and a branch whose braces do not
balance (a signature or a `{` that differs per configuration, body shared).
Each ends in a K&R-shaped `if(cond) { … }` that the grammar reads as an
implicit-int function definition named `if`; the extractor then files it —
and every function after it — under the enclosing function, or the enclosing
function runs to the end of the file and the ones after it vanish (265 such
nested "functions" in 72 files on a betaflight tree, the remainder after
colbymchenry#1729's designated-initializer blank). A second, smaller cause was an
all-caps block macro in statement position (`ATOMIC_BLOCK(NVIC_PRIO_MAX) {`,
`PG_FOREACH(reg) {`), which the statement-macro blank skipped by design
because it only matched lowercase iterator macros.
blankCUnbalancedConditionalBranches runs at the tail of preParseCSource,
after restoreDirectiveLines, because it edits directive lines on purpose:
for a group where any branch shows one of the three symptoms, the first
branch not written `#if 0` is kept verbatim and every other branch plus the
group's directive lines (`\`-continuations included) are blanked to spaces,
newlines and `\r` kept — what the preprocessor would hand a compiler for
that configuration. Balanced groups, the vast majority, are untouched.
Groups nest innermost-first. blankCStatementMacroCalls now also accepts an
ALL_CAPS name (PascalCase — a constructor, if the file is C++ — still
excluded). The C++ grammar parses both shapes cleanly (verified on the same
snippets as .cpp), so preParseCppSource is unchanged; the pre-parse runs
before the kernel route point, so the kernel arm needs no change.
Measured on betaflight (2,109 .c files), against the colbymchenry#1729 build: nested C
functions 265 → 5 (left: `if MACRO {` conditions with no parentheses, and a
`STATIC_DMA_DATA_AUTO union { … } x;` local); C function nodes 39,993 →
39,772 — 260 phantoms gone, ~39 real functions back (spiInternalInitStream,
spiInternalStartDMA, spiInternalStopDMA, processSmartPortTelemetry,
esc4wayProcess, arm_mat_mult_q15, …). Edges: 1,263 lost / 729 gained; the
lost rows are calls previously attributed to the file node or to a phantom
`if` (709 exact-match calls, 490 file-level `contains` of what were really
locals), the gained rows are the same calls re-attributed to the real
function (671 exact-match). Fuzzy-resolved calls 154 → 37.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Keeping the first branch dropped `main` from jq's main.c: `#ifdef WIN32
int umain(…) { #else int main(…) { #endif` kept the WIN32 head. A group
that opens on a feature test (`#ifdef X`, `#if defined(X)`, `#if X`) and
has an `#else` now keeps the `#else` branch — the build without the
feature; `#ifndef` / `#if !…` keep the first, `#if 0` the live one.
Two more damage shapes, both from betaflight: a branch that is a
fragment of an expression (`|| (h4 == NULL)` inside an `if (…)`,
`isEmpty(s) ||` before the rest of a condition) and one that opens with
`} else if (…) {` — closing a block it did not open. Both left phantom
`if` functions behind (7 on that tree).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
d047047 to
f3db172
Compare
…mchenry#2876; 11 landed PRs yield to main; EXTRACTION_VERSION 55) Main landed eleven of our PRs as our own commits with the maintainer's additions (colbymchenry#2847, colbymchenry#2848, colbymchenry#2850, colbymchenry#2851, colbymchenry#2853, colbymchenry#2854, colbymchenry#2855, colbymchenry#2856, colbymchenry#2868, colbymchenry#2874, colbymchenry#2875), the traversal half of colbymchenry#2839 (colbymchenry#2860) and the trail hardening of colbymchenry#1667 (colbymchenry#2869). Where the integration carried our original version, main's wins: play.ts, routes.ts, the tsjs kernel extractor, import-resolver's PHP alias lookup, isDecisiveSameFileMatch, matchNewReceiverCall and the tests of those PRs. Kept beside main: colbymchenry#1952's enclosing-scope member pick, colbymchenry#1936's closest-copy confidence, colbymchenry#2258's Kotlin imported-type block, colbymchenry#1875's commitHdlProfile on indexFiles and hdlAccess, colbymchenry#1583's impact budget (the type-hierarchy walk itself is main's), colbymchenry#1736's C #if tests. 22 changelog bullets of closed PRs removed where main has its own wording. EXTRACTION_VERSION 55: Rust receiver chains (colbymchenry#2846), `new C().m` shapes (colbymchenry#2854), Java records (colbymchenry#2868), Go method exportedness (colbymchenry#2857), generic supertypes (colbymchenry#2856). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… walk's edge dedup The merge dropped the `finally` tail shared between colbymchenry#1736's last test and main's new Go package test, so extraction.test.ts no longer parsed. The type-hierarchy walk keeps the diamond edges and the cycle dedup its graph.test.ts cases expect, on top of main's colbymchenry#2860. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… named `if`
tree-sitter-c parses every branch of a `#if … #elif … #else … #endif`
group as a run of block items, so a group is harmless only when each
branch is complete on its own. Three shapes in real C are not: a branch
that begins with `else` (an arm of an if-chain kept behind `#ifdef`), a
branch that ends with a bare `if (…)` header whose body follows the
`#endif` (ST's HAL latency tables), and a branch whose braces don't
balance (a function head that differs per build). Each leaves a
K&R-shaped `if (…) { … }` that the grammar reads as a function named
`if`, filed inside the enclosing function, and the functions after it
were nested under it or lost. Separately, a block macro in capitals in
statement position (`ATOMIC_BLOCK(NVIC_PRIO_MAX) {`) was skipped by the
statement-macro blank, which matched lowercase iterator macros only.
blankCUnbalancedConditionalBranches runs at the end of preParseCSource,
after the directive restore, since it edits directive lines. A group
where a branch shows one of those shapes, or is a fragment of an
expression or a list (`|| (h4 == NULL)` inside an `if (…)`), loses its
directive lines, offsets kept. When a branch can't stand next to the
others, one branch is kept verbatim: the `#else` of a positive test
(`#ifdef X`, `#if defined(X)`), the first of `#ifndef`, the live one of
`#if 0`. The statement-macro blank also takes ALL_CAPS names (PascalCase
still excluded, for C++ constructors). Both engines see it: the kernel
gets the pre-parsed bytes from the TS route point.
Ours on top:
- A group on a flag whose branches each balance (braces and parens)
keeps every branch, only its directive lines blanked: read in a row
they are still C (`} else if (a) {…} else if (b) {…}`, list entries,
`|| x || y`). Dropping one lost real code of builds the project uses:
betaflight's failsafe.c arms under `#if ENABLE_RESCUE_PLAN` (default-on
in common_post.h), and adc_stm32h7xx.c's adcTagMap, whose inner MCU
chain kept its `#else #error` branch. Unbalanced and literal groups
keep one branch, as above. The PR's first unit test now expects both
arms of its balanced else-led group.
- The group is read on a copy with comments and literals masked. A brace
inside a block comment over several lines counted as code, so Lua's
`** {======` banners made loadlib.c's platform group look damaged, and
its dlopen, Windows and dyld implementations were dropped for the stub;
on betaflight it dropped 19 CMSIS-DSP MVE functions.
- A line splice is read on the raw line: a `\` at the end of a comment
line inside a #define body still continues the #define (masking hid
it, which made jemalloc's tsd.h include guard look damaged).
- Only a directive's own `\` continuations are blanked with it. The PR
blanked every directive line right under a collapsed group's `#if` or
`#else`, so a kept branch lost a leading `#include` or `#define`.
- `#if 1` keeps its first branch; it kept the dead `#else`.
- Unit tests for each, mutation-checked; CHANGELOG bullet rewritten
without numbers.
- Tried and dropped: keeping the feature branch of a flag the file
#defines itself (the same-file evidence PR #2615 uses). Applied per
group, it broke dtoa's coordinated groups in jq (`#ifdef
Avoid_Underflow` kept its branch while a later `#ifndef
Avoid_Underflow` still kept its `}`), cutting jvp_strtod in half.
Validation, main b6379e6 vs this:
- Phantom functions (named if or (…)) removed / real functions added:
betaflight 49679ccd 257 / 40 (spiInternalStartDMA, gyroDetect,
processSmartPortTelemetry, esc4wayProcess, the CMSIS matrix
routines), stm32h7xx_hal_driver 7e541d92 68 / 6 (LL_SetFlashLatency,
HAL_ADC_AnalogWDGConfig), stm32g4xx_hal_driver a6001282 30 / 2, redis 7f440372 8 / 16 (tre_parse, je_mallocx,
lua_cmsgpack), jq 49fbca12 1 / 11 (strptime, gethex, jvp_dtoa). No
real function lost except jq's WIN32-only wmain. Enum members added:
betaflight 701, stm32h7 8.
- Calls move to the real function: on betaflight, 746 calls that sat on
a phantom or on the file node are gone, and 770 now sit on real
functions (369 on restored ones, 401 on functions that existed but
had lost them to a phantom). Lost: references from `#if`
conditions inside an initializer (betaflight TIM_N x12), test-macro
references (redis TEST x10), and a few calls in feature branches of
unbalanced groups.
- leveldb (C++) and gin (Go) byte-identical.
- kernel-parity (betaflight, stm32h7, redis, jq): 0 files with diffs on
main and here; fewer files defer to wasm (betaflight 1216 -> 1095,
redis 169 -> 150, jq 21 -> 15, stm32h7 156 -> 146).
- Cost: C preParse CPU over betaflight 5.7 -> 8.6 s (the PR as written
8.4 s), redis 1.2 -> 1.6 s.
Whole-index CPU, median of 3 interleaved: redis 41.2 vs 40.0 s,
stm32h7xx_hal_driver 21.4 vs 21.9 s (noise).
Original PR #1736 by @danusha2345.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks @danusha2345! Your pass landed in #2940 with you as the commit author, and it clears every phantom |
…mchenry#2972; 10 landed PRs yield to main; EXTRACTION_VERSION 56) Main landed ten more of our PRs as our own commits with the maintainer's additions (colbymchenry#2923 ← colbymchenry#1936, colbymchenry#2924 ← colbymchenry#1951, colbymchenry#2925 ← colbymchenry#1939, colbymchenry#2930 ← colbymchenry#2921, colbymchenry#2940 ← colbymchenry#1736, colbymchenry#2942 ← colbymchenry#1947, colbymchenry#2950 ← colbymchenry#1952, colbymchenry#2961 ← colbymchenry#1954, colbymchenry#2965 ← colbymchenry#1942, colbymchenry#2972 ← colbymchenry#1933) and the module-path half of colbymchenry#2308 (colbymchenry#2955). Where the integration carried our version, main's wins: the enclosing-scope member pick and its tests, the Go and Java receiver typers, the delegating-call self-loop rule, the duplicated-module copy confidence, the C #if collapse, the module-path resolution of Rust paths, the trail/path hardening in utils.ts. Kept beside main: colbymchenry#1875's Verilog hooks and status fields, colbymchenry#2839's implementations command, colbymchenry#1923, colbymchenry#2922, colbymchenry#2258, colbymchenry#1761, colbymchenry#1583, the bare-call half of colbymchenry#2308 (rust-use-bindings.ts). CHANGELOG is main's plus the open PRs' bullets. EXTRACTION_VERSION 56: the C #if group handling (colbymchenry#2940) and the kernel change. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…tests, colbymchenry#1947's cls exemption, locale-tiebreak test) Our copies of the C #if tests colbymchenry#2940 superseded are dropped (one of them was main's own test twice); the library-method rule no longer exempts a receiver named `cls`, as colbymchenry#2942 landed it; ts-this-field-call.test.ts is main's (colbymchenry#2927 landed the same code-unit tiebreak the integration carried from the earlier PR). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Stacked on #1733 (its commit comes first; rebases to one commit once that lands). Together they close the C extraction extent problem behind #1729: after #1733 a 2,109-file betaflight tree still had 265 C functions the graph filed inside another function; after this, 5.
What the 265 were
Not one cause, and mostly not real functions. 192 of them are literally named
if, the rest(NVIC_PRIO_TIMER),(reg),ATMEL_DEVICE_MATCH— phantoms. Verified shape by shape with the shippedtree-sitter-c.wasm:#ifbranch that begins withelse. tree-sitter-c parses each preprocessor branch as a run of block items, andelsecannot start one, so theif (cond) { … }that follows is read as a K&R implicit-int function definition namedif, nested in the enclosing function:current.c,stm32*_ll_usart.c,_ll_tim.c,_ll_gpio.c,flash_w25n.c, …)if (…)/else if (…)header whose{body sits after#endif— the STM32H7ll_utils.clatency tables, 24 phantoms in one file.arm_mat_*,smartport.c,bus_spi_ll.c): here the enclosing function vanished entirely and its locals leaked to file scope —arm_mat_mult_q15.chad zero function nodes.ATOMIC_BLOCK(NVIC_PRIO_MAX) {,PG_FOREACH(reg) {— whichblankCStatementMacroCallsmatched in lowercase only.The fix
blankCUnbalancedConditionalBranches: for a#if … #else/#elif … #endifgroup where any branch shows one of shapes 1–3, keep the first branch not written#if 0verbatim and blank every other branch and the group's directive lines (continuations included) to spaces, newlines kept. Balanced groups are untouched; innermost groups first. It runs at the tail ofpreParseCSource, afterrestoreDirectiveLines, for the same reasonblankCNamedVariadicDefineDotsdoes — it edits directive lines deliberately.blankCStatementMacroCallsalso accepts[A-Z_][A-Z0-9_]*now (PascalCase still excluded, for the C++-constructor reason in its comment). The C++ grammar parses both snippets cleanly, so the C++ path is left alone; the pre-parse precedes the kernel route point, so the kernel needs no change.Measured
betaflight fork, 2,109
.cfiles, wasm arm:fuzzycallsEdge-set delta against the #1733 index, rows keyed with
resolvedBy: 1,263 lost / 729 gained. Lost: 709exact-matchcalls whose source was a phantom (if@…) or a file node holding leaked locals, 490 file-levelcontainsfor those locals (pInA,row,flag), 53 references, 11 function-refs. Gained: 671exact-matchcalls from the recovered functions (USBD_LL_Init → HAL_PCD_Init,LL_TIM_DeInit → …,processSmartPortTelemetry → erpmToRpm) and 43containsfor them. Functions that now exist with their real extents:spiInternalInitStream/StartDMA/StopDMA(bus_spi_ll.c),processSmartPortTelemetry(522–903),esc4wayProcess(429–929),arm_mat_mult_q15. Ten lost and ten gained rows read against source: every lost row is a re-attribution from a phantom or a file node, not a real call gone.The 5 that remain: four
if SILABS_DEVICE_MATCH {/else if ATMEL_DEVICE_MATCH {inserial_4way.c(a paren-less macro as anifcondition) and oneSTATIC_DMA_DATA_AUTO union { … } testBuffer;inflashfs.c— each its own narrow blank, not folded in here.Tests
extraction.test.ts: a unit test of the blank (length and line count preserved, a continuation line blanked, the first branch kept, the second branch and the directives blanked, a balanced#ifdefuntouched,#if 0keeps the live branch, identity on a file without groups) and an end-to-end index where a function after a damaged group used to be a phantom(NVIC_PRIO_MAX)and now has its own extent and top-level name; the end-to-end test fails without the change.extraction.test.ts+resolution.test.ts: 815,tscclean.🤖 Generated with Claude Code