Skip to content

fix(extraction): collapse a damaged C #if group to its first branch before parsing - #1736

Closed
danusha2345 wants to merge 2 commits into
colbymchenry:mainfrom
danusha2345:fix/c-preprocessor-brace-damage
Closed

danusha2345 wants to merge 2 commits into
colbymchenry:mainfrom
danusha2345:fix/c-preprocessor-brace-damage

Conversation

@danusha2345

Copy link
Copy Markdown
Contributor

Stacked on #1733 (its commit comes first; rebases to one commit once that lands). Together they close the C extraction extent problem behind #1729: after #1733 a 2,109-file betaflight tree still had 265 C functions the graph filed inside another function; after this, 5.

What the 265 were

Not one cause, and mostly not real functions. 192 of them are literally named if, the rest (NVIC_PRIO_TIMER), (reg), ATMEL_DEVICE_MATCH — phantoms. Verified shape by shape with the shipped tree-sitter-c.wasm:

  1. A #if branch that begins with else. tree-sitter-c parses each preprocessor branch as a run of block items, and else cannot start one, so the if (cond) { … } that follows is read as a K&R implicit-int function definition named if, nested in the enclosing function:
    void f(int id) {
        if (id == 1) { a(id); }
    #ifdef USE_V
        else if (id == 2) { b(id); }
    #endif
        else { c(id); }
    }
    (current.c, stm32*_ll_usart.c, _ll_tim.c, _ll_gpio.c, flash_w25n.c, …)
  2. A branch that ends with a bare if (…) / else if (…) header whose { body sits after #endif — the STM32H7 ll_utils.c latency tables, 24 phantoms in one file.
  3. Branches that leave braces unbalanced (CMSIS arm_mat_*, smartport.c, bus_spi_ll.c): here the enclosing function vanished entirely and its locals leaked to file scope — arm_mat_mult_q15.c had zero function nodes.
  4. All-caps block macros in statement position — ATOMIC_BLOCK(NVIC_PRIO_MAX) {, PG_FOREACH(reg) { — which blankCStatementMacroCalls matched in lowercase only.

The fix

blankCUnbalancedConditionalBranches: for a #if … #else/#elif … #endif group where any branch shows one of shapes 1–3, keep the first branch not written #if 0 verbatim and blank every other branch and the group's directive lines (continuations included) to spaces, newlines kept. Balanced groups are untouched; innermost groups first. It runs at the tail of preParseCSource, after restoreDirectiveLines, for the same reason blankCNamedVariadicDefineDots does — it edits directive lines deliberately. blankCStatementMacroCalls also accepts [A-Z_][A-Z0-9_]* now (PascalCase still excluded, for the C++-constructor reason in its comment). The C++ grammar parses both snippets cleanly, so the C++ path is left alone; the pre-parse precedes the kernel route point, so the kernel needs no change.

Measured

betaflight fork, 2,109 .c files, wasm arm:

#1733 this
C functions nested inside another C function 265 5
C function nodes 39,993 39,772 (260 phantoms gone, ~39 real functions back)
fuzzy calls 154 37

Edge-set delta against the #1733 index, rows keyed with resolvedBy: 1,263 lost / 729 gained. Lost: 709 exact-match calls whose source was a phantom (if@…) or a file node holding leaked locals, 490 file-level contains for those locals (pInA, row, flag), 53 references, 11 function-refs. Gained: 671 exact-match calls from the recovered functions (USBD_LL_Init → HAL_PCD_Init, LL_TIM_DeInit → …, processSmartPortTelemetry → erpmToRpm) and 43 contains for them. Functions that now exist with their real extents: spiInternalInitStream/StartDMA/StopDMA (bus_spi_ll.c), processSmartPortTelemetry (522–903), esc4wayProcess (429–929), arm_mat_mult_q15. Ten lost and ten gained rows read against source: every lost row is a re-attribution from a phantom or a file node, not a real call gone.

The 5 that remain: four if SILABS_DEVICE_MATCH { / else if ATMEL_DEVICE_MATCH { in serial_4way.c (a paren-less macro as an if condition) and one STATIC_DMA_DATA_AUTO union { … } testBuffer; in flashfs.c — each its own narrow blank, not folded in here.

Tests

extraction.test.ts: a unit test of the blank (length and line count preserved, a continuation line blanked, the first branch kept, the second branch and the directives blanked, a balanced #ifdef untouched, #if 0 keeps the live branch, identity on a file without groups) and an end-to-end index where a function after a damaged group used to be a phantom (NVIC_PRIO_MAX) and now has its own extent and top-level name; the end-to-end test fails without the change. extraction.test.ts + resolution.test.ts: 815, tsc clean.

🤖 Generated with Claude Code

@danusha2345

Copy link
Copy Markdown
Contributor Author

Rebased onto current main (2f1a99d): head 9ce9e30, one commit. The #1733 commit this was stacked on is dropped since #1755 landed the same blanking; the #if collapse applies cleanly on top of it. tsc clean; the C extraction tests pass.

@danusha2345

Copy link
Copy Markdown
Contributor Author

Second commit, b0ed0d1, from running the branch on two more C trees.

jq lost main. src/main.c has #ifdef WIN32 int umain(…) { #else int main(…) { #endif — two function heads in one group. Keeping the first branch kept the WIN32 head, and main (289–730) was gone from the index. The survivor is now the default-build branch: a group that opens on a feature test (#ifdef X, #if defined(X), #if X) and has an #else keeps the #else branch; #ifndef / #if !… keep the first; #if 0 the live one. jq: main back, phantom if functions 1 → 0, fuzzy 23 → 0.

Two more damage shapes (betaflight, the 7 phantom ifs the first cut left): a branch that is a fragment of an expression — || (hopamp4 == NULL) inside an if (…) in the STM32 HAL, isEmpty(cmdline) || before the rest of a condition in cli.c — and a branch that opens with } else if (…) {, closing a block it did not open. Betaflight (2,109 C files, main at cd4e65b which has #1755): phantom if 203 → 0; edges −1,339 / +1,429 against main, the removals re-attributions off file-level nodes and the additions the members of enums that now parse (IRQn_Type). Fixtures for all three shapes are in the test; the first test's expectations flipped to the #else branch.

@danusha2345

Copy link
Copy Markdown
Contributor Author

Merged current main (1f4379d, post-#1804) into this branch: head 1c261bc. One conflict, the tail of the C pre-parse pipeline: #1804 wraps it in the raw-string restore, this PR adds the #if-group collapse. Composed as restore ∘ collapse ∘ variadic-define ∘ directive-restore — the collapse is length-preserving, and it counts braces with raw-string bodies still masked, so a brace inside R"(...)" can't trip it. tsc clean; extraction, cpp-raw-string-delimiter-haserror and cpp-raw-string-preparse-1505 pass (697 tests).

@danusha2345

Copy link
Copy Markdown
Contributor Author

Merged current main (7a23648) into this branch: head 8dff394. The only conflict was the CHANGELOG entry; no source conflict. tsc clean, the branch's tests pass.

@danusha2345
danusha2345 force-pushed the fix/c-preprocessor-brace-damage branch 3 times, most recently from 5cb0e8a to 291ad0e Compare October 6, 2026 09:53
@danusha2345
danusha2345 force-pushed the fix/c-preprocessor-brace-damage branch 3 times, most recently from 695943f to 009912e Compare October 10, 2026 11:36
@danusha2345
danusha2345 force-pushed the fix/c-preprocessor-brace-damage branch 6 times, most recently from 922640b to d047047 Compare October 22, 2026 10:53
danusha2345 and others added 2 commits October 24, 2026 16:44
… before parsing

tree-sitter-c keeps every branch of a `#if … #elif … #else … #endif` group
and parses each as a run of block items, so a group is only harmless when
each branch is complete on its own. Three shapes in real C are not: a branch
that begins with `else` (`} #ifdef X  else if (…) { … } #endif`), a branch
that ends with a bare `if (…)` header whose body follows the `#endif` (the
ST HAL's per-device flash-latency tables), and a branch whose braces do not
balance (a signature or a `{` that differs per configuration, body shared).
Each ends in a K&R-shaped `if(cond) { … }` that the grammar reads as an
implicit-int function definition named `if`; the extractor then files it —
and every function after it — under the enclosing function, or the enclosing
function runs to the end of the file and the ones after it vanish (265 such
nested "functions" in 72 files on a betaflight tree, the remainder after
colbymchenry#1729's designated-initializer blank). A second, smaller cause was an
all-caps block macro in statement position (`ATOMIC_BLOCK(NVIC_PRIO_MAX) {`,
`PG_FOREACH(reg) {`), which the statement-macro blank skipped by design
because it only matched lowercase iterator macros.

blankCUnbalancedConditionalBranches runs at the tail of preParseCSource,
after restoreDirectiveLines, because it edits directive lines on purpose:
for a group where any branch shows one of the three symptoms, the first
branch not written `#if 0` is kept verbatim and every other branch plus the
group's directive lines (`\`-continuations included) are blanked to spaces,
newlines and `\r` kept — what the preprocessor would hand a compiler for
that configuration. Balanced groups, the vast majority, are untouched.
Groups nest innermost-first. blankCStatementMacroCalls now also accepts an
ALL_CAPS name (PascalCase — a constructor, if the file is C++ — still
excluded). The C++ grammar parses both shapes cleanly (verified on the same
snippets as .cpp), so preParseCppSource is unchanged; the pre-parse runs
before the kernel route point, so the kernel arm needs no change.

Measured on betaflight (2,109 .c files), against the colbymchenry#1729 build: nested C
functions 265 → 5 (left: `if MACRO {` conditions with no parentheses, and a
`STATIC_DMA_DATA_AUTO union { … } x;` local); C function nodes 39,993 →
39,772 — 260 phantoms gone, ~39 real functions back (spiInternalInitStream,
spiInternalStartDMA, spiInternalStopDMA, processSmartPortTelemetry,
esc4wayProcess, arm_mat_mult_q15, …). Edges: 1,263 lost / 729 gained; the
lost rows are calls previously attributed to the file node or to a phantom
`if` (709 exact-match calls, 490 file-level `contains` of what were really
locals), the gained rows are the same calls re-attributed to the real
function (671 exact-match). Fuzzy-resolved calls 154 → 37.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Keeping the first branch dropped `main` from jq's main.c: `#ifdef WIN32
int umain(…) { #else int main(…) { #endif` kept the WIN32 head. A group
that opens on a feature test (`#ifdef X`, `#if defined(X)`, `#if X`) and
has an `#else` now keeps the `#else` branch — the build without the
feature; `#ifndef` / `#if !…` keep the first, `#if 0` the live one.

Two more damage shapes, both from betaflight: a branch that is a
fragment of an expression (`|| (h4 == NULL)` inside an `if (…)`,
`isEmpty(s) ||` before the rest of a condition) and one that opens with
`} else if (…) {` — closing a block it did not open. Both left phantom
`if` functions behind (7 on that tree).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@danusha2345
danusha2345 force-pushed the fix/c-preprocessor-brace-damage branch from d047047 to f3db172 Compare October 24, 2026 14:07
danusha2345 pushed a commit to danusha2345/codegraph that referenced this pull request Oct 24, 2026
…mchenry#2876; 11 landed PRs yield to main; EXTRACTION_VERSION 55)

Main landed eleven of our PRs as our own commits with the maintainer's additions
(colbymchenry#2847, colbymchenry#2848, colbymchenry#2850, colbymchenry#2851, colbymchenry#2853, colbymchenry#2854, colbymchenry#2855, colbymchenry#2856, colbymchenry#2868, colbymchenry#2874, colbymchenry#2875), the
traversal half of colbymchenry#2839 (colbymchenry#2860) and the trail hardening of colbymchenry#1667 (colbymchenry#2869). Where the
integration carried our original version, main's wins: play.ts, routes.ts, the
tsjs kernel extractor, import-resolver's PHP alias lookup, isDecisiveSameFileMatch,
matchNewReceiverCall and the tests of those PRs. Kept beside main: colbymchenry#1952's
enclosing-scope member pick, colbymchenry#1936's closest-copy confidence, colbymchenry#2258's Kotlin
imported-type block, colbymchenry#1875's commitHdlProfile on indexFiles and hdlAccess, colbymchenry#1583's
impact budget (the type-hierarchy walk itself is main's), colbymchenry#1736's C #if tests.
22 changelog bullets of closed PRs removed where main has its own wording.

EXTRACTION_VERSION 55: Rust receiver chains (colbymchenry#2846), `new C().m` shapes (colbymchenry#2854),
Java records (colbymchenry#2868), Go method exportedness (colbymchenry#2857), generic supertypes (colbymchenry#2856).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
danusha2345 pushed a commit to danusha2345/codegraph that referenced this pull request Oct 24, 2026
… walk's edge dedup

The merge dropped the `finally` tail shared between colbymchenry#1736's last test and main's new Go
package test, so extraction.test.ts no longer parsed. The type-hierarchy walk keeps the
diamond edges and the cycle dedup its graph.test.ts cases expect, on top of main's colbymchenry#2860.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
colbymchenry pushed a commit that referenced this pull request Oct 27, 2026
… named `if`

tree-sitter-c parses every branch of a `#if … #elif … #else … #endif`
group as a run of block items, so a group is harmless only when each
branch is complete on its own. Three shapes in real C are not: a branch
that begins with `else` (an arm of an if-chain kept behind `#ifdef`), a
branch that ends with a bare `if (…)` header whose body follows the
`#endif` (ST's HAL latency tables), and a branch whose braces don't
balance (a function head that differs per build). Each leaves a
K&R-shaped `if (…) { … }` that the grammar reads as a function named
`if`, filed inside the enclosing function, and the functions after it
were nested under it or lost. Separately, a block macro in capitals in
statement position (`ATOMIC_BLOCK(NVIC_PRIO_MAX) {`) was skipped by the
statement-macro blank, which matched lowercase iterator macros only.

blankCUnbalancedConditionalBranches runs at the end of preParseCSource,
after the directive restore, since it edits directive lines. A group
where a branch shows one of those shapes, or is a fragment of an
expression or a list (`|| (h4 == NULL)` inside an `if (…)`), loses its
directive lines, offsets kept. When a branch can't stand next to the
others, one branch is kept verbatim: the `#else` of a positive test
(`#ifdef X`, `#if defined(X)`), the first of `#ifndef`, the live one of
`#if 0`. The statement-macro blank also takes ALL_CAPS names (PascalCase
still excluded, for C++ constructors). Both engines see it: the kernel
gets the pre-parsed bytes from the TS route point.

Ours on top:
- A group on a flag whose branches each balance (braces and parens)
  keeps every branch, only its directive lines blanked: read in a row
  they are still C (`} else if (a) {…} else if (b) {…}`, list entries,
  `|| x || y`). Dropping one lost real code of builds the project uses:
  betaflight's failsafe.c arms under `#if ENABLE_RESCUE_PLAN` (default-on
  in common_post.h), and adc_stm32h7xx.c's adcTagMap, whose inner MCU
  chain kept its `#else #error` branch. Unbalanced and literal groups
  keep one branch, as above. The PR's first unit test now expects both
  arms of its balanced else-led group.
- The group is read on a copy with comments and literals masked. A brace
  inside a block comment over several lines counted as code, so Lua's
  `** {======` banners made loadlib.c's platform group look damaged, and
  its dlopen, Windows and dyld implementations were dropped for the stub;
  on betaflight it dropped 19 CMSIS-DSP MVE functions.
- A line splice is read on the raw line: a `\` at the end of a comment
  line inside a #define body still continues the #define (masking hid
  it, which made jemalloc's tsd.h include guard look damaged).
- Only a directive's own `\` continuations are blanked with it. The PR
  blanked every directive line right under a collapsed group's `#if` or
  `#else`, so a kept branch lost a leading `#include` or `#define`.
- `#if 1` keeps its first branch; it kept the dead `#else`.
- Unit tests for each, mutation-checked; CHANGELOG bullet rewritten
  without numbers.
- Tried and dropped: keeping the feature branch of a flag the file
  #defines itself (the same-file evidence PR #2615 uses). Applied per
  group, it broke dtoa's coordinated groups in jq (`#ifdef
  Avoid_Underflow` kept its branch while a later `#ifndef
  Avoid_Underflow` still kept its `}`), cutting jvp_strtod in half.

Validation, main b6379e6 vs this:
- Phantom functions (named if or (…)) removed / real functions added:
  betaflight 49679ccd 257 / 40 (spiInternalStartDMA, gyroDetect,
  processSmartPortTelemetry, esc4wayProcess, the CMSIS matrix
  routines), stm32h7xx_hal_driver 7e541d92 68 / 6 (LL_SetFlashLatency,
  HAL_ADC_AnalogWDGConfig), stm32g4xx_hal_driver a6001282 30 / 2, redis 7f440372 8 / 16 (tre_parse, je_mallocx,
  lua_cmsgpack), jq 49fbca12 1 / 11 (strptime, gethex, jvp_dtoa). No
  real function lost except jq's WIN32-only wmain. Enum members added:
  betaflight 701, stm32h7 8.
- Calls move to the real function: on betaflight, 746 calls that sat on
  a phantom or on the file node are gone, and 770 now sit on real
  functions (369 on restored ones, 401 on functions that existed but
  had lost them to a phantom). Lost: references from `#if`
  conditions inside an initializer (betaflight TIM_N x12), test-macro
  references (redis TEST x10), and a few calls in feature branches of
  unbalanced groups.
- leveldb (C++) and gin (Go) byte-identical.
- kernel-parity (betaflight, stm32h7, redis, jq): 0 files with diffs on
  main and here; fewer files defer to wasm (betaflight 1216 -> 1095,
  redis 169 -> 150, jq 21 -> 15, stm32h7 156 -> 146).
- Cost: C preParse CPU over betaflight 5.7 -> 8.6 s (the PR as written
  8.4 s), redis 1.2 -> 1.6 s.
  Whole-index CPU, median of 3 interleaved: redis 41.2 vs 40.0 s,
  stm32h7xx_hal_driver 21.4 vs 21.9 s (noise).

Original PR #1736 by @danusha2345.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@colbymchenry

Copy link
Copy Markdown
Owner

Thanks @danusha2345! Your pass landed in #2940 with you as the commit author, and it clears every phantom if function on betaflight, the ST HALs, redis and jq. We changed one thing in how a damaged group is opened: when each branch balances on its own, all branches now stay and only the directive lines go, because keeping one branch dropped real code, like betaflight's rescue-plan arms in failsafe.c and adc_stm32h7xx.c's adcTagMap. Groups that don't balance, and #if 0/#if 1, still keep one branch as you designed. We also fixed a few edge cases: braces inside multi-line comments (Lua's banners), a \ splice inside a comment, a directive right under a kept branch, and #if 1. The CHANGELOG line is reworded without numbers. It ships in the next release.

danusha2345 pushed a commit to danusha2345/codegraph that referenced this pull request Oct 28, 2026
…mchenry#2972; 10 landed PRs yield to main; EXTRACTION_VERSION 56)

Main landed ten more of our PRs as our own commits with the maintainer's additions
(colbymchenry#2923 ← colbymchenry#1936, colbymchenry#2924 ← colbymchenry#1951, colbymchenry#2925 ← colbymchenry#1939, colbymchenry#2930 ← colbymchenry#2921, colbymchenry#2940 ← colbymchenry#1736,
colbymchenry#2942 ← colbymchenry#1947, colbymchenry#2950 ← colbymchenry#1952, colbymchenry#2961 ← colbymchenry#1954, colbymchenry#2965 ← colbymchenry#1942, colbymchenry#2972 ← colbymchenry#1933) and the
module-path half of colbymchenry#2308 (colbymchenry#2955). Where the integration carried our version, main's
wins: the enclosing-scope member pick and its tests, the Go and Java receiver typers,
the delegating-call self-loop rule, the duplicated-module copy confidence, the C #if
collapse, the module-path resolution of Rust paths, the trail/path hardening in
utils.ts. Kept beside main: colbymchenry#1875's Verilog hooks and status fields, colbymchenry#2839's
implementations command, colbymchenry#1923, colbymchenry#2922, colbymchenry#2258, colbymchenry#1761, colbymchenry#1583, the bare-call half of
colbymchenry#2308 (rust-use-bindings.ts). CHANGELOG is main's plus the open PRs' bullets.

EXTRACTION_VERSION 56: the C #if group handling (colbymchenry#2940) and the kernel change.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
danusha2345 pushed a commit to danusha2345/codegraph that referenced this pull request Oct 28, 2026
…tests, colbymchenry#1947's cls exemption, locale-tiebreak test)

Our copies of the C #if tests colbymchenry#2940 superseded are dropped (one of them was main's
own test twice); the library-method rule no longer exempts a receiver named `cls`,
as colbymchenry#2942 landed it; ts-this-field-call.test.ts is main's (colbymchenry#2927 landed the same
code-unit tiebreak the integration carried from the earlier PR).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants