Skip to content

fix(core): compile analysis workloads eagerly with sparkplug - #104

Open
not-matthias wants to merge 1 commit into
mainfrom
cod-3936-add-always-sparkplug-to-codspeed-node-analysis-mode-v8-flags
Open

not-matthias wants to merge 1 commit into
mainfrom
cod-3936-add-always-sparkplug-to-codspeed-node-analysis-mode-v8-flags

Conversation

@not-matthias

Copy link
Copy Markdown
Member

Add --always-sparkplug to analysis-mode V8 flags so eligible functions compile directly to baseline machine code instead of depending on tiering and shared compilation batches. This reduces sensitivity to unrelated benchmark setup changing when compilation happens.

Simulation (including the instrumentation alias) and memory mode are affected. TurboFan and Maglev remain disabled; walltime is unchanged. Existing measurements will shift baseline, and inline-cache feedback remains shared.

Verified on Node 24.21.0 using the actual generated flags: a first-call function has BASELINE code in all three analysis-mode aliases. Walltime and disabled mode do not receive the flag. Core typecheck, lint, and formatting passed; the core unit-test command found no tests. The full build was blocked by missing native instrument-hook headers (hooks/includes/core.h). Real-runner variance measurements are still needed before merging; this PR does not claim to reproduce the reported benchmark improvement.

Avoid dependence on Sparkplug tiering and compilation batches during analysis measurements. Keep optimizing compilers disabled and walltime unchanged. Document the baseline shift and remaining inline-cache sensitivity.
@greptile-apps

greptile-apps Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

RetriggerView in GreptileConfidence Score: 4/5

[Medium impact] The change affects CodSpeed’s analysis-mode benchmark runs.

The change appears safe to merge, with non-blocking follow-ups for consistent runner flags and regression tests.

Fix All in Claude CodeFindings

  1. P2 Runner compilation settings diverge ▶
  2. P2 New flag lacks direct tests ▶
Fix with agent prompt
### Issue 1
packages/core/src/introspection.ts:23
Adding `--always-sparkplug` here leaves the companion `codspeed exec` wrapper using the old analysis flags. Its [flag list](https://github.com/codspeedhq/codspeed/blob/HEAD/crates/exec-harness/src/runtime_env/node/node.sh#L47-L73) mirrors `getV8Flags()` but does not read it. Update that wrapper alongside this change so `codspeed exec` and plugin-driven runs use the same compilation settings.

### Issue 2
packages/core/src/introspection.ts:23
Existing tests compare configured flags with another call to `getV8Flags()`, so they would still pass if this line disappeared or moved into the wrong mode. Add focused assertions that simulation, instrumentation, and memory include `--always-sparkplug`, while walltime and disabled mode exclude it.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

Adds --always-sparkplug to analysis runs and documents the changed measurement baseline.

  • Analysis runs compile eligible functions to baseline code sooner.

not-matthias explicitly identifies the changed measurement baseline and shared feedback as expected limitations. not-matthias also states that real-runner variance measurements remain needed before merging.

Reviews (1) · Last reviewed commit: "fix(core): compile analysis workloads ea..." · Reviewed by Greptile

"--hash-seed=1",
"--random-seed=1",
"--no-opt",
"--always-sparkplug",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Runner compilation settings diverge

Adding --always-sparkplug here leaves the companion codspeed exec wrapper using the old analysis flags. Its flag list mirrors getV8Flags() but does not read it. Update that wrapper alongside this change so codspeed exec and plugin-driven runs use the same compilation settings.

Prompt To Fix With AI
This is a comment left during a code review.
Path: packages/core/src/introspection.ts
Line: 23

Comment:
**Runner compilation settings diverge**

Adding `--always-sparkplug` here leaves the companion `codspeed exec` wrapper using the old analysis flags. Its [flag list](https://github.com/codspeedhq/codspeed/blob/HEAD/crates/exec-harness/src/runtime_env/node/node.sh#L47-L73) mirrors `getV8Flags()` but does not read it. Update that wrapper alongside this change so `codspeed exec` and plugin-driven runs use the same compilation settings.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Fix in Claude Code Fix in Codex

"--hash-seed=1",
"--random-seed=1",
"--no-opt",
"--always-sparkplug",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 New flag lacks direct tests

Existing tests compare configured flags with another call to getV8Flags(), so they would still pass if this line disappeared or moved into the wrong mode. Add focused assertions that simulation, instrumentation, and memory include --always-sparkplug, while walltime and disabled mode exclude it.

Prompt To Fix With AI
This is a comment left during a code review.
Path: packages/core/src/introspection.ts
Line: 23

Comment:
**New flag lacks direct tests**

Existing tests compare configured flags with another call to `getV8Flags()`, so they would still pass if this line disappeared or moved into the wrong mode. Add focused assertions that simulation, instrumentation, and memory include `--always-sparkplug`, while walltime and disabled mode exclude it.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Fix in Claude Code Fix in Codex

@codspeed

codspeed Bot commented Oct 23, 2026

Copy link
Copy Markdown

Merging this PR will regress 44 benchmarks

3 warnings, 1 note

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚠️ 13 benchmarks are dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

⚠️ Base and head ran on different runner types

Some benchmarks with significant performance changes were compared across different runner types, so the changes might not come from the code. Run base and head on the same runner type, or use CodSpeed Macro Runners for more stable results.

Learn more about Macro Runners

Open the report in CodSpeed to investigate

ℹ️ 16 benchmarks have no flamegraph

The flamegraph wasn't generated: most of the time was spent in runtime internals, which would leave nothing useful to analyze.

See affected benchmarks

⚡ 74 improved benchmarks
❌ 44 regressed benchmarks
✅ 146 untouched benchmarks
⏩ 56 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
❌ CPU Simulation switch 1 205.1 µs 590.4 µs -65.26%
❌ CPU Simulation switch 1 244.6 µs 657.1 µs -62.78%
❌ CPU Simulation recursive fibo 10 300.1 µs 685.3 µs -56.22%
❌ Memory test_iterative_fibo_100 656 B 1,312 B -50%
❌ Memory test_iterative_fibo_10 656 B 1,312 B -50%
❌ Memory test_recursive_cached_fibo_20 656 B 1,312 B -50%
❌ Memory test_recursive_fibo_20 656 B 1,312 B -50%
❌ Memory test_recursive_cached_fibo_20 656 B 1,312 B -50%
❌ Memory test sync baz 10 656 B 1,312 B -50%
❌ Memory switch 2 656 B 1,312 B -50%
❌ Memory test_recursive_cached_fibo_30 656 B 1,312 B -50%
❌ Memory test_recursive_fibo_20 656 B 1,312 B -50%
❌ Memory test_iterative_fibo_100 656 B 1,312 B -50%
❌ Memory test_iterative_fibo_10 656 B 1,312 B -50%
❌ Memory test sync baz 100 656 B 1,312 B -50%
❌ Memory test_recursive_cached_fibo_30 656 B 1,312 B -50%
❌ Memory switch 2 656 B 1,312 B -50%
❌ Memory test sync baz 100 656 B 1,312 B -50%
❌ Memory recursive fibo 15 with hooks 656 B 1,312 B -50%
❌ Memory recursive fibo 15 with hooks 656 B 1,312 B -50%
... ... ... ... ... ...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing cod-3936-add-always-sparkplug-to-codspeed-node-analysis-mode-v8-flags (8dcd85b) with main (d15b97d)

Open in CodSpeed

Footnotes

  1. 56 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant