[dougqh]

What Does This Do

Optimizations on top of the dense tag store (#13004), mostly for that store, which this PR also turns on by default (trace.experimental.dense.tags.enabled=false switches back to the hash store). With the flag off, spans allocate exactly what #13004's do.

• Presize each span's dense store from a per-operation sizing hint. A port of Store known span tags densely, behind an experimental flag (phase 3b - dense tag storage) #12856's SizingHint / SizingHintTable: a bounded, racy table of hints, keyed by operation name, with separate lanes for entry spans and child spans.

  • Where the hint lives: on the span's DenseIdStore, not on DDSpanContext, so flag-off spans carry no extra field.

  • When spans record their size:

    • a published span, when it's serialized, with every tag present;
    • a span in a dropped trace, in CoreTracer.write, after the early post-processors and the root's sampling decision have added their tags.

    Recording at span finish would miss those late tags, and every later span of the operation would grow its store to twice the hint.

• One reference store per dense write path. store and getAndPut resolve the slot index first, then store the value once. On JDK 17 and earlier, C2 expands each reference-store site's G1 write barriers inline, at about 250 B apiece.
• Seed a SpanPrototype's known tags in bulk. The new method IdStore.seedAbsent appends a frozen, parentless, dense source with two array copies, provided the source's occupancy words share no bit with the target's live, tombstoned or inherited words, checked per level. If they share none, no prototype tag can already be present, so "absent only" holds without a per-tag check. If they share any, it copies nothing and the caller falls back to seeding tag by tag.
  • The plan: whether to use the bulk path is decided once per prototype and cached on it. It records whether the store supports bulk seeding, whether the interceptor needs any known tag other than span.kind, and span.kind's ordinal, which is set directly.
  • Where it applies: both at span construction and in applyOverwriting, the path BaseDecorator.afterStart takes today.
  • Custom prototype tags always go one at a time.
• Sizing hints cached on the prototype. A span that uses its prototype's own operation name takes the hint cached on the prototype and skips the table lookup. A span given any other operation name gets that name's hint from the table, so the cache never shares a hint the table wouldn't.
• singleSpanBuilder(SpanPrototype, …), so the prototype entry points mirror the plain ones (startSpan, buildSpan, singleSpanBuilder).
• Public forEach on HashStore and IdStore, so the prototype paths can walk a prototype's custom store directly.

Motivation

Without sizing, every fresh span grew its dense store on its first tag. C2 then inlined grow() and its two allocations into DenseIdStore.store, which compiled to 2816 B, over InlineSmallCode, so every setTag called store out of line. With the hints converged, grow() is cold:

Fresh spans, dense on, JDK 17	before	after
DenseIdStore.store	2816 B	768 B, inlined again
DDSpanContext.setTag(long, int)	3648 B	1792 B
DDSpanContext.setTag(long, CharSequence)	3328 B	1408 B

The id-keyed setters now compile under the 2500 B limit with the store inlined, which C2 needs before it can coarsen the locks in a decorator's run of setTag calls.

Measurements

Measured before the restack. The numbers below were taken against #12918's tip. #12918 has since been split into #13006, #13007, #13003 and #13004. That stack differs mainly in flag-off layout: the mask words moved into DenseIdStore, and the app-thread view is gone. Fresh numbers against #13004 will follow.

Whole-trace allocation (TraceShapeBenchmark, from #12992). Each op builds one web-request-shaped trace: a server root, five jdbc prepared-statement children and a @Trace span, all finished, with ports and status outside Integer's box cache. Bytes per trace (7 spans), JDK 17, -prof gc, single-threaded, error under ±1 B:

arm	master	#12918, dense on	this PR, dense on (default)
legacy: every tag by name	8152	5904	5720 (−30%)
normal: the calls today's decorators make	7480	5912	5728 (−23%)
newApi: prototype at construction, tags by id	7504	6192	5728 (−24%)

• On the decorators' real path, a trace allocates 23% less than on master. The sizing hints take 184 B per trace off Split TagMap into id and name stores, owned directly by the span #12918's dense store, and construction-time seeding stops costing extra (newApi −464 B).
• The benchmark's writer feeds the hints. They learn when a published trace is serialized, or when a dropped trace is written, and a benchmark writer that just drops traces does neither: the hints then stay at their initial size, and every span is presized small and then grows. This PR's copy of the benchmark records each span's size before dropping the trace, as serialization would.
• Application thread only. Serialization and client-side stats aren't included, so end-to-end percentages will be lower.

Earlier measurements. The numbers below are from SpanCreationBenchmark before the above was found. It also drops traces, so the hints there may not have converged; treat its byte figures as superseded by the whole-trace table. The throughput comparisons between entry points still hold, since every arm ran under the same condition.

SpanCreationBenchmark on an M1 Max with Zulu 17, -Xms2g -Xmx2g, 4 forks, 5 × 5 s iterations, -prof gc. Experiment jars carried a LongAdder fix for SpanMetricsImpl's counters, which isn't part of this PR; without it, counter contention dominates at 8 threads.

Bytes per span, dense on: jdbc-shaped spans drop by −130 to −196 B (about 12%) at 1 to 8 threads. Web-shaped spans barely change, because they already fit their first allocation. Flag off, bytes are identical to #12918.

Throughput, dense on, vs. #12918: flat. Single-threaded it's within ±1%, except bare spans, which are about 4% slower from the per-span hint lookup. With the LongAdder fix, this branch is ahead in 13 of 16 cells at 1 to 8 threads (+0.5% to +4.5%), but only one cell clears the error bars. Scaling to 8 threads is limited by per-trace costs outside the tag store, so the byte saving doesn't show up as throughput yet.

Prototype entry points (jdbc, dense on, with the LongAdder fix, ops/µs):

entry point	1 thread	8 threads
direct setTags	2.16 ± 0.04	7.43 ± 0.34
startSpan + applyOverwriting (decorators today)	2.27 ± 0.02	7.72 ± 0.14
buildSpan(prototype)	2.36 ± 0.02	7.72 ± 0.20
singleSpanBuilder(prototype)	2.29 ± 0.02	7.21 ± 0.24
startSpan(prototype)	2.36 ± 0.02	7.73 ± 0.16

• Construction vs. after-start stamping: building the span from the prototype beats stamping it after start by about 4% single-threaded. That's two lock acquisitions fewer and no hint lookup. At 8 threads the two are even.
• The bulk seed itself is within noise, because prototypes carry only two or three known tags today. It pays off as prototypes take on more constants.
• singleSpanBuilder(prototype) measures slower than buildSpan(prototype), whose builder escape analysis already removes. It's kept for API symmetry.

Additional Notes

• Stacked on Add a dense known-tag store behind an experimental flag (dense store - part 8) #13004, and now a single commit: the PR's history was mostly merges of Split TagMap into id and name stores, owned directly by the span #12918, which has been split up and closed.
• Reviews: /techdebt and /perf-review ran over these commits, and their findings are addressed. One was a real default-path regression: an extra lock acquisition in applyOverwriting with the flag off, which now has a test guarding it. One remains open: the hint only ever grows, so one unusually large span presizes every later span of its operation. That needs a benchmark with mixed span sizes before deciding on decay.
• Tests:
  • SizingHintTable{,HashOnly}ForkedTest;
  • seedAbsent cases in TagStoresTest;
  • prototype construction, overwrite and custom-tag cases on both stores: SpanPrototypeConstructionDenseForkedTest reruns the suite with the dense store on and asserts that it is.
• Follow-ups, kept separate on purpose:
  • SpanMetricsImpl counters to LongAdder: +28% at 8 threads in these runs, a separate small PR off master;
  • making read-through cheaper (it keeps spans small, so the aim is to shrink its per-span bookkeeping, not remove it);
  • cheap span snapshots, and moving post-processing off the application thread.

Where this is going

This PR is one step in replacing the span's single TagMap with storage that knows each tag's kind. The goal is to cut what every span allocates on the application thread, without changing behavior.

The end state

• A tag is classified once, at the boundary. A known tag becomes an id (from the tag registry, Resolve direction-scoped OpenTelemetry names at the OpenTelemetry boundary (otlp - tag registry - phase 3) #12731), and a custom tag stays a name. Instrumentation sets known tags by id (Route intercepted tags by id instead of by name (otlp - tag registry - phase 3) #12817), so the JIT can constant-fold the routing.
• The span owns its tag stores directly. Known tags are kept by id and custom tags by name. TagMap becomes a thin view for code outside the span.
• Known tags live in a dense store by default. Parallel arrays of serials and values replace an Entry object per tag, and each span's store is presized from a per-operation hint. With the flag on, a web-request-shaped trace allocates about 30% less on the application thread than on master.
• Core reads and writes a span's stored tags through SpanStorage, not a TagMap. That covers post-processors now and serializers later, so no TagMap view gets built for them. SpanStorage has its own verbs, load and store, apart from TagMap's get/set and the span's intercepted getTag/setTag, so DDSpanContext can offer both. It starts with four methods and grows id, typed-load and primitive-store variants. Later, an intercepted counterpart (the setTag vocabulary) becomes the type setTagsFrom, TagContributor and TagExtractor take, with one lock per batch.

Later, not in this stack: cheaper read-through to parent spans, trace-level tags in a per-trace holder (TraceSegment), and primitive tag values without boxing.

The stack. This replaces #12918, split into pieces that can each be reviewed, and measured, on their own.

PR	Step	Allocation per 7-span trace (flag off unless noted)
#12999	A null tag value removes the tag in TagMap. This is the stack's only behavior change, kept separate on purpose.	unchanged
#12992	Bucket known tags by slot in TagMap, and add TraceShapeBenchmark	−96 B vs master
#13001	SpanStorage for the eager post-processors	unchanged
#13002	Extract TagMap's bucket mechanics into HashSupport (a pure move)	unchanged
#13006	Add the id and name tag stores, not yet used	unchanged
#13007	Make TagMap a facade over the two stores	unchanged
#13003	Have the span own its tag stores directly	+104 B (+1.3%)
#13004	Dense id store behind trace.experimental.dense.tags.enabled (off by default)	flag on: −30.6%
#12926	Turn the dense store on by default, add sizing hints and bulk prototype seeding	flag on: about 3% less again

Every PR after #12999 is behavior-preserving.

The pieces' allocation figures come from a deterministic per-thread probe (ThreadMXBean allocated bytes) that builds the same 7-span trace as TraceShapeBenchmark. The #12992 and #12926 figures come from TraceShapeBenchmark itself.

🤖 Generated with Claude Code

[datadog-official]

🎯 Code Coverage (details)
• Patch Coverage: 76.22%
• Overall Coverage: 60.17% (+0.01%)

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 1877eec | Docs | Give us feedback!}

[dd-octo-sts]

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite	Status
Startup	🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	14.01 s	13.98 s	[-0.4%; +0.8%] (no difference)
startup:insecure-bank:tracing:Agent	12.93 s	12.99 s	[-1.2%; +0.2%] (no difference)
startup:petclinic:appsec:Agent	17.26 s	16.95 s	[+0.8%; +3.0%] (maybe worse)
startup:petclinic:iast:Agent	16.94 s	17.07 s	[-1.7%; +0.2%] (no difference)
startup:petclinic:profiling:Agent	16.67 s	16.93 s	[-2.5%; -0.6%] (maybe better)
startup:petclinic:sca:Agent	17.03 s	16.98 s	[-0.6%; +1.3%] (no difference)
startup:petclinic:tracing:Agent	16.11 s	16.19 s	[-1.3%; +0.2%] (no difference)

Commit: 1877eec3 · CI Pipeline · Benchmarking Platform UI

---

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.