Skip to content

cosmo_val: pair-count shape noise in the OneCovariance covariance - #432

Open
cailmdaley wants to merge 4 commits into
developfrom
cosmo-val-paircount-noise
Open

cailmdaley wants to merge 4 commits into
developfrom
cosmo-val-paircount-noise

Conversation

@cailmdaley

@cailmdaley cailmdaley commented Oct 26, 2026 •

Copy link
Copy Markdown
Collaborator

Full-area v2 analytic shape noise is 14–19% below measured-pair noise over 12–83′; cosmo_val now uses measured effective pairs by default.
On the fixed 2075.3 deg² B vectors, the noise-only correction changes the fiducial COSEBIs B PTE from 0.43798 to 0.64339.
Closes #431.

Mechanism and convention

For unordered TreeCorr auto pairs, W_b = Σ w_i w_j and Q_b = Σ (w_i w_j)² give V_b = 2 s⁴Q_b/W_b², with per-component dispersion s.
OneCovariance c8755d6 accepts [tabulated inputs files] npair_mm_file and uses ordered counts: supply 2 W_b²/Q_b, making its 4 s⁴/N_ordered equal V_b.
The total two-component RMS is √2 s; mixing conventions gives a factor of four.

A separate cached NN rule reads only positions and weights, counts with w and w², and persists W, Q and nominal geometric centres on each exact covariance grid.
This avoids OneCovariance's unsafe rebinning of effective counts and requires no shape measurement or blind access.
covariance.shape_noise: analytic restores the old uniform-area path; other workflows and implicit tomographic targets retain analytic defaults.
No changes to blinding, signal vectors or other covariance terms.

Full-area B-only evidence

Controlled substitution, not a fresh full-area OneCovariance run: replace only the analytic ξ noise diagonal, then propagate with the original operators and fixed cached B vectors.
The analytic/pair noise ratio over 12–83′ is 0.80879–0.85565 (median 0.83931); it is 0.84051 near 30′ and 0.82209 near 60′.
The measured W reproduces the integration-ξ pair metadata to 1.13×10⁻¹³ maximum relative difference.

Fixed 12–83′ statistic dof Analytic-noise PTE Pair-noise PTE
COSEBIs B, 20 modes 20 0.43798 0.64339
Pure-B ξ₊ 7 0.34732 0.48005
Pure-B ξ₋ 7 0.071519 0.12853
Pure-B combined 14 0.15656 0.29359

The covariance operators reproduce the cached COSEBIs/pure-B covariances to relative error 3.51×10⁻¹⁶ / zero at checked precision; all reported B sub-covariances pass Cholesky validation.
No E-mode χ²/PTE or cosmological inference was computed.
This is not catalogue acceptance or validation of mask-dependent mixed/sample-variance terms; the measured correction includes selection and weight variation as well as boundaries.

Validation and figures

  • Slurm 25030338: 26 passed, 2 site-dependent skips, plus real OneCovariance analytic and tabulated-count noise-only probes matching 4 s⁴/N_ordered to its text precision.
  • Full positions/weights count 25028179: 24 CPUs, 128000 MB, 1h32m41s; B-only evidence 25028544: 4 CPUs, 16 GiB, 58 s, in sp_validation-ddd58fe0.sif with checkout-first PYTHONPATH.
  • Ruff check/format, git diff --check and independent static review pass.
    Derivation, interface and resource/migration notes: docs/pair-count-noise.md.

Analytic / measured-pair shape-noise variance vs θ, for the full v2 area (2075 deg², solid) and the 33.5 deg² pilot (dashed). Below ~1′ the analytic term is too large: more close pairs are measured than a uniform field predicts. Beyond ~7′ the full-area ratio settles at about 0.85 and stays nearly flat to 100′. The pilot instead falls steeply past 30′, the signature of its patch edges. A flat full-area deficit points to mask holes and weight/depth variation rather than survey edges, so the correction does not shrink with area.

noise ratio

B-mode PTEs at the fixed 12–83′ cut, analytic vs pair-count noise. All pass either way at full area. The correction moves each PTE up by roughly 1.3–2×, as expected for a noise term that was ~16% low.

B-mode PTEs

Figures live in docs/validation/pair-noise/ on this branch. The same directory holds report.html, bmode-noise-results.json, noise-per-bin.csv, REPRODUCE.md, scripts and logs.
run/, latest/, production and #430's branch are untouched.

— AI assistant on behalf of Cail.

🤖 Generated with Claude Code

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L5YaFeu6oHriuZyFwNAbPd
@cailmdaley
cailmdaley marked this pull request as ready for review October 28, 2026 18:48
cailmdaley added a commit that referenced this pull request Oct 28, 2026
Resolve covariance.smk against #432: keep #436's per-component pass-through
(sigma_e_convention from #432 is superseded); adapt #432's pair-noise test
and doc to the single convention.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

cosmo_val: pair-count shape noise in the OneCovariance covariance

1 participant