Photoz catalogue handling by martinkilbinger · Pull Request #409 · CosmoStat/sp_validation

martinkilbinger · 2026-10-08T14:42:34Z

This PR handles the reading of GaaP photometry and photo-z catalogues, and their merging with ShapePipe catalogues. The code corresponds to v1.X formats, not yet v2.0 compatible. It is used to create the joint ShapePipe+ PhotoPipe v1.6.y catalogues for current tomographic studies.

It includes the special handling of the two z-band masks.

Merge 6ec03ae committed scripts/fill_photoz_bands.py and scripts/check_filled_fields.py with their conflict markers still in place, so neither file could be imported or run. Both were added independently on the two sides, which is why git conflicted the whole file. Upstream's version is a superset: it adds the per-tile row-order check (check_tile_row_order, n_check_rows, check_tol_arcsec, FITS_RADEC_CANDIDATES) and the Z_ML key, and differs otherwise only in formatting. Taken as-is, with the one local change upstream lacks re-applied: reading TILE_ID with dset.fields() so the scan chunk holds one column instead of all of them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013NzWDtdQbsK1VhvYbDnmTo

Pushed by the lint gate.

github-actions · 2026-10-08T14:42:50Z

✅ ruff is clean — nothing to fix here.

cailmdaley

Nice progress, Martin. The two-band z coverage logic (measured_z & covered_z) | (measured_z2 & covered_z2) looks right. A few integration snags turned up while tracing the v1.X path, flagged inline; none of them are a verdict on the approach:

the PSF-size export column (one-word fix)
binned_mask needs output_path for the calibration script
footprint_only mode in masking.py can't evaluate the new composite cut yet, and that needs a decision about what "footprint" means there

One pointer for later, not for this PR: v2 support probably lives mostly in the enrichment I/O. fill_photoz_bands.py currently pairs PhotoPipe and ShapePipe rows by their order within each tile, so a reliable row-matching rule is the main thing to settle.

— Claude on behalf of Cail

cailmdaley · 2026-10-10T10:30:19Z

    "MAGERR_AUTO",
    "NGMIX_T_NOSHEAR",
-    "NGMIX_T_PSF_RECONV_NOSHEAR",
+    "NGMIX_Tpsf_NOSHEAR",


read_cat() already maps v1 catalogues onto the canonical names (through grammar), and it hides NGMIX_Tpsf_NOSHEAR in the process. So this lookup raises KeyError even on v1.6 inputs. The canonical column also carries the adapter's NGMIX_Tpsf_1P correction, so it's the one we want anyway:

Suggested change

"NGMIX_Tpsf_NOSHEAR",

"NGMIX_T_PSF_RECONV_NOSHEAR",

(The first_present() fallback in fill_cat_gal() then isn't needed for this.)

cailmdaley · 2026-10-10T10:30:19Z

+  good: False
+
+  # Output file path base
+  output_base: "footprint_binned" 


With this section present, calibrate_comprehensive_cat.py reads cbm["output_path"] (line 170), so it'll KeyError before writing anything. output_base is what create_binned_mask_comprehensive.py reads, so keeping both is the smallest fix:

Suggested change

output_base: "footprint_binned"

output_base: "footprint_binned"

output_path: "footprint_binned_8192.fits"

cailmdaley · 2026-10-10T10:30:19Z

+    mag_col: MAG_GAAP_0p7_z
+    mag_col2: MAG_GAAP_0p7_z2
+    label: "z-band imaging"
+    kind: covered_2bands


MASK_256_z is in SPATIAL_CUTS in scripts/masking.py, so with footprint_only=True this cut gets sent to apply_condition(), which only handles single-column cuts. The result is ValueError: Unknown mask condition kind: 'covered_2bands'.

It's worth deciding what footprint-only should mean for this cut. One option reproduces the catalogue selection, magnitude checks included. The other is purely geometric: the union of the two imaging footprints, with bad = bad_z & bad_z2. demo_create_footprint_mask.py has the same gap, since it ignores col_name2.

martinkilbinger · 2026-10-10T12:04:05Z

Nice progress, Martin.

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

…tion, test the guarantees (#420) * Refuse blinded tomography at direct pseudo-Cl producers * Check catalogue blind stamps before consuming SACC parts * Resolve physical shear paths for same-file blind declarations * Test blind determinism, envelope, multi-tracer shifts and secret-free outputs * Validate blind stamps in SACC loads, including paper-workflow readers * Observe actual CCL tracer pairs in the multi-tracer blind test * Reject hidden cosmology values in saved outputs and captured logs * Keep the blind gate to the cosmo_val path; Paper II readers stay as they were papers/bmodes runs only on public (blind: none) catalogues, so threading a blind through its readers and adding Snakemake adapters is out of scope here. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ruff autofix (format + safe lint fixes) Pushed by the lint gate. * Run CI on the lint autofix head The lint gate's autofix commit is pushed with GITHUB_TOKEN, which triggers no workflows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ay8NWFocUojrbRVfYoC5nH --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

* implement cosmocov * ruff check * ruff format * ruff format * edits after review * edits after review * updated path files * update snakemake files with tomographic option * ruff autofix (format + safe lint fixes) Pushed by the lint gate. * OneCovariance inputs: per-component sigma_e and the footprint FITS mask cat_config's cov_th.sigma_e sums both ellipticity components; OneCovariance's ellipticity_dispersion is per component, so divide by sqrt(2) at the boundary. mask_file_lensing must be a healpix FITS map: pass the catalogue's mask: key instead of the ASCII mask-Cl text, which OneCovariance could not read. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NbbToHtxp4TtFr3KEEiNkZ --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Cail Daley <cail.daley@cea.fr> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

#423) * Preserve logical-mask booleans when writing HDF5 catalogues * ruff autofix (format + safe lint fixes) Pushed by the lint gate. --------- Co-authored-by: Cail Daley <cdaley@l1.nibi.sharcnet> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

Default external-mask cut excludes the star halos

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

The catalogue reader presents v1 NGMIX_Tpsf_NOSHEAR under its v2 name, so requesting the v1 name fails in get_col after calibration. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

martinkilbinger and others added 19 commits February 25, 2026 16:23

additive bias calculation for paper updated to v1.4.6.3

c90ae4f

added config

392461e

Merge remote-tracking branch 'upstream/develop' into photoz

8cfe153

added fill_photoz script

0ec3e3f

fill photoz bands fixes

7f29a8b

adding mag errors to fill_photoz

7e88b05

added more flags to fill_photoz

9798047

added 0p7 and 1p0 aperture magnitudes to PhotoPipe + SP output

7cdcfdf

Fixed fill photoz script with new MP_NAME type

bfae665

Merge branch 'develop' of github.com:CosmoStat/sp_validation into photoz

cd9a13e

comprehensive cat backwards compatible, maybe not use for merge

82b7b15

Using all photometric bands and masks

055a1d5

Merge remote-tracking branch 'upstream/develop' into photoz

6ec03ae

added forgotten PhotoPipe columns for calibrated cat

dfe6ee0

Merge remote-tracking branch 'upstream/develop' into photoz

d80e3ef

fixed v1.X.6 pp mask

eaeb5e9

Merge remote-tracking branch 'upstream/develop' into photoz

a3cdbab

ruff autofix (format + safe lint fixes)

b2208d4

Pushed by the lint gate.

martinkilbinger requested a review from cailmdaley October 8, 2026 14:45

martinkilbinger self-assigned this Oct 8, 2026

cailmdaley reviewed Oct 10, 2026

View reviewed changes

renovate Bot and others added 6 commits October 12, 2026 04:39

Update actions/upload-artifact digest to cf430e0 (#411)

fcc1295

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

Update all non-major dependencies (#412)

3f1827f

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

Update all non-major dependencies (#414)

e052df5

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

Lock file maintenance (#415)

4954b97

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

cailmdaley and others added 7 commits October 25, 2026 22:33

Exclude star halos from default mask cut

b35383f

Merge pull request #424 from CosmoStat/fix/default-mask-no-halos

058da2e

Default external-mask cut excludes the star halos

Update docker/login-action digest to 81c6f93 (#426)

158c6d3

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

added v1.X.14 config

4705ed8

Merge remote-tracking branch 'upstream/photoz' into v.1.6.14

be3c91f

Use v2 name NGMIX_T_PSF_RECONV_NOSHEAR for PSF size column

44f502c

The catalogue reader presents v1 NGMIX_Tpsf_NOSHEAR under its v2 name, so requesting the v1 name fails in get_col after calibration. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Navigation Menu

Repository navigation

Photoz catalogue handling - #409

Photoz catalogue handling#409
martinkilbinger wants to merge 32 commits into
developfrom
photoz

martinkilbinger commented Oct 8, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 8, 2026 •

edited

Loading

Uh oh!

cailmdaley left a comment

Uh oh!

cailmdaley Oct 10, 2026

Uh oh!

cailmdaley Oct 10, 2026

Uh oh!

cailmdaley Oct 10, 2026

Uh oh!

martinkilbinger commented Oct 10, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	output_base: "footprint_binned"
	output_base: "footprint_binned"
	output_path: "footprint_binned_8192.fits"

Conversation

martinkilbinger commented Oct 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cailmdaley left a comment

Choose a reason for hiding this comment

Uh oh!

cailmdaley Oct 10, 2026

Choose a reason for hiding this comment

Uh oh!

cailmdaley Oct 10, 2026

Choose a reason for hiding this comment

Uh oh!

cailmdaley Oct 10, 2026

Choose a reason for hiding this comment

Uh oh!

martinkilbinger commented Oct 10, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

martinkilbinger commented Oct 8, 2026 •

edited

Loading

github-actions Bot commented Oct 8, 2026 •

edited

Loading