Skip to content

Pairing check warns and records DR6-only losses; empty tiles produce empty products - #981

Open
cailmdaley wants to merge 4 commits into
developfrom
fix/match-guard-warn
Open

cailmdaley wants to merge 4 commits into
developfrom
fix/match-guard-warn

Conversation

@cailmdaley

@cailmdaley cailmdaley commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

External DR6-only pairing losses now warn and record instead of rejecting a tile. Zero-object tiles also have a valid workflow partition, so the scheduled stages can publish empty products.

Sparse edge tiles 058.220 and 207.238 still exceeded the external pairing budget after #975.
The SExtractor-side wrong-pixels check remains a hard failure.
External-only objects never gain a shape.

  • Record for downstream cuts: every newly joined tile's published tile_make_cat.json carries catalogue_match: n_sextractor, n_external, n_paired, n_external_only, n_dropped, external_paired_fraction, and external_warning_budget.
    Join the manifest's unit, converted to float, to final-catalogue TILE_ID.
  • Warning threshold: retain max(20, 0.02 N + 50), without the 50 when no pairs exist.
    The external allowance argument/config key is removed; the threshold no longer controls acceptance.
  • Empty products: explicit empty stamp shapes, empty EPOCH handling, and safe make-cat reductions support zero-row catalogues.
    The workflow retains its configured chunk count and gives every empty chunk the bounded interval (2, 1), avoiding ngmix's unbounded upper-limit sentinel.
    Ordinary chunk, merge, completeness, and publication paths then receive all expected empty files.

Nonzero partitions and catalogue measurements are unchanged.
A before/after probe found byte-identical partition documents for 512 nonzero cases.

Tests at c94fbd06: 1153 not-slow and 51 workflow tests pass on a clean checkout in the campaign SIF, inside a four-core Slurm allocation.
The workflow regression resolves the production DAG, executes its partition writer and all 16 chunk readers, and feeds their bounds through the real module-to-publication path.
Restoring zero-object rejection or the unbounded empty range each makes that workflow regression fail.
The matching, durable-record, SExtractor-failure, and empty-module regressions remain covered.

A wholly empty batch still fails the separate external checker's science-selection requirement; this PR does not change that gate policy.

Refs #974.

— OpenAI coding agent on behalf of Cail

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant