Repository navigation
Pairing check warns and records DR6-only losses; empty tiles produce empty products - #981
Open
cailmdaley wants to merge 4 commits into
Open
cailmdaley wants to merge 4 commits into
cailmdaley wants to merge 4 commits into
Conversation
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
External DR6-only pairing losses now warn and record instead of rejecting a tile. Zero-object tiles also have a valid workflow partition, so the scheduled stages can publish empty products.
Sparse edge tiles 058.220 and 207.238 still exceeded the external pairing budget after #975.
The SExtractor-side wrong-pixels check remains a hard failure.
External-only objects never gain a shape.
tile_make_cat.jsoncarriescatalogue_match:n_sextractor,n_external,n_paired,n_external_only,n_dropped,external_paired_fraction, andexternal_warning_budget.Join the manifest's
unit, converted to float, to final-catalogueTILE_ID.max(20, 0.02 N + 50), without the 50 when no pairs exist.The external allowance argument/config key is removed; the threshold no longer controls acceptance.
The workflow retains its configured chunk count and gives every empty chunk the bounded interval
(2, 1), avoiding ngmix's unbounded upper-limit sentinel.Ordinary chunk, merge, completeness, and publication paths then receive all expected empty files.
Nonzero partitions and catalogue measurements are unchanged.
A before/after probe found byte-identical partition documents for 512 nonzero cases.
Tests at
c94fbd06: 1153 not-slow and 51 workflow tests pass on a clean checkout in the campaign SIF, inside a four-core Slurm allocation.The workflow regression resolves the production DAG, executes its partition writer and all 16 chunk readers, and feeds their bounds through the real module-to-publication path.
Restoring zero-object rejection or the unbounded empty range each makes that workflow regression fail.
The matching, durable-record, SExtractor-failure, and empty-module regressions remain covered.
A wholly empty batch still fails the separate external checker's science-selection requirement; this PR does not change that gate policy.
Refs #974.
— OpenAI coding agent on behalf of Cail