Fuse partition and metrics filtering into manifest entry deserialization #3657
Description
Feature Request / Improvement
When scanning a table with row filter, _open_manifest currently deserializes all live manifest entries via ManifestEntry#fetch_manifest_entry() and then applies partition and metrics filters as a second pass. This materialises the full list of entries before discarding non-matching ones.
The intermediate list and second iteration pass add overhead that scales with total entry count, not matched entry count.
Proposal:
Add ManifestFile#prune_manifest_entry() that applies the partition/metrics evaluation as the entries are deserialized, avoiding the intermediate list allocation for non-matching entries. Update _open_manifest() to use it.
Benchmark results (across multiple tables and filter expression):
| Total Manifest Entries | Entries Matching Filter | Two-pass Time | Fused Time | Improvement |
|---|---|---|---|---|
| 116k | 78 (0.07%) | 9.13s | 7.18s | -21.3% |
| 438k | 1 (~0%) | 35.12s | 27.81s | -20.8% |
| 823k | 4,962 (0.6%) | 54.38s | 41.38s | -23.9% |
| 18k | 269 (1.5%) | 1.23s | 0.90s | -27.0% |
| 8k | 185 (2.2%) | 0.57s | 0.40s | -30.2% |
No behavioral change - correctness validated across all runs.
Activity
SaiNikhil02 commentedon Aug 30, 2026
Hi @zenfenan / maintainers, I’m interested in working on this issue as my first contribution to PyIceberg.
I noticed that PR #3658 was previously opened and is now closed. Is this issue still available to work on, and would you be okay with me picking it up? I’d also be happy to follow any guidance from the previous PR review before implementing the change.
zenfenan commentedon Aug 31, 2026
@SaiNikhil02 Thanks for your message! I bumped it up in the Slack channel for the project maintainers to re-open the PR and review it. I am keen on re-opening the PR and merging it.