Repository navigation
perf(calcite): count COUNT(DISTINCT col) in the scan; top-N ties in scan order - #562
Merged
Merged
Conversation
…te does The top-N heap kept the right rows of a tie (the first ones scanned) but returned them in heap order, while Calcite's sort is stable and returns them in scan order. SQL leaves the order of ties open, but a query should not change its answer depending on whether the push-down fired. Rows now rank by scan position after their keys. The tests missed it because their oracle wrapped the sort key in "+ 0", which Calcite simplifies back to the bare column, so the oracle was pushed down too and compared the heap with itself. The oracle is now CASE WHEN id > -1 THEN key END, and the tests assert it really scans. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WtuVGQ5h5Lk6Zn4JnkxF1K
An ungrouped COUNT(DISTINCT col) was left to Calcite, which hashes a boxed value per row. The aggregate rule now answers it, alongside the other calls it can answer, from VortexTable.distinctCount: a pass over the rows the WHERE keeps that counts integers, temporals, doubles and booleans by their bits in a primitive LongIntMap (for doubles the same equality as Double.equals, which Calcite's DISTINCT uses), dictionary strings by marking the codes a chunk uses and decoding each once, and other strings through a set. Decimals, binary and nested columns, and a grouped or FILTERed COUNT(DISTINCT), stay with Calcite. Raincloud appian-order (4.28M rows), best of 3: count(distinct order_customerId) 194 -> 46 ms Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WtuVGQ5h5Lk6Zn4JnkxF1K
dfa1
force-pushed
the
feat/count-distinct
branch
from
October 18, 2026 11:17
527d459 to
e3d27cb
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #561 (rebased onto main once it merges). Two commits:
ORDER BY ... LIMIT(perf(calcite): answer ORDER BY ... LIMIT from a bounded heap over the scan #557) returned tied rows in heap order; Calcite's sort is stable, so ties now come out in scan order. The tests missed it because their oracle (order by x + 0) is simplified by Calcite back toxand was pushed down too; oracles are nowCASE WHEN id > -1 THEN x ENDand the tests assert they really scan.COUNT(DISTINCT col)is answered byVortexTable.distinctCount: integers, temporals, doubles (same equality asDouble.equals, which Calcite'sDISTINCTuses) and booleans by their bits in a primitiveLongIntMap, dictionary strings by the codes a chunk uses, other strings through a set. Grouped/FILTERed distincts, decimals, binary and nested stay with Calcite.Raincloud appian-order (4.28M rows):
count(distinct order_customerId)194 → 46 ms.Tests
CountDistinctTest: integer with nulls, doubles with -0.0/0.0/NaN, dictionary strings, two at once; each against a scanning oracle.SortPushDownTestand the jni sort cross-check use scanning oracles.count(distinct g)and a filteredcount(distinct n)over a Rust-written file.🤖 Generated with Claude Code
https://claude.ai/code/session_01WtuVGQ5h5Lk6Zn4JnkxF1K