Repository navigation
feat(calcite): read ORDER BY without LIMIT lazily on big tables - #570
Merged
Merged
Conversation
dfa1
added a commit
that referenced
this pull request
Oct 19, 2026
…ALUES cap Review of #570 found two flaws in the cap: - the sort rule counted fetch + offset rows, but only the rows after the offset become literals, so LIMIT 100 OFFSET 900 over a wide table fell back to Calcite's slow sort though it fits (it was pushed down before the cap); - the grouped rule checked the cap after groupedFold had scanned the whole file, so 2,000 to 10,000 groups paid for two scans. groupedFold now takes the limit and gives up as soon as it is exceeded. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
dfa1
force-pushed
the
feat/lazy-sort-scan
branch
from
October 19, 2026 04:57
5152554 to
2aa6184
Compare
dfa1
added a commit
that referenced
this pull request
Oct 19, 2026
…ALUES cap Review of #570 found two flaws in the cap: - the sort rule counted fetch + offset rows, but only the rows after the offset become literals, so LIMIT 100 OFFSET 900 over a wide table fell back to Calcite's slow sort though it fits (it was pushed down before the cap); - the grouped rule checked the cap after groupedFold had scanned the whole file, so 2,000 to 10,000 groups paid for two scans. groupedFold now takes the limit and gives up as soon as it is exceeded. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Calcite's own sort buffers every row of the table before it returns the first. Clicking a column header in an IDE grid sends exactly that (SELECT t.* ... ORDER BY col), and on a 100M-row table it ran the JDBC process out of memory. Rewrite a limitless sort over a table of more than 100,000 rows to a scan of a derived, already-ordered table (VortexOrderedTable). It returns the first 1,024 rows from the existing bounded top-N pass and, when the client reads past them, repeats the pass for twice as many and skips what was returned. A grid page costs one pass and holds one page; a client that reads everything holds what Calcite's sort would. The table publishes its collation so the planner drops the sort the query asked for, and the planner's row estimate for the sort input so a filter does not make the plain sort look cheaper. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- count the rows handed out when the cursor advances, not when a column is read: a client that only steps through rows got earlier rows again on the next pass; - grow each pass eightfold and say what a pass costs: every pass reads the whole table, so a client reading to the end of 100M rows pays seven, not two; - drop the previous batch before the next pass runs; - make topN report through the table's chunksScannedLastQuery counter; - trim the CHANGELOG entry to one sentence. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
dfa1
force-pushed
the
feat/lazy-sort-scan
branch
from
October 19, 2026 05:04
2aa6184 to
d1045e6
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #569 (it touches the same rule); retarget to
mainonce that merges.Problem
Clicking a column header in an IDE data grid sends
SELECT t.* FROM t ORDER BY colwith noLIMIT. The plan isEnumerableSortover the full scan, which buffers every row as a boxedObject[]before the first one is returned. On a 100M-row table that ran IntelliJ's JDBC process out of memory ("Database client process needs more memory"), for every column.Change
A limitless
ORDER BYover a table of more than 100,000 rows is rewritten to a scan of a derived, already-ordered table (VortexOrderedTable):VortexTable#topN);A grid page costs one pass and holds one page. Every pass reads the whole table (the heap bounds memory, not I/O), and each later pass is eight times longer, so a client that reads a 100M-row table to the end pays seven full scans and holds what Calcite's sort would. Smaller tables keep Calcite's sort.
Two planner details that were needed: the derived table publishes its collation (otherwise the planner adds the sort back), and its row estimate is the planner's estimate for the sort input (otherwise a filtered plain sort looks cheaper).
Measured
100M-row, 9-column table, JVM with
-Xmx512m, first rows ofSELECT t.* ... ORDER BY:eventType DESC4.2 s,ISIN2.9 s,created_at DESC2.7 s,eventIdentifier DESC(long unique strings) 28 s. All failed withOutOfMemoryErrorbefore.Tests
LazySortScanTest: the plan has no sort for a big table and keeps it for a small one; for 7ORDER BYshapes (ties, nulls, nulls first/last, descending, two keys) reading every row, across several batch doublings, equals a stable sort; reading only a page returns the top rows; a filter plus projection takes the same path.No vortex-jni cross-check: this changes how the Java adapter plans a query, nothing is read or written differently.
🤖 Generated with Claude Code