Repository navigation
fix(calcite): keep push-down VALUES under the JVM's 64 KB method limit - #569
Merged
Merged
Conversation
The sort and grouped-aggregate rules answer with a literal VALUES, which
Calcite compiles into one Java method. Past about 5,000 cells the JVM
refuses it ("Code grows beyond 64 KB") and the query failed outright: a
GROUP BY with 3,000 groups, or ORDER BY ... LIMIT 1000 over 9 columns, as an
IDE grid issues. Cap the push-down at 4,000 cells (rows x columns, measured:
4,500 compile, 6,000 do not) and leave bigger results to Calcite's own scan.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
dfa1
force-pushed
the
fix/values-64kb-limit
branch
from
October 18, 2026 20:23
e223da1 to
c598d9d
Compare
…ALUES cap Review of #570 found two flaws in the cap: - the sort rule counted fetch + offset rows, but only the rows after the offset become literals, so LIMIT 100 OFFSET 900 over a wide table fell back to Calcite's slow sort though it fits (it was pushed down before the cap); - the grouped rule checked the cap after groupedFold had scanned the whole file, so 2,000 to 10,000 groups paid for two scans. groupedFold now takes the limit and gives up as soon as it is exceeded. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Found while trying the
jdbc:vortex:driver (#568) in IntelliJ against a 100M-row file.Bug
The sort (
ORDER BY ... LIMIT) and grouped-aggregate push-downs answer with a literalVALUES. Calcite compiles the wholeVALUESinto one Java method, and the JVM refuses a method over 64 KB of bytecode, so the query failed withCode grows beyond 64 KB. The grouped cap (10,000 groups) and the sort cap (1,000 rows) counted rows only, not columns.group bya string column with 3,000 groups andcount(*)failed; 2,000 groups passed.select * ... order by x limit 1000over 9 columns failed;limit 500passed. IntelliJ's grid sorts withorder by ... limit 500.Fix
A shared budget of 4,000 literal cells (rows x columns). Measured on a mixed-type table: 4,500 cells compile, 6,000 do not. Above the budget the rule leaves the plan to Calcite's own scan/sort, which is slower but correct.
Tests
ValuesCodegenLimitTest: over the cap the query falls back and returns the full, correct result; under the cap it is still pushed down. 4 of its 6 cases fail without the fix.No vortex-jni cross-check: this is Calcite code generation in the Java adapter, nothing is read or written differently.
🤖 Generated with Claude Code