Repository navigation
SQLite: Support signed and decimal numbers in type modifiers - #2557
LucaCappelletti94 wants to merge 5 commits into
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #2557 +/- ##
==========================================
+ Coverage 81.52% 81.56% +0.03%
==========================================
Files 42 42
Lines 34205 34267 +62
Branches 34205 34267 +62
==========================================
+ Hits 27885 27949 +64
Misses 2770 2770
+ Partials 3550 3548 -2 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
00bbf26 to
f5fbdb0
Compare
| let _guard = self.recursion_counter.try_decrease()?; | ||
|
|
||
| let dialect = self.dialect; | ||
| if let Some(custom) = self.maybe_parse(|p| p.parse_type_with_signed_modifier())? { |
There was a problem hiding this comment.
Suggestion on where this lives.
maybe_parse(|p| p.parse_type_with_signed_modifier()) at the top of parse_data_type is effectively a dialect hook: the dialect gets first chance to produce a DataType, and maybe_parse rolls back when it declines. But it's wired through a boolean flag inside the function, which has two side effects:
- every data type in every dialect now goes through
maybe_parse, which formats anexpected_ref("")error string and discards it, just to say "not applicable"; - ~70 lines of SQLite-specific grammar sit in the shared parser instead of in
sqlite.rs.
Could we make it a real hook, the same way parse_prefix / parse_infix / parse_statement work?
// Dialect trait
/// Dialect-specific data type parsing. Returning `None` falls back to the standard parser.
fn parse_data_type(&self, _parser: &mut Parser) -> Option<Result<DataType, ParserError>> {
None
}
// Parser::parse_data_type
if let Some(result) = self.dialect.parse_data_type(self) {
return result.map(|t| (t, false.into()));
}SQLiteDialect then implements parse_data_type and owns the signed-modifier logic, including its own maybe_parse for the rollback. Other dialects pay nothing, supports_signed_type_modifier goes away, and the SQLite code moves next to the other SQLite quirks.
With the logic in one place it may also simplify: parse_object_name plus parse_optional_type_modifiers already produce DataType::Custom, so the dialect impl could reuse them and only add the +/- handling, instead of the lookahead pass and the second parse loop.
There was a problem hiding this comment.
So first of all, thank you for pointing out the discarded error allocation. Now supports_signed_type_modifier() is checked before anything else and replaces maybe_parse with a non-allocating peek, so other dialects pay one flag call.
I kept the flag because supports_* methods are preferably for grammar, and GenericDialect can enable them, which a hook in SQLiteDialect would not allow. parse_optional_type_modifiers rejects signs and has no two-modifier limit, so reusing it would change shared behaviour.
There was a problem hiding this comment.
Thanks for the update.
The location concern still stands, and the new code made it clearer why.
The flag names one case but enables three
| Case | Example | Today's failure |
|---|---|---|
| signed numbers | CHAR(-1), CHAR(+10) |
Expected: literal int, found: - |
| fractional modifiers | VARCHAR(2.5), NUMERIC(10.5) |
Could not parse '2.5' as u64 |
| oversized precision | DECIMAL(99999999999999999999999, 2) |
number too large to fit in target type |
Only the first is new syntax. The other two already tokenize as a single number and fail inside parse_literal_uint, because DataType stores precision as u64. A boolean flag has
to describe that slice in its name. A parse hook does not.
Why I'd still prefer the parse_data_type hook
- The lookahead and custom parse move to
sqlite.rsnext to theparse_statementoverride. The common parser keeps one line. supports_signed_type_modifiergoes away, so there is no naming problem.- Every helper the two functions use is already
pub, so nothing else changes.
Scope note
This path produces DataType::Custom, so INT(10) and INT(-1) get different AST shapes. That is fine for SQLite, which ignores modifier values, but it is not a base for other
dialects. Supporting these modifiers generally means a precision type richer than u64 in DataType, which is a separate AST refactor. Keeping this SQLite-only behind a hook avoids
committing the common parser to Custom before that decision.
There was a problem hiding this comment.
Went for the hook
f5fbdb0 to
5022ece
Compare
5022ece to
939df88
Compare
s5dsn-eqee
left a comment
There was a problem hiding this comment.
Looks great, love it. Just needs a few minor changes, a bit more tests, and a fuzz seed file.
| } | ||
|
|
||
| fn peek_nonstandard_type_modifiers(parser: &Parser) -> bool { | ||
| if !matches!(parser.peek_token_ref().token, Token::Word(_)) |
There was a problem hiding this comment.
SQLite allows multi-word type names before the modifiers (type-name grammar, datatypes), but
the peek only accepts a single word before (, so these still fail while sqlite3 accepts them:
CREATE TABLE t (a DOUBLE PRECISION(-1));
CREATE TABLE t (a CHARACTER VARYING(-1));
Could the peek skip a run of words and join them into the custom name?| let _guard = self.recursion_counter.try_decrease()?; | ||
|
|
||
| let dialect = self.dialect; | ||
| if let Some(data_type) = dialect.parse_data_type(self) { |
There was a problem hiding this comment.
The new hook also runs during the speculative parse_data_type in parse_prefix_inner, so abs(-1) now comes back as DataType::Custom and relies on the Custom arm to rewind.
It works, but nothing pins it. Could you add a regression test?
sqlite().verified_stmt("SELECT abs(-1), round(1, 1.5), char(+1, -2)");There was a problem hiding this comment.
Added the test
| ); | ||
| } | ||
|
|
||
| #[test] |
There was a problem hiding this comment.
Could you extend the test to cover the edges? These all parse and round-trip on the branch, and sqlite3 3.51 accepts
them:
for sql in [
"CREATE TABLE t (a CHAR(0.0))",
"CREATE TABLE t (a CHAR(-0))",
"CREATE TABLE t (a CHAR(1.))",
"CREATE TABLE t (a CHAR(1e3))",
"CREATE TABLE t (a CHAR(+1.5e-3))",
"CREATE TABLE t (a CHAR(+1, +2))",
"CREATE TABLE t (a CHAR(-1) NOT NULL DEFAULT NULL)",
"CREATE TABLE t (a CHAR(1.5), b CHAR(10))",
"CREATE TABLE t (a char(-1))",
r#"CREATE TABLE t (a "CHAR"(-1))"#,
"CREATE TABLE t (a [CHAR](-1))",
"SELECT 1::CHAR(-1)",
] {
sqlite().verified_stmt(sql);
}
sqlite().one_statement_parses_to("CREATE TABLE t (a CHAR ( - 1 ))", "CREATE TABLE t (a CHAR(-1))");
sqlite().one_statement_parses_to("CREATE TABLE t (a CHAR(-1,-2))", "CREATE TABLE t (a CHAR(-1, -2))");
Negative cases: CHAR(1.5,), CHAR(+-1), CHAR(NULL). Also worth asserting that CHAR(10) still yields DataType::Char, not Custom, and that all_dialects_except(|d|
d.is::<SQLiteDialect>()) rejects CHAR(+10).There was a problem hiding this comment.
Plus, AGENTS.md asks for a fuzz seed per fix: fuzz/fuzz_seeds/sqlite_2557_signed_type_modifiers.sql.
There was a problem hiding this comment.
Yeah, will have to add them throughout.
7c629aa to
7c466b5
Compare
This PR lets
SQLiteDialectparse signed, decimal and arbitrarily large numbers as type modifiers on single-word type names, keeping them verbatim. Plain unsigned modifiers such asCHAR(10)produce the same AST as before.SQLite's type name grammar takes a signed number in each modifier.