Repository navigation
fix(bigquery-jdbc): request int64 timestamps so the REST path is lossless - #14890
keshavdandeva wants to merge 3 commits into
Conversation
There was a problem hiding this comment.
Code Review
This pull request implements support for lossless int64-encoded timestamps when picosecond precision is disabled. It configures DataFormatOptions to request int64 timestamps by default, preserves FieldValue wrappers for TIMESTAMP columns to identify these encoded values, and decodes them to epoch microseconds during type conversion and range formatting. The review feedback suggests improving the robustness of the range bound parsing in BigQueryTemporalUtility by wrapping the Long.valueOf conversions in a try-catch block to handle potential NumberFormatExceptions and throw a BigQueryJdbcException.
| || isStruct(field) | ||
| || (field.getType() != null | ||
| && field.getType().getStandardType() == StandardSQLTypeName.RANGE); | ||
| && (field.getType().getStandardType() == StandardSQLTypeName.RANGE |
There was a problem hiding this comment.
Should we move RANGE/TIMESTAMP checks to the unpackRow? (the only place it is used now).
I don't think RANGE/TIMESTAMP fit definition of 'complex' type given that SDK attributes them to primitives, so rather than hijacking what is complex, we'd better to special case it where it matters
| /** Returns int64-encoded TIMESTAMP values as epoch micros, as the Arrow path delivers them. */ | ||
| private static Object unwrap(FieldValue fv, StandardSQLTypeName bqType) | ||
| throws BigQueryJdbcException { | ||
| if (bqType != StandardSQLTypeName.TIMESTAMP |
There was a problem hiding this comment.
nit: I'd reverse the logic, default to fv.getValue() and special-case TIMESTAMP with useInt64
b/577457377
Summary
On the REST path, BigQuery returns
TIMESTAMPvalues as float64 seconds by default. That loses microseconds for dates far from 1970, and9999-12-31 23:59:59.999999 UTCcomes back as year10000. The value is rounded by the server, so the driver can't fix it on the client.Fix
useInt64Timestamp(true), which returns exact epoch microseconds. WithEnableTimestampPicos=1we still requestISO8601_STRING, because BigQuery rejects both together.TIMESTAMPvalues as epoch micros, as the Arrow path already does. This covers top-level, STRUCT/ARRAY and RANGE values.Tests
BigQueryConnectionTest: default connections request int64 timestamps.BigQueryJsonResultSetTest: int64TIMESTAMPandRANGE<TIMESTAMP>values decode exactly.ITTimestampTimeZoneTest: adds9999-12-31 23:59:59.999999 UTCto the existinggetTimestamp/getStringchecks across 4 JVM zones. These checks fail without the fix (+10000-01-01) and pass with it.