Repository navigation
Conversation
…st() and JSON text (#228) NativeAsyncExecutor.append_samples called .tolist() on every column, walked the timestamps again with int(), and sent the chunk as one JSON text to arcadedb_n_async. A one-dimensional numpy int, float or bool column and an int or datetime64[ns] timestamp array now go through the new arcadedb_n_async_bin: the JSON holds a kind marker per column ("q" int64, "d" float64) and the sample count, the data crosses in an address table of (address, count) longs, the convention of arcadedb_n_insert_colp. The host tag column and anything the text path converts differently (lists, float timestamps, other datetime64 units, 2-D, empty, uint64 past int64, a column of another length) keep the text. Java builds the same long[] and boxed Object[] columns and calls the same appendSamples, so the stored samples are bit-identical (NaN is the canonical NaN of the text). A library without the entry, or ARCADEDB_NATIVE_TS_BIN=0, keeps the text path; primitive=True is the same length check. Tests (native only, run in CI): the lane's shape takes the entry and reads back by count, sums and a window; every dtype is stored as the text path stores it; the entry choice per shape; the Java checks of a wrong table. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Collaborator
Author
|
CI on head The test results will be added here when the dispatch run finishes. The timed A/B (DECISIONS #240, wheel versus wheel) is prepared and not started. |
…test file (#228) CI run 41716062800 failed only these three, all in tests/test_native_timeseries_binary.py, all in the test code: the window of the lane test ended at index 1900 so host_7 had 9 points, not 10 (end now 1999); an int32 array into a DOUBLE column is an int64 column ("q"), the expected kinds listed "d"; datetime64 + datetime64 is not defined in numpy, the datetime64[ms] timestamps are now datetime64 + timedelta64. No source change: every other parity file of the run passed (2036 passed). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ay may take it there The parity leg on windows-x64 py3.14 failed one case (values11): the test expected None for a np.longdouble array, which holds only on platforms where longdouble is 80-bit extended. Where it is float64 the picker is right to take it (tolist reproduces the values). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #228
On the native wheel,
append_samplesturned every column into a list with.tolist(), walked the timestamps again withint(), and sent the chunk as one JSON text. In the untimed probe of the l4 lane shape (100,000-point chunks, numpy columns,primitive=True), glue plus encode was 46.7% of the load wall (numbers in the issue). This PR sends the numeric columns by address.What changes: a new library entry
arcadedb_n_async_bin(handle, op, json, table, n_table)next toarcadedb_n_async. Forappend_samples, a one-dimensional numpy int, float or bool column, and an int ordatetime64[ns]timestamp array, cross in an address table of(address, count)longs (the convention ofarcadedb_n_insert_colp), and the JSON holds only a kind marker per column ("q"int64,"d"float64) and the sample count. The host tag column and every shape the text path converts differently keep the text: lists, float timestamps, otherdatetime64units, 2-D and empty arrays,uint64past the int64 range, a column whose length differs from the timestamps. Python side:codec.sample_arraydecides per column,NativeAsyncExecutor.append_samplesbuilds the table. Java side:NativeExtras.asyncImpl(the old body ofasync, now shared by both entries) builds the samelong[]and boxedObject[]columns the text gave and calls the sameappendSamples.What does not change: the stored samples are bit-identical to the text path (a float32 or float16 array widens to float64 exactly, a bool array is 0 and 1, a NaN is the canonical NaN the text gives, an int array into a DOUBLE column and a float array into a LONG column go through the same
Numberconversions).primitive=Truekeeps its meaning on the native build: the same length check, before the crossing, with the same message. A library without the entry (the loader setslib.TS_BINfalse) orARCADEDB_NATIVE_TS_BIN=0keeps the text path. The JVM wheel is not touched.Tests (
tests/test_native_timeseries_binary.py, native only, collected throughconftest.pylike the other native observers, no skips): the lane's shape takesarcadedb_n_async_binonce per chunk and its text holds no number of a numeric column, and the stored samples are read back by count, sums and a window; a matrix of dtypes (float64 with -0.0, NaN, infinities, 1e300 and 5e-324, float32, float16, int32 into DOUBLE, int64 extremes, uint8, bool, float into LONG) is stored bit-equal to the text path of the same call; timestamp arrays of int32, uint16, uint64,datetime64[ns]and a strided view are stored equal; non-contiguous and big-endian columns are stored as values; the entry choice for lists, float timestamps,datetime64[ms](still aTypeError), a column of another length,primitive=Trueand a library without the entry;sample_arrayfor 24 dtype cases; and the Java checks of a wrong address table (nothing is written). The CI run ids are in the first comment.Not claimed: no speed number. A wheel-versus-wheel timed A/B (DECISIONS #240) runs on the laptop after the CI wheel exists; its verdict goes on this PR. The Java side still boxes each value once, the engine's
TimeSeriesBatchroute is a possible follow-up. This PR and #224 both touchlib.pyandextras.py(a flag, a signature line, an import); the conflicts are textual.🤖 Generated with Claude Code