Skip to content

Native append_samples: numeric columns cross by address, not as .tolist() and JSON text - #234

Draft
tae898 wants to merge 3 commits into
mainfrom
perf/native-timeseries-binary
Draft

tae898 wants to merge 3 commits into
mainfrom
perf/native-timeseries-binary

Conversation

@tae898

@tae898 tae898 commented Nov 3, 2026

Copy link
Copy Markdown
Collaborator

Closes #228

On the native wheel, append_samples turned every column into a list with .tolist(), walked the timestamps again with int(), and sent the chunk as one JSON text. In the untimed probe of the l4 lane shape (100,000-point chunks, numpy columns, primitive=True), glue plus encode was 46.7% of the load wall (numbers in the issue). This PR sends the numeric columns by address.

What changes: a new library entry arcadedb_n_async_bin(handle, op, json, table, n_table) next to arcadedb_n_async. For append_samples, a one-dimensional numpy int, float or bool column, and an int or datetime64[ns] timestamp array, cross in an address table of (address, count) longs (the convention of arcadedb_n_insert_colp), and the JSON holds only a kind marker per column ("q" int64, "d" float64) and the sample count. The host tag column and every shape the text path converts differently keep the text: lists, float timestamps, other datetime64 units, 2-D and empty arrays, uint64 past the int64 range, a column whose length differs from the timestamps. Python side: codec.sample_array decides per column, NativeAsyncExecutor.append_samples builds the table. Java side: NativeExtras.asyncImpl (the old body of async, now shared by both entries) builds the same long[] and boxed Object[] columns the text gave and calls the same appendSamples.

What does not change: the stored samples are bit-identical to the text path (a float32 or float16 array widens to float64 exactly, a bool array is 0 and 1, a NaN is the canonical NaN the text gives, an int array into a DOUBLE column and a float array into a LONG column go through the same Number conversions). primitive=True keeps its meaning on the native build: the same length check, before the crossing, with the same message. A library without the entry (the loader sets lib.TS_BIN false) or ARCADEDB_NATIVE_TS_BIN=0 keeps the text path. The JVM wheel is not touched.

Tests (tests/test_native_timeseries_binary.py, native only, collected through conftest.py like the other native observers, no skips): the lane's shape takes arcadedb_n_async_bin once per chunk and its text holds no number of a numeric column, and the stored samples are read back by count, sums and a window; a matrix of dtypes (float64 with -0.0, NaN, infinities, 1e300 and 5e-324, float32, float16, int32 into DOUBLE, int64 extremes, uint8, bool, float into LONG) is stored bit-equal to the text path of the same call; timestamp arrays of int32, uint16, uint64, datetime64[ns] and a strided view are stored equal; non-contiguous and big-endian columns are stored as values; the entry choice for lists, float timestamps, datetime64[ms] (still a TypeError), a column of another length, primitive=True and a library without the entry; sample_array for 24 dtype cases; and the Java checks of a wrong address table (nothing is written). The CI run ids are in the first comment.

Not claimed: no speed number. A wheel-versus-wheel timed A/B (DECISIONS #240) runs on the laptop after the CI wheel exists; its verdict goes on this PR. The Java side still boxes each value once, the engine's TimeSeriesBatch route is a possible follow-up. This PR and #224 both touch lib.py and extras.py (a flag, a signature line, an import); the conflicts are textual.

🤖 Generated with Claude Code

…st() and JSON text (#228)

NativeAsyncExecutor.append_samples called .tolist() on every column, walked the timestamps again with int(), and sent the chunk as one JSON text to arcadedb_n_async. A one-dimensional numpy int, float or bool column and an int or datetime64[ns] timestamp array now go through the new arcadedb_n_async_bin: the JSON holds a kind marker per column ("q" int64, "d" float64) and the sample count, the data crosses in an address table of (address, count) longs, the convention of arcadedb_n_insert_colp. The host tag column and anything the text path converts differently (lists, float timestamps, other datetime64 units, 2-D, empty, uint64 past int64, a column of another length) keep the text. Java builds the same long[] and boxed Object[] columns and calls the same appendSamples, so the stored samples are bit-identical (NaN is the canonical NaN of the text). A library without the entry, or ARCADEDB_NATIVE_TS_BIN=0, keeps the text path; primitive=True is the same length check.

Tests (native only, run in CI): the lane's shape takes the entry and reads back by count, sums and a window; every dtype is stored as the text path stores it; the entry choice per shape; the Java checks of a wrong table.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@tae898

tae898 commented Nov 3, 2026

Copy link
Copy Markdown
Collaborator Author

CI on head 4cb904dbdb, read at 06:16 KST on 2026-11-04. The native build has compiled the new entry: the dispatch run of test-native.yml (41716062800) built and gated the wheels for linux-x86_64, linux-aarch64 and windows-x64; macos-arm64 was still building, and the parity tests (which include tests/test_native_timeseries_binary.py) start after the wheel jobs, so they have not run yet. The pull_request JVM run (41716066631) is green: the new test file is collected on the native backend only. The pull_request native run (41716066558) was skipped because the PR is a draft.

The test results will be added here when the dispatch run finishes. The timed A/B (DECISIONS #240, wheel versus wheel) is prepared and not started.

tae898 and others added 2 commits November 4, 2026 06:40
…test file (#228)

CI run 41716062800 failed only these three, all in tests/test_native_timeseries_binary.py, all in the test code: the window of the lane test ended at index 1900 so host_7 had 9 points, not 10 (end now 1999); an int32 array into a DOUBLE column is an int64 column ("q"), the expected kinds listed "d"; datetime64 + datetime64 is not defined in numpy, the datetime64[ms] timestamps are now datetime64 + timedelta64. No source change: every other parity file of the run passed (2036 passed).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ay may take it there

The parity leg on windows-x64 py3.14 failed one case (values11): the test expected None for a
np.longdouble array, which holds only on platforms where longdouble is 80-bit extended. Where it is
float64 the picker is right to take it (tolist reproduces the values).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Native append_samples: send numeric columns as binary buffers, not .tolist() and one JSON text per chunk

1 participant