Repository navigation
Conversation
…239) iter_json_batches drained the whole result into dicts (_drain_dicts), converted every row, and only then cut the list into batches, so batch_size bounded nothing and the peak held every row twice. It now holds at most one reply (_NEXT_BATCH rows) and one output batch, and reads the cursor when the next batch is asked for. Rows, order, types and batch boundaries are those of to_json_list cut at batch_size. A result closed between two batches raises the closed-before-read error, as the other iterators do. to_columns, to_arrow and to_polars are unchanged (a column type is decided over all rows). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Test runs for this PR, read on 2026-11-04:
The four that fail on main: One gap: the buffer-bound test passes on main too, so it does not catch the unbounded drain this PR fixes. The memory claim rests on the timed study's headline (peak RSS of a 1M-row |
|
Timed A/B (native wheel; arm A = main's CI native wheel, arm B = the same tree with only this PR's files, A/A control; laptop timed lane,
Clean pairs per figure: 10; every run passed its arm check, and the output differential (iter_json_batches against to_json_list at ten batch sizes, arm A against arm B) passed before the run. By the rule written before the run this is a SHIP: peak memory is far below 1.00, the first batch and the total time are within their guards (both faster), and the control is flat. The buffer-bound unit test now has a main-failing form ( The merge waits for the end of the current batch of timed studies that pin main (merging now would move their pins); the PR leaves draft at that point. Edited 2026-11-04: the main-failing test line now records the CI proof. |
…r-bound test counts rows read ahead so it fails on main The old test watched the result set's buffer, which the drain on main empties as it goes, so it passed there. The new one counts rows read from the cursor against rows delivered at every batch: main has read all 5000 after the first batch of 100, the change at most one reply ahead. Docs carry the measured ratios with 95% intervals from the laptop study (copied by script from ab.txt). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Closes #239
On the native build
iter_json_batchesdrained the whole result into dicts, converted every row, and only then cut the list into batches, sobatch_sizebounded nothing. It now reads the cursor one batch at a time: at most one reply (_NEXT_BATCH, 2048 rows) and one output batch are held, and the first batch is returned after reading only the rows it needs.The cut is
src/arcadedb/_native/database.py: a newNativeResultSet._iter_dict_batches(size)(reads_bufand_fill()as_drain_dictsdoes, without collecting the result), anditer_json_batchesnow converts each batch withjson_nativeas it is yielded. The batch size is still validated when the first batch is asked for, as before.Same output: rows, order, types and batch boundaries are those of
to_json_listcut atbatch_size(every batch full except the last; an empty result yields nothing). One behaviour differs on purpose and follows__next__: the cursor is read when the next batch is asked for, so a result closed between two batches (byclose()or by leaving itswithblock) raises the "closed before all its rows were read" error, where before the batches already built kept coming. An engine error met while reading surfaces after the batches before it. A generator that is dropped half-way leaves the cursor to the result set (close(),with,__del__), as dropping an iteration does.Not changed:
to_columns,to_arrowandto_polarsstill drain first, because a column's type is decided over all its values. The peak RSS of those paths is measured with this one in the study below, so a follow-up can be judged on a figure.Tests (
tests/test_native_streaming_batches.py, native backend only, collected throughconftest.pylike the other native-observing files): equal toto_json_listfor ten batch sizes over a 5000-row result of several replies; the buffer never holds more than one reply plus a batch; the first batch is returned after at most one read of the cursor and leaves the cursor open;close()between two batches raises and closes the cursor; leaving awithblock mid-iteration; a dropped generator; a batch size below one. The existingiter_json_batchestests (test_core.py,test_resultset.py,test_resultset_read_errors.py,test_nonfinite_floats.py) run on both builds and are unchanged.Overlap with #223 (
perf/native-json-list-untagged,to_json_listskips the walk when no reply was tagged): both touchNativeResultSetin_native/database.py, in different hunks (#223:__init__,_lazy,_fill,to_json_list,_exec; this PR: a new method after_drain_dictsanditer_json_batches). Neither hunk overlaps, so the two should merge in either order without a conflict. Merge order: #223 first is natural, and theniter_json_batcheshere can skipjson_nativefor a result that never restored a tagged value, with the same one-lineself._taggedcheck thatto_json_listgets there; that follow-up is not in this PR because_taggeddoes not exist on main.Draft: no merge until the peak-RSS figure is in. Peak RSS of a 1M-row read, native against the JVM wheel, comes from a probe (dropped if the native peak is not clearly above the JVM wheel's), then a timed A/B on the maintainer's laptop with peak RSS as the gating figure and a wall guard.
Docs and test updated in 255a7b3: the buffer-bound test now counts rows read from the cursor against rows delivered at every batch (main has read all 5000 rows after the first batch of 100, so it fails there), and the docs and docstrings say that
batch_sizebounds memory on the native build, with the measured laptop figures and 95% intervals.Edited 2026-11-04 to add this note.
🤖 Generated with Claude Code