Skip to content

Native to_json_list: skip the walk for untagged results; RIDs and map keys JSON-safe - #223

Open
tae898 wants to merge 2 commits into
mainfrom
perf/native-json-list-untagged
Open

tae898 wants to merge 2 commits into
mainfrom
perf/native-json-list-untagged

Conversation

@tae898

@tae898 tae898 commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #219.

Native NativeResultSet.to_json_list now walks the rows with codec.json_native only when a reply of the result restored a tagged value, and json_native returns JSON-safe values for an RID and for map keys. Draft: the timed A/B (one-library overlay, ab.py) is prepared and not run, so the speed claim is the untimed probe in #219 until it is.

What changes

  • codec.loads_tagged(text, floats) is loads plus a flag, true when a tagged value was restored. NativeResultSet keeps _tagged across every reply it decodes: the first batch in _exec, every arcadedb_n_next and arcadedb_n_next_bin batch in _fill, and a lazy command result starts false. The flag is sticky, so a result with an untagged first batch and a later tagged one (or the reverse) walks. to_json_list returns the decoded rows as they are when the flag is false. A caller that builds a result set without saying (NativeResultSet(db, reply)) gets true, the old behaviour.
  • A reply whose only tags are ["B", offset, count] (the float tail of vectors) decodes to plain lists of Python floats, so it does not set the flag and is not walked. This is tested, and a reply with vectors and a date does walk.
  • json_native returns the text of an RID ("#10:5"), also in a list and in a map. A map key that is not a string becomes a string: an RID its text, True "true", None "null", a number its str.
  • to_list is unchanged: it still returns RID objects and the Python key types (tested).
  • Tagged results give the same output as before for DATE and DATETIME (epoch milliseconds), DECIMAL (float) and sets (lists).

The value for a link, and the one deliberate difference. The JVM wheel's to_json_list returns the RID text for a link property, a list of links, SELECT @rid and a map value (read on the 26.11.1.dev0 JVM wheel, 2026-11-04), and string keys for a map stored with integer keys ({'1': 'a'}; JSON objects have string keys). The native wheel now returns the same, and tests/test_to_json_list_links.py states those values and runs on both backends (it passes on the JVM wheel too). The one change to an output that was JSON-safe: a tagged map with integer keys used to come back with int keys ({1: 'a'}) and now has str keys, as the JVM wheel has. json.dumps of either gives the same text.

Tests (all new, none skipped; tests/test_native_json_list.py is native only through conftest.py, tests/test_to_json_list_links.py runs on both)

I ran the new files against the unmodified CI native wheel (tree of arcadedb-26.11.1.dev0-py3-none-manylinux_2_28_x86_64.whl, commit 413f54f, whose codec.py and database.py equal origin/main): 25 failed, 5 passed in 0.45s. The tests that fail on the old code are the ones that state the change (an untagged result is not walked, a float-tail-only reply is not walked, an RID and a non-string key become strings, the loads_tagged flag, the links on both builds). The five that pass on the old code are guards for what must not change (an empty result, a lazy command result that must still walk, to_list keeping the Python types, to_list keeping RID objects and integer keys, and the other json_native conversions). The tests of a tagged or mixed result also fail on the old code, but only through their premise check (the old code has no flag to read); their walk-count asserts hold on the old code by construction, and the mutants below are what shows they guard the new code. I also checked the guards with two mutants of the new code: a non-sticky flag fails the tagged-first-then-untagged test, and counting a "B" tag as restored fails the float-tail tests.

How I ran the tests. The CI native wheel installed as in test-native.yml (uv run --no-project --python 3.12 --with <wheel> --with pytest --with numpy --with requests --with pyarrow --with jpype1 --with setuptools --with pandas python -m pytest), with the extracted wheel tree first on PYTHONPATH and the two changed modules copied over it (libarcadedb.so is the wheel's), taskset -c 8-15. 519 passed and 23 skipped (all NA-JVM or NOTYET as the native parity run reports them) over test_native_json_list.py, test_to_json_list_links.py, test_native_codec_loads.py, test_native_result_transport.py, test_native_point_read.py, test_resultset.py, test_nonfinite_floats.py, test_results_after_close.py, test_resultset_read_errors.py, test_columnar_readers.py, test_float_vector_bulk.py, test_bulk_json_paths.py and test_core.py. Pre-commit (black 26.10.1, isort, the name check), compile, pyflakes (only the existing # noqa imports) and Bandit 1.9.4 over src tests are clean.

Not in this PR. The timed A/B. The overlay study only changes _native/codec.py and _native/database.py over the main CI wheel, so libarcadedb.so is identical in both arms.

🤖 Generated with Claude Code

Docs updated in 7c17592 (docs and docstrings only).

Edited 2026-11-04: added the docs line above.

…ed; RIDs and map keys JSON-safe (closes #219)

A reply without the escaped tag decodes to json.loads types, which json_native returns unchanged, so the walk over every value was
overhead (51 to 95 percent of the to_json_list wall on paper-shaped results up to 100 rows in the untimed probe). loads_tagged says
whether a tagged value was restored; the result set keeps the flag across every reply it decodes (sticky) and to_json_list walks only
when it is set. A reply whose only tags are float-tail "B" tags decodes to plain float lists and does not set it.

json_native returned the RID object for a link, which json.dumps refuses, where the JVM build writes the text. It now returns the text
and writes map keys that are not strings as the JVM build's JSON object does (RID text, "1", "true", "null").

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@tae898

tae898 commented Nov 3, 2026

Copy link
Copy Markdown
Collaborator Author

Timed A/B (native build; arm A = main 413f54f, arm B = A's tree with only _native/database.py and _native/codec.py from this PR (PR 223 88aafa5), A/A control; laptop timed lane, ab.py --noise, ABBA pairs; the rule written before the run). Each figure is the statement plus to_json_list() on a result shaped like the benchmark's graph and sparse reads, time ratio B/A:

figure median B/A 95% CI clean pairs
one-row point read (jl_point_us) 0.958 [0.942, 0.964] 12
top 10, three columns (jl_top10_3col_us) 0.987 [0.970, 0.999] 12
100-row histogram (jl_hist100_us) 0.976 [0.971, 0.980] 12
100-row scan, four columns (jl_scan4_r100_us) 0.953 [0.946, 0.959] 12
sparse search top 10 (jl_sparse10_us) 0.984 [0.979, 0.987] 12
control: to_list() on the 100-row scan 0.993 [0.990, 1.001] 12

Result by the rule: SHIP (every gating interval below 1.00, the to_list() control not moved, the A/A control clean). The gain is the second walk that untagged results no longer take; it is a few percent of the whole call because the engine's statement time dominates these small results. The same PR makes RID values (links, @rid, map keys) JSON-safe in to_json_list(), as on the JVM build.

Marking this ready. It merges after the other pending timed A/B that pins src/arcadedb/_native has run, so its pins stay valid.

…n both builds; native skips the walk for untagged results (PR 223)

Results how-to and the to_json_list docstrings (JVM ResultSet, NativeResultSet). Docs and docstrings only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Native to_json_list walks untagged rows again, and json_native returns RID objects json.dumps refuses

1 participant