Skip to content

JVM GraphBatch: list, dict and date-time properties stay on the JSON bulk path - #244

Draft
tae898 wants to merge 4 commits into
mainfrom
perf/jvm-gb-tagged-shapes
Draft

tae898 wants to merge 4 commits into
mainfrom
perf/jvm-gb-tagged-shapes

Conversation

@tae898

@tae898 tae898 commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #243

On the JVM wheel one list, tuple, dict, date-time, date, Decimal or bytes property sent the whole GraphBatch.create_vertices call, and the whole GraphBatch.new_edges call with properties, down per-element Python (_to_java_property_matrix, or new_edge per edge). This carries those shapes through json_bulk_dumps_tagged, the tagged-value JSON #82 added for insert_many.

What changes:

  • json_bulk_shape_ok (type_conversion.py) is the cheap per-value gate for what json_bulk_scalar_ok refuses: the exact types list, tuple, dict, datetime, date, Decimal, bytes, bytearray. A non-empty list or tuple of only int and float is refused and keeps main's path: the untimed step-0 probe measured 117 us per row through the text against 31 us on the property matrix for a list of 128 doubles (one double[]). Subclasses (pandas Timestamp), numpy values, sets and pandas NA stay refused as before.
  • An input whose values are all scalars runs main's code: the same gate, the untagged json.dumps, and in Java the same per-key loop (DocumentBatcher.value adds two type tests per value). The tagged writer and the shape gate are only reached when a row holds a shape.
  • For a shape, _create_vertices_json_bulk and new_edges write the text with json_bulk_dumps_tagged. A chunk of vertices that holds a value the text cannot carry (a NaN, an integer beyond 64 bits or a date-time inside a list or map, a subnormal, a non-str dict key) goes through the property-matrix path on its own. For edges every chunk is written before the first one crosses, so the same values start the per-edge path over with nothing buffered. A declared ARRAY_OF_FLOATS property whose list the float matrix cannot carry still sends its chunk to the matrix path, as in #496.
  • VertexBatcher and EdgeBatcher decode through DocumentBatcher.value: a nested object or array becomes a plain Map or List (as DocumentBatcher.fill does) and a tag is decoded by DocumentBatcher.untag (now package-private).

Tests (tests/test_graph_batch_tagged_shapes.py, JVM only, listed in NATIVE_NOT_APPLICABLE): every shape, declared and schemaless, is stored by create_vertices and new_edges as the property-matrix path and new_edge per edge store it (Python type and repr of every property compared); spies on _to_java_property_matrix and new_edge prove which path ran; a list of only numbers keeps the matrix path (alone and beside a dict); an input of scalars never reaches the shape gate or the tagged writer; values the text cannot carry keep the old path and store the same; chunk-level fallback with _BULK_CHUNK lowered; declared vector next to list, dict and date-time properties; the gates. tests/test_graph_batch.py used a datetime to reach the property-matrix path in two tests, and now uses a NaN.

Known difference inside a nested container: a small integer inside a list or map arrives from the JSON text as a Java Integer where the per-element path builds a Long. It reads back as the same Python int (the tests compare it); scalar integers already behaved this way on the bulk path.

Overlap with #221 (perf/jvm-create-vertices-jarray): #221 changes only is_vector_candidate in type_conversion.py; this PR adds functions elsewhere in that file and edits graph_batch.py and the bridge classes, so the two merge without a conflict. A JPype float[] or double[] on an undeclared property is not a shape and keeps the old path here; on a declared ARRAY_OF_FLOATS property #221 sends it to the float matrix.

Status: draft. The step-0 probe (untimed, one run each, per row, PR/main) for the first head showed list_strings 0.46 and 0.43 (vertices, edges), dict 0.33 and 0.33, datetime 0.74 and 0.54, and the two regressions this head removes (list of doubles 3.8 times slower, scalar rows 1.18 and 1.12). Nothing is timed yet. A timed wheel-against-wheel A/B (pre-registered, DECISIONS #240; list_strings, dict and datetime gate it, list of doubles and scalar rows are no-harm guards) decides whether the PR leaves draft.

Docs updated in 4d6c9c0: a section in how-to/bulk-loads.adoc (shapes on the bulk path, numeric lists keep the older path, the fallback rule, the laptop A/B figures with 95% intervals) and the new_edges docstring. Edited 2026-11-04 to add this line.

🤖 Generated with Claude Code

tae898 and others added 3 commits November 4, 2026 06:50
…hrough the JSON bulk path (#243)

create_vertices and new_edges refused a whole call when one property was a list, dict, date-time, date, Decimal or bytes, and went per element
through JPype. They now use json_bulk_dumps_tagged (insert_many's format from #82) and VertexBatcher/EdgeBatcher decode with
DocumentBatcher.untag after toMap(). A chunk (vertices) or call (edges) holding a value the text cannot carry still takes the old path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…243)

Step-0 probe (untimed): a list of 128 doubles cost 117 us per row through the text against 31 us on the property matrix (one double[]), and a scalar row 6.18 against 5.22 us. A list or tuple of only int/float now keeps main's path, and an input whose values are all scalars runs main's untagged json.dumps and gate (no shape test, no tagged writer); Java reads scalars as before (DocumentBatcher.value adds two type tests).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… the NaN (CI run 41731925082)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@tae898

tae898 commented Nov 4, 2026

Copy link
Copy Markdown
Collaborator Author

Test runs for this PR, read on 2026-11-04:

Run Branch Code Result
41741256851 perf/jvm-gb-tagged-shapes (head 690d0143c0) this PR every job green (all 34 checks)
41741270826 tmp/gb-shapes-tests-only (head c62a814bb8) main plus only the new test file linux/amd64 py3.12: 32 failed and 3 errors, all in tests/test_graph_batch_tagged_shapes.py; 2155 other tests pass

Every test that fails on main is one of the new shape tests (the shape gate, the tagged writer and the bridge decode do not exist there). The tests-only branch is deleted after this read. The timed A/B (one-library overlay on main's CI JVM wheel, rule written before the run) is queued on the laptop's timed lane.

@tae898

tae898 commented Nov 4, 2026

Copy link
Copy Markdown
Collaborator Author

Timed A/B on head 690d014 (JVM build; arm A = main's CI JVM wheel, arm B = the same tree with this PR's graph_batch.py, type_conversion.py and bridge jar, A/A control; laptop timed lane, tools/ab.py --noise, ABBA, round 0 dropped). Per-row time, B/A median with 95% CI (below 1 = this PR is faster):

property shape role create_vertices new_edges wall
list of strings gating 0.403 [0.378, 0.420] 0.409 [0.389, 0.420] 0.480 [0.456, 0.501]
dict gating 0.265 [0.243, 0.286] 0.278 [0.269, 0.298] 0.336 [0.314, 0.358]
datetime gating 0.515 [0.503, 0.550] 0.482 [0.469, 0.518] 0.591 [0.557, 0.618]
list of floats (keeps main's path) no-harm guard 1.024 [0.963, 1.109] 1.010 [0.979, 1.027] 1.009 [0.944, 1.037]
scalars only control 1.017 [0.995, 1.077] 0.989 [0.948, 1.038] 0.996 [0.979, 1.039]

Clean pairs per figure: 10, 11, 12; every A/A median between 0.968 and 1.043. By the rule written before the run (revised after the untimed probe and before any timed run, as the study README records) this is a SHIP: every gating interval is below 0.90, the list-of-floats guard shows no harm, and the scalar control is flat.

The merge waits for the end of the current batch of timed studies that pin main (merging now would move their pins); the PR leaves draft at that point, with its docs.

… fallback rule, the laptop A/B figures (#243)

Docs and docstring only. Figures copied by script from the study ab.txt.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

JVM GraphBatch: a list, dict or date-time property sends create_vertices and new_edges to per-element Python

1 participant