JVM GraphBatch: list, dict and date-time properties stay on the JSON bulk path

docs/modules/ROOT/pages/how-to/bulk-loads.adoc

@@ -1,5 +1,5 @@
 = Bulk loads and commit size
-:description: The default commit size of insert_many, graph_batch and create_vertices on the arcadedb (native) and arcadedb-jvm packages, what it means for a failed load, what to do after an interrupted graph batch, how to change it, and a known limitation: a graph batch does not reuse the space of deleted edges.
+:description: The default commit size of insert_many, graph_batch and create_vertices on the arcadedb (native) and arcadedb-jvm packages, which property shapes a graph batch loads in bulk on the JVM build, what it means for a failed load, what to do after an interrupted graph batch, how to change it, and a known limitation: a graph batch does not reuse the space of deleted edges.
 
 `insert_many`, `graph_batch` and `GraphBatch.create_vertices` load data in transactions. The commit size is the number of rows in one transaction. It decides how much work the engine holds in memory before it commits, and how many rows stay in the database when a load fails.
 
@@ -67,6 +67,36 @@ with db.graph_batch(commit_every=5000) as batch:  # also sets the create_vertice
     ...
 ----
 
+== Property shapes in a graph batch (`arcadedb-jvm`)
+
+On the `arcadedb-jvm` package, `GraphBatch.create_vertices` and `GraphBatch.new_edges` load these property values through the bulk path (the same tagged JSON text as `insert_many`): a list, a tuple, a dict, a `datetime`, a `date`, a `Decimal` and `bytes` (also `bytearray`). A value is stored as the one-by-one path stores it. A list or tuple of only `int` and `float` keeps the older path, which crosses it as one `double[]`.
+
+A value that the text cannot carry sends work to the older path on its own: a `NaN`, an integer beyond 64 bits, a `datetime` inside a list or a dict, or a dict key that is not a `str`. `create_vertices` falls back for the chunk of at most 100,000 rows that holds the value, and `new_edges` for the whole call. A value of another type (a `set`, a numpy or pandas value, a subclass of the types above) also takes the older path. Nothing is lost on a fallback, and the data is the same. Only the speed changes.
+
+The table is one laptop study of the `arcadedb-jvm` package: per-row time of this build against the build before it, 20,000 vertices and 20,000 edges for each shape in chunks of 5,000, one property on every row. Each cell is the median of the pair ratios (below 1 means faster) with its 95 % interval, from 11 to 12 clean ABBA pairs for each figure of `ab.py --noise`, with a clean A/A control. The `arcadedb` (native) package does not change.
+
+[cols="3,2,2"]
+|===
+| Property shape | `create_vertices` | `new_edges`
+
+| List of strings
+| 0.403 (0.378 to 0.420)
+| 0.409 (0.389 to 0.420)
+
+| Dict
+| 0.265 (0.243 to 0.286)
+| 0.278 (0.269 to 0.298)
+
+| Datetime
+| 0.515 (0.503 to 0.550)
+| 0.482 (0.469 to 0.518)
+
+| List of floats (keeps the old path)
+| 1.024 (0.963 to 1.109)
+| 1.010 (0.979 to 1.027)
+
+|===
+
 == Graph batches next to deletes (fixed in ArcadeDB 26.11.1)
 
 NOTE: *Fixed in ArcadeDB 26.11.1.* Before 26.11.1, a graph batch did not reuse the space of deleted edges (https://github.com/ArcadeData/arcadedb/issues/10746[ArcadeData/arcadedb#10746]). The engine wrote the edge records of each `graph_batch()` flush onto new pages and never looked at the free space that deletes left on older pages, so the edge bucket gained one 64 KiB page per flush while the live data stayed flat. From 26.11.1 (https://github.com/ArcadeData/arcadedb/pull/10836[ArcadeData/arcadedb#10836]) each edge goes where an ordinary insert would put it while an existing page has room, and only the rest of a flush goes onto new pages, so a large import stays as fast as before. The 26.11.1 packages ship this fix. The `26.11.1.devN` pre-releases embed engine 26.10.2 and still show the growth described below.

src/arcadedb/graph_batch.py

@@ -15,7 +15,9 @@
     convert_python_to_java,
     declared_float_vectors,
     is_vector_candidate,
+    json_bulk_dumps_tagged,
     json_bulk_scalar_ok,
+    json_bulk_shape_ok,
     split_float_vectors,
 )
 
@@ -190,6 +192,11 @@ def create_vertices(
                 ``ARRAY_OF_FLOATS`` whose value in every row is a list or a numpy
                 array of numbers of one length crosses as one binary float32
                 matrix instead of JSON text.
+                A list (not one of only numbers), tuple, dict, date-time, date,
+                Decimal or bytes value also stays on the bulk path: it crosses
+                inside the JSON as a tagged value and is stored as the
+                per-element path stores it. A list of only numbers keeps the
+                property-matrix path, which crosses it as one double[].
 
         Large inputs cross in chunks of 100,000 rows on this build. On the native build (the ``arcadedb`` package) the chunk is the
         batch's ``commit_every``, 1000 unless ``graph_batch()`` got another value, and each chunk is its own transaction. Chunks that
@@ -244,14 +251,23 @@ def _create_vertices_json_bulk(self, type_name, rows):
         string (two bulk copies instead of per-value marshaling, measured
         ~2.4x). The float vectors of ARRAY_OF_FLOATS properties cross as one
         binary matrix beside the JSON instead of as text (split_float_vectors).
-        Returns None when unavailable/unsuitable so the caller falls back to
-        the property-matrix path."""
+        A list (not one of only numbers), tuple, dict, date-time, date, Decimal
+        or bytes value crosses inside the JSON as the tagged value
+        json_bulk_dumps_tagged writes and VertexBatcher decodes (the stored
+        value is what the property-matrix path stores). An input whose values
+        are all scalars takes the untagged json.dumps path as before. A chunk
+        holding a value the text cannot carry (a NaN, an integer beyond 64
+        bits or a date-time inside a list, a subnormal) goes through the
+        property-matrix path on its own. Returns None when
+        unavailable/unsuitable so the caller falls back to the property-matrix
+        path for the whole input."""
         try:
             vertex_batcher = jpype.JClass("com.arcadedb.python.VertexBatcher")
             declared = set(declared_float_vectors(self._java_db, type_name))
         except Exception:
             return None
 
+        tagged = False  # a value that needs the tagged text; False keeps the untagged code of before
         for row in rows:
             if row:
                 for key, value in row.items():
@@ -263,10 +279,14 @@ def _create_vertices_json_bulk(self, type_name, rows):
                     if key in declared and is_vector_candidate(value):
                         continue  # the vector split below takes it, or its chunk goes the matrix way
                     if not json_bulk_scalar_ok(value):
-                        # e.g. datetime/bytes (the matrix path preserves types), or a
-                        # value JSON would change: an integer beyond 64 bits, NaN or
-                        # Infinity, a lone surrogate
-                        return None
+                        if not json_bulk_shape_ok(value):
+                            # a type the tagged JSON does not carry (a set, a numpy
+                            # or pandas value, any subclass, a list of only numbers:
+                            # the matrix path converts those), or a scalar JSON
+                            # would change: an integer beyond 64 bits, NaN or
+                            # Infinity, a lone surrogate
+                            return None
+                        tagged = True
 
         import json
 
@@ -276,15 +296,26 @@ def _create_vertices_json_bulk(self, type_name, rows):
             chunk = rows[start : start + self._BULK_CHUNK]
             split = split_float_vectors(self._java_db, type_name, chunk, declared)
             rest = split[0] if split else chunk
-            if declared and not self._json_bulk_ok(rest):
+            if not tagged:
+                text = None
+                if not declared or self._json_bulk_ok(rest):
+                    text = json.dumps([row or {} for row in rest])
+            else:
+                text = None
+                if self._json_bulk_declared_ok(rest, declared):
+                    try:
+                        text = json_bulk_dumps_tagged([row or {} for row in rest])
+                    except (TypeError, ValueError):
+                        text = None  # a value the text cannot carry, see the docstring
+            if text is None:
                 # a list the matrix cannot carry (ragged, a null, not numbers)
-                # is left in this chunk: the property-matrix path takes the chunk
+                # is left in this chunk, or a value the text would change: the
+                # property-matrix path takes the chunk
                 java_rids = self._java_graph_batch.createVertices(
                     type_name, self._to_java_property_matrix(chunk)
                 )
                 rids.extend(str(rid) for rid in java_rids)
                 continue
-            text = json.dumps([row or {} for row in rest])
             if split:
                 joined = vertex_batcher.createVerticesJson(
                     self._java_graph_batch, type_name, text, split[1], split[2]
@@ -307,6 +338,16 @@ def _json_bulk_ok(rows):
                         return False
         return True
 
+    @staticmethod
+    def _json_bulk_declared_ok(rows, declared):
+        """False when a declared ARRAY_OF_FLOATS property still holds a list (or any non-scalar) after the vector split."""
+        for row in rows:
+            if row:
+                for key in declared:
+                    if key in row and not json_bulk_scalar_ok(row[key]):
+                        return False
+        return True
+
     def new_edge(
         self,
         source_vertex_or_rid,
@@ -354,8 +395,13 @@ def new_edges(
             edge_type: Edge type name.
             destination_rids: Sequence of destination RIDs (same length).
             properties: Optional sequence of per-edge property dicts (same
-                length). JSON-representable values take the bulk path;
-                anything else falls back to per-edge buffering.
+                length). Scalars and a list (not one of only numbers), tuple,
+                dict, date-time, date, Decimal or bytes value take the bulk
+                path (the last ones as tagged values inside the JSON). A list
+                of only numbers, a value the text cannot carry (a NaN, an
+                integer beyond 64 bits, a date-time inside a list or dict, a
+                non-str dict key) or any other type sends the whole call to
+                per-edge buffering.
         """
         self._check_not_closed()
         from .results import _bridge_class
@@ -419,18 +465,20 @@ def new_edges(
                 return self
 
             rows = []
+            tagged = False  # a value that needs the tagged text; False keeps the untagged code of before
             for src, dst, p in zip(source_rids, destination_rids, properties):
                 row = {"_src": str(src), "_dst": str(dst)}
                 for key, value in (p or {}).items():
-                    if (
-                        type(key) is not str
-                        or key in ("_src", "_dst")
-                        or not json_bulk_scalar_ok(value)
-                    ):
-                        # non-JSON value (datetime/bytes), or one JSON would change
-                        # (an integer beyond 64 bits, NaN or Infinity, a lone
-                        # surrogate), or a property name that is not a str (JSON
-                        # would write 1 as "1"): per-edge fallback preserves
+                    ok = type(key) is str and key not in ("_src", "_dst")
+                    if ok and not json_bulk_scalar_ok(value):
+                        ok = json_bulk_shape_ok(value)
+                        tagged = tagged or ok
+                    if not ok:
+                        # a type the tagged JSON does not carry (a set, a numpy or
+                        # pandas value, a list of only numbers), or a scalar JSON
+                        # would change (an integer beyond 64 bits, NaN or Infinity,
+                        # a lone surrogate), or a property name that is not a str
+                        # (JSON would write 1 as "1"): per-edge fallback preserves
                         # types exactly and refuses the name
                         props_iter = properties
                         for s2, d2, p2 in zip(
@@ -441,14 +489,32 @@ def new_edges(
                     row[key] = value
                 rows.append(row)
 
-            import json
+            if not tagged:
+                import json
 
-            for start in range(0, len(rows), self._BULK_CHUNK):
-                edge_batcher.newEdgesJson(
-                    self._java_graph_batch,
-                    edge_type,
-                    json.dumps(rows[start : start + self._BULK_CHUNK]),
-                )
+                for start in range(0, len(rows), self._BULK_CHUNK):
+                    edge_batcher.newEdgesJson(
+                        self._java_graph_batch,
+                        edge_type,
+                        json.dumps(rows[start : start + self._BULK_CHUNK]),
+                    )
+                return self
+
+            # Every chunk is written before the first one crosses, so a value the
+            # text cannot carry (a NaN, an integer beyond 64 bits or a date-time
+            # inside a list or map, a subnormal) can still start the per-edge
+            # path over with nothing buffered.
+            try:
+                texts = [
+                    json_bulk_dumps_tagged(rows[start : start + self._BULK_CHUNK])
+                    for start in range(0, len(rows), self._BULK_CHUNK)
+                ]
+            except (TypeError, ValueError):
+                for s2, d2, p2 in zip(source_rids, destination_rids, properties):
+                    self.new_edge(s2, edge_type, d2, **(p2 or {}))
+                return self
+            for text in texts:
+                edge_batcher.newEdgesJson(self._java_graph_batch, edge_type, text)
             return self
         except Exception as e:
             raise ArcadeDBError(

src/arcadedb/type_conversion.py

@@ -697,6 +697,41 @@ def json_bulk_scalar_ok(value: Any) -> bool:
     return False
 
 
+_BULK_TAGGED_SHAPES = frozenset(
+    (list, tuple, dict, datetime, date, Decimal, bytes, bytearray)
+)
+
+
+_NUMBER_TYPES = frozenset((int, float))
+
+
+def json_bulk_shape_ok(value: Any) -> bool:
+    """Whether a value that json_bulk_scalar_ok refused may still go through json_bulk_dumps_tagged in the graph batches (the cheap, per-value gate).
+
+    True for the exact types the tagged JSON carries: a list, tuple or dict, a date-time, date, Decimal, bytes or bytearray. The deep
+    check (a NaN, an integer beyond 64 bits or a date-time inside a list, a subnormal) is json_bulk_dumps_tagged's own, which raises
+    ValueError or TypeError so the caller sends the rows down the per-element path. False for a non-empty list or tuple of nothing but
+    int and float: the per-element path crosses it as one double[] and measured about 3.8 times faster than the text for 128 doubles
+    (step-0 probe, 2026-11-04), so it keeps that path. A subclass (pandas.Timestamp), a numpy scalar or array, a set and pandas NA stay
+    refused: the per-element path converts those.
+    """
+    kind = type(value)
+    if kind not in _BULK_TAGGED_SHAPES:
+        return False
+    if (
+        (kind is list or kind is tuple)
+        and value
+        and set(map(type, value)) <= _NUMBER_TYPES
+    ):
+        return False
+    return True
+
+
+def json_bulk_tagged_ok(value: Any) -> bool:
+    """A scalar json_bulk_scalar_ok accepts, or a shape json_bulk_shape_ok accepts."""
+    return json_bulk_scalar_ok(value) or json_bulk_shape_ok(value)
+
+
 def _json_bulk_check(value: Any) -> None:
     """Raise ValueError for a value JSON would change on the way to the engine.
 

src/java/com/arcadedb/python/DocumentBatcher.java

@@ -294,7 +294,20 @@ public static Object[] boxLongs(final long[] a) {
    * A tagged value from the Python side, {"\u0001": [kind, payload]}: T LocalDateTime (UTC wall clock), D LocalDate, n BigDecimal,
    * b byte[] (base64): the objects the per-row path (type_conversion.convert_python_to_java) builds. Anything else is returned unchanged.
    */
-  private static Object untag(final Object v) {
+  /**
+   * One value of a parsed JSON row, as the engine stores it: a nested object or array becomes a plain Map or List (a parsed
+   * JSONArray stored as it is reads back as a JSONArray inside the transaction that wrote it) and a tagged value is decoded. A
+   * scalar comes back as it is, after two type tests: the graph batches call this per value, so it adds nothing to a scalar row.
+   */
+  static Object value(final Object v) {
+    if (v instanceof JSONObject o)
+      return untag(o.toMap());
+    if (v instanceof JSONArray a)
+      return a.toList();
+    return v;
+  }
+
+  static Object untag(final Object v) {
     if (v instanceof Map<?, ?> m && m.size() == 1 && m.get("\u0001") instanceof java.util.List<?> t && t.size() == 2
         && t.get(0) instanceof String kind && t.get(1) instanceof String s) {
       switch (kind) {

src/java/com/arcadedb/python/EdgeBatcher.java

@@ -6,8 +6,8 @@
  * whole edge list as String[] RIDs (marshaled in one bulk array copy) and
  * loops Java-side, so Python pays one crossing per batch.
  *
- * Property-less edges only — the common bulk-ingest shape; edges with
- * properties keep the per-edge path.
+ * newEdgesJson takes edges with properties too, in the value shapes of
+ * VertexBatcher.createVerticesJson (JSON values and tagged date-times).
  */
 package com.arcadedb.python;
 
@@ -44,8 +44,8 @@ public static void newEdgesJoined(final GraphBatch batch, final String joinedSou
 
   /**
    * Bulk edges WITH properties: one JSON array of rows, each
-   * {"_src": "#1:0", "_dst": "#1:1", ...properties}. JSON-representable
-   * property values only; the Python side falls back per-edge otherwise.
+   * {"_src": "#1:0", "_dst": "#1:1", ...properties}. Property values are JSON values and the tagged values
+   * DocumentBatcher.untag decodes; the Python side falls back per-edge otherwise.
    */
   public static void newEdgesJson(final GraphBatch batch, final String edgeType, final String jsonRows) {
     final com.arcadedb.serializer.json.JSONArray rows = new com.arcadedb.serializer.json.JSONArray(jsonRows);
@@ -59,7 +59,7 @@ public static void newEdgesJson(final GraphBatch batch, final String edgeType, f
         if (key.equals("_src") || key.equals("_dst"))
           continue;
         props[j++] = key;
-        props[j++] = row.isNull(key) ? null : row.get(key);
+        props[j++] = DocumentBatcher.value(row.isNull(key) ? null : row.get(key));
       }
       if (props.length == 0)
         batch.newEdge(src, edgeType, dst);

src/java/com/arcadedb/python/VertexBatcher.java

@@ -7,9 +7,10 @@
  * ~0.2us/vertex Python-side) and returns the created RIDs as ONE
  * semicolon-joined string, so the whole batch costs two bulk string copies.
  *
- * JSON-representable property values only (str/int/float/bool/null and
- * nested lists/maps thereof) — the Python side falls back to the matrix
- * path for anything else (e.g. datetime, bytes).
+ * Property values are str/int/float/bool/null, nested lists and maps thereof,
+ * and the tagged values DocumentBatcher.untag decodes (date-time, date,
+ * BigDecimal, byte[]) sitting directly in a row; the Python side falls back to
+ * the matrix path for anything else.
  */
 package com.arcadedb.python;
 
@@ -44,7 +45,7 @@ public static String createVerticesJson(final GraphBatch batch, final String typ
       int j = 0;
       for (final String key : row.keySet()) {
         flat[j++] = key;
-        flat[j++] = row.isNull(key) ? null : row.get(key);
+        flat[j++] = DocumentBatcher.value(row.isNull(key) ? null : row.get(key));
       }
       for (int c = 0; c < extra; c++) {
         flat[j++] = vectorNames[c];