From f80eeb357097aa2bd88ec372e82ec37462cb5e59 Mon Sep 17 00:00:00 2001 From: Taewoon Kim Date: Wed, 4 Nov 2026 06:07:43 +0900 Subject: [PATCH 1/2] insert_many (JVM build): pause the cyclic garbage collector for a call with a list or tuple input (closes #235) The collector ran inside the call and re-traversed the caller's live row list again and again (14.5% of the calling thread's samples in a profile). A/B on one CI wheel (python-gc-ingest-ab, DECISIONS #238 addendum TA-2 and #239): insert wall 95% CI [0.819, 0.903] of the default, whole load wall [0.861, 0.923], peak RSS [0.924, 1.057], A/A control clean. Only a list or a tuple is paused (no user code runs between the library's chunk reads); a generator or other lazy input keeps the collector, since cycles built between chunks would pile up. The previous state is restored in finally. The native build is not changed (no collection ran inside its call). Co-Authored-By: Claude Opus 5.5 --- .../modules/ROOT/pages/how-to/bulk-loads.adoc | 12 +- src/arcadedb/core.py | 35 +++ tests/conftest.py | 1 + tests/test_insert_many_gc_pause.py | 230 ++++++++++++++++++ 4 files changed, 277 insertions(+), 1 deletion(-) create mode 100644 tests/test_insert_many_gc_pause.py diff --git a/docs/modules/ROOT/pages/how-to/bulk-loads.adoc b/docs/modules/ROOT/pages/how-to/bulk-loads.adoc index 148c077..cb4f427 100644 --- a/docs/modules/ROOT/pages/how-to/bulk-loads.adoc +++ b/docs/modules/ROOT/pages/how-to/bulk-loads.adoc @@ -1,5 +1,5 @@ = Bulk loads and commit size -:description: The default commit size of insert_many, graph_batch and create_vertices on the arcadedb (native) and arcadedb-jvm packages, what it means for a failed load, what to do after an interrupted graph batch, how to change it, and a known limitation: a graph batch does not reuse the space of deleted edges. +:description: The default commit size of insert_many, graph_batch and create_vertices on the arcadedb (native) and arcadedb-jvm packages, what it means for a failed load, what to do after an interrupted graph batch, how to change it, how insert_many treats Python's cyclic garbage collector on arcadedb-jvm, and a known limitation: a graph batch does not reuse the space of deleted edges. `insert_many`, `graph_batch` and `GraphBatch.create_vertices` load data in transactions. The commit size is the number of rows in one transaction. It decides how much work the engine holds in memory before it commits, and how many rows stay in the database when a load fails. @@ -67,6 +67,16 @@ with db.graph_batch(commit_every=5000) as batch: # also sets the create_vertice ... ---- +== The cyclic garbage collector during `insert_many` (`arcadedb-jvm`) + +On `arcadedb-jvm`, `insert_many` pauses Python's cyclic garbage collector for the length of a call when `rows` is a list or a tuple, and puts it back as it was when the call ends, also when the call raises. The call allocates many objects that the collector tracks. Without the pause, the collector goes through your whole row list again and again during the load, which makes the load slower. The call itself creates no reference cycles from a list or a tuple of plain rows, and the peak memory of the load did not rise in our measurement. + +A generator or any other lazy input is not paused. Your own code runs between the chunks that the generator yields, and that code can create reference cycles that only the collector frees. If your generator builds plain dicts, you can call `gc.disable()` around the load yourself. + +`gc.disable()` is process-wide. While a call with a list runs, other Python threads do not collect cyclic garbage automatically. When calls from several threads overlap, the pause does not last for the whole of each call: the collector comes back when the call that paused it ends. The collector is never left disabled by a call, and a collector that you disabled yourself stays disabled. + +The `arcadedb` (native) package does not change the collector. In our measurements, no collection ran inside a native `insert_many` call, so there is nothing to pause. + == Graph batches next to deletes (fixed in ArcadeDB 26.11.1) NOTE: *Fixed in ArcadeDB 26.11.1.* Before 26.11.1, a graph batch did not reuse the space of deleted edges (https://github.com/ArcadeData/arcadedb/issues/10746[ArcadeData/arcadedb#10746]). The engine wrote the edge records of each `graph_batch()` flush onto new pages and never looked at the free space that deletes left on older pages, so the edge bucket gained one 64 KiB page per flush while the live data stayed flat. From 26.11.1 (https://github.com/ArcadeData/arcadedb/pull/10836[ArcadeData/arcadedb#10836]) each edge goes where an ordinary insert would put it while an existing page has room, and only the rest of a flush goes onto new pages, so a large import stays as fast as before. The 26.11.1 packages ship this fix. The `26.11.1.devN` pre-releases embed engine 26.10.2 and still show the growth described below. diff --git a/src/arcadedb/core.py b/src/arcadedb/core.py index 08fb8e1..97966bf 100644 --- a/src/arcadedb/core.py +++ b/src/arcadedb/core.py @@ -4,6 +4,7 @@ Database and DatabaseFactory classes for embedded database access. """ +import gc import itertools from collections.abc import Mapping from datetime import date @@ -719,6 +720,17 @@ def insert_many( property to float. A vector that is ragged, missing or null in a row stays in the JSON. + When ``rows`` is a list or a tuple, Python's cyclic garbage collector + is paused for the duration of the call and put back as it was when the + call ends, on success and on an exception alike. The call allocates + many objects the collector tracks, and without the pause it re-traverses + the caller's whole row list again and again, which costs a measurable + part of the load time on this build (see the bulk-loads docs page). A + generator or any other lazy iterable is not paused, because user code + that runs between the chunks can create reference cycles that only the + collector frees. ``gc.disable()`` is process wide, so other threads do + not collect automatically while a call with a list runs. + Args: type_name: Target type (must exist): a document type gets documents, a vertex type gets vertices (use ``graph_batch()`` @@ -761,6 +773,29 @@ def insert_many( call waits for them before raising. """ self._check_not_closed() + if not isinstance(rows, (list, tuple)): + # A lazy input runs user code between chunks that may create + # reference cycles, so the collector keeps running. + return self._insert_many_nogc( + type_name, rows, commit_every=commit_every, parallel=parallel + ) + # A sized input: no user code runs between the library's chunk reads, so + # the cyclic collector is paused for the call (the caller's live row list + # is not re-traversed while the call allocates) and the previous state is + # restored in finally: re-enabled only if it was enabled on entry. + # Measured in ArcadeData/arcadedb-python#235 (A/B: python-gc-ingest-ab). + gc_was_enabled = gc.isenabled() + gc.disable() + try: + return self._insert_many_nogc( + type_name, rows, commit_every=commit_every, parallel=parallel + ) + finally: + if gc_was_enabled: + gc.enable() + + def _insert_many_nogc(self, type_name, rows, commit_every=10_000, parallel=False): + """The body of insert_many, without the pause of the cyclic collector.""" # Rows are read from the iterable one chunk at a time and each chunk # crosses as its own JSON text, so the memory a load holds is bounded # by the chunk, not by the input. Reading the whole input into a list diff --git a/tests/conftest.py b/tests/conftest.py index 01ab332..c9b2f27 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -97,6 +97,7 @@ "test_build_no_uv_refresh.py": "build scripts of the JVM wheel", "test_build_wheel_check.py": "build scripts of the JVM wheel", "test_query_vector_params_jvm.py": "the java.util objects the JVM build's parameter conversion builds", + "test_insert_many_gc_pause.py": "the cyclic collector pause is the JVM build's (the native insert_many allocates nothing the collector tracks)", } if BACKEND == "native" and not os.environ.get("NATIVE_PARITY_ALL"): collect_ignore.extend(NATIVE_NOT_APPLICABLE) diff --git a/tests/test_insert_many_gc_pause.py b/tests/test_insert_many_gc_pause.py new file mode 100644 index 0000000..9c8e1b3 --- /dev/null +++ b/tests/test_insert_many_gc_pause.py @@ -0,0 +1,230 @@ +"""insert_many pauses CPython's cyclic garbage collector for the call when the input is a list or a tuple (JVM build). + +The call allocates many objects the collector tracks (JPype crossings, chunk lists), and without the pause the collector re-traverses the +caller's live row list again and again (ArcadeData/arcadedb-python#235). A lazy input (a generator or any other iterator) is not paused: +user code that runs between the chunks can create reference cycles that only the collector frees. The previous state is restored in +``finally``: the collector comes back on only when it was on at entry, also after an exception. + +The state is observed from inside the call: ``Database._insert_many_chunk`` (and ``_insert_many_parallel``) are wrapped to record +``gc.isenabled()`` and then run unchanged, so the rows are really stored and counted. +""" + +import gc + +import pytest + +from arcadedb.core import Database +from arcadedb.exceptions import ArcadeDBError + +ROWS = 25 + + +def _count(db, type_name): + q = f"SELECT count(*) AS n FROM {type_name}" # nosec B608 - test-controlled + return int(db.query("sql", q).to_list()[0]["n"]) + + +@pytest.fixture +def collector_state(): + """Remember the collector's state and put it back, so a failing test cannot leave it off for the rest of the run.""" + was = gc.isenabled() + yield + if was: + gc.enable() + else: + gc.disable() + + +@pytest.fixture +def seen(monkeypatch): + """gc.isenabled() as seen inside each chunk insert and inside the parallel writer path of insert_many.""" + states = [] + real_chunk = Database._insert_many_chunk + real_parallel = Database._insert_many_parallel + + def chunk(self, *args, **kwargs): + states.append(gc.isenabled()) + return real_chunk(self, *args, **kwargs) + + def parallel(self, *args, **kwargs): + states.append(gc.isenabled()) + return real_parallel(self, *args, **kwargs) + + monkeypatch.setattr(Database, "_insert_many_chunk", chunk) + monkeypatch.setattr(Database, "_insert_many_parallel", parallel) + return states + + +@pytest.fixture +def doc_type(temp_db): + temp_db.command("sql", "CREATE DOCUMENT TYPE GcDoc") + return "GcDoc" + + +def _rows(): + return [{"id": i, "name": f"row_{i}"} for i in range(ROWS)] + + +@pytest.mark.parametrize("make", [list, tuple], ids=["list", "tuple"]) +def test_the_collector_is_paused_during_a_call_with_a_sized_input( + temp_db, doc_type, seen, collector_state, make +): + gc.enable() + + written = temp_db.insert_many(doc_type, make(_rows())) + + assert written == ROWS + assert _count(temp_db, doc_type) == ROWS + assert seen == [False] + + +@pytest.mark.parametrize("make", [list, tuple], ids=["list", "tuple"]) +def test_the_collector_is_enabled_again_after_the_call( + temp_db, doc_type, seen, collector_state, make +): + gc.enable() + + temp_db.insert_many(doc_type, make(_rows())) + + assert gc.isenabled() + + +def test_a_call_that_uses_several_chunks_is_paused_for_all_of_them( + temp_db, doc_type, seen, collector_state +): + gc.enable() + rows = [{"id": i} for i in range(25)] + + # commit_every=10 splits the 25 rows into chunks of 10, 10 and 5 + assert temp_db.insert_many(doc_type, rows, commit_every=10) == 25 + + assert seen == [False, False, False] + assert gc.isenabled() + + +def test_a_parallel_call_with_a_list_is_paused_too( + temp_db, doc_type, seen, collector_state +): + gc.enable() + + written = temp_db.insert_many(doc_type, _rows(), parallel=True) + + assert written == ROWS + assert _count(temp_db, doc_type) == ROWS + assert seen == [False] + assert gc.isenabled() + + +def test_the_collector_is_enabled_again_after_an_exception( + temp_db, doc_type, monkeypatch, collector_state +): + gc.enable() + states = [] + + def failing_chunk(self, *args, **kwargs): + states.append(gc.isenabled()) + raise RuntimeError("chunk failed") + + monkeypatch.setattr(Database, "_insert_many_chunk", failing_chunk) + + with pytest.raises(RuntimeError, match="chunk failed"): + temp_db.insert_many(doc_type, _rows()) + + # the pause was in force when the failure happened, and the exception did not leave it on + assert states == [False] + assert gc.isenabled() + assert not temp_db.is_transaction_active() + + +def test_a_collector_the_caller_disabled_stays_disabled( + temp_db, doc_type, seen, collector_state +): + gc.disable() + + written = temp_db.insert_many(doc_type, _rows()) + + assert written == ROWS + assert seen == [False] + assert not gc.isenabled() + + +def test_a_collector_the_caller_disabled_stays_disabled_after_an_exception( + temp_db, doc_type, monkeypatch, collector_state +): + gc.disable() + + def failing_chunk(self, *args, **kwargs): + raise RuntimeError("chunk failed") + + monkeypatch.setattr(Database, "_insert_many_chunk", failing_chunk) + + with pytest.raises(RuntimeError, match="chunk failed"): + temp_db.insert_many(doc_type, _rows()) + + assert not gc.isenabled() + + +def test_a_generator_is_not_paused(temp_db, doc_type, seen, collector_state): + gc.enable() + in_generator = [] + + def rows(): + for row in _rows(): + in_generator.append(gc.isenabled()) + yield row + + written = temp_db.insert_many(doc_type, rows()) + + assert written == ROWS + assert _count(temp_db, doc_type) == ROWS + assert in_generator == [True] * ROWS + assert seen == [True] + assert gc.isenabled() + + +@pytest.mark.parametrize( + "make", [iter, lambda rows: (r for r in rows)], ids=["iterator", "genexp"] +) +def test_other_lazy_inputs_are_not_paused( + temp_db, doc_type, seen, collector_state, make +): + gc.enable() + + written = temp_db.insert_many(doc_type, make(_rows())) + + assert written == ROWS + assert seen == [True] + assert gc.isenabled() + + +def test_a_parallel_generator_is_not_paused(temp_db, doc_type, seen, collector_state): + gc.enable() + + written = temp_db.insert_many(doc_type, iter(_rows()), parallel=True) + + assert written == ROWS + assert seen == [True] + assert gc.isenabled() + + +def test_an_empty_list_leaves_the_collector_as_it_was( + temp_db, doc_type, collector_state +): + gc.enable() + assert temp_db.insert_many(doc_type, []) == 0 + assert gc.isenabled() + gc.disable() + assert temp_db.insert_many(doc_type, []) == 0 + assert not gc.isenabled() + + +def test_a_closed_database_raises_before_the_collector_is_touched( + temp_db, collector_state +): + gc.enable() + temp_db.close() + + with pytest.raises(ArcadeDBError, match="closed"): + temp_db.insert_many("GcDoc", _rows()) + + assert gc.isenabled() From fdc16ea51f16d8c971ab1928defe250ad6ed8d48 Mon Sep 17 00:00:00 2001 From: Taewoon Kim Date: Wed, 4 Nov 2026 11:06:00 +0900 Subject: [PATCH 2/2] Docs: bulk-loads gc pause section carries the study's ratios with intervals instead of 'did not rise' (PR 236) Figures generated from python-gc-ingest-ab-laptop-20261025/ab/ab-score.md (arm C). Docs only. Co-Authored-By: Claude Opus 5.5 --- docs/modules/ROOT/pages/how-to/bulk-loads.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/modules/ROOT/pages/how-to/bulk-loads.adoc b/docs/modules/ROOT/pages/how-to/bulk-loads.adoc index cb4f427..84c5737 100644 --- a/docs/modules/ROOT/pages/how-to/bulk-loads.adoc +++ b/docs/modules/ROOT/pages/how-to/bulk-loads.adoc @@ -69,7 +69,7 @@ with db.graph_batch(commit_every=5000) as batch: # also sets the create_vertice == The cyclic garbage collector during `insert_many` (`arcadedb-jvm`) -On `arcadedb-jvm`, `insert_many` pauses Python's cyclic garbage collector for the length of a call when `rows` is a list or a tuple, and puts it back as it was when the call ends, also when the call raises. The call allocates many objects that the collector tracks. Without the pause, the collector goes through your whole row list again and again during the load, which makes the load slower. The call itself creates no reference cycles from a list or a tuple of plain rows, and the peak memory of the load did not rise in our measurement. +On `arcadedb-jvm`, `insert_many` pauses Python's cyclic garbage collector for the length of a call when `rows` is a list or a tuple, and puts it back as it was when the call ends, also when the call raises. The call allocates many objects that the collector tracks. Without the pause, the collector goes through your whole row list again and again during the load, which makes the load slower. The call itself creates no reference cycles from a list or a tuple of plain rows. A laptop A/B on the `arcadedb-jvm` wheel measured the pause with `ab.py --noise` (500,000 dict rows in slices of 100,000, ABBA order, 12 clean pairs for each figure, clean A/A control). Against the default, the call took 0.860 of the time [0.819, 0.903] (95 % interval of the median pair ratio), the whole load 0.876 [0.861, 0.923], and the peak RSS 0.976 [0.924, 1.057], which is no detectable change. A generator or any other lazy input is not paused. Your own code runs between the chunks that the generator yields, and that code can create reference cycles that only the collector frees. If your generator builds plain dicts, you can call `gc.disable()` around the load yourself.