Repository navigation
Conversation
…l with a list or tuple input (closes #235) The collector ran inside the call and re-traversed the caller's live row list again and again (14.5% of the calling thread's samples in a profile). A/B on one CI wheel (python-gc-ingest-ab, DECISIONS #238 addendum TA-2 and #239): insert wall 95% CI [0.819, 0.903] of the default, whole load wall [0.861, 0.923], peak RSS [0.924, 1.057], A/A control clean. Only a list or a tuple is paused (no user code runs between the library's chunk reads); a generator or other lazy input keeps the collector, since cycles built between chunks would pile up. The previous state is restored in finally. The native build is not changed (no collection ran inside its call). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tae898
added a commit
that referenced
this pull request
Nov 3, 2026
…calls it (as the JVM change in #236), no re-entry through the public method Re-entering self.insert_many made an instance-level spy or a subclass override run twice per call. insert_many keeps its signature, __defaults__ and docstring, checks the database is open, pauses the collector for a list or tuple (restoring in finally), and passes commit_every explicitly to the body, so the columnar fixture's patched default still reaches it. A test asserts that a spy on insert_many sees one call for a list. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ervals instead of 'did not rise' (PR 236) Figures generated from python-gc-ingest-ab-laptop-20261025/ab/ab-score.md (arm C). Docs only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #235
On the
arcadedb-jvmbuild,insert_manynow pauses Python's cyclic garbage collector for the duration of a call whenrowsis a list or a tuple, and restores the previous state infinally(re-enabled only if it was enabled on entry). A generator, an iterator, a DataFrame or any other lazy input is not paused. The native build is not changed.The change is the arm C of the study
python-gc-ingest-ab-laptop-20261025(HumemAI benchmark notes), adopted by the decision in DECISIONS #238 addendum on TA-2 and #239: the study's rule was written before the first timed run and C passed it. Wheel measured:arcadedb-jvm26.11.1.dev0 from CI run 41646452646 on main413f54ffe1; 500,000 dict rows in slices of 100,000, ABBA order,ab.py --noise, 95% CI of the median ratio against the default:gc.disable()gc.disable()gc.disable()The shipped form follows the risk section of the study README: the guard to a list or a tuple is what keeps a generator that builds reference cycles from piling up garbage for the whole load.
gc.disable()is process wide, so with overlapping calls from several threads the pause does not last for the whole of each call, and it is never left stuck off.Code:
Database.insert_manyinsrc/arcadedb/core.pyrecordsgc.isenabled(), callsgc.disable()for a list or tuple, runs the unchanged body (moved to_insert_many_nogc) and restores infinally. Docs: the docstring and a new section on the bulk-loads page, with no speed figure. Tests:tests/test_insert_many_gc_pause.py(JVM only, listed inNATIVE_NOT_APPLICABLE) observesgc.isenabled()from inside the call for a list and a tuple, several chunks, the parallel mode, an exception, a collector the caller had disabled, and a generator, an iterator and a generator expression.🤖 Generated with Claude Code
Docs updated in fdc16ea (docs and docstrings only).
Edited 2026-11-04: added the docs line above.