Repository navigation
fix(tests): stop Windows xdist workers from sharing DuckDB and sushi files - #6184
nicksspirit wants to merge 5 commits into
Conversation
- tests/core/engine_adapter/test_duckdb.py: three tests that call
duck_conn.install_extension("ducklake") now share the same
xdist_group("ducklake_extension"). On Windows, parallel xdist workers
all downloading and renaming the extension binary at once triggers
'Could not move file: Access is denied'. Grouping them forces the
three tests onto a single worker, so extension installation is always
sequential.
- tests/core/linter/test_helpers.py: both sushi-context tests now use
the copy_to_temp_path fixture instead of pointing directly at
examples/sushi. The shared .cache/optimized_query directory is
excluded by the copytree ignore rules, so each test gets its own
writable cache root and Windows file-lock contention is eliminated.
Signed-off-by: Nick Muoh <nickspirit3+github@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the
Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
No actionable comments were generated in the recent review. Walkthrough
Merge Risk:
|
… locks On Windows, a loaded DuckDB extension (.dll) holds an OS file handle that blocks install_extension() from atomically moving the newly-downloaded file into place. The duck_conn fixture previously only returned the connection (no explicit close), relying on CPython reference-counting to tear it down between tests. With the adapter fixture holding the connection via a lambda closure, and pytest-xdist running three ducklake tests sequentially on the same worker, two stale connections can still hold the ducklake.dll open by the time the third test calls install_extension(), producing: _duckdb.IOException: IO Error: Could not move file: Access is denied. Change the fixture to yield + conn.close() so the connection — and any DLLs it has loaded — is fully released before the next test begins. Signed-off-by: Nick Muoh <nickspirit3+github@gmail.com> Co-authored-by: Cursor <cursoragent@cursor.com>
Windows keeps ducklake.dll mapped after the connection closes, so a later test's install_extension cannot move the file into place. Signed-off-by: Nick Muoh <nickspirit3+github@gmail.com> Co-authored-by: Cursor <cursoragent@cursor.com>
A fixture loads DuckLake once per test without a second install call while the DLL is still mapped, which on Windows causes 'Access is denied' when install_extension tries to move ducklake.dll. Signed-off-by: Nick Muoh <nickspirit3+github@gmail.com> Co-authored-by: Cursor <cursoragent@cursor.com>
|
The sushi Blocking
Non-blocking
The 3.9 failure ( |
2514d48 to
b7aa915
Compare
Summary
Parallel pytest-xdist workers were sharing the DuckDB extension install and the sushi
.cache/optimized_querydirectory. This unblocks #6176.DuckLake is loaded first.
install_extensionruns only when that load fails, and at most once per process.duckdb_extensions().installedstays false on a new connection after an earlier connection in the same process has already loaded the DLL, so a per-connection install check still callsinstall_extension. On Windows that replacesducklake.duckdb_extensionwhile the DLL is mapped and fails withCould not move file: Access is denied. Closing the connection does not unmap the DLL. Auto-install is turned off soLOADcannot download and move the file either. The Windows job also installs DuckLake once before pytest, so the test processes only load it.Other tests still construct
Contextdirectly onexamples/sushiand share that cache. This PR only isolates the two that failed. Worth a follow-up if more of them flake on Windows.Evidence
FAILED test_drop_table_ducklake_no_cascade_mixed_case_alias(Access is denied);FAILED test_get_range_of_a_key_in_model_block_testing_on_sushi(Permission denied on the sushi cache). 2 failed, 2768 passed.close()(run 41478055750) failed intest_drop_table_ducklake_no_cascade_mixed_case_alias. Theduckdb_extensions().installedcheck (run 41636441149) still calledinstall_extensionand failed intest_drop_table_ducklake_no_cascade. The sushi cache failure did not return.cicd-tests-windowspassed, including the DuckLake pre-install step and fast unit tests.Merge Danger
Door: two-way
Revert the commit. Test isolation only.
Blast Radius: CI
These tests no longer share files across xdist workers. Production engines are unchanged.