Skip to content

fix(tests): stop Windows xdist workers from sharing DuckDB and sushi files - #6184

Open
nicksspirit wants to merge 5 commits into
mainfrom
fix/windows-ci-file-locks
Open

nicksspirit wants to merge 5 commits into
mainfrom
fix/windows-ci-file-locks

Conversation

@nicksspirit

@nicksspirit nicksspirit commented Nov 2, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

 tests/core/engine_adapter/test_duckdb.py
-  duck_conn.install_extension("ducklake")
+  ensure_ducklake(duck_conn)
     test_ducklake_partitioning
     test_drop_table_ducklake_no_cascade
     test_drop_table_ducklake_no_cascade_mixed_case_alias

 tests/core/linter/test_helpers.py
-  Context(paths=["examples/sushi"])
+  Context(paths=copy_to_temp_path("examples/sushi"))
     test_get_position_of_model_block
     test_get_range_of_a_key_in_model_block_testing_on_sushi

 .github/workflows/pr.yaml
+  install ducklake once before Windows pytest

Parallel pytest-xdist workers were sharing the DuckDB extension install and the sushi .cache/optimized_query directory. This unblocks #6176.

DuckLake is loaded first. install_extension runs only when that load fails, and at most once per process. duckdb_extensions().installed stays false on a new connection after an earlier connection in the same process has already loaded the DLL, so a per-connection install check still calls install_extension. On Windows that replaces ducklake.duckdb_extension while the DLL is mapped and fails with Could not move file: Access is denied. Closing the connection does not unmap the DLL. Auto-install is turned off so LOAD cannot download and move the file either. The Windows job also installs DuckLake once before pytest, so the test processes only load it.

Other tests still construct Context directly on examples/sushi and share that cache. This PR only isolates the two that failed. Worth a follow-up if more of them flake on Windows.

Evidence

  • Before: Windows CI on PR 6176 (job 125057540476): FAILED test_drop_table_ducklake_no_cascade_mixed_case_alias (Access is denied); FAILED test_get_range_of_a_key_in_model_block_testing_on_sushi (Permission denied on the sushi cache). 2 failed, 2768 passed.
  • This branch, previous heads: Windows CI failed three times. Grouping alone (run 41408530674) and explicit close() (run 41478055750) failed in test_drop_table_ducklake_no_cascade_mixed_case_alias. The duckdb_extensions().installed check (run 41636441149) still called install_extension and failed in test_drop_table_ducklake_no_cascade. The sushi cache failure did not return.
  • Local: macOS, the two install-once unit tests and the three DuckLake tests passed.
  • After: Windows CI on 2514d48 (job 126298523258): cicd-tests-windows passed, including the DuckLake pre-install step and fast unit tests.

Merge Danger

Door: two-way

Revert the commit. Test isolation only.

Blast Radius: CI

These tests no longer share files across xdist workers. Production engines are unchanged.

- tests/core/engine_adapter/test_duckdb.py: three tests that call
  duck_conn.install_extension("ducklake") now share the same
  xdist_group("ducklake_extension"). On Windows, parallel xdist workers
  all downloading and renaming the extension binary at once triggers
  'Could not move file: Access is denied'. Grouping them forces the
  three tests onto a single worker, so extension installation is always
  sequential.

- tests/core/linter/test_helpers.py: both sushi-context tests now use
  the copy_to_temp_path fixture instead of pointing directly at
  examples/sushi. The shared .cache/optimized_query directory is
  excluded by the copytree ignore rules, so each test gets its own
  writable cache root and Windows file-lock contention is eliminated.

Signed-off-by: Nick Muoh <nickspirit3+github@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
@coderabbitai

coderabbitai Bot commented Nov 2, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • Resume reviews
  • Trigger review

No actionable comments were generated in the recent review.

Walkthrough

DuckLake tests now share extension setup, and the DuckDB connection fixture closes connections during teardown. Windows CI installs and loads DuckLake before fast unit tests. Two linter helper tests use temporary copies of the sushi examples.

Changes: DuckLake test setup and connection cleanup

Layer / File(s) Summary
Set up DuckLake tests
tests/core/engine_adapter/test_duckdb.py
A shared helper attempts to load DuckLake before installing it. Tests cover load success and install fallback, and DuckLake integration tests use the shared fixture.
Close connections and prepare Windows CI
tests/conftest.py, .github/workflows/pr.yaml
The duck_conn fixture closes its connection during teardown. Windows CI installs and loads DuckLake before running fast unit tests.

Linter test paths

Layer / File(s) Summary
Use temporary example copies
tests/core/linter/test_helpers.py
Two helper tests pass a temporary copy of examples/sushi to Context.

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Other

Possibly related issues

  • SQLMesh/sqlmesh#6182: The issue reports Windows xdist workers racing during DuckLake installation. This PR changes the setup used by the same DuckLake tests.

Possibly related PRs

  • SQLMesh/sqlmesh#6183: Both PRs address concurrent DuckLake extension installation in the same tests.

Suggested reviewers: kingjaiteh

Merge Risk: Minimal · previous review 2514d

No material code risk was identified. The reported DCO check failure still needs attention.

Pre-merge checks: Passed 3 · Failed 2

Failed Failed checks: 2 warnings

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 25.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documenta… Write docstrings for the functions missing them to satisfy the coverage threshold.
Description check Warning The description gives a detailed, relevant summary of the test-isolation changes and includes test evidence. It does not follow the repository template because it omits the required Description, Test … Add the required template sections. Rename or supplement Summary with Description, add a Test Plan that lists the commands and environments tested, and include the complete Checklist with each applicable item marked.
Passed Passed checks: 3 passed
Check name Status Explanation
Title check Passed The title clearly identifies the main change: preventing Windows xdist workers from sharing DuckDB and sushi files.
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Docstring Coverage

Explanation

Docstring coverage is 25.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 4 functions across 3 files. (1 skipped: 1 unsupported.)

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

Full details: Description check

Explanation

The description gives a detailed, relevant summary of the test-isolation changes and includes test evidence. It does not follow the repository template because it omits the required Description, Test Plan, and Checklist sections.

Resolution

Add the required template sections. Rename or supplement Summary with Description, add a Test Plan that lists the commands and environments tested, and include the complete Checklist with each applicable item marked.

Finishing Touches
Generate docstrings
  • Commit to this branch
  • Create a new PR
Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Tip

🎯 Customize your risk assessments

Guide how CodeRabbit assesses merge risk and security architecture risk. Highlight sensitive changes and explain your security boundaries using Merge Risk Instructions and Security Risk Instructions in your review settings. Security architecture risk assessments require the Advanced plan or above.

✅ Automatic approvals, with your rules

Let CodeRabbit approve pull requests once required threads are resolved, pre-merge checks pass, and your risk limits are met. Enable Automatic approvals in review settings, with overrides for teams or file paths.

Recent review info
Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 1afdf145-67a4-4651-a6fc-ffdb77b95e4c
Commits

Reviewing files that changed from the base of the PR and between b7aa915 and 2514d48.

Files selected for processing: 3
  • .github/workflows/pr.yaml
  • tests/conftest.py
  • tests/core/engine_adapter/test_duckdb.py
Files skipped from review as they are similar to previous changes: 1
  • tests/conftest.py

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 9 remain after this review.

nicksspirit and others added 4 commits November 2, 2026 08:03
… locks

On Windows, a loaded DuckDB extension (.dll) holds an OS file handle that
blocks install_extension() from atomically moving the newly-downloaded file
into place.  The duck_conn fixture previously only returned the connection
(no explicit close), relying on CPython reference-counting to tear it down
between tests.  With the adapter fixture holding the connection via a lambda
closure, and pytest-xdist running three ducklake tests sequentially on the
same worker, two stale connections can still hold the ducklake.dll open by
the time the third test calls install_extension(), producing:

  _duckdb.IOException: IO Error: Could not move file: Access is denied.

Change the fixture to yield + conn.close() so the connection — and any DLLs
it has loaded — is fully released before the next test begins.

Signed-off-by: Nick Muoh <nickspirit3+github@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Windows keeps ducklake.dll mapped after the connection closes, so a later
test's install_extension cannot move the file into place.

Signed-off-by: Nick Muoh <nickspirit3+github@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
A fixture loads DuckLake once per test without a second install call
while the DLL is still mapped, which on Windows causes 'Access is
denied' when install_extension tries to move ducklake.dll.

Signed-off-by: Nick Muoh <nickspirit3+github@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
@mday-io

mday-io commented Nov 3, 2026

Copy link
Copy Markdown
Collaborator

The sushi copy_to_temp_path change looks good: on main those two tests write examples/sushi/.cache into the repo, and with this PR they don't. The DuckLake part still fails on Windows, though.

Blocking

  1. The DuckLake fix doesn't hold on Windows. On the current head (b7aa915), cicd-tests-windows fails with:

    ERROR at setup of test_drop_table_ducklake_no_cascade
    >           duck_conn.install_extension("ducklake")
    E           _duckdb.IOException: IO Error: Could not move file: Access is denied.
    

    So the installed check in duckdb_extensions() said ducklake wasn't installed, and the install ran anyway, even with all three tests pinned to one worker via loadgroup. The earlier commits (grouping alone, then close()) also failed on Windows. A sturdier approach might be to try load_extension("ducklake") first and only install if that raises, or to pre-install ducklake once in a CI step before pytest. Either way, please show it green on Windows CI before we merge, since Linux and macOS can't reproduce this.

  2. The description is out of date. It says Windows CI hasn't re-run this branch, but it has run three times and failed each time. Please update the evidence section once there's a passing run.

Non-blocking

  1. The comment on duck_conn in tests/conftest.py says closing the connection releases loaded extension DLLs. DuckDB doesn't unload extensions when a connection closes, and the CI result fits that. The yield/close() change is fine to keep, but please reword or drop that explanation. The same goes for the ducklake_conn docstring if the approach changes.
  2. About 40 other tests still use Context(paths=["examples/sushi"]) directly and share the same cache. It's fine to fix only the two that flaked here, but worth a note in the description or a follow-up issue.

The 3.9 failure (test_freeze_time_concurrent) looks unrelated to this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants