Skip to content

fix(tests): isolate DuckLake extension caches per worker - #6193

Open
mday-io wants to merge 1 commit into
SQLMesh:mainfrom
mday-io:michael-day_css/ducklake-cache-isolation
Open

mday-io wants to merge 1 commit into
SQLMesh:mainfrom
mday-io:michael-day_css/ducklake-cache-isolation

Conversation

@mday-io

@mday-io mday-io commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Concurrent pytest workers can install DuckLake into the same default DuckDB extension cache and interfere with each other. Give each worker a cache under its pytest base temp directory, use it for the three DuckLake adapter tests, and verify both the configured directory and the installed extension path.

Extracted from #6173 so this test repair can land independently. The patch contains no provider or cancellation changes.

Test Plan

  • All four DuckLake cases passed serially and with two pytest workers.
  • The complete DuckDB adapter module passed all 16 tests.
  • make style passed. The initial run found the declared httpx2 dependency missing; installing it in the private venv resolved the failure.
  • All six make fast-test phases passed (3,010 passed, 13 skipped), with automatic worker count limited to two.

Checklist

  • I have run make style and fixed any issues
  • I have added tests for my changes (if applicable)
  • All existing tests pass (make fast-test)
  • My commits are signed off (git commit -s) per the DCO

Summary by CodeRabbit

  • Tests
    • Expanded automated coverage for DuckLake extension installation and catalog behavior, including checks for worker-specific extension paths and existing partitioning and cascade-drop scenarios. These changes affect test coverage only; no end-user functionality changes are noted.

Signed-off-by: mday-io <mdaytn@gmail.com>
@coderabbitai

coderabbitai Bot commented Nov 4, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

No actionable comments were generated in the recent review.

Walkthrough

DuckLake tests now use a fixture that installs and loads the extension in a directory under the pytest worker’s temporary directory. A new test checks the configured directory and DuckLake’s install path.

Changes: DuckLake test setup

Layer / File(s) Summary
Worker-local extension fixture
tests/core/engine_adapter/test_duckdb.py
The fixture configures an extension directory under the worker’s temporary directory, installs and loads DuckLake, and provides the connection. A test checks the directory and install path. Partitioning and cascade-drop tests now use the fixture instead of installing the extension themselves.

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Other · Severity of issue fixed: Medium

Possibly related issues

  • SQLMesh/sqlmesh issue 6182 describes concurrent DuckLake extension installs failing under pytest-xdist. This change configures the extension directory under the worker’s temporary directory.

Possibly related PRs

  • SQLMesh/sqlmesh#6184: Both PRs change setup for the DuckLake tests to address concurrent extension installation, using different approaches.
  • SQLMesh/sqlmesh#6183: Both PRs address the concurrent installation issue in the same tests, using different approaches.

Suggested reviewers: kingjaiteh, stuffbyyuki

Merge Risk: Minimal · up to 50b6c

This change only affects test setup. Each test worker now installs DuckLake into its own temporary directory, which avoids cache collisions between parallel workers. Offline local runs that depended on a preinstalled DuckLake may now fail, but CI is unaffected. No merge-blocking risk remains.

Pre-merge checks: Passed 5
Check name Status Explanation
Title check Passed The title names the main change: per-worker DuckLake extension caches for the test suite. The raw summary confirms the fixture sets a worker-specific extension directory under pytest's base temp direc…
Description check Passed The description states the problem (concurrent pytest workers sharing the default DuckDB extension cache), the fix (a per-worker cache under the pytest base temp directory for the three DuckLake adapt…
Docstring Coverage Passed No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentat…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Title check

Explanation

The title names the main change: per-worker DuckLake extension caches for the test suite. The raw summary confirms the fixture sets a worker-specific extension directory under pytest's base temp directory, and the PR objectives describe the same change. The title is specific and accurate.

Full details: Description check

Explanation

The description states the problem (concurrent pytest workers sharing the default DuckDB extension cache), the fix (a per-worker cache under the pytest base temp directory for the three DuckLake adapter tests), and verification (configured directory and installed extension path). The Test Plan lists the commands run and their results. The Checklist follows the template and is marked complete. The description does not mention the new test explicitly beyond the verification sentence, but it covers the required sections and the key information.

Full details: Docstring Coverage

Explanation

No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 0 functions across 1 files.

Finishing Touches
Generate unit tests (beta)
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Recent review info
Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 64d99249-561c-47bc-a672-e5c1cac2fc96
Commits

Reviewing files that changed from the base of the PR and between c4dbac4 and 50b6cf2.

Files selected for processing: 1
  • tests/core/engine_adapter/test_duckdb.py

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 9 remain after this review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant