sqlmesh run keeps reporting "environment is being updated" after the updating plan has failed #6157

Activity

1. lafirm commented on Oct 21, 2026

   

   lafirm

   on Oct 21, 2026

   ContributorAuthor

   More actions

   MRE

   The following standalone DuckDB example intentionally fails during virtual-layer promotion, after the environment record is updated but before finalization. So that we can avoid needing a long-running concurrent query: a subsequent run command should encounter the same persisted unfinalized state.

   Use a fresh, disposable directory with SQLMesh 0.236.3 and DuckDB support installed.

   Create config.yaml:

   gateways:
     local:
       connection:
         type: duckdb
         database: repro.duckdb
   
   default_gateway: local
   
   model_defaults:
     dialect: duckdb
   
   disable_anonymized_analytics: true
   
   run:
     environment_check_interval: 2
     environment_check_max_wait: 20

   Create models/example.sql:

   MODEL (
     name repro.example,
     kind FULL
   );
   
   SELECT 1::INT AS id;

   Apply the initial successful plan:

   sqlmesh plan prod --auto-apply

   Then replace models/example.sql with:

   MODEL (
     name repro.example,
     kind FULL
   );
   
   SELECT 2::INT AS id;
   
   ON_VIRTUAL_UPDATE_BEGIN;
   SELECT CAST('intentional_promotion_failure' AS INTEGER);
   ON_VIRTUAL_UPDATE_END;

   Apply the changed plan:

   sqlmesh plan prod --auto-apply

   The virtual-update statement intentionally raises a conversion error.

   After the plan exits unsuccessfully, execute:

   sqlmesh run prod --skip-janitor

   Observe repeated warnings referencing the failed plan:

   Environment 'prod' is being updated by plan '<failed_plan_id>'. Retrying in 2 seconds...
   

   With this config, the run should eventually fail after approximately 20 seconds. Without the override, the default maximum wait is 6 hours.
2. StuffbyYuki

   added

   ImprovementImproves existing functionalityImproves existing functionality

   on Oct 22, 2026
3. bosprimigenious commented on Nov 3, 2026

   

   bosprimigenious

   on Nov 3, 2026

   More actions

   I independently reproduced the failed-promotion example on pristine main c4dbac4 with real local DuckDB. After the plan exits with the deliberate conversion error, the stored environment remains unfinalized and run waits for the configured 20 seconds. The warning cannot establish that the plan process is still alive.

   Draft PR #6191 changes the diagnostics and Run documentation: report the unfinalized state, show the configured wait, and explain inspecting the plan logs, fixing the error, and successfully reapplying sqlmesh plan <environment>. It preserves the guard, state records and retry/timeout behavior; it does not clear state or bypass finalization.

   The added real DuckDB regression verifies that scheduling is blocked, the stored failed state is unchanged, the run times out, and fixing/reapplying the plan restores scheduling. It fails against the original message and passes after the change (1 passed, 108 existing Pydantic deprecation warnings). I also repeated the full fresh CLI sequence before/after: initial plan/run success, intentional promotion failure, timeout, successful recovery plan/run; socket interception recorded no connection attempts.

   The PR remains draft/NOT READY: full style still has the same 125 normalized mypy errors as baseline, and full fast-test collection has 23 optional dependency errors. Trino/PostgreSQL and full engine CI remain unverified. The PR discloses Codex implementation and agent-only verification/review.