Skip to content

Support shadow deployment history in bundle generate - #7204

Merged
shreyas-goenka merged 1 commit into
mainfrom
shreyas-goenka_data/stack/deployment-history-generate
Oct 28, 2026
Merged

Support shadow deployment history in bundle generate#7204
shreyas-goenka merged 1 commit into
mainfrom
shreyas-goenka_data/stack/deployment-history-generate

Conversation

@shreyas-goenka

@shreyas-goenka shreyas-goenka commented Oct 27, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Remove the dependence on DMS for bundle generate. The command should purely rely on resources.json to generate configuration.

@github-actions

github-actions Bot commented Oct 27, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Commit: 2532ede22 | Workflow run

Regression test report

Selected 3 changed tests (limit=20, 0 not selected)

6 tests: regression=2 unreleased=1 coverage=3 failing=0

test branch main (9c74505) latest (v1.22.1)
TestAccept/bundle/generate/auto-bind/DMS= ✅ ✅ ✅
TestAccept/bundle/generate/auto-bind/DMS=true ✅ ✅ ❌
TestAccept/bundle/generate/dashboard-inplace/DMS= ✅ ✅ ✅
TestAccept/bundle/generate/dashboard-inplace/DMS=true ✅ ❌ ❌
TestAccept/bundle/generate/genie_space_inplace/DMS= ✅ ✅ ✅
TestAccept/bundle/generate/genie_space_inplace/DMS=true ✅ ❌ ❌
TestAccept/bundle/generate/auto-bind/DMS=true: branch ✅ | main (9c74505) ✅ | latest (v1.22.1) ❌

latest (v1.22.1):

acceptance_test.go:1341: Diff:
        --- bundle/generate/auto-bind/output.txt
        +++ /tmp/TestAcceptbundlegenerateauto-bindDMS=true1945611196/002/output.txt
        @@ -10,37 +10,12 @@
         >>> [CLI] bundle generate job --key test_job --existing-job-id [JOB_ID] --config-dir resources --source-dir src --bind
         File successfully saved to src/test.py
         Job configuration successfully saved to resources/test_job.job.yml
        -Successfully bound job with an id '[JOB_ID]'
        +Error: bind is not supported for a bundle target that records deployment history
         
        ->>> ls src/
        -test.py
         
        ->>> cat resources/test_job.job.yml
        -      name: auto-bind-job-[UNIQUE_NAME]
        -
        -=== Deploy the bound job:
        ->>> [CLI] bundle deploy
        -Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/auto-bind-test-[UNIQUE_NAME]/files...
        -Updated jobs.test_job
        -Files: 5 uploaded, 0 deleted
        -Resources: 0 created, 1 changed, 0 deleted, 0 unchanged
        -
        -=== Destroy the bundle:
        ->>> [CLI] bundle destroy --auto-approve
        -The following resources will be deleted:
        -  delete resources.jobs.test_job
        -
        -All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/auto-bind-test-[UNIQUE_NAME]
        -
        -Destroy: 1 deleted
        -
        -=== Check that job is bound and does not exist after bundle is destroyed:
        ->>> errcode [CLI] jobs get [JOB_ID] --output json
        -Error: Job [JOB_ID] does not exist.
        -
        -Exit code: 1
        -
         === Delete the tmp folder:
         >>> [CLI] workspace delete /Workspace/Users/[USERNAME]/python-[UNIQUE_NAME]/test
         
         >>> [CLI] workspace delete /Workspace/Users/[USERNAME]/python-[UNIQUE_NAME]
        +
        +Exit code: 1
TestAccept/bundle/generate/dashboard-inplace/DMS=true: branch ✅ | main (9c74505) ❌ | latest (v1.22.1) ❌

main (9c74505):

acceptance_test.go:1341: Diff:
        --- bundle/generate/dashboard-inplace/output.txt
        +++ /tmp/TestAcceptbundlegeneratedashboard-inplaceDMS=true1663952589/002/output.txt
        @@ -28,6 +28,7 @@
         
         === update the dashboard file using bundle generate
         >>> [CLI] bundle generate dashboard --resource test_dashboard --force
        +Warn: Could not read deployment history: Fault injected by test.
         Writing dashboard to dash.lvdash.json
         
         >>> cat dash.lvdash.json
        @@ -36,3 +37,4 @@
         }
         
         >>> print_requests.py --dms //api/2.0/bundle --get --oneline
        +{"method": "GET", "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]"}

latest (v1.22.1):

acceptance_test.go:1341: Diff:
        --- bundle/generate/dashboard-inplace/output.txt
        +++ /tmp/TestAcceptbundlegeneratedashboard-inplaceDMS=true1804081324/002/output.txt
        @@ -28,11 +28,12 @@
         
         === update the dashboard file using bundle generate
         >>> [CLI] bundle generate dashboard --resource test_dashboard --force
        -Writing dashboard to dash.lvdash.json
        +Error: Fault injected by test. (403 PERMISSION_DENIED)
         
        ->>> cat dash.lvdash.json
        -{
        -  "a": "b"
        -}
        +Endpoint: GET [DATABRICKS_URL]/api/2.0/bundle/deployments/[DEPLOYMENT_ID]?
        +HTTP Status: 403 Forbidden
        +API error_code: PERMISSION_DENIED
        +API message: Fault injected by test.
         
        ->>> print_requests.py --dms //api/2.0/bundle --get --oneline
        +
        +Exit code: 1
        
FAIL
TestAccept/bundle/generate/genie_space_inplace/DMS=true: branch ✅ | main (9c74505) ❌ | latest (v1.22.1) ❌

main (9c74505):

acceptance_test.go:1341: Diff:
        --- bundle/generate/genie_space_inplace/output.txt
        +++ /tmp/TestAcceptbundlegenerategenie_space_inplaceDMS=true3201097330/002/output.txt
        @@ -24,6 +24,7 @@
         
         === update the genie space file using bundle generate
         >>> [CLI] bundle generate genie-space --resource test_genie_space --force
        +Warn: Could not read deployment history: Fault injected by test.
         Writing genie space to space.geniespace.json
         
         >>> cat space.geniespace.json
        @@ -32,3 +33,4 @@
         }
         
         >>> print_requests.py --dms //api/2.0/bundle --get --oneline
        +{"method": "GET", "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]"}
        
FAIL

latest (v1.22.1):

acceptance_test.go:1341: Diff:
        --- bundle/generate/genie_space_inplace/output.txt
        +++ /tmp/TestAcceptbundlegenerategenie_space_inplaceDMS=true3832044095/002/output.txt
        @@ -24,11 +24,12 @@
         
         === update the genie space file using bundle generate
         >>> [CLI] bundle generate genie-space --resource test_genie_space --force
        -Writing genie space to space.geniespace.json
        +Error: Fault injected by test. (403 PERMISSION_DENIED)
         
        ->>> cat space.geniespace.json
        -{
        -  "a": "b"
        -}
        +Endpoint: GET [DATABRICKS_URL]/api/2.0/bundle/deployments/[DEPLOYMENT_ID]?
        +HTTP Status: 403 Forbidden
        +API error_code: PERMISSION_DENIED
        +API message: Fault injected by test.
         
        ->>> print_requests.py --dms //api/2.0/bundle --get --oneline
        +
        +Exit code: 1

@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Oct 27, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Integration test report

Commit: 2532ede

Run: 40676666884

Env 🔄​flaky ✅​pass 🙈​skip Time
✅​ aws linux-2core-8gb 284 17 7:05
✅​ aws-windows-latest-4core-16gb 286 15 4:00
🔄​ azure linux-2core-8gb 2 281 17 6:48
✅​ azure-windows-latest-4core-16gb 285 15 3:58
✅​ gcp linux-2core-8gb 284 17 7:18
✅​ gcp-windows-latest-4core-16gb 286 15 4:50
Test Name azure linux-2core-8gb
🔄​ TestFilerWorkspaceNotebook 🔄​f
🔄​ TestFilerWorkspaceNotebook/rNb.r 🔄​f
Top 6 slowest tests (at least 2 minutes):
duration env testname
4:07 gcp-windows-latest-4core-16gb TestAccept
3:56 gcp linux-2core-8gb TestAccept
3:48 aws linux-2core-8gb TestAccept
3:42 azure linux-2core-8gb TestAccept
3:20 aws-windows-latest-4core-16gb TestAccept
3:14 azure-windows-latest-4core-16gb TestAccept

@shreyas-goenka
shreyas-goenka force-pushed the shreyas-goenka_data/stack/deployment-history-generate branch from cb1e384 to 81cdcc7 Compare October 27, 2026 23:11
@shreyas-goenka shreyas-goenka mentioned this pull request Oct 27, 2026
Merged
@shreyas-goenka
shreyas-goenka force-pushed the shreyas-goenka_data/stack/deployment-history-generate branch from 81cdcc7 to 3e9a047 Compare October 27, 2026 23:22
@shreyas-goenka
shreyas-goenka force-pushed the shreyas-goenka_data/stack/deployment-history-generate branch from 3e9a047 to 17d8805 Compare October 27, 2026 23:27
@shreyas-goenka
shreyas-goenka force-pushed the shreyas-goenka_data/stack/deployment-history-generate branch from 17d8805 to 28af6bc Compare October 27, 2026 23:40
@shreyas-goenka
shreyas-goenka force-pushed the shreyas-goenka_data/stack/deployment-history-generate branch from 28af6bc to 0f2c7cf Compare October 27, 2026 23:54
@shreyas-goenka
shreyas-goenka force-pushed the shreyas-goenka_data/stack/deployment-history-generate branch from 0f2c7cf to 9ae37b8 Compare October 28, 2026 00:07
@shreyas-goenka
shreyas-goenka force-pushed the shreyas-goenka_data/stack/deployment-history-generate branch from 9ae37b8 to 94c5579 Compare October 28, 2026 00:18
@shreyas-goenka
shreyas-goenka removed this pull request from stack #7208 October 28, 2026 00:53
@shreyas-goenka
shreyas-goenka force-pushed the shreyas-goenka_data/stack/deployment-history-generate branch from 94c5579 to 6e74279 Compare October 28, 2026 00:53
@shreyas-goenka
shreyas-goenka changed the base branch from shreyas-goenka_data/stack/deployment-history-open-readonly to shreyas-goenka_data/stack/deployment-history-bind-unbind October 28, 2026 00:53
@shreyas-goenka shreyas-goenka mentioned this pull request Oct 28, 2026
Draft
@shreyas-goenka
shreyas-goenka marked this pull request as ready for review October 28, 2026 01:01
@shreyas-goenka
shreyas-goenka requested review from a team as code owners October 28, 2026 01:01
@shreyas-goenka
shreyas-goenka force-pushed the shreyas-goenka_data/stack/deployment-history-generate branch from 6e74279 to 1fb582c Compare October 28, 2026 10:06
@github-actions github-actions Bot added the DABs DABs related issues label Oct 28, 2026
Base automatically changed from shreyas-goenka_data/stack/deployment-history-bind-unbind to main October 28, 2026 10:58
@shreyas-goenka
shreyas-goenka force-pushed the shreyas-goenka_data/stack/deployment-history-generate branch from 1fb582c to 2532ede Compare October 28, 2026 10:58
@shreyas-goenka
shreyas-goenka added this pull request to the merge queue Oct 28, 2026
Merged via the queue into main with commit d27fd75 Oct 28, 2026
42 of 44 checks passed
@shreyas-goenka
shreyas-goenka deleted the shreyas-goenka_data/stack/deployment-history-generate branch October 28, 2026 11:50
@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Oct 28, 2026

Copy link
Copy Markdown
Collaborator

Integration test report

Commit: d27fd75

Run: 40683078199

Env 🔄​flaky ✅​pass 🙈​skip Time
🔄​ aws linux-2core-8gb 3 1496 1220 161:16
✅​ aws-windows-latest-4core-16gb 1424 1245 181:19
🔄​ azure linux-2core-8gb 2 1344 1272 139:59
✅​ azure-windows-latest-4core-16gb 1271 1297 141:38
✅​ gcp linux-2core-8gb 1333 1277 147:03
✅​ gcp-windows-latest-4core-16gb 1258 1302 155:11
Test Name aws linux-2core-8gb azure linux-2core-8gb
🔄​ TestAccept 🔄​f ✅​p
🔄​ TestAccept/bundle/config-remote-sync/multiple_resources/DMS= ✅​p 🔄​f
🔄​ TestAccept/bundle/config-remote-sync/multiple_resources/DMS=true ✅​p 🔄​f
🔄​ TestAccept/bundle/resources/postgres_synced_tables/recreate 🔄​f 🙈​s
🔄​ TestAccept/bundle/resources/postgres_synced_tables/recreate/DMS= 🔄​f
Top 50 slowest tests (at least 2 minutes):
duration env testname
10:17 azure linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
10:16 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
9:52 azure linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=
9:33 aws-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
9:10 aws linux-2core-8gb TestAccept/bundle/invariant/no_drift/DMS=true/INPUT_CONFIG=cluster_libraries.yml.tmpl/READPLAN=
8:51 aws-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/resize-terminated-fallback/DMS=true
8:40 aws-windows-latest-4core-16gb TestAccept/bundle/deploy/spark-jar-task/DMS=true
8:38 aws-windows-latest-4core-16gb TestAccept/bundle/deploy/spark-jar-task/DMS=
8:23 gcp linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
8:08 azure-windows-latest-4core-16gb TestAccept/bundle/invariant/delete_idempotent/DMS=/INPUT_CONFIG=cluster.yml.tmpl/READPLAN=
7:57 aws-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
7:53 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DMS=
7:50 aws linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
7:47 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
7:35 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DMS=true
7:33 aws linux-2core-8gb TestAccept/bundle/invariant/no_drift/DMS=true/INPUT_CONFIG=cluster_libraries.yml.tmpl/READPLAN=1
7:32 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
7:28 azure-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
7:26 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
7:24 azure-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
7:22 aws-windows-latest-4core-16gb TestAccept/bundle/invariant/no_drift/DMS=true/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=
7:21 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DMS=
7:19 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DMS=true
7:17 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
7:15 aws linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
7:14 aws-windows-latest-4core-16gb TestAccept/bundle/invariant/no_drift/DMS=true/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=1
7:12 aws-windows-latest-4core-16gb TestAccept/bundle/invariant/no_drift/DMS=true/INPUT_CONFIG=cluster.yml.tmpl/READPLAN=1
7:10 azure linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
7:09 aws-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
7:05 aws linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
7:03 azure linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
7:02 azure linux-2core-8gb TestAccept/bundle/config-remote-sync/multiple_resources/DMS=
6:57 aws-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=true
6:53 azure-windows-latest-4core-16gb TestAccept/bundle/invariant/delete_idempotent/DMS=/INPUT_CONFIG=cluster_libraries.yml.tmpl/READPLAN=
6:49 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
6:42 aws-windows-latest-4core-16gb TestAccept/bundle/invariant/no_drift/DMS=true/INPUT_CONFIG=cluster_libraries.yml.tmpl/READPLAN=
6:38 aws-windows-latest-4core-16gb TestAccept/bundle/invariant/no_drift/DMS=true/INPUT_CONFIG=cluster.yml.tmpl/READPLAN=
6:34 aws linux-2core-8gb TestAccept/bundle/config-remote-sync/multiple_resources/DMS=
6:34 gcp linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
6:33 gcp linux-2core-8gb TestAccept/bundle/config-remote-sync/multiple_resources/DMS=true
6:29 gcp linux-2core-8gb TestAccept/bundle/config-remote-sync/multiple_resources/DMS=
6:25 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=
6:15 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/deploy/update-after-create/DMS=true
6:09 aws linux-2core-8gb TestAccept/bundle/config-remote-sync/multiple_resources/DMS=true
6:03 aws linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
5:56 azure linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
5:52 aws-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=
5:51 azure-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
5:48 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=
5:44 aws-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@denik denik denik approved these changes
@renaudhartert-db renaudhartert-db Awaiting requested review from renaudhartert-db renaudhartert-db is a code owner automatically assigned from databricks/eng-deco-cli
@lennartkats-db lennartkats-db Awaiting requested review from lennartkats-db lennartkats-db is a code owner automatically assigned from databricks/eng-deco-dabs

Assignees

No one assigned

Labels

DABs DABs related issues

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@shreyas-goenka @eng-dev-ecosystem-bot @denik