Repository navigation
Back up Cloudberry incremental materialized views - #131
Open
talmacschen-arch wants to merge 1 commit into
Open
talmacschen-arch wants to merge 1 commit into
talmacschen-arch wants to merge 1 commit into
Conversation
A Cloudberry incremental materialized view (IVM, pg_class.relisivm) was backed
up as a plain `CREATE MATERIALIZED VIEW`, so on restore it lost its incremental
behaviour. Worse, the IVM maintenance triggers the kernel installs on the base
tables (named IVM_trigger_*, calling pg_catalog.ivm_immediate_before /
ivm_immediate_maintenance) are not internal (tgisinternal = false), so
GetTriggers backed them up too. Their names and function arguments embed the
matview's OID, which changes on restore, and an incremental matview re-creates
these triggers itself when it is repopulated -- so restoring the backed-up
triggers would duplicate them with stale OIDs.
This change:
- selects relisivm in GetAllViews (Cloudberry only -- GPDB has no such column)
and exposes it as View.IsIncremental;
- emits `CREATE INCREMENTAL MATERIALIZED VIEW` for such views, so the kernel
re-establishes the maintenance triggers when the matview is refreshed;
- excludes the IVM maintenance triggers from GetTriggers (by trigger function,
which is precise and a no-op on systems without IVM).
Matview data handling is unchanged: matviews are still emitted `WITH NO DATA`;
a manual `REFRESH MATERIALIZED VIEW` after restore repopulates the matview, at
which point the kernel re-creates the correct maintenance triggers.
Verified end to end on a live Apache Cloudberry 2.1.0 (SynxDB4 4.5.4-3) cluster
with a rebuilt gpbackup/gprestore:
- metadata.sql emits `CREATE INCREMENTAL MATERIALIZED VIEW` for the IVM and a
plain `CREATE MATERIALIZED VIEW` for a normal matview, and contains none of
the IVM_trigger_* statements;
- after restore the IVM exists unpopulated (relisivm = t) with zero triggers;
- after `REFRESH MATERIALIZED VIEW`, the kernel re-creates the 8 maintenance
triggers and a subsequent INSERT into the base table is reflected in the
matview (incremental maintenance works end to end).
Add a unit test for the INCREMENTAL print path and integration tests for the
relisivm flag and the IVM-trigger exclusion (both Cloudberry-gated).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A Cloudberry incremental materialized view (IVM,
pg_class.relisivm) is backed up incorrectly in two ways:GetAllViewsdoes not readrelisivm, and the matview is emitted as a plainCREATE MATERIALIZED VIEW. On restore it silently becomes a non-incremental matview.IVM_trigger_*, callingpg_catalog.ivm_immediate_before/ivm_immediate_maintenance) are not internal (tgisinternal = false), soGetTriggersbacks them up. Their names and function arguments embed the matview's OID (which changes on restore), and an incremental matview re-creates these triggers itself when repopulated — so restoring the backed-up triggers would duplicate them with stale OIDs.Reproduced on a live Apache Cloudberry 2.1.0 cluster: stock
gpbackup2.2.0 emitsCREATE MATERIALIZED VIEW ... USING heapplus 8CREATE TRIGGER "IVM_trigger_..._<oid>"statements for a single IVM.Fix
relisivminGetAllViews— Cloudberry only, since GPDB has no such column — and expose it asView.IsIncremental.CREATE INCREMENTAL MATERIALIZED VIEWfor such views, so the kernel re-establishes the maintenance triggers when the matview is refreshed.GetTriggers, matching on the trigger function (precise, and a no-op on systems without IVM).Matview data handling is unchanged: matviews are still emitted
WITH NO DATA; a manualREFRESH MATERIALIZED VIEWafter restore repopulates them, at which point the kernel re-creates the correct maintenance triggers.Verification
End to end on a live Apache Cloudberry 2.1.0 (SynxDB4 4.5.4-3) cluster with a rebuilt
gpbackup/gprestore:metadata.sqlfor the IVMCREATE INCREMENTAL MATERIALIZED VIEW; a normal matview staysCREATE MATERIALIZED VIEWIVM_trigger_*inmetadata.sqlrelisivm = t, unpopulated, 0 triggersREFRESH MATERIALIZED VIEWINSERTinto base table after refreshAdded a unit test for the
INCREMENTALprint path (fails if printed as a plain matview) and integration tests for therelisivmflag and the IVM-trigger exclusion (both Cloudberry-gated).