Skip to content

Back up Cloudberry incremental materialized views - #131

Open
talmacschen-arch wants to merge 1 commit into
apache:mainfrom
talmacschen-arch:fix/backup-incremental-matviews
Open

talmacschen-arch wants to merge 1 commit into
apache:mainfrom
talmacschen-arch:fix/backup-incremental-matviews

Conversation

@talmacschen-arch

Copy link
Copy Markdown

Problem

A Cloudberry incremental materialized view (IVM, pg_class.relisivm) is backed up incorrectly in two ways:

  1. GetAllViews does not read relisivm, and the matview is emitted as a plain CREATE MATERIALIZED VIEW. On restore it silently becomes a non-incremental matview.
  2. The IVM maintenance triggers the kernel installs on the base tables (IVM_trigger_*, calling pg_catalog.ivm_immediate_before / ivm_immediate_maintenance) are not internal (tgisinternal = false), so GetTriggers backs them up. Their names and function arguments embed the matview's OID (which changes on restore), and an incremental matview re-creates these triggers itself when repopulated — so restoring the backed-up triggers would duplicate them with stale OIDs.

Reproduced on a live Apache Cloudberry 2.1.0 cluster: stock gpbackup 2.2.0 emits CREATE MATERIALIZED VIEW ... USING heap plus 8 CREATE TRIGGER "IVM_trigger_..._<oid>" statements for a single IVM.

Fix

  • Select relisivm in GetAllViews — Cloudberry only, since GPDB has no such column — and expose it as View.IsIncremental.
  • Emit CREATE INCREMENTAL MATERIALIZED VIEW for such views, so the kernel re-establishes the maintenance triggers when the matview is refreshed.
  • Exclude the IVM maintenance triggers from GetTriggers, matching on the trigger function (precise, and a no-op on systems without IVM).

Matview data handling is unchanged: matviews are still emitted WITH NO DATA; a manual REFRESH MATERIALIZED VIEW after restore repopulates them, at which point the kernel re-creates the correct maintenance triggers.

Verification

End to end on a live Apache Cloudberry 2.1.0 (SynxDB4 4.5.4-3) cluster with a rebuilt gpbackup/gprestore:

Step Result
metadata.sql for the IVM ✅ CREATE INCREMENTAL MATERIALIZED VIEW; a normal matview stays CREATE MATERIALIZED VIEW
IVM_trigger_* in metadata.sql ✅ none (excluded)
after restore ✅ IVM exists, relisivm = t, unpopulated, 0 triggers
after REFRESH MATERIALIZED VIEW ✅ kernel re-creates the 8 maintenance triggers
INSERT into base table after refresh ✅ reflected in the matview — incremental maintenance works

Added a unit test for the INCREMENTAL print path (fails if printed as a plain matview) and integration tests for the relisivm flag and the IVM-trigger exclusion (both Cloudberry-gated).

Note: touches the same GetAllViews query as #130 (PAX matview reloptions). The two are independent fixes; whichever merges second needs a trivial rebase on that query block.

A Cloudberry incremental materialized view (IVM, pg_class.relisivm) was backed
up as a plain `CREATE MATERIALIZED VIEW`, so on restore it lost its incremental
behaviour. Worse, the IVM maintenance triggers the kernel installs on the base
tables (named IVM_trigger_*, calling pg_catalog.ivm_immediate_before /
ivm_immediate_maintenance) are not internal (tgisinternal = false), so
GetTriggers backed them up too. Their names and function arguments embed the
matview's OID, which changes on restore, and an incremental matview re-creates
these triggers itself when it is repopulated -- so restoring the backed-up
triggers would duplicate them with stale OIDs.

This change:

  - selects relisivm in GetAllViews (Cloudberry only -- GPDB has no such column)
    and exposes it as View.IsIncremental;
  - emits `CREATE INCREMENTAL MATERIALIZED VIEW` for such views, so the kernel
    re-establishes the maintenance triggers when the matview is refreshed;
  - excludes the IVM maintenance triggers from GetTriggers (by trigger function,
    which is precise and a no-op on systems without IVM).

Matview data handling is unchanged: matviews are still emitted `WITH NO DATA`;
a manual `REFRESH MATERIALIZED VIEW` after restore repopulates the matview, at
which point the kernel re-creates the correct maintenance triggers.

Verified end to end on a live Apache Cloudberry 2.1.0 (SynxDB4 4.5.4-3) cluster
with a rebuilt gpbackup/gprestore:

  - metadata.sql emits `CREATE INCREMENTAL MATERIALIZED VIEW` for the IVM and a
    plain `CREATE MATERIALIZED VIEW` for a normal matview, and contains none of
    the IVM_trigger_* statements;
  - after restore the IVM exists unpopulated (relisivm = t) with zero triggers;
  - after `REFRESH MATERIALIZED VIEW`, the kernel re-creates the 8 maintenance
    triggers and a subsequent INSERT into the base table is reflected in the
    matview (incremental maintenance works end to end).

Add a unit test for the INCREMENTAL print path and integration tests for the
relisivm flag and the IVM-trigger exclusion (both Cloudberry-gated).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant