Repository navigation
Conversation
aljoscha
force-pushed
the
decoupled-coordination
branch
from
September 7, 2026 12:02
9705b3c to
d1afd3d
Compare
aljoscha
changed the title
aljoscha
force-pushed
the
decoupled-coordination
branch
13 times, most recently
from
September 9, 2026 06:43
cfbf09d to
9d5e1a9
Compare
Contributor
Author
|
Commit 9d5e1a9 connects storage-backed catalog read protection to committed delivery. Bounds and maintained requirements have separate durable records, final-state transaction validation rejects incompatibility and regression, and the adapter returns admission errors before its fatal commit boundary. Committed bounds reach storage through implications. Dry runs model collection births and drops without controller effects. This is an intermediate milestone 1 change, not activation. SQL creation still needs to produce initial bounds and logical-input requirements with secured readability, including shared-shard initialization. Recovery and durable-progress publication remain unwired. Validation: formatting, targeted compilation, and the durable admission/reopen and adapter admission/dry-run tests pass. Independent review findings are resolved. Regular PR CI is still pending: https://buildkite.com/materialize/test/builds/134033 |
aljoscha
force-pushed
the
decoupled-coordination
branch
13 times, most recently
from
September 12, 2026 20:30
281b6c4 to
697e983
Compare
Qualify protection ownership by adapter identity and deployment so replacing or dropping one adapter cannot revoke a peer. Create the default declaration only during fresh initialization and preserve an empty inventory on reopen. Use schema v96 and reject the unsupported singleton-catalog conversion.
Followers reconstruct catalog state and enact committed effects without republishing derived builtin rows. Keep selected-plan and legacy notice metadata local on followers while reserving publication for the committer. Out-of-order and failed projection publication remain scoped M3 limitations. Startup seeding and shared-table reset ownership are separate work.
Record whether this writable handle committed initial catalog contents. Reopens, joins and savepoints cannot claim authority to seed shared projections. Resample initialization on every transaction attempt.
Native execution probes and retirement fixtures must inspect the realization variant rather than the removed optional replica_id field. Keep the existing assertions and deadlines unchanged.
Distinguish reconstructed snapshots from originating commit rows before publication. Route selected-plan notices to their commit owner and preserve replacement-shard backfill from both sources. The table reset and seeding policy remains unchanged in this preparation.
Publish sessions and subscriptions to owner-qualified backing tables. Public views retain their columns and OIDs while checking live adapter incarnation membership, so predecessor writes cannot become successor observations. Keep protected peers out of startup resets and preserve the session-write barrier through the public view. Retired-row reclamation remains separate.
Match the existing 42704 error contract for dropped SUBSCRIBE dependencies. Assert the exact SQLSTATE and message rather than the obsolete internal-error wording.
Use the bootstrap WAL snapshot and conditional append to reconcile active configuration atomically. Concurrent writes require fresh snapshots, and the existing catalog authority check orders publication against promotion. Prewarming keeps its private replacement-shard backfill without overwriting active shared configuration. Planning still uses deployment-local settings. Keep reconciliation local to bootstrap rather than adding a background task or making readiness depend on compute queries.
Regenerate catalog-server coverage and plans for the session/subscription views. Preserve public OIDs and account for their restricted raw tables in table listings, while keeping the public names in view listings.
Keep process-local mz_system credentials separate from durable role-auth state. Authentication uses the override while shared projections and catalog following use only durable credentials. Install local credentials on the serving open path, including joins, without publishing startup rows.
Carry the successful initializer provenance into bootstrap. Protected joins and restarts retain shared tables and discard reconstruction snapshots, without discarding originating transaction diffs or private replacement backfill. Configuration reconciliation and local authentication remain separate from derived projection publication.
Allow deployment initialization to establish shard identities and birth permissions without constructing a storage runtime or starting finalization. Joined catalogs validate committed mappings without initializing them. Keep existing runtime bootstrap on the shared initialization path.
Move the existing monotonic Persist append operation to storage-client. Keep timestamp assignment and definitive conflict retries unchanged, so replica-owned history can reuse it without depending on a controller.
Allow the adapter and replicas to share an independently managed relay. External mode uses the existing remote client and does not bind a local pubsub listener. Keep the in-process default unchanged.
Publish original health transitions independently of adapter subscriptions. Keep current health in one private replica-owned relation, fenced by execution, so late predecessor events and history pruning cannot restore stale health. Preserve existing history schemas and retention. Only successful catalog committers publish native Paused/Dropped history. Followers retain local effects without duplicating lifecycle events, while catalog placement supplies current existence and zero-replica status. Missing or unordered lifecycle history remains a scoped M3 limitation.
Keep historical drop reasons intact, while checking current zero-replica health independently of those history events.
Observe terminal Persist output without starting a producer or manufacturing historical transitions. Keep completion observations qualified by the replica incarnation and separate from retired execution events. Synchronize the constant-sink restart fixture on recovery publication and preserve its running-status contract with an unknown transition time.
Exercise real output frontiers and catalog placement at sink installation. Synchronize absence assertions through the existing publication queue.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Maintained dataflows, sources, sinks and compaction should not depend on a SQL
adapter staying alive. This draft moves maintained lifecycle into catalog-following
replicas and is now implementing independent SQL adapters and process management.
M2 is closed within its approved scope. M3 is in progress, not complete.
Approach
use the query protocol rather than lifecycle installation RPCs.
DROP and retirement release it. Silence does not expire holds.
with a 60s default. New or stronger grants remain immediate and commit-before-use.
pubsubd. Controllerd owns process management, not shared database maintenance.
The design
and current handoff
define contracts, scope and remaining work.
Current M3 state
Implemented foundations include packaged pubsubd with verified external-relay
wiring, retained local supervision, replica launch specifications, stable adapter
declarations and execution identities, joined catalog reconstruction, and finite
storage-metadata initialization. Shared projection seeding is fresh-only. Active
configuration is transactionally reconciled before readiness, and shared credentials
exclude process-local authentication overrides.
Native current health now uses one private replica-owned relation, independent of
unchanged retained history. Execution fencing prevents predecessor events from
restoring stale health. Catalog placement supplies zero-replica paused state.
Only successful catalog committers publish native Paused/Dropped history, while
followers retain local effects and joins do not manufacture history. Completed
sinks reconstruct current health from actual terminal Persist output, with an
unknown transition time and no producer startup or historical replay.
Originator-only derived projections and catalog-driven lifecycle history permit
unordered publication and missing updates after publisher failure for M3. This
exception does not apply to current existence, shared configuration, user data,
protection or runtime ownership. Catalog-backed projection conversion is separately
owned.
Controllerd/adapterd, finite table/WAL completion and adoption wiring, SQL/HTTP
adapter management, retired-row cleanup and complete backend/deployment integration
remain open. Two-adapter independence, zero-adapter recovery and controller-restart
demos have not run. Schema v96 creates the default adapter only for fresh
environments. Empty adapter inventory does not recreate it, and there is no
M2-to-M3 conversion.
The status baseline passed 534 existing catalog SQL assertions and native SQL
lifecycle smoke. The completed-sink correction passes strict Clippy, 33 focused
regressions, an additional real-Persist negative admission/placement regression,
and the strengthened restart workflow under the aggregate 8-CPU/16-GiB budget.
The restart preserves history counts and verifies unknown transition time with
unchanged SQL deadlines. The exact SHOW inventory assertion also passes after
including the private source. These checks are not full M3 acceptance.
CI140760 finished with 93
passing jobs, including both original fixed-seed parallel shards, all cluster
shards and testdrive 6. Testdrive 4 still reports the now-corrected SHOW inventory
and an unresolved terminal-source status contract. The earlier MV-frontier timing
failure did not recur, but its cause remains unknown. Upgrade conversion and
merge skew remain excluded, not passing. Combined M3 acceptance remains open.
Closed M2 evidence and exclusions
with the 60s default, including native warm promotion, hydration/read-protection
restart, all cluster shards, both restart shards, Cargo and both parallel shards.
Upstream merge skew, existing-environment conversion and Kafka RTR explicit-DDL
invalidation are verified exclusions, not passing checks.
fixed-seed parallel shards pass on actual 8-CPU/16-GiB hosts after the bounded
Iceberg minting correction, with unchanged backends, concurrency and deadlines.
No heap observer was enabled. Controlled frontier checks reproduced the same
gap-proportional defect upstream and on this branch. No memory-policy tuning
is included, and this is not a general memory-bound or starvation-freedom claim.
and warm handover
establish maintained execution, physical retention and warmed execution continuity.
M2 covers fresh protected environments and compatible deployments. Pre-feature
binary compatibility, managed replica-targeted MV warm handover and comprehensive
rollout cancellation recovery remain excluded. Explicit-DDL invalidation is
separate work. Revision checks and acknowledged-DDL semantics are not weakened.
Version-overlap fixtures use separately compiled binaries from the same source,
not a general compatibility matrix.
The retired delayed-oracle fixture leaves a narrow response-ordering coverage gap.
The existing optimizer-recursion timeout and earlier runtime/metric-series flakes
remain unexplained, not repaired by relaxed deadlines. Interval-specific tests
use explicit short publication intervals or a test-only pause with synchronization.
The abandoned async catalog-owner refactor remains out of scope. This PR stays draft.