Repository navigation
Persist parent and child procedures atomically before scheduling - #19007
Open
d-wang-commit wants to merge 1 commit into
Open
d-wang-commit wants to merge 1 commit into
d-wang-commit wants to merge 1 commit into
Conversation
Commit parent progress and new children in one ConfigNode consensus entry. Retry failed writes before scheduling and preserve recovery/snapshot consistency. Add regression coverage for restart, retry, interruption, and serialization.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Persist the parent's progress together with newly submitted children
When a Procedure returns new children,
ProcedureExecutor.initializeChildrenadvances the parent's execution state to WAITING. The existing persistence path writes only the children before scheduling them. A later lock release can persist the parent, but a crash before that write leaves an old RUNNABLE parent alongside its already persisted children. Procedures that hold their lock across child execution, such asRemoveDataNodesProcedure, cannot rely on that later lock-release write either.On recovery, the executor can schedule both the stale parent and its unfinished children. Re-executing the parent's submission step creates duplicate children with different procedure IDs. For region migration, competing add-peer procedures can advance and roll back the same replica independently.
Persist the parent's next state and all new children before scheduling any child.
ConfigProcedureStorewrites this batch as oneUpdateProceduresPlanconsensus entry, checks its returned status, and lets the executor retry failures with the existing backoff helper. An interrupted persistence retry must not fall through to scheduling children.Preserve recovery and snapshot consistency
Each procedure in the batch is length-delimited and uses the existing single-procedure encoding, so optional trailing fields cannot consume the next record. Apply the batch under the same
ProcedureInfomonitor used by recovery reads and snapshot creation, preventing either operation from observing a partially applied batch. Existing single-procedure entries and snapshot file formats remain readable.Tests
ProcedureChildPersistenceTest: restart after one or several children are persisted, retry a failed batch before executing children, and stop during a failing retry without starting children. It exercises the real executor and serializers with a recording store and a controlled stop at the persistence boundary.UpdateProceduresPlanTest: batch serialization and snapshot recovery, preservation of the old single-entry reader, one consensus write per batch, and rejection-status propagation.ProcedureChildPersistenceTest,UpdateProceduresPlanTest,TestProcedureExecutor,STMProcedureTest,StateMachineProcedureTest,ProcedureInfoTest,ConfigPhysicalPlanSerDeTest, andConfigPhysicalPlanTypeTest.IoTDBRegionMigrateConfigNodeCrashForRatisIT#cnCrashDuringCreatePeerTestandIoTDBRegionMigrateConfigNodeCrashIoTV1IT#cnCrashDuringCreatePeerTest, using the unmodified upstream IT assertions on this independent branch.mvn verify -pl integration-test -am -PDailyIT,with-integration-testswith the above selections and no-specified-test flags for upstream reactor modules. Spotless passed.mvn package -am -DskipTestsandmvn package -am -DskipTests -Pwith-zh-localeboth passed.Side effects and risks
This introduces ConfigNode plan type 502. New readers retain support for old entries, but old ConfigNodes cannot decode the new batch type; mixed-version rolling upgrade and downgrade are not established by these tests. A batch entry grows with the parent and child records, and snapshot creation now excludes batch updates for its duration. Large child batches and slow snapshot disks were not stress-tested.
The change prevents duplicate child submission caused by mismatched persisted parent/child states. It does not make external DataNode actions exactly-once, repair existing inconsistent procedure records, or replace individual procedures' retry and rollback logic. Persistent store failures keep the children unscheduled until the batch succeeds or the executor is interrupted.
This PR has:
Key changed/added classes (or packages if there are too many classes) in this PR
ProcedureExecutor: persist the parent and new children before scheduling, retry store failures, and stop after interruption.ConfigProcedureStore/IProcedureStore: use and document atomic batch persistence.UpdateProceduresPlan/ConfigPhysicalPlan/ConfigPhysicalPlanType/ConfigPlanExecutor: encode, decode, and apply the batch consensus entry.ProcedureInfo: exclude recovery reads and snapshots from partially applied batches.ProcedureChildPersistenceTest/UpdateProceduresPlanTest: exercise recovery, failures, serialization, and consensus submission.