Skip to content

Fix leader distribution and RegionGroup directory assertions - #19006

Open
d-wang-commit wants to merge 1 commit into
apache:masterfrom
d-wang-commit:fix-region-leader-and-migration-checks
Open

d-wang-commit wants to merge 1 commit into
apache:masterfrom
d-wang-commit:fix-region-leader-and-migration-checks

Conversation

@d-wang-commit

Copy link
Copy Markdown
Contributor

Description

Count the full DataRegion leader population

The leader-distribution fixture creates three or six user DataRegionGroups. Internal database groups participate in the same balancing round, but the assertions removed those groups and then required the remaining user-only subset to be balanced. A balanced cluster can therefore fail with a user-only distribution such as {1=1, 3=2}.

Count the fixture's DataRegionGroups including internal databases, without mixing in independently balanced SchemaRegion leaders. Poll for exactly one Running leader per group, the expected number of leader-holding nodes, and a maximum per-node count difference of one. Keep the ReadOnly and stopped-node transitions and verify the same conditions after them.

Check only the migrated consensus group

A migration removes one RegionGroup replica, so other groups' consensus directories may remain on the source DataNode. Replace the whole-directory emptiness check and hardcoded 1_1 exception with checks for the selected group ID. Resolve Ratis group directories through the existing Raft group UUID conversion. Add a regression that leaves an unrelated three-replica database on the source node and verifies that its directories survive the migration.

Tests

  • UT: none added; this PR only changes integration-test fixtures and assertions.
  • IT: updated IoTDBRegionGroupLeaderDistributionIT and the shared migration assertions; added migrationShouldPreserveOtherRegionFiles in IoTDBRegionMigrateNormalForIoTV2StreamIT.
  • Passed all three methods in IoTDBRegionGroupLeaderDistributionIT and all three methods in IoTDBRegionMigrateNormalForIoTV2StreamIT with a clean reactor build: mvn clean verify -pl integration-test -am -PClusterIT,with-integration-tests -DskipUTs -Dit.test=IoTDBRegionGroupLeaderDistributionIT,IoTDBRegionMigrateNormalForIoTV2StreamIT -DfailIfNoTests=false -Dfailsafe.failIfNoSpecifiedTests=false.
  • Also passed IoTDBRegionMigrateNormalITForIoTV2BatchIT#normal1C2DTest and IoTDBRegionMigrateAddingPeerCrashForRatisIT#addingPeerCrashShouldFailAndRollback: 8 selected IT methods passed in total.
  • The same six leader/stream methods passed in the downstream edition. Both builds used the existing Procedure persistence implementation; no production fix was included.
  • Spotless and the clean English reactor build passed. No production or localized messages were changed.

Side effects and risks

No production code, persistence format, leader-balancing policy, or node-state restriction changes. The leader checks retain the existing 50-second bound and now reject missing leaders and non-Running leader replicas. Migration checks still require removal of the selected source directory and existence of the destination directory; unrelated RegionGroups no longer cause a false failure. A forced-down DataNode retains the existing skip behavior for local file checks.


This PR has:

  • been self-reviewed.
  • added comments explaining the assertion scope.
  • added integration tests.
  • been tested in a test IoTDB cluster.

Key changed/added classes (or packages if there are too many classes) in this PR
  • IoTDBRegionGroupLeaderDistributionIT: include internal DataRegion leaders and reject incomplete or unhealthy distributions.
  • IoTDBRegionOperationReliabilityITFramework: check directories for the selected RegionGroup.
  • IoTDBRegionMigrateITFrameworkForRatis: map the selected group to its Ratis UUID directory.
  • IoTDBRegionMigrateNormalForIoTV2StreamIT: verify unrelated region files survive migration.

Count all DataRegion leaders and require a complete, healthy distribution.
Check only the migrated consensus group and preserve unrelated region files.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant