Skip to content

CH-298 fix nfs server regression after a failed volume - #884

Merged
filippomc merged 2 commits into
developfrom
feature/CH-298-fix-nfs-locked-after-failed-volume
Oct 22, 2026
Merged

filippomc merged 2 commits into
developfrom
feature/CH-298-fix-nfs-locked-after-failed-volume

Conversation

@filippomc

@filippomc filippomc commented Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator

Closes CH-298

Implemented solution

  • Create:
    • Cleans up leftovers with discardPartial, which never deletes data it can't vouch for: it refuses a mounted volume that isn't empty, and moves a file at the mountpoint aside to <mp>.discarded-<ts>.
    • Rolls back on failure.
    • Creates the mountpoint only after mkfs.
  • Exports:
    • mount-all skips non-directories.
    • Create and remount succeed when the volume is in /var/lib/nfs/etab, even if exportfs -r exits 1 because of another entry. nfs-utils only warns about a bad entry, then still writes etab and flushes the cache.
  • Mounting: mountOne no longer creates missing mountpoints, and the watchdog skips volumes that don't have one.
  • Delete: unmounts until nothing is mounted (or fails), and keeps a regular file at the mountpoint.
  • loop.Attach: serialized within the process, and retries on EBUSY/ENXIO.
  • Provisioner: volumeExists also checks for <mp>.quota before skipping a delete.

How to test this PR

  • Go unit and scenario tests (no root needed) in nfsvol/internal/mount/lifecycle_test.go, loop/loop_test.go and provisioner_test.go. Each fix was checked by undoing it: all 10 are caught by at least one test.
  • test/failover-test.sh:
    • New 4f: export table clean after restart.
    • New 8: provisioning while another export is broken; reproduces this outage.
    • New 9: PVC delete after an interrupted nfsvol delete.
    • Extended 7: no .quota left after PVC delete; image archived when the storage class says so.
    • Not yet run on a cluster. It needs ≥3 nodes; run it on dev before prod.

Sanity checks:

  • The pull request is explicitly linked to the relevant issue(s)
  • The issue is well described: clearly states the problem and the general proposed solution(s)
  • In this PR it is explicitly stated how to test the current change
  • The issue seta the scope and the type of issue (bug, story, task, etc.)
  • The relevant components are indicated in the issue (if any)
  • All the automated test checks are passing
  • All the linked issues are included in one Sprint
  • All the linked issues are in the Review state
  • All the linked issues are assigned

Breaking changes (select one):

  • The present changes do not change the preexisting api in any way
  • This PR and the issue are tagged as a breaking-change and the migration procedure is well described above

Possible deployment updates issues (select one):

  • There is no reason why deployments based on CloudHarness may break after the current update
  • This PR and the issue are tagged as alert:deployment

Test coverage (select one):

  • Tests for the relevant cases are included in this pr
  • The changes included in this pr are out of the current test coverage scope

Documentation (select one):

  • The documentation has been updated to match the current changes
  • The changes included in this PR are out of the current documentation scope

Nice to have (if relevant):

  • Screenshots of the changes
  • Explanatory video/animated gif

if err != nil {
return "", fmt.Errorf("open backing file %s: %w", backingFile, err)
}
defer bf.Close()
run_merge = (
app_builder.handle_merge
if (app_builder := self.app_builders.get(template, None))
if (app_builder: = self.app_builders.get(template, None))
@filippomc
filippomc merged commit 62d26c5 into develop Oct 22, 2026
7 of 11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant