Skip to content

🐛 OPRUN-4791: Enforce revision-level progression deadlines in COD - #3055

Open
fao89 wants to merge 3 commits into
operator-framework:mainfrom
fao89:oprun-4791-progress-deadline
Open

fao89 wants to merge 3 commits into
operator-framework:mainfrom
fao89:oprun-4791-progress-deadline

Conversation

@fao89

@fao89 fao89 commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Description

The progression timeout covers how long a ClusterObjectDeployment (COD) takes
to progress to its desired new revision. The timer starts when that revision's
ClusterObjectSet (COS) is created and ends when the COS records its first
status.completedAt, meaning the entire revision has completed rollout. The COD
then reports Progressing=False with reason NewClusterObjectSetProgressed.

All phases share one budget. A five-phase revision with a ten-minute deadline
gets ten minutes total; completing a phase does not reset the timer. Time spent
waiting to create the COS is excluded, and each new revision gets a fresh budget.

Enforce the deadline directly in the COD controller so it reports
Progressing=False with reason ProgressDeadlineExceeded when the budget expires
before completion, even if the COS reports PreconditionsNotMet,
InvalidSpecification, or no readiness condition. Schedule a deadline check even
when the COS emits no further events, and cap error backoff at the remaining
budget.

Reconciliation continues after timeout, allowing late revision completion to
recover. Completed revisions are not timed again after health regressions.
Availability from an older ready revision is preserved.

Jira: OPRUN-4791

Validation:

  • Regression reproduced against the previous implementation, then passed with
    direct COD enforcement.
  • Focused controller and deadline tests, including retry scheduling, creation
    retries, late completion, health regression, and retained availability.
  • make test-unit: all 51 test packages passed with race detection.
  • make lint: zero issues.
  • make verify: passed with no generated-file changes.
  • make lint-api-diff BASELINE_BRANCH=upstream/main: no new findings.

Reviewer Checklist

  • API Go Documentation
  • Tests: Unit Tests (and E2E Tests, if appropriate)
  • Comprehensive Commit Messages
  • Links to related GitHub Issue(s)

Summary by CodeRabbit

  • New Features
    • Rollout progress deadlines now apply across all phases without resetting and begin when the corresponding revision is created; time spent waiting for revision creation is excluded.
    • If the deadline expires before completion, the deployment reports stalled progress while readiness failures remain visible. Once a revision completes, its deadline no longer applies, even if it later becomes unhealthy; an older ready revision can remain available.
    • Deadline checks are scheduled based on the time remaining, and omitting the deadline setting disables enforcement.
  • Documentation
    • Clarified deadline timing, expiry behavior, and the setting’s location.

@openshift-ci
openshift-ci Bot requested review from perdasilva and tmshort October 22, 2026 16:37
@openshift-ci

openshift-ci Bot commented Oct 22, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign perdasilva for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@netlify

netlify Bot commented Oct 22, 2026 •

Copy link
Copy Markdown

✅ Deploy Preview for olmv1 ready!

Name Link
🔨 Latest commit d97b31b
🔍 Latest deploy log https://app.netlify.com/projects/olmv1/deploys/6adb41cabbcfbffe92ba0e6b
😎 Deploy Preview https://deploy-preview-3055--olmv1.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.
🤖 Make changes Run an agent on this branch

To edit notification comments on pull requests, go to your Netlify project configuration.

@coderabbitai

coderabbitai Bot commented Oct 22, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Enterprise
  • Run ID: 1d222f74-7bfe-4cae-8390-a5d43c6d0e68
📥 Commits

Reviewing files that changed from the base of the PR and between c01bfa4 and d97b31b.

📒 Files selected for processing: 14
  • api/v1/clusterobjectdeployment_types.go
  • applyconfigurations/api/v1/clusterobjectdeploymentstatus.go
  • applyconfigurations/api/v1/clusterobjectdeploymenttemplatespec.go
  • docs/api-reference/olmv1-api-reference.md
  • docs/draft/concepts/clusterobjectsets.md
  • helm/olmv1/base/object-controller/crd/experimental/olm.operatorframework.io_clusterobjectdeployments.yaml
  • internal/object-controller/controllers/clusterobjectdeployment_controller.go
  • internal/object-controller/controllers/clusterobjectdeployment_controller_test.go
  • internal/object-controller/controllers/clusterobjectdeployment_deadline_test.go
  • internal/object-controller/controllers/progress_deadline.go
  • internal/object-controller/controllers/progress_deadline_test.go
  • manifests/experimental-e2e.yaml
  • manifests/experimental.yaml
  • test/e2e/features/cod_progression_timeout.feature
🚧 Files skipped from review as they are similar to previous changes: 1
  • api/v1/clusterobjectdeployment_types.go

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

📝 Walkthrough

ClusterObjectDeployment now enforces a progress deadline from ClusterObjectSet creation across revision phases. Reconciliation reports expired deadlines and schedules work based on remaining time. API descriptions, manifests, tests, and concept documentation describe the deadline behavior.

Changes: ClusterObjectDeployment progress deadlines

Layer / File(s) Summary
Define revision deadline behavior
api/v1/clusterobjectdeployment_types.go, applyconfigurations/api/v1/clusterobjectdeploymenttemplatespec.go, applyconfigurations/api/v1/clusterobjectdeploymentstatus.go, docs/api-reference/olmv1-api-reference.md, docs/draft/concepts/clusterobjectsets.md, helm/olmv1/base/object-controller/crd/experimental/olm.operatorframework.io_clusterobjectdeployments.yaml, manifests/experimental*.yaml
API descriptions and concept documentation define when the deadline starts and stops, how it applies across phases, and how completion affects enforcement.
Enforce deadlines during reconciliation
internal/object-controller/controllers/clusterobjectdeployment_controller.go, internal/object-controller/controllers/clusterobjectdeployment_controller_test.go, internal/object-controller/controllers/clusterobjectdeployment_deadline_test.go, test/e2e/features/cod_progression_timeout.feature
Reconciliation reports expired deadlines for incomplete revisions and returns a requeue duration when a deadline remains. New revisions do not receive the deployment deadline in their specs. Tests cover phase progress, creation retries, completion, availability, and separate ClusterObjectSet readiness reporting.
Bound retries by remaining deadline
internal/object-controller/controllers/clusterobjectdeployment_controller.go, internal/object-controller/controllers/progress_deadline.go, internal/object-controller/controllers/progress_deadline_test.go
The rate limiter caps retry backoff by the active revision’s remaining deadline. It wakes immediately after expiration and uses delegate backoff when no applicable deadline exists.

Priority: ➖ Normal

Estimated code review effort: 3 (Moderate) | ~25 minutes

Change: Bug fix

Sequence Diagram(s)

sequenceDiagram
  participant ControllerRuntime
  participant ClusterObjectDeploymentReconciler
  participant ClusterObjectSet
  ControllerRuntime->>ClusterObjectDeploymentReconciler: Reconcile deployment
  ClusterObjectDeploymentReconciler->>ClusterObjectSet: Create or inspect revision
  ClusterObjectSet-->>ClusterObjectDeploymentReconciler: Creation timestamp and readiness status
  ClusterObjectDeploymentReconciler-->>ControllerRuntime: Return status and deadline requeue
Loading

Suggested reviewers: fgiudici, pedjak

Merge Risk: ⚪ Minimal · up to d97b3

No actionable issue remains identified for this change; it is ready for normal merge checks.

Pre-merge checks: Passed 4 · Failed 1

Failed Failed checks: 1 warning

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 20.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 25 functions across 8 files. (6 skipped: … Write docstrings for the functions missing them to satisfy the coverage threshold.
Passed Passed checks: 4 passed
Check name Status Explanation
Title check Passed The title accurately names the main change: the COD controller now enforces revision-level progression deadlines. It follows the 🐛 bug-fix prefix convention in the description template and includes th…
Description check Passed The description covers the change in detail. It explains the deadline start and end points, the shared phase budget, the exclusion of COS creation wait time, the direct COD enforcement with ProgressDe…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Title check

Explanation

The title accurately names the main change: the COD controller now enforces revision-level progression deadlines. It follows the 🐛 bug-fix prefix convention in the description template and includes the Jira key. The title is 66 characters, which is within a normal title length.

Full details: Description check

Explanation

The description covers the change in detail. It explains the deadline start and end points, the shared phase budget, the exclusion of COS creation wait time, the direct COD enforcement with ProgressDeadlineExceeded, backoff capping, continued reconciliation after timeout, and preserved availability. It also lists validation steps and links the Jira issue. The Reviewer Checklist is kept but none of its items are ticked, and the 🐛 prefix from the template is not stated as a prefix on the title line. These are minor gaps; the core content is complete and matches the template's Description section.

Full details: Docstring Coverage

Explanation

Docstring coverage is 20.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 25 functions across 8 files. (6 skipped: 6 unsupported.)

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@fao89
fao89 force-pushed the oprun-4791-progress-deadline branch from f0e33ed to 8293ba7 Compare October 22, 2026 17:41
Document one deadline for the entire revision, starting at COS creation.
Time spent waiting to create the COS is outside the MVP budget.

Cover a deployment older than its deadline, five phases sharing one budget,
late completion, later health regressions, and a fresh budget for the next
revision without depending on phase completion timestamps.

Signed-off-by: Fabricio Aguiar <fabricio.aguiar@gmail.com>

rh-pre-commit.version: 2.3.2
rh-pre-commit.check-secrets: ENABLED
@fao89
fao89 force-pushed the oprun-4791-progress-deadline branch from 8293ba7 to c01bfa4 Compare October 23, 2026 08:19

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's also remove this. If you want to move the rest of the clean up (removing checks for Ready=False/ProgressDeadlineExceeded on the COS) for a later PR to keep this one small, I think that's fine

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. COD no longer copies progressDeadlineMinutes into new COS revisions or consults their Ready=False/ProgressDeadlineExceeded condition. It calculates the budget from its template and the revision's creation and completion timestamps. COS deadline support remains for ClusterExtension installations until that path is migrated.

Comment on lines 53 to 56
if !hasDeadline {
r.pastDeadline.Delete(item)
return backoff
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if we should distinguish between errors and no deadline, something like:

Suggested change
if !hasDeadline {
r.pastDeadline.Delete(item)
return backoff
}
remaining, hasDeadline, err := r.remaining(item)
if err != nil {
// Couldn't determine the deadline (transient read error). Do NOT touch the
// past-deadline marker — deleting it here is lets intermittent errors
// re-arm the one-shot immediate requeue. We have no remaining
// value to cap against, so fall back to the delegate backoff.
return backoff
}
if !hasDeadline {
r.pastDeadline.Delete(item)
return backoff
}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. Deadline lookups now return errors separately from the absence of a deadline. A transient Get or List failure uses the delegate backoff without clearing the expired-deadline marker. Regression tests cover intermittent failures and verify that recovery does not re-arm the immediate retry.

Comment on lines +267 to +283
remaining: func(req ctrl.Request) (time.Duration, bool) {
ctx := context.Background()
cod := &ocv1.ClusterObjectDeployment{}
if err := reader.Get(ctx, req.NamespacedName, cod); err != nil || !cod.DeletionTimestamp.IsZero() {
return 0, false
}
var revisions ocv1.ClusterObjectSetList
if err := reader.List(ctx, &revisions, client.MatchingFields{clusterObjectSetGroupIndex: cod.Name}); err != nil {
return 0, false
}
latest := latestActiveRevision(revisions.Items)
if latest == nil || latest.Labels[labels.TemplateHashKey] != templateHash(cod.Spec.Template) {
// Creating the desired revision is outside the MVP deadline.
return 0, false
}
return durationUntilDeploymentDeadline(clk, latest)
},

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe we should also consider the deletion case?

Suggested change
remaining: func(req ctrl.Request) (time.Duration, bool) {
ctx := context.Background()
cod := &ocv1.ClusterObjectDeployment{}
if err := reader.Get(ctx, req.NamespacedName, cod); err != nil || !cod.DeletionTimestamp.IsZero() {
return 0, false
}
var revisions ocv1.ClusterObjectSetList
if err := reader.List(ctx, &revisions, client.MatchingFields{clusterObjectSetGroupIndex: cod.Name}); err != nil {
return 0, false
}
latest := latestActiveRevision(revisions.Items)
if latest == nil || latest.Labels[labels.TemplateHashKey] != templateHash(cod.Spec.Template) {
// Creating the desired revision is outside the MVP deadline.
return 0, false
}
return durationUntilDeploymentDeadline(clk, latest)
},
remaining: func(req ctrl.Request) (time.Duration, bool, error) {
ctx := context.Background()
cod := &ocv1.ClusterObjectDeployment{}
if err := reader.Get(ctx, req.NamespacedName, cod); err != nil {
return 0, false, err // unknown
}
if !cod.DeletionTimestamp.IsZero() {
return 0, false, nil // deleting: definitively no deadline
}
var revisions ocv1.ClusterObjectSetList
if err := reader.List(ctx, &revisions, client.MatchingFields{clusterObjectSetGroupIndex: cod.Name}); err != nil {
return 0, false, err // unknown
}
latest := latestActiveRevision(revisions.Items)
if latest == nil || latest.Labels[labels.TemplateHashKey] != templateHash(cod.Spec.Template) {
return 0, false, nil // creating desired revision: outside MVP deadline
}
d, has := durationUntilDeploymentDeadline(clk, latest)
return d, has, nil
},

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. Missing or deleting deployments and missing or deleting desired revisions definitively have no active deadline and clear the retry marker. Transient lookup errors preserve it. Added regression coverage for both cases.

fao89 added 2 commits October 23, 2026 12:13
Calculate the deployment deadline from the desired COS creation timestamp
through its first successful completion. Enforce the deadline even when COS
readiness remains Pending, PreconditionsNotMet, or another non-deadline reason.

Schedule the COD at the deadline and cap retry backoff so a quiet or failing
revision cannot leave the deployment progressing indefinitely. Reuse the
revision completion latch to allow late recovery without timing completed
revisions again. Keep time waiting for COS creation outside the MVP budget.

Cover fixed budgets across phases, blocked revisions, retry scheduling,
creation retries, late completion, and availability from an older revision.

Signed-off-by: Fabricio Aguiar <fabricio.aguiar@gmail.com>

rh-pre-commit.version: 2.3.2
rh-pre-commit.check-secrets: ENABLED
Regenerate apply configuration comments, API reference documentation, the
COD CRD, and experimental manifests from the clarified deadline contract.

Signed-off-by: Fabricio Aguiar <fabricio.aguiar@gmail.com>

rh-pre-commit.version: 2.3.2
rh-pre-commit.check-secrets: ENABLED
@fao89
fao89 force-pushed the oprun-4791-progress-deadline branch from c01bfa4 to d97b31b Compare October 23, 2026 11:15
@fao89

fao89 commented Oct 23, 2026

Copy link
Copy Markdown
Contributor Author

/cc @fgiudici @dtfranz

@openshift-ci
openshift-ci Bot requested review from dtfranz and fgiudici October 23, 2026 15:05

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants