Repository navigation
Conversation
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
✅ Deploy Preview for olmv1 ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
f0e33ed to
8293ba7
Compare
Document one deadline for the entire revision, starting at COS creation. Time spent waiting to create the COS is outside the MVP budget. Cover a deployment older than its deadline, five phases sharing one budget, late completion, later health regressions, and a fresh budget for the next revision without depending on phase completion timestamps. Signed-off-by: Fabricio Aguiar <fabricio.aguiar@gmail.com> rh-pre-commit.version: 2.3.2 rh-pre-commit.check-secrets: ENABLED
8293ba7 to
c01bfa4
Compare
There was a problem hiding this comment.
let's also remove this. If you want to move the rest of the clean up (removing checks for Ready=False/ProgressDeadlineExceeded on the COS) for a later PR to keep this one small, I think that's fine
There was a problem hiding this comment.
Done. COD no longer copies progressDeadlineMinutes into new COS revisions or consults their Ready=False/ProgressDeadlineExceeded condition. It calculates the budget from its template and the revision's creation and completion timestamps. COS deadline support remains for ClusterExtension installations until that path is migrated.
| if !hasDeadline { | ||
| r.pastDeadline.Delete(item) | ||
| return backoff | ||
| } |
There was a problem hiding this comment.
I wonder if we should distinguish between errors and no deadline, something like:
| if !hasDeadline { | |
| r.pastDeadline.Delete(item) | |
| return backoff | |
| } | |
| remaining, hasDeadline, err := r.remaining(item) | |
| if err != nil { | |
| // Couldn't determine the deadline (transient read error). Do NOT touch the | |
| // past-deadline marker — deleting it here is lets intermittent errors | |
| // re-arm the one-shot immediate requeue. We have no remaining | |
| // value to cap against, so fall back to the delegate backoff. | |
| return backoff | |
| } | |
| if !hasDeadline { | |
| r.pastDeadline.Delete(item) | |
| return backoff | |
| } |
There was a problem hiding this comment.
Done. Deadline lookups now return errors separately from the absence of a deadline. A transient Get or List failure uses the delegate backoff without clearing the expired-deadline marker. Regression tests cover intermittent failures and verify that recovery does not re-arm the immediate retry.
| remaining: func(req ctrl.Request) (time.Duration, bool) { | ||
| ctx := context.Background() | ||
| cod := &ocv1.ClusterObjectDeployment{} | ||
| if err := reader.Get(ctx, req.NamespacedName, cod); err != nil || !cod.DeletionTimestamp.IsZero() { | ||
| return 0, false | ||
| } | ||
| var revisions ocv1.ClusterObjectSetList | ||
| if err := reader.List(ctx, &revisions, client.MatchingFields{clusterObjectSetGroupIndex: cod.Name}); err != nil { | ||
| return 0, false | ||
| } | ||
| latest := latestActiveRevision(revisions.Items) | ||
| if latest == nil || latest.Labels[labels.TemplateHashKey] != templateHash(cod.Spec.Template) { | ||
| // Creating the desired revision is outside the MVP deadline. | ||
| return 0, false | ||
| } | ||
| return durationUntilDeploymentDeadline(clk, latest) | ||
| }, |
There was a problem hiding this comment.
maybe we should also consider the deletion case?
| remaining: func(req ctrl.Request) (time.Duration, bool) { | |
| ctx := context.Background() | |
| cod := &ocv1.ClusterObjectDeployment{} | |
| if err := reader.Get(ctx, req.NamespacedName, cod); err != nil || !cod.DeletionTimestamp.IsZero() { | |
| return 0, false | |
| } | |
| var revisions ocv1.ClusterObjectSetList | |
| if err := reader.List(ctx, &revisions, client.MatchingFields{clusterObjectSetGroupIndex: cod.Name}); err != nil { | |
| return 0, false | |
| } | |
| latest := latestActiveRevision(revisions.Items) | |
| if latest == nil || latest.Labels[labels.TemplateHashKey] != templateHash(cod.Spec.Template) { | |
| // Creating the desired revision is outside the MVP deadline. | |
| return 0, false | |
| } | |
| return durationUntilDeploymentDeadline(clk, latest) | |
| }, | |
| remaining: func(req ctrl.Request) (time.Duration, bool, error) { | |
| ctx := context.Background() | |
| cod := &ocv1.ClusterObjectDeployment{} | |
| if err := reader.Get(ctx, req.NamespacedName, cod); err != nil { | |
| return 0, false, err // unknown | |
| } | |
| if !cod.DeletionTimestamp.IsZero() { | |
| return 0, false, nil // deleting: definitively no deadline | |
| } | |
| var revisions ocv1.ClusterObjectSetList | |
| if err := reader.List(ctx, &revisions, client.MatchingFields{clusterObjectSetGroupIndex: cod.Name}); err != nil { | |
| return 0, false, err // unknown | |
| } | |
| latest := latestActiveRevision(revisions.Items) | |
| if latest == nil || latest.Labels[labels.TemplateHashKey] != templateHash(cod.Spec.Template) { | |
| return 0, false, nil // creating desired revision: outside MVP deadline | |
| } | |
| d, has := durationUntilDeploymentDeadline(clk, latest) | |
| return d, has, nil | |
| }, |
There was a problem hiding this comment.
Done. Missing or deleting deployments and missing or deleting desired revisions definitively have no active deadline and clear the retry marker. Transient lookup errors preserve it. Added regression coverage for both cases.
Calculate the deployment deadline from the desired COS creation timestamp through its first successful completion. Enforce the deadline even when COS readiness remains Pending, PreconditionsNotMet, or another non-deadline reason. Schedule the COD at the deadline and cap retry backoff so a quiet or failing revision cannot leave the deployment progressing indefinitely. Reuse the revision completion latch to allow late recovery without timing completed revisions again. Keep time waiting for COS creation outside the MVP budget. Cover fixed budgets across phases, blocked revisions, retry scheduling, creation retries, late completion, and availability from an older revision. Signed-off-by: Fabricio Aguiar <fabricio.aguiar@gmail.com> rh-pre-commit.version: 2.3.2 rh-pre-commit.check-secrets: ENABLED
Regenerate apply configuration comments, API reference documentation, the COD CRD, and experimental manifests from the clarified deadline contract. Signed-off-by: Fabricio Aguiar <fabricio.aguiar@gmail.com> rh-pre-commit.version: 2.3.2 rh-pre-commit.check-secrets: ENABLED
c01bfa4 to
d97b31b
Compare
Description
The progression timeout covers how long a ClusterObjectDeployment (COD) takes
to progress to its desired new revision. The timer starts when that revision's
ClusterObjectSet (COS) is created and ends when the COS records its first
status.completedAt, meaning the entire revision has completed rollout. The CODthen reports
Progressing=Falsewith reasonNewClusterObjectSetProgressed.All phases share one budget. A five-phase revision with a ten-minute deadline
gets ten minutes total; completing a phase does not reset the timer. Time spent
waiting to create the COS is excluded, and each new revision gets a fresh budget.
Enforce the deadline directly in the COD controller so it reports
Progressing=Falsewith reasonProgressDeadlineExceededwhen the budget expiresbefore completion, even if the COS reports
PreconditionsNotMet,InvalidSpecification, or no readiness condition. Schedule a deadline check evenwhen the COS emits no further events, and cap error backoff at the remaining
budget.
Reconciliation continues after timeout, allowing late revision completion to
recover. Completed revisions are not timed again after health regressions.
Availability from an older ready revision is preserved.
Jira: OPRUN-4791
Validation:
direct COD enforcement.
retries, late completion, health regression, and retained availability.
make test-unit: all 51 test packages passed with race detection.make lint: zero issues.make verify: passed with no generated-file changes.make lint-api-diff BASELINE_BRANCH=upstream/main: no new findings.Reviewer Checklist
Summary by CodeRabbit