[Quinn-With-Two-Ns]

What changed?

• Track replaying mutable-side-effect invocations separately from the command queue and resolve skipped invocations after their workflow task's command events have been applied.
• Reuse the existing cancellation/cached-result transition, preserving callback workflow time without sweeping unrelated commands.
• Introduce MUTABLE_SIDE_EFFECT_REPLAY_RECOVERY, disabled by default for its first release. Register opt-in usage at the mutableSideEffect call site and honor recorded flags during replay.
• Add cache-eviction, callback-time, flag-registration, and negative nondeterminism-detection tests. Retain only the commandless-task and callback-time history fixtures.

Skip-count calculation and marker matching are unchanged.

Why?

Refs #3163.

A skipped mutable side effect has no corresponding marker event. During full replay, its internal placeholder can remain pending when there is no later command event to trigger cleanup, and an empty RecordMarker command can reach the server.

Unconditionally notifying the pending invocation at the next task start changes the workflow time observed by its continuation and can break replay. The added timing regression covers that case; flagged cleanup instead runs at the completed task's boundary, before the next task advances time or delivers signals.

Breaking changes?

None to public APIs. The new SDK flag remains disabled by default (Enabled: pending) for rollout compatibility; tests opt in explicitly. Unflagged histories retain their existing replay behavior.

Server PR

None required.

Validation

• ./gradlew --offline spotlessApply
• Retained state-machine and mutable-side-effect tests: 14 passed.

  ./gradlew --offline :temporal-sdk:test \
    --tests io.temporal.internal.statemachines.MutableSideEffectStateMachineTest \
    --tests io.temporal.workflow.MutableSideEffectReplayTest \
    --tests io.temporal.workflow.MutableSideEffectTest

• Real-server configuration: 9 passed.

  ./gradlew --offline -PtestServer=dev-server :temporal-sdk:test \
    --tests io.temporal.workflow.MutableSideEffectReplayTest \
    --tests io.temporal.workflow.MutableSideEffectTest

• One-off compatibility validation generated 20 histories with the unmodified baseline SDK (adf0fc8) on the test and real servers. Baseline replay passed first, followed by 40 patched replay cases comparing exact callback values, order, and workflow times with opt-in off/on. The bulk corpus and generator are intentionally not included in this PR.
• Broader focused state-machine, side-effect, versioning, query, sticky-worker, and SDK-flag tests passed during development. The full repository suite has not been run.

---

Note

Medium Risk
Changes workflow replay state-machine behavior in a determinism-sensitive path, though the new behavior is gated behind a default-off SDK flag with extensive replay coverage.

Overview
Fixes replay after cache eviction or worker restart when a Workflow.mutableSideEffect call is skipped (no marker recorded): pending replay placeholders no longer stick around and emit invalid RecordMarker commands.

The change adds opt-in SDK flag MUTABLE_SIDE_EFFECT_REPLAY_RECOVERY (introduced in 1.41.0, disabled by default). When enabled, replay tracks eligible invocations and, at workflow task completion (after that task’s command events), cancels still-pending placeholders and delivers the cached value so callbacks run at the original task time—not at the next task start, which would shift Workflow.currentTimeMillis() and break determinism. Flag use is recorded at the mutableSideEffect call site, and eligibility is fixed per invocation so a flag recorded on a later task cannot retime an earlier skipped call.

CHANGELOG, state-machine unit tests, workflow replay tests (including captured histories), and negative nondeterminism checks are included; marker matching and skip-count logic are unchanged.

^{Reviewed by Cursor Bugbot for commit dcac469. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 9e5c687. Configure here.}