[abelonogov-ld]

Requirements

• I have added test coverage for new or changed functionality
• I have followed the repository's pull request submission guidelines
• I have validated my changes against all supported platform versions

Related issues

Tier 3 of the Event Durability spec, "Tier 3 — persistence". Stacked on #403 (tier 2) and targets that branch, so this diff shows only tier 3. It will be retargeted once the branches below it merge. The iOS counterpart is launchdarkly/ios-client-sdk#538, and both write the same on-disk format.

Opened as a draft: a multi-threading and readability review turned up items still being worked through (see Known open items).

Describe the solution you've provided

Recorded events go to an append-only log on disk rather than only an in-memory buffer, so an application that dies shortly after recording an event still reports it: the next run delivers what the previous one left behind. That covers what tier 2 cannot: SIGKILL, ANR kills, native crashes, and the system reclaiming a backgrounded process.

• EventPersistence (new public enum) and EventProcessorBuilder.eventPersistence(...):
  • DISABLED (default): events stay in memory and the SDK does not touch the filesystem for events.
  • DEFERRED: writes happen on the SDK's commit thread, a moment after recording.
  • IMMEDIATE: track and identify write on the caller's thread before returning, so there is no window.
• EventStore: an append-only log under the no-backup files directory, one directory per mobile key. It handles framing, capacity, torn-tail recovery of a log interrupted by a crash, and a fallback to memory when a write fails (for example, a full disk), so a disk failure never takes the application down. Each process appends to its own open-<process> log; closed ready-<payloadId> batches are shared, so a batch left by a process that never runs again is still delivered. The byte format lives in EventLogFormat and the log this process appends to in DiskEventLog, mirroring the iOS split.
• Recording and commits (DirectEventProcessor): an evaluation appends to a pending run and updates a counter, with no encoding on the caller's thread. A commit encodes the run and its summaries together and stages them in the store. track and identify are commit points. Evaluations commit once a threshold accumulates, on flush, and before delivery.
• Delivery reads batches oldest first. A batch keeps its payload ID across retries and process restarts, so LaunchDarkly discards a repeated delivery instead of double counting. Delivery stops at the first retryable failure, schedules a single retry instead of sleeping on the events thread, and removes a batch only once it has been accepted or permanently refused. A batch whose read fails transiently is kept for a later attempt.
• Loss reporting: flushAsync() answers false when events the SDK accepted were lost for good: an event or summary that could not be serialized, or that was too large to store. Each loss is reported to exactly one caller, the one whose delivery closed its batch after the loss, as on iOS. Events dropped for capacity are counted in diagnostics, not reported as losses.
• close() commits first, then delivers with what is left of one two-second budget. Neither step waits on the caller's thread past the budget, so a stalled disk write cannot become an ANR. Work that outlasts the budget finishes in the background.
• AnalyticsEventSender posts a pre-assembled body under an explicit payload ID.
• DEFAULT_CAPACITY goes from 100 to 1000, matching iOS, since capacity now also bounds what is held on disk.

Describe alternatives you've considered

SQLite. Measured as a store and rejected: it costs more per event than an append to a log the SDK already frames, for durability properties the log gets from write alone. See sqlite-event-persistence-findings.md in the spec folder.

Fsync on every commit. What durability is defending against is the process dying, not the device losing power. Bytes handed to the kernel survive the former, and an fsync per event would dominate the cost of track.

One shared log for all processes. That is simpler, but one process's startup recovery could rename a log another live process is still appending to. A log per process avoids that without file locks.

Additional context

Tests. 879 unit tests pass locally (testDebugUnitTest). New coverage includes EventStoreMultiProcessTest (several stores on one directory, recovery, transient read failures, events still counted against capacity while being written), DirectEventProcessorTest (commit points, retry, capacity across memory and disk, persistence failing partway through, oversized frames reported as losses, a loss reported to the right flush across a retry, close() within budget on a stalled disk write), OutboundEventBufferSerializationTest, and SameContextPerContextEventSummarizerTest.

Benchmarks. EventPersistenceBenchmark (JVM and instrumented) measures recording cost per mode. It runs only when LD_EVENT_BENCH=1, so ordinary test runs are unaffected.

Test app. Now configures EventPersistence.IMMEDIATE, so the events recorded by Eval+Kill now, which tier 2 loses, are recovered and delivered on the next launch. It also gains Over-Refresh Eval, a main-thread stress loop that compares evaluation cost while filling capacity and after it is full.

Known open items (from the threading review, to be resolved before this leaves draft):

• flushAsync(), and so flushAndWait, does not commit on the caller's thread as flush() does, so a flushAndWait that times out can leave events only in memory, even at IMMEDIATE.
• Batches left by a previous run are not counted against capacity until the first listing, which an offline start postpones.
• The store and the processor both schedule commits, so a threshold commit is followed by a redundant no-op pass.

Version. EventPersistence and EventProcessorBuilder.eventPersistence have no @since tag yet; one should be added once the release version is known.

---

Note

Overview
Tier 3 event durability replaces the in-memory DefaultEventProcessor wrapper with a new DirectEventProcessor stack that writes to an on-disk EventStore (append-only logs under getNoBackupFilesDir(), per mobile key), so events survive process death and are delivered on the next launch with a stable payload ID via AnalyticsEventSender.

Applications opt in through EventPersistence (DISABLED / DEFERRED / IMMEDIATE) on EventProcessorBuilder. IMMEDIATE commits track/identify on the caller thread; delivery adds retry/backoff, launch gating and pacing that defer to in-flight flag loads (FlagRequestListener wired from ConnectivityManager), and flushAsync / flushAndWait semantics (including multi-env parallel flush and crash-handler delivery tests).

Supporting pieces include BatchLocks / DiskEventLog, recovery of interrupted logs, DeliveryRetryState, default event capacity raised to 1000, launchdarkly-java-sdk-internal 1.12.2, optional LD_EVENT_BENCH device benchmarks, and the example app switching to blocking LDClient.init(..., timeout).

^{Reviewed by Cursor Bugbot for commit 51aa35e. Bugbot is set up for automated code reviews on this repo. Configure here.}

[tanderson-ld]

Do we still need this Math.min check if we have in.getChannel().size?

Semi related, where do we impose a maximum size we are willing to read that matches the 2MB max file size we're expecting these to be?

[abelonogov-ld]

        try (SeekableByteChannel sbc = Files.newByteChannel(path);
             InputStream in = Channels.newInputStream(sbc)) {
            long size = sbc.size();
            if (size > (long)MAX_BUFFER_SIZE)
                throw new OutOfMemoryError("Required array size too large");

            return read(in, (int)size);
        }```

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 33af63a. Configure here.}

[tanderson-ld]

This seems hacky, and also a bit of a lie since it is saying a request was answered, but in this case no request was sent.

[tanderson-ld]

This is a hack to use in real code.

[tanderson-ld]

I'm not sure this should be communicating the request/response information to other parts of the code if avoidable.

Why not just communicate that one component wants network layer priority? Then the component can hold network layer priority across multiple requests and we don't need to have the concept of loaded, failed, stopped permeating out of this API into other parts of the code.

[tanderson-ld]

This seems like way too much logic / pollution of concerns to be embedded in the direct event processor itself. This is approaching a QoS system and that is too much to be in this class.

[tanderson-ld]

We should ideally not reveal from one component needing network priority that it has requests in flight. It really should just be that it needs the network layer bandwidth and what it will do with that bandwidth is up to it. It may make a thousand requests, who knows, other portions of our code shouldn't need to know.