Skip to content

feat: Add the file-based override source - #529

Draft
kinyoklion wants to merge 6 commits into
rlamb/overrides-python-eventsfrom
rlamb/overrides-python-file-source
Draft

kinyoklion wants to merge 6 commits into
rlamb/overrides-python-eventsfrom
rlamb/overrides-python-file-source

Conversation

@kinyoklion

@kinyoklion kinyoklion commented Sep 25, 2026 •

Copy link
Copy Markdown
Member

Summary

This is the fifth step of the flag overrides port described by the OVERRIDE specification. It is based on the events branch because the phases are stacked; retarget to feat/overrides once that branch merges.

ldclient.integrations.overrides.FileOverrideSourceBuilder is the public entry point. It is passed to datasystem.ConfigBuilder.overrides(...) and builds an override source that reads one or more local files in the file data source document format (optional flags, flagValues, and segments members, JSON or YAML) and supplies each successful load to the SDK's override sink as a full snapshot. flagValues entries expand into full flag definitions, so the layer holds only full entities; as the specification describes for a value-only override, the expansion is a flag that is on and serves its single variation as the fallthrough, so the evaluation reason kind is FALLTHROUGH. The existing file data sources keep their current behavior, including their own flagValues expansion.

Behavior:

  • Files are combined in the configured order. duplicate_keys_handling is fail by default, which rejects the reload and keeps the previously loaded overrides, or ignore, which keeps the first configured file's entry.
  • A configured file that does not exist contributes no overrides and is not an error: it can be created later, deleting a file removes its overrides, and deleting the last file clears the layer.
  • Change detection is in place before the initial load, so a change made between the two is not missed.
  • A file that exists but cannot be read or parsed fails that whole reload. The last good overrides stay in effect, the failure is logged, and the load is retried after a bounded delay and on the next detected change, so a file observed mid-write recovers on its own.
  • change_detection is one of two alternatives. polling (the default) compares modification time and size on an interval, one second by default with a one second minimum; a lower interval is raised to the minimum with a warning, and a value that is not a finite number, or longer than the runtime can wait, is a construction error. watching uses the watchdog package and is a construction error when the package is not installed. Notifications are debounced so a burst from one edit produces one reload.
  • The initial load runs synchronously inside start, which the data system calls during client construction, so an override present at startup takes effect from the first evaluation.
  • Every applied change is logged at Info level, for example Flag overrides in effect: 2 flags, 1 segment (/etc/ld/a.json: 2 flags, 1 segment; /etc/ld/b.json: absent), or Flag overrides: none in effect (...).
  • A builder with no paths is a construction error. Unknown change_detection or duplicate_keys_handling values raise ValueError when set. This is stricter than the Go builder, which treats an unrecognized duplicate keys handling as fail. A path listed more than once is logged at warning level and the repeats are ignored; the first occurrence keeps its place in the order. Paths are compared the way the platform compares file names (os.path.normcase): without regard to case on Windows, exactly elsewhere.

The module is added to the API reference (docs/api-integrations.rst) and every public docstring carries the experimental note.

Tests cover the builder validation and defaults, synchronous initial load, YAML, multi-file order and duplicate handling, absent files appearing and disappearing, the Info log lines, both change detection modes, last-good retention across a malformed edit in both modes, the automatic retry with no change signal, close semantics, and an end-to-end run through LDClient where a file is added, changed, and emptied while the client never receives LaunchDarkly data.

SDK-3250


Note

Overview
Adds experimental file-based flag overrides via FileOverrideSourceBuilder, wired through datasystem with overrides(...).

The source reads one or more local JSON/YAML files (same shape as the file data source: flags, flagValues, segments), merges them in path order, and pushes full snapshots to the SDK override layer on each successful load. The initial load runs synchronously in start, so startup overrides apply before the client is used; missing files contribute nothing, while read/parse failures keep the last good snapshot and retry.

Operators can choose polling (default, 1s interval with a minimum) or watchdog-based watching, plus duplicate-key handling (fail vs ignore). Each applied change logs an Info line summarizing flags/segments per file. Public API and Sphinx docs document ldclient.integrations.overrides; a large integration test suite covers builder validation, reload behavior, failure retention, and end-to-end LDClient evaluation.

Reviewed by Cursor Bugbot for commit 99b851f. Bugbot is set up for automated code reviews on this repo. Configure here.

Opened from termag

@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-events branch from 042fce2 to 0e55650 Compare September 28, 2026 20:29
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-file-source branch from 021778c to 2074b74 Compare September 28, 2026 20:29
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-events branch from 0e55650 to c79f244 Compare September 30, 2026 20:31
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-file-source branch from 2074b74 to 1950d45 Compare September 30, 2026 20:31
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-events branch from c79f244 to 352d360 Compare October 1, 2026 23:43
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-file-source branch from 1950d45 to 3e47045 Compare October 1, 2026 23:43
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-events branch from 352d360 to 9f4108c Compare October 3, 2026 01:18
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-file-source branch from 3e47045 to cfb781a Compare October 3, 2026 01:18
@kinyoklion

Copy link
Copy Markdown
Member Author

bugbot review

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-events branch from 9f4108c to 051cf7d Compare October 13, 2026 21:02
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-file-source branch from cfb781a to ea4dc19 Compare October 13, 2026 21:02
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-events branch from 051cf7d to 0124739 Compare October 13, 2026 22:40
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-file-source branch from ea4dc19 to 3de11ab Compare October 13, 2026 22:40
@kinyoklion

Copy link
Copy Markdown
Member Author

bugbot review

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-events branch from 0124739 to 224677f Compare October 16, 2026 21:34
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-file-source branch from bb5a108 to 5e42ebc Compare October 16, 2026 21:34
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-events branch from 224677f to 2b471ae Compare October 28, 2026 22:17
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-file-source branch from 5e42ebc to 314dea2 Compare October 28, 2026 22:17
@kinyoklion

Copy link
Copy Markdown
Member Author

bugbot review

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-events branch from 2b471ae to c5a69c9 Compare October 28, 2026 22:40
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-file-source branch from 314dea2 to 4bace41 Compare October 28, 2026 22:40
@kinyoklion

Copy link
Copy Markdown
Member Author

bugbot review

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 99b851f. Configure here.

@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-events branch from c5a69c9 to 70ada89 Compare October 29, 2026 16:48
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-file-source branch from 99b851f to fac9ec8 Compare October 29, 2026 16:48
Adds ldclient.integrations.overrides.FileOverrideSourceBuilder, the
file-based override source described by the OVERRIDE specification. The
source reads one or more JSON or YAML files in the file data source document
format, with optional flags, flagValues, and segments members, and supplies
each successful load to the SDK's override sink as a full snapshot.

Files are combined in the configured order. The duplicate keys handling is
fail by default, which rejects the reload and keeps the previously loaded
overrides, or ignore, which keeps the first file's entry. A configured file
that does not exist contributes no overrides, so a file can be created later
and deleting a file removes its overrides. A file that exists but cannot be
read or parsed fails that reload, the last good overrides stay in effect, the
failure is logged, and the load is retried after a bounded delay and on the
next detected change. Change detection is one of two modes: polling, the
default, examines the files once per second by default with a one second
minimum, and watching reacts to file system notifications through the
watchdog package. Watching without the watchdog package and a builder with no
paths are construction errors. The initial load completes during client
construction. Every applied change is logged at Info level with the overrides
in effect and what each configured file supplied.
…l of the file override source

Change detection is set up before the initial load, so an edit made while the files are first read is picked up. Close stops the reloader even when the change detector fails to close. The builder rejects a poll interval that is not a number or not finite.
…ntries as duplicate keys

A path listed more than once would make every entry of that file a duplicate of itself: under the
default fail handling the load failed, and under ignore the first listing won by accident. The
builder now drops the repeats, keeps the first occurrence in its place in the order, and logs a
warning once at build time. This is the behavior the Node SDK has.
…t on Windows

The builder dropped repeated paths by exact string match, so on Windows
a path listed twice in different case survived as two paths, and every
key of that file failed as a duplicate of itself under the default fail
handling. Paths are now compared with os.path.normcase, which ignores
case on Windows and compares exactly elsewhere. The first spelling keeps
its place in the order.
… SDKs

A flagValues entry becomes a flag without client-side availability, so
an all-flags state for client-side SDKs leaves it out even when the flag
it overrides is client-side. A full definition with clientSide is the
way to override such a flag. A test pins both.
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-events branch from 70ada89 to cad8f0b Compare October 29, 2026 17:16
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-file-source branch from fac9ec8 to 0415ec1 Compare October 29, 2026 17:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant