Skip to content

feat: Add the file-based override source - #529

Draft
kinyoklion wants to merge 1 commit into
rlamb/overrides-python-eventsfrom
rlamb/overrides-python-file-source
Draft

kinyoklion wants to merge 1 commit into
rlamb/overrides-python-eventsfrom
rlamb/overrides-python-file-source

Conversation

@kinyoklion

Copy link
Copy Markdown
Member

Summary

This is the fifth step of the flag overrides port described by the OVERRIDE specification. It is based on the events branch because the phases are stacked; retarget to feat/overrides once that branch merges.

ldclient.integrations.overrides.FileOverrideSourceBuilder is the public entry point. It is passed to datasystem.ConfigBuilder.overrides(...) and builds an override source that reads one or more local files in the file data source document format (optional flags, flagValues, and segments members, JSON or YAML) and supplies each successful load to the SDK's override sink as a full snapshot. flagValues entries expand into full flag definitions through the shared file data code, so the layer holds only full entities.

Behavior:

  • Files are combined in the configured order. duplicate_keys_handling is fail by default, which rejects the reload and keeps the previously loaded overrides, or ignore, which keeps the first configured file's entry.
  • A configured file that does not exist contributes no overrides and is not an error: it can be created later, deleting a file removes its overrides, and deleting the last file clears the layer.
  • A file that exists but cannot be read or parsed fails that whole reload. The last good overrides stay in effect, the failure is logged, and the load is retried after a bounded delay and on the next detected change, so a file observed mid-write recovers on its own.
  • change_detection is one of two alternatives. polling (the default) compares modification time and size on an interval, one second by default with a one second minimum; a lower interval is raised to the minimum with a warning. watching uses the watchdog package and is a construction error when the package is not installed. Notifications are debounced so a burst from one edit produces one reload.
  • The initial load runs synchronously inside start, which the data system calls during client construction, so an override present at startup takes effect from the first evaluation.
  • Every applied change is logged at Info level, for example Flag overrides in effect: 2 flags, 1 segment (/etc/ld/a.json: 2 flags, 1 segment; /etc/ld/b.json: absent), or Flag overrides: none in effect (...).
  • A builder with no paths is a construction error. Unknown change_detection or duplicate_keys_handling values raise ValueError when set. This is stricter than the Go builder, which treats an unrecognized duplicate keys handling as fail.

The module is added to the API reference (docs/api-integrations.rst) and every public docstring carries the experimental note.

Tests cover the builder validation and defaults, synchronous initial load, YAML, multi-file order and duplicate handling, absent files appearing and disappearing, the Info log lines, both change detection modes, last-good retention across a malformed edit in both modes, the automatic retry with no change signal, close semantics, and an end-to-end run through LDClient where a file is added, changed, and emptied while the client never receives LaunchDarkly data.

Adds ldclient.integrations.overrides.FileOverrideSourceBuilder, the
file-based override source described by the OVERRIDE specification. The
source reads one or more JSON or YAML files in the file data source document
format, with optional flags, flagValues, and segments members, and supplies
each successful load to the SDK's override sink as a full snapshot.

Files are combined in the configured order. The duplicate keys handling is
fail by default, which rejects the reload and keeps the previously loaded
overrides, or ignore, which keeps the first file's entry. A configured file
that does not exist contributes no overrides, so a file can be created later
and deleting a file removes its overrides. A file that exists but cannot be
read or parsed fails that reload, the last good overrides stay in effect, the
failure is logged, and the load is retried after a bounded delay and on the
next detected change. Change detection is one of two modes: polling, the
default, examines the files once per second by default with a one second
minimum, and watching reacts to file system notifications through the
watchdog package. Watching without the watchdog package and a builder with no
paths are construction errors. The initial load completes during client
construction. Every applied change is logged at Info level with the overrides
in effect and what each configured file supplied.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant