Skip to content

feat: Add file data loading code with reload, retry, and polling - #525

Draft
kinyoklion wants to merge 1 commit into
feat/overridesfrom
rlamb/overrides-python-filedata-reloader
Draft

kinyoklion wants to merge 1 commit into
feat/overridesfrom
rlamb/overrides-python-filedata-reloader

Conversation

@kinyoklion

@kinyoklion kinyoklion commented Sep 25, 2026 •

Copy link
Copy Markdown
Member

Summary

Adds ldclient.impl.integrations.files.filedata, the file reading, parsing, and merging logic that the file-based override source described by the OVERRIDE specification is built on. The specification requires ordered multi-file merging with duplicate keys handling, a reload that keeps the last good data across a malformed edit and retries, debouncing of change notifications, a polling change detector as well as a watching one, and correct handling of a configured file that does not exist yet. This module provides those pieces; the override source itself is added in a later PR of the stack.

The existing file data sources are not changed. The FDv1 file update processor (Files.new_data_source) and the FDv2 file initializer and synchronizer (Files.new_data_source_v2, datasystem.file_ds_builder) keep their current implementation and behavior in every respect: the flagValues expansion (an on flag with fallthrough variation 0 and version 1, evaluated with a FALLTHROUGH reason), the version fallback that stamps version 1 on entries without one, the failure messages and their levels, the FDv2 synchronizer's OFF state on a failed initial load and its INVALID_DATA error kind, and the polling and watching rules. A new test module, test_file_data_sources_pinned_behavior.py, pins each of those behaviors so the override feature stays purely additive. The baseline test files for the existing sources are unmodified.

What the new module provides for the override source:

  • Document parsing. A document that starts with an opening brace is parsed as JSON and any other document as YAML. Flag and segment definitions are decoded into the model classes while the file is read, so an invalid definition fails that load. A definition may omit its own key and version, which are filled in from the map key and a version of 1.
  • Merging. Files are combined in the configured order. The duplicate keys handling is fail (the load fails) or ignore (the first file's entry is kept). The result records how many entries each file supplied.
  • Reloader. Serializes reloads, debounces change signals with a settle window that each signal extends, keeps the last good data by not applying a failed load, retries a failed load after a bounded delay so a file observed mid-write recovers without another notification, reports an identical failure once, skips an application whose file contents are byte-identical to the last applied contents, and applies a success after a failure even when the contents did not change. It can treat a configured file that does not exist as a file with no content. Closing does not wait for an in-flight reload.
  • Poller. Compares modification time and size on an interval, so a same-size rewrite and a same-time size change are both detected, as are files that appear or disappear.
  • Watcher. Uses the watchdog package. It watches the directory of each file so an absent file is picked up when it appears, matches the destination of a move event so a file written by rename is detected, retries a directory that does not exist yet, and reacts only to notifications that can change a file's content or presence (the watchdog inotify mask also reports a file being opened or read).

Tests cover parsing, merging, loading, the reloader (initial load, failure retention, debounce coalescing and window extension, retry with and without further signals, identical-failure reporting, skip-unchanged and recovery, serialization of concurrent reloads, close semantics, worker thread lifecycle), the poller, and the watcher, plus the pinned behavior of the existing sources.

Adds ldclient.impl.integrations.files.filedata, the file reading, parsing,
and merging logic that the file-based override source is built on. A
document that starts with an opening brace is parsed as JSON and any other
document as YAML. Definitions are decoded into the flag and segment models
while the file is read, so an invalid definition fails that load. Files are
merged in the configured order with a duplicate keys handling of fail or
ignore, and the result records how many entries each file supplied.

The Reloader owns the reload cycle: it serializes reloads, debounces change
signals with a settle window, keeps the last good data by not applying a
failed load, retries a failed load after a bounded delay, reports an identical
failure once, and skips an application whose file contents did not change. It
can treat a configured file that does not exist as a file with no content.
The Poller detects changes by comparing modification time and size on an
interval, including files that appear or disappear. The Watcher uses the
watchdog package, watches the directory of each file so an absent file is
picked up when it appears, matches the destination of a move so a file
written by rename is detected, retries a directory that does not exist yet,
and reacts only to notifications that can change a file's content or
presence.

The existing file data sources are not changed and keep their current
behavior. A new test module pins that behavior: the flagValues expansion and
its evaluation reason, the version fallback, the failure messages, the FDv2
status and error kinds, and the polling and watching rules.
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-filedata-reloader branch from c478bd1 to 7e64242 Compare September 28, 2026 20:28
@kinyoklion kinyoklion changed the title feat: Add shared file data code with reload, retry, and polling feat: Add file data loading code with reload, retry, and polling Sep 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant