Skip to content

[2/4] Stateful shared memory and reactive online processing - #146

Merged
eeddy merged 5 commits into
latestfrom
pr142/02-stateful-core
Nov 10, 2026
Merged

eeddy merged 5 commits into
latestfrom
pr142/02-stateful-core

Conversation

@ECEEvanCampbell

@ECEEvanCampbell ECEEvanCampbell commented Nov 9, 2026 •

Copy link
Copy Markdown
Collaborator

Polling/copying entire buffers just to detect arrivals is replaced with published shared-memory state and notifications. This part adds atomic commit/apply operations, state snapshots, window/delta reads, notifier pools, reactive criteria/hooks/executors, event logging, and the corresponding streamer/predictor/adaptation integration.

The existing five core regression modules are included here, before the GUI work. Latest's Emager v3 path is retained; its writer and the upstream-only MindRove/Muovi Plus writers publish state through the same atomic contract. Their existing sample ordering and counts are covered by integration tests.

Validation: 153 passed across the stateful/reactive/adaptation/event-log suites, feature and metric checks, and retained-upstream integration tests.

Part 2/4, split from #142 and rebuilt on current latest (06ba120). Review this part alone: single feature commit · incremental comparison.

Upstream overlap removed: libemg/datasets.py, libemg/_datasets/, libemg/offline_metrics.py, and libemg/_discrete_models/ are unchanged from latest in every part. Existing formula tests, CI, README, and upstream-only assets are also preserved. The rewritten history contains one feature commit per part; it does not replay the older copied/squashed upstream history.

Implementation provenance: replayed feature delta 378d6c1..4f65170 from the original branch, with conflict resolutions for newer upstream behavior. The original compose_emgpredictor branch remains available; local pre-rebase backup refs are retained.

All four PRs still target latest. Review and merge in order: #145 → #146 → #147 → #148. Until preceding parts merge, GitHub's PR-wide diff is cumulative; the commit/comparison links above isolate this part. Use merge commits to preserve the chain, or rebase the remaining parts after a squash merge.

Test environment: Python 3.13.17 / NumPy 2.4.6, with latest's device/discrete dependencies installed in an isolated test directory. Draft status is retained for review and hardware validation.

The combined Sphinx HTML documentation build succeeds; documentation warnings remain.

ECEEvanCampbell and others added 2 commits November 9, 2026 14:34
Replay only the feature delta 265d0ae..378d6c1. Preserve latest datasets, metrics formulas, discrete models, device additions, channel-last windows and spacebar collection. Keep NumPy compatibility fixes without the unrelated metric-fixture rename. Defer reactive and pipeline docs to their implementations; omit generated environment locks.

Co-authored-by: Christian Morrell <cmorrell@unb.ca>
Replayed from 4f65170 with the five existing core test modules brought forward from 16cdc6b. Preserve latest Myo address and Emager version support, multi-rate/discrete offline window parsing, and integrate the upstream MindRove and Muovi Plus writers with atomic state publication. Includes the reactive documentation alongside its implementation.
@eeddy

eeddy commented Nov 10, 2026

Copy link
Copy Markdown
Collaborator

@copilot can you rebase on latest?

Copilot AI and others added 2 commits November 10, 2026 03:46
# Conflicts:
#	docs/source/emg_toolbox.rst
#	libemg/_streamers/_oymotion_streamer.py
#	libemg/_streamers/_sifi_bridge_streamer.py
#	libemg/adaptation/managers.py
#	libemg/emg_predictor.py
#	libemg/environments/__init__.py
#	libemg/output_writer.py
#	libemg/shared_memory_manager.py
#	libemg/streamers.py
#	requirements.txt
#	setup.py

Co-authored-by: eeddy <31893850+eeddy@users.noreply.github.com>
Co-authored-by: eeddy <31893850+eeddy@users.noreply.github.com>

Copilot AI commented Nov 10, 2026

Copy link
Copy Markdown

@copilot can you rebase on latest?

Done — I merged latest into this branch (instead of rebasing, since rebasing would require a force-push). Addressed in 5909c52, with follow-up workflow permission hardening in 52a93a4.

@eeddy

eeddy commented Nov 10, 2026

Copy link
Copy Markdown
Collaborator

@ECEEvanCampbell A couple high level questions before I do a full review:

(1) Does this overwrite all of the non-reactive functionality from before or is it just additive?
(2) Do we have an idea of whether there is any latency that can ever be introduced?

Generally I like and get the concept but I wonder if it adds un-needed complexity if its forced? LMK what you think,

@ECEEvanCampbell

ECEEvanCampbell commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator Author
  1. In the codebase it overwrote the non-reactive functionality from before in the sense that any logic that would have polled (e.g., waiting for a window size) would now be a callback that is triggered when the hook triggers (Online streamers driven by hooks rather than polling went from 93.7% to 10.7% CPU). Nothing is stopping you from writing the functions that run in loops checking for a window increment being accrued the old way (the shared memory counter is under a different tag though "emg_meta" is a structure now with the sample counts in it among many other things).
  2. There isn't any latency gained or lost. It just stops the CPU from pinning from the core running the polling. Waking from a hook has mean wake latency of 62 µs, so it adds hardly anything (probably less than polling and querying the old style had).

We do sort of require this kind of rewrite when we move to supporting many of our capabilities in the gui. Instead of needing to spawn a process for each of the components that might trigger off live data coming in, we are able to hook into the inputs and update them when their inputs have changed in a meaningful order in one process (rather than having 10 processes polling).

@eeddy

eeddy commented Nov 10, 2026

Copy link
Copy Markdown
Collaborator

@ECEEvanCampbell Okay this sounds good to me! Thanks for the in depth answer.

@eeddy

eeddy commented Nov 10, 2026

Copy link
Copy Markdown
Collaborator

@copilot I shouldn't have a setup.py anymore since I deleted it in favour of toml in latest. Also requirements.txt updates seem odd?

Co-authored-by: eeddy <31893850+eeddy@users.noreply.github.com>

Copilot AI commented Nov 10, 2026

Copy link
Copy Markdown

@copilot I shouldn't have a setup.py anymore since I deleted it in favour of toml in latest. Also requirements.txt updates seem odd?

Fixed in c42bc12: removed setup.py and restored requirements.txt to the latest version so this PR no longer carries those packaging drifts.

@eeddy
eeddy marked this pull request as ready for review November 10, 2026 15:45
@eeddy
eeddy merged commit 09b9321 into latest Nov 10, 2026
2 of 8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants