Skip to content

Compose emgpredictor - #142

Closed
ECEEvanCampbell wants to merge 313 commits into
latestfrom
compose_emgpredictor
Closed

ECEEvanCampbell wants to merge 313 commits into
latestfrom
compose_emgpredictor

Conversation

@ECEEvanCampbell

@ECEEvanCampbell ECEEvanCampbell commented Jul 5, 2026 •

Copy link
Copy Markdown
Collaborator

Superseded by four draft PRs, all targeting latest, rebuilt on 06ba120 to exclude upstream overlap. Review and merge in this order:

Each part is now one feature commit above the preceding part. Datasets, offline-metric formulas, discrete-model modules, existing formula tests, CI, and README remain unchanged from latest. Individual commit links and validation results are in each replacement PR. The original compose_emgpredictor branch remains unchanged; the replacements retain its unique feature work alongside newer upstream implementations.


Original description:

Superseded by four draft PRs, all targeting latest. Review and merge in this order:

  1. [1/4] Streaming fixes, atomic snapshots, and compatibility updates

  2. [2/4] Stateful shared memory and reactive online processing

  3. [3/4] Pipeline editor, GUI workflows, and sessions

  4. [4/4] Hand-animation screen-guided training and choreography

The replacements preserve this PR's final source tree and all original commits. Core regression tests were moved into part 2 so they can be validated before the GUI changes. Each PR documents its incremental comparison, validation results, and existing integration conflicts with latest.


Original description:

This branch composes the online EMGPredictor/adaptation workflow. The most recent work on it (this session) adds three things:

  1. Atomic shared-memory reads ( + _count share a lock)

The streamer writes each modality buffer and its sample counter under separate locks, and readers copied them in two separate critical sections. A batch landing between the two reads made the counter run ahead of the copied buffer, so log_to_file spliced dropped/duplicated samples into the saved signal (visible as "noise" that visualize never showed, because it is stateless per frame).

SharedMemoryManager.get_variables([...]) — copies several variables under a single held lock (one acquire when they share a lock; deadlock-safe id-ordered acquisition otherwise).

assign_shared_memory_locks(...) — gives and _count the same lock; used by all streamer factories in streamers.py.

OnlineDataHandler.get_data and MemoryManager.process_data now take an atomic (data, count) snapshot. SharedMemoryOutputWriter shares the buffer's lock with its _count.

Because every writer updates data before count, an atomic read can only ever see the count lagging the data (self-correcting) — never leading it, which is the failure mode.

  1. Online Adaptation tutorial

New Sphinx page (docs/source/documentation/adaptation/) documenting the adaptation suite: a within-subject SGT initialization followed by an online environment that adapts the model in the loop. Grounded in the CoAdaptUnderCurriculum reference workflow and trimmed to the minimal wiring (the four cooperating processes, get_edil_adaptation_objects, the model/memory/feedback interfaces the user supplies, and the save_dir/file_path/load_dir linkages). Wired into the Modules toctree and the API reference (emg_toolbox.rst).

  1. Doc toolchain bump

requirements.txt # For Docs pins bumped to Sphinx>=8.1.3, myst-parser>=4.0.0, sphinx-rtd-theme>=3.0.2. Sphinx 5.0.0 fails to import on Python 3.13 (removed stdlib imghdr) and the old myst/rtd-theme pins are incompatible with modern Sphinx. The full docs (including the new tutorial) now build cleanly.

Reviewer notes

The docs build was verified locally end-to-end (sphinx-build docs/source) — build succeeds; remaining warnings are all pre-existing in other pages.

The lock/snapshot change touches the shared-memory contract used by every streamer; the shared-lock convention is applied uniformly so existing streamers keep working.

This branch also carries the earlier compose_emgpredictor commits (incremental-learning updates, SiFi streamer, numpy>2.0).

cbmorrell and others added 30 commits September 13, 2024 16:19
Removed a duplicate method that parent class already had.
.gitignore didn't ignore all .pyc files.
Controller had many methods that the user shouldn't call.

Converted these methods to private.
Added class for Fitts task using regression.

Needs to be modified for classification.
SocketController used a deque, which isn't thread-safe and meant that the SocketController's data parameter didn't update in all threads when it was updated in another.

Replaced deque with multiprocessing queue, which is thread safe.
ClassifierController was not parsing all messages properly.

Added parameters to allow for definite parsing and some checks to tell the user if they set something up incorrectly.
OnlineEMGClassifier.visualize hadn't been updated since shared memory changes.

Modified to use new ClassifierController.
Regression worked in Fitts environment, but not classification.

Modified data parsing to handle classification properly. Also added a parameter that the user can pass in to customize which commands cause which directions.
Replaced abstract run method with abstract _run_helper method.
Description for the legend parameterr didn't fully explain what user should pass in.
Moved shared functionality into common parent class and added some helpful parameters to IsoFitts.
Having commands be the keys made it a bit tedious to grab the commands.

Reverted so the controller outputs are the keys and we grab the commands from that.
memory items now carry a state block (generation, total samples, closed)
and a notifier pool, so a write announces itself and each observer pairs
those published facts with its own criterion, instead of copying whole
buffers just to learn whether a counter moved:

  - shared_memory_manager: Snapshot, commit/apply, subscribe/unsubscribe,
    read_window/read_since
  - reactive: hooks, criteria and executors over stateful shared memory
  - event_log: cross-process record of every propagation decision, cheap
    enough to leave switched on
  - adaptation/hooks: environment_feedback -> memory_flag -> adapt_flag
    expressed as reactions; Memory/AdaptationManager wait rather than spin
  - EMGClassifier/EMGRegressor gain a reactive run path and fall back to
    the polled loop when no notifier pool is available
  - streamers hand the default notifier pool to the child process
GUI. pipeline.py exposes the same document, registry and compiler the
editor drives, and runs a saved pipeline headless from the command line.
each generated from streamer/registry signatures so a new device or node
appears without edits here. gui.py runs a manual render loop on every
path so panels get a per-frame slice on the render thread, reuses an
already-open panel instead of building a second one over the same tags,
and no longer requires an online data handler up front.
spectrum, wavelet decomposition, TDPSD and Hjorth moments) so each is
computed once per window batch. An unrecognised feature name now raises
ValueError instead of being silently dropped, which previously surfaced
much later as a dimension mismatch.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants