Skip to content

[1/4] Streaming fixes, atomic snapshots, and compatibility updates - #145

Merged
eeddy merged 3 commits into
latestfrom
pr142/01-streaming-fixes
Nov 10, 2026
Merged

eeddy merged 3 commits into
latestfrom
pr142/01-streaming-fixes

Conversation

@ECEEvanCampbell

@ECEEvanCampbell ECEEvanCampbell commented Nov 9, 2026 •

Copy link
Copy Markdown
Collaborator

A reader could observe the sample counter ahead of the copied signal because data and count were locked separately. This part adds shared buffer/count locks and atomic snapshots, together with the predictor/output-writer and adaptation foundation they use. It also retains the SiFi sampling configuration and Bluetooth recovery work, NumPy 2 compatibility fixes, Python 3.13 docs tooling, and incremental animation writes.

Integration preserves latest's dataset catalog, discrete training/classifier support, Myo address and Emager version options, and channel-last windows. Spacebar collection uses the new logger while retaining its press/release behavior.

Validation: 28 passed against the exact first-part tree: python -m pytest -q tests/test_feature_extractor.py tests/test_offline_metrics.py tests/feature_tests tests/test_upstream_integration.py. The metric fixtures are retained; the former missing-fixture errors are gone.

Part 1/4, split from #142 and rebuilt on current latest (06ba120). Review this part alone: single feature commit · incremental comparison.

Upstream overlap removed: libemg/datasets.py, libemg/_datasets/, libemg/offline_metrics.py, and libemg/_discrete_models/ are unchanged from latest in every part. Existing formula tests, CI, README, and upstream-only assets are also preserved. The rewritten history contains one feature commit per part; it does not replay the older copied/squashed upstream history.

Implementation provenance: replayed feature delta 265d0ae..378d6c1 from the original branch, with conflict resolutions for newer upstream behavior. The original compose_emgpredictor branch remains available; local pre-rebase backup refs are retained.

All four PRs still target latest. Review and merge in order: #145 → #146 → #147 → #148. Until preceding parts merge, GitHub's PR-wide diff is cumulative; the commit/comparison links above isolate this part. Use merge commits to preserve the chain, or rebase the remaining parts after a squash merge.

Test environment: Python 3.13.17 / NumPy 2.4.6, with latest's device/discrete dependencies installed in an isolated test directory. Draft status is retained for review and hardware validation.

The combined Sphinx HTML documentation build succeeds; documentation warnings remain.

Replay only the feature delta 265d0ae..378d6c1. Preserve latest datasets, metrics formulas, discrete models, device additions, channel-last windows and spacebar collection. Keep NumPy compatibility fixes without the unrelated metric-fixture rename. Defer reactive and pipeline docs to their implementations; omit generated environment locks.

Co-authored-by: Christian Morrell <cmorrell@unb.ca>
@ECEEvanCampbell
ECEEvanCampbell force-pushed the pr142/01-streaming-fixes branch from 378d6c1 to 698637a Compare November 9, 2026 18:53
Comment thread pyproject.toml Resolved
Comment thread .python-version Resolved
Comment thread uv.lock Resolved
Comment thread libemg/output_writer.py
@@ -0,0 +1,148 @@
from abc import ABC, abstractmethod

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ECEEvanCampbell Could we add some comments to this file so that it is clear what its purpose is?

with open(loc, 'wb') as f:
pickle.dump(self.model, f)

def ensure_directory(directory: str) -> None:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably not a huge deal but this should likely be in a genearl utils file. But probably fine

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah we can refactor that down the road. it only is used by the adaptation pipeline currently though.

Comment thread libemg/_streamers/_oymotion_streamer.py Resolved
Comment thread libemg/environments/__init__.py Outdated
@@ -1 +1 @@
from libemg.environments import _base, controllers, emg_hero, fitts No newline at end of file
from libemg.environments import _base, controllers, emg_hero, fitts,curricular_fitts No newline at end of file

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NIT: Add space

Comment thread libemg/utils.py Resolved
Comment thread setup.py Outdated
@@ -25,7 +25,7 @@
long_description_content_type="text/markdown",

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we just remove setup.py in favour of using toml (which is the new way of doing things I guess?)

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd be fine with that.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you want to do this or me? I'm happy to but youll have to rebase again

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's leave that for an isolated PR later

@eeddy

eeddy commented Nov 10, 2026

Copy link
Copy Markdown
Collaborator

@copilot resolve the merge conflicts in this pull request

…fixes

# Conflicts:
#	requirements.txt
#	setup.py

Co-authored-by: eeddy <31893850+eeddy@users.noreply.github.com>

Copilot AI commented Nov 10, 2026

Copy link
Copy Markdown

@copilot resolve the merge conflicts in this pull request

Resolved. I merged latest into this branch and fixed the merge conflicts in commit 03fd1c4.

Copilot AI requested a review from eeddy November 10, 2026 03:24
@eeddy
eeddy marked this pull request as ready for review November 10, 2026 03:26
@eeddy

eeddy commented Nov 10, 2026

Copy link
Copy Markdown
Collaborator

@ECEEvanCampbell all tests passed :)

@eeddy
eeddy merged commit 30ddefd into latest Nov 10, 2026
8 checks passed
@eeddy
eeddy deleted the pr142/01-streaming-fixes branch November 10, 2026 03:43
eeddy added a commit that referenced this pull request Nov 10, 2026
…ge config

CI reported "all tests failing". In fact no tests ran at all.

tests/test_figure_release.py called matplotlib.use("TkAgg", force=True) at
module scope. On a headless runner that raises, which is a *collection* error,
and a single collection error aborts the whole pytest session. 2077 collected
tests never executed, coverage measured only import-time lines at 17.92%, and
--cov-fail-under=44 then failed the job. The backend switch and the tkinter
import are both guarded now, so the module skips where it cannot run. It is
about Tk's finalisation behaviour specifically, so skipping headless is the
correct outcome rather than something to work around.

Restore [tool.pytest.ini_options] and [tool.coverage.run], both of which went
missing in the #145-#148 merge:

  * The marker registration went with them, which is why `download` and
    `hardware` reported as unknown marks. The flags and skip logic behind them
    live in tests/conftest.py and were never lost, so the markers still
    worked -- they were just undeclared, and --strict-markers was gone too.
  * The coverage omit list went with them. Vendor streamers and GUI panels
    need a device or a display and are covered only by the import smoke test;
    counting them makes the headline figure a measure of how much GUI code
    exists. With the list restored, coverage is 66% rather than 18%.

Raise the coverage floor to 55, measured at 66%.

Regenerate tests/data/golden/: #147 fixed getCOMPfeat, which omitted axis=2 on
the fourth moment, so np.sum collapsed the entire batch to one scalar and
every window and channel was divided by the same global numerator -- one
window's data changed another window's feature, and the value depended on
whatever else was in the batch. COMP has no MATLAB reference, so the snapshot
was the only thing watching it. The numbers moved for a correct reason, so the
fixture is regenerated rather than the test relaxed.

Note in the workflow that the 3.10-3.12 matrix contradicts
requires-python = ">=3.13". It passes only because requirements.txt installs a
flat dependency list and never installs libemg itself, so pip never checks
requires-python. Which of the two should move is a support policy call and is
left open.
eeddy added a commit that referenced this pull request Nov 11, 2026
* Revert #144 metrics in favour of #147; drop setup.py for pyproject.toml

Revert "Added formulas and tests (#144)"

#147 added libemg/_feature_selection_metrics.py, a registry covering a strict
superset of the metrics #144 put on OfflineMetrics: 34 codes against 25,
adding rPU, rNS, rCE, rC, UD, TT, MAV, CA and AE. Nothing in the library
called #144's methods -- the only reference anywhere was the test module
added alongside them.

#147's implementations also fix three defects those tests documented:

  * _get_KL_divergence hardcoded `term3 = 24` where the feature
    dimensionality belongs, so KLD went negative on low-dimensional data.
    #147 uses x.shape[1].
  * get_IIF ended in an unguarded `num_sum / dnum_sum` and raised
    ZeroDivisionError on perfectly separable data. #147 has an explicit
    ratio() guard.
  * get_BD looped over the repetition-segment count where every sibling
    metric loops over the class count. #147 loops class by class.

#147's tests are stronger too: closed-form values (SI == sqrt(3),
MSA == sqrt(4/3), C == 0.75, rC == 5/6, CDM on XOR == 0) rather than the
ordering assertions these had. Its docs also carry fuller attribution --
Nawfel 2021, the Nawfel 2020 thesis, Singh 2003 for PRISM and Kohn 1996 for
CDM, each with a note on what remains unverified.

Test fallout:

  * test_offline_metrics_extended.py keeps the regression-metric and
    extract_* coverage and drops the seven classes that exercised #144.
  * test_extension_contracts.py lowers its get_available_metrics() floor,
    which returns 14 again rather than 37.
  * test_extension_contracts.py now imports CLASSIFIER_MODELS and
    REGRESSOR_MODELS instead of parsing them out of a dict literal inside
    each __init__. #145-#148 moved them to module level, which broke the
    parser and with it collection of two test modules.
  * test_data_handler.py retires the strict xfail for a file shorter than the
    window: the _get_mode_windows rewrite in #145-#148 fixed it, so the
    marker had started failing as an unexpected pass. Verified that a
    10-sample file now yields no windows and leaves metadata aligned.

Remove setup.py; pyproject.toml is now the only build configuration. It was
missing [build-system], authors, keywords, classifiers, license and a
packages.find section, so the two files together were already dropping
metadata: Keywords and Classifier were absent from the built distribution.
Those are ported here, along with project URLs and the tests/docs/examples
exclusion that find_packages(exclude=["*tests*"]) used to provide. The
Development Status classifier also loses its trailing whitespace, which made
it an invalid trove classifier that PyPI rejects on upload.

Verified with python -m build and twine check: both artifacts pass, the wheel
carries all 13 packages and the 17 libemg.resources choreography files, and
nothing from tests/, docs/, examples/ or demos/ leaks in.

* Stop the event-log test writing unused.log into the repository root

test_a_real_log_hands_a_child_what_it_needs_and_nothing_it_should_not_have
constructed EventLog(path="unused.log"), and start() opens that path, so
every run of the suite left an untracked unused.log in the working tree. The
name suggests the path was expected never to be opened.

Every other path-taking test in the module already builds its path from
tmp_path; this one is brought in line.

* Unblock the test session: guard the Tk backend, restore pytest/coverage config

CI reported "all tests failing". In fact no tests ran at all.

tests/test_figure_release.py called matplotlib.use("TkAgg", force=True) at
module scope. On a headless runner that raises, which is a *collection* error,
and a single collection error aborts the whole pytest session. 2077 collected
tests never executed, coverage measured only import-time lines at 17.92%, and
--cov-fail-under=44 then failed the job. The backend switch and the tkinter
import are both guarded now, so the module skips where it cannot run. It is
about Tk's finalisation behaviour specifically, so skipping headless is the
correct outcome rather than something to work around.

Restore [tool.pytest.ini_options] and [tool.coverage.run], both of which went
missing in the #145-#148 merge:

  * The marker registration went with them, which is why `download` and
    `hardware` reported as unknown marks. The flags and skip logic behind them
    live in tests/conftest.py and were never lost, so the markers still
    worked -- they were just undeclared, and --strict-markers was gone too.
  * The coverage omit list went with them. Vendor streamers and GUI panels
    need a device or a display and are covered only by the import smoke test;
    counting them makes the headline figure a measure of how much GUI code
    exists. With the list restored, coverage is 66% rather than 18%.

Raise the coverage floor to 55, measured at 66%.

Regenerate tests/data/golden/: #147 fixed getCOMPfeat, which omitted axis=2 on
the fourth moment, so np.sum collapsed the entire batch to one scalar and
every window and channel was divided by the same global numerator -- one
window's data changed another window's feature, and the value depended on
whatever else was in the batch. COMP has no MATLAB reference, so the snapshot
was the only thing watching it. The numbers moved for a correct reason, so the
fixture is regenerated rather than the test relaxed.

Note in the workflow that the 3.10-3.12 matrix contradicts
requires-python = ">=3.13". It passes only because requirements.txt installs a
flat dependency list and never installs libemg itself, so pip never checks
requires-python. Which of the two should move is a support policy call and is
left open.

* Move the display-dependent GUI tests out of the per-PR run

The run aborted with SIGABRT (exit 134) at 87%, in
test_pipeline_training.py::test_the_editor_can_be_closed_and_reopened, inside
dearpygui's show_viewport(). With no display DearPyGui calls abort() in native
code, so the process dies outright and every remaining test goes with it. The
three modules that do this already wrapped the call in
`try/except Exception: pytest.skip(...)`, which cannot work -- there is no
Python exception to catch.

Add an opt-in `gui` marker alongside --slow / --download / --hardware, and apply
it to the three modules that create a viewport: test_pipeline_training,
test_streamer_panel and test_visualization_panel (49 tests). They are
deselected per-PR and run in a new nightly job under xvfb-run, which gives
them a real display so they genuinely execute rather than skip.

Also add tests/conftest.py::requires_display() as a safety net for anyone
running without a display and without xvfb. It has to check before the call,
for the same reason the existing except clause did not help.

The GUI tests that never create a viewport -- choreography, data collection,
the pipeline editor's document and viewer panels -- are unaffected and still
run per-PR. Coverage is 64% with the viewport modules deselected, against 66%
with them, because libemg/_gui/* is omitted from the report anyway. The floor
stays at 55.

* Fixes for failing tests

* Fixed the tests

* Removed GUI tests
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants