Repository navigation
[1/4] Streaming fixes, atomic snapshots, and compatibility updates - #145
Conversation
Replay only the feature delta 265d0ae..378d6c1. Preserve latest datasets, metrics formulas, discrete models, device additions, channel-last windows and spacebar collection. Keep NumPy compatibility fixes without the unrelated metric-fixture rename. Defer reactive and pipeline docs to their implementations; omit generated environment locks. Co-authored-by: Christian Morrell <cmorrell@unb.ca>
378d6c1 to
698637a
Compare
| @@ -0,0 +1,148 @@ | |||
| from abc import ABC, abstractmethod | |||
There was a problem hiding this comment.
@ECEEvanCampbell Could we add some comments to this file so that it is clear what its purpose is?
| with open(loc, 'wb') as f: | ||
| pickle.dump(self.model, f) | ||
|
|
||
| def ensure_directory(directory: str) -> None: |
There was a problem hiding this comment.
Probably not a huge deal but this should likely be in a genearl utils file. But probably fine
There was a problem hiding this comment.
Yeah we can refactor that down the road. it only is used by the adaptation pipeline currently though.
| @@ -1 +1 @@ | |||
| from libemg.environments import _base, controllers, emg_hero, fitts No newline at end of file | |||
| from libemg.environments import _base, controllers, emg_hero, fitts,curricular_fitts No newline at end of file | |||
| @@ -25,7 +25,7 @@ | |||
| long_description_content_type="text/markdown", | |||
There was a problem hiding this comment.
Can we just remove setup.py in favour of using toml (which is the new way of doing things I guess?)
There was a problem hiding this comment.
I'd be fine with that.
There was a problem hiding this comment.
Do you want to do this or me? I'm happy to but youll have to rebase again
There was a problem hiding this comment.
Let's leave that for an isolated PR later
|
@copilot resolve the merge conflicts in this pull request |
…fixes # Conflicts: # requirements.txt # setup.py Co-authored-by: eeddy <31893850+eeddy@users.noreply.github.com>
Resolved. I merged |
|
@ECEEvanCampbell all tests passed :) |
…ge config
CI reported "all tests failing". In fact no tests ran at all.
tests/test_figure_release.py called matplotlib.use("TkAgg", force=True) at
module scope. On a headless runner that raises, which is a *collection* error,
and a single collection error aborts the whole pytest session. 2077 collected
tests never executed, coverage measured only import-time lines at 17.92%, and
--cov-fail-under=44 then failed the job. The backend switch and the tkinter
import are both guarded now, so the module skips where it cannot run. It is
about Tk's finalisation behaviour specifically, so skipping headless is the
correct outcome rather than something to work around.
Restore [tool.pytest.ini_options] and [tool.coverage.run], both of which went
missing in the #145-#148 merge:
* The marker registration went with them, which is why `download` and
`hardware` reported as unknown marks. The flags and skip logic behind them
live in tests/conftest.py and were never lost, so the markers still
worked -- they were just undeclared, and --strict-markers was gone too.
* The coverage omit list went with them. Vendor streamers and GUI panels
need a device or a display and are covered only by the import smoke test;
counting them makes the headline figure a measure of how much GUI code
exists. With the list restored, coverage is 66% rather than 18%.
Raise the coverage floor to 55, measured at 66%.
Regenerate tests/data/golden/: #147 fixed getCOMPfeat, which omitted axis=2 on
the fourth moment, so np.sum collapsed the entire batch to one scalar and
every window and channel was divided by the same global numerator -- one
window's data changed another window's feature, and the value depended on
whatever else was in the batch. COMP has no MATLAB reference, so the snapshot
was the only thing watching it. The numbers moved for a correct reason, so the
fixture is regenerated rather than the test relaxed.
Note in the workflow that the 3.10-3.12 matrix contradicts
requires-python = ">=3.13". It passes only because requirements.txt installs a
flat dependency list and never installs libemg itself, so pip never checks
requires-python. Which of the two should move is a support policy call and is
left open.
* Revert #144 metrics in favour of #147; drop setup.py for pyproject.toml Revert "Added formulas and tests (#144)" #147 added libemg/_feature_selection_metrics.py, a registry covering a strict superset of the metrics #144 put on OfflineMetrics: 34 codes against 25, adding rPU, rNS, rCE, rC, UD, TT, MAV, CA and AE. Nothing in the library called #144's methods -- the only reference anywhere was the test module added alongside them. #147's implementations also fix three defects those tests documented: * _get_KL_divergence hardcoded `term3 = 24` where the feature dimensionality belongs, so KLD went negative on low-dimensional data. #147 uses x.shape[1]. * get_IIF ended in an unguarded `num_sum / dnum_sum` and raised ZeroDivisionError on perfectly separable data. #147 has an explicit ratio() guard. * get_BD looped over the repetition-segment count where every sibling metric loops over the class count. #147 loops class by class. #147's tests are stronger too: closed-form values (SI == sqrt(3), MSA == sqrt(4/3), C == 0.75, rC == 5/6, CDM on XOR == 0) rather than the ordering assertions these had. Its docs also carry fuller attribution -- Nawfel 2021, the Nawfel 2020 thesis, Singh 2003 for PRISM and Kohn 1996 for CDM, each with a note on what remains unverified. Test fallout: * test_offline_metrics_extended.py keeps the regression-metric and extract_* coverage and drops the seven classes that exercised #144. * test_extension_contracts.py lowers its get_available_metrics() floor, which returns 14 again rather than 37. * test_extension_contracts.py now imports CLASSIFIER_MODELS and REGRESSOR_MODELS instead of parsing them out of a dict literal inside each __init__. #145-#148 moved them to module level, which broke the parser and with it collection of two test modules. * test_data_handler.py retires the strict xfail for a file shorter than the window: the _get_mode_windows rewrite in #145-#148 fixed it, so the marker had started failing as an unexpected pass. Verified that a 10-sample file now yields no windows and leaves metadata aligned. Remove setup.py; pyproject.toml is now the only build configuration. It was missing [build-system], authors, keywords, classifiers, license and a packages.find section, so the two files together were already dropping metadata: Keywords and Classifier were absent from the built distribution. Those are ported here, along with project URLs and the tests/docs/examples exclusion that find_packages(exclude=["*tests*"]) used to provide. The Development Status classifier also loses its trailing whitespace, which made it an invalid trove classifier that PyPI rejects on upload. Verified with python -m build and twine check: both artifacts pass, the wheel carries all 13 packages and the 17 libemg.resources choreography files, and nothing from tests/, docs/, examples/ or demos/ leaks in. * Stop the event-log test writing unused.log into the repository root test_a_real_log_hands_a_child_what_it_needs_and_nothing_it_should_not_have constructed EventLog(path="unused.log"), and start() opens that path, so every run of the suite left an untracked unused.log in the working tree. The name suggests the path was expected never to be opened. Every other path-taking test in the module already builds its path from tmp_path; this one is brought in line. * Unblock the test session: guard the Tk backend, restore pytest/coverage config CI reported "all tests failing". In fact no tests ran at all. tests/test_figure_release.py called matplotlib.use("TkAgg", force=True) at module scope. On a headless runner that raises, which is a *collection* error, and a single collection error aborts the whole pytest session. 2077 collected tests never executed, coverage measured only import-time lines at 17.92%, and --cov-fail-under=44 then failed the job. The backend switch and the tkinter import are both guarded now, so the module skips where it cannot run. It is about Tk's finalisation behaviour specifically, so skipping headless is the correct outcome rather than something to work around. Restore [tool.pytest.ini_options] and [tool.coverage.run], both of which went missing in the #145-#148 merge: * The marker registration went with them, which is why `download` and `hardware` reported as unknown marks. The flags and skip logic behind them live in tests/conftest.py and were never lost, so the markers still worked -- they were just undeclared, and --strict-markers was gone too. * The coverage omit list went with them. Vendor streamers and GUI panels need a device or a display and are covered only by the import smoke test; counting them makes the headline figure a measure of how much GUI code exists. With the list restored, coverage is 66% rather than 18%. Raise the coverage floor to 55, measured at 66%. Regenerate tests/data/golden/: #147 fixed getCOMPfeat, which omitted axis=2 on the fourth moment, so np.sum collapsed the entire batch to one scalar and every window and channel was divided by the same global numerator -- one window's data changed another window's feature, and the value depended on whatever else was in the batch. COMP has no MATLAB reference, so the snapshot was the only thing watching it. The numbers moved for a correct reason, so the fixture is regenerated rather than the test relaxed. Note in the workflow that the 3.10-3.12 matrix contradicts requires-python = ">=3.13". It passes only because requirements.txt installs a flat dependency list and never installs libemg itself, so pip never checks requires-python. Which of the two should move is a support policy call and is left open. * Move the display-dependent GUI tests out of the per-PR run The run aborted with SIGABRT (exit 134) at 87%, in test_pipeline_training.py::test_the_editor_can_be_closed_and_reopened, inside dearpygui's show_viewport(). With no display DearPyGui calls abort() in native code, so the process dies outright and every remaining test goes with it. The three modules that do this already wrapped the call in `try/except Exception: pytest.skip(...)`, which cannot work -- there is no Python exception to catch. Add an opt-in `gui` marker alongside --slow / --download / --hardware, and apply it to the three modules that create a viewport: test_pipeline_training, test_streamer_panel and test_visualization_panel (49 tests). They are deselected per-PR and run in a new nightly job under xvfb-run, which gives them a real display so they genuinely execute rather than skip. Also add tests/conftest.py::requires_display() as a safety net for anyone running without a display and without xvfb. It has to check before the call, for the same reason the existing except clause did not help. The GUI tests that never create a viewport -- choreography, data collection, the pipeline editor's document and viewer panels -- are unaffected and still run per-PR. Coverage is 64% with the viewport modules deselected, against 66% with them, because libemg/_gui/* is omitted from the report anyway. The floor stays at 55. * Fixes for failing tests * Fixed the tests * Removed GUI tests
A reader could observe the sample counter ahead of the copied signal because data and count were locked separately. This part adds shared buffer/count locks and atomic snapshots, together with the predictor/output-writer and adaptation foundation they use. It also retains the SiFi sampling configuration and Bluetooth recovery work, NumPy 2 compatibility fixes, Python 3.13 docs tooling, and incremental animation writes.
Integration preserves latest's dataset catalog, discrete training/classifier support, Myo address and Emager version options, and channel-last windows. Spacebar collection uses the new logger while retaining its press/release behavior.
Validation: 28 passed against the exact first-part tree:
python -m pytest -q tests/test_feature_extractor.py tests/test_offline_metrics.py tests/feature_tests tests/test_upstream_integration.py. The metric fixtures are retained; the former missing-fixture errors are gone.Part 1/4, split from #142 and rebuilt on current
latest(06ba120). Review this part alone: single feature commit · incremental comparison.Upstream overlap removed:
libemg/datasets.py,libemg/_datasets/,libemg/offline_metrics.py, andlibemg/_discrete_models/are unchanged fromlatestin every part. Existing formula tests, CI, README, and upstream-only assets are also preserved. The rewritten history contains one feature commit per part; it does not replay the older copied/squashed upstream history.Implementation provenance: replayed feature delta
265d0ae..378d6c1from the original branch, with conflict resolutions for newer upstream behavior. The originalcompose_emgpredictorbranch remains available; local pre-rebase backup refs are retained.All four PRs still target
latest. Review and merge in order: #145 → #146 → #147 → #148. Until preceding parts merge, GitHub's PR-wide diff is cumulative; the commit/comparison links above isolate this part. Use merge commits to preserve the chain, or rebase the remaining parts after a squash merge.Test environment: Python 3.13.17 / NumPy 2.4.6, with latest's device/discrete dependencies installed in an isolated test directory. Draft status is retained for review and hardware validation.
The combined Sphinx HTML documentation build succeeds; documentation warnings remain.