Repository navigation
Phase 2: fix all three modes, Mute Mode v2 (33% -> 64.5%), live demo console - #1
Merged
Merged
Conversation
…console Blind Mode - New decision engine (blind_mode/engine.py): closing speed from measured timestamps instead of a hard coded 0.1 s, median filter, range rejection, speed smoothing with a dead band, and proximity alerts for static obstacles - Non blocking HapticDriver; the ModeManager adapter no longer freezes the loop while a motor buzzes - Echo timing uses perf_counter; mock GPIO simulates each sensor separately and starts echoes on the trigger's falling edge Deaf Mode - ModeManager runs the app in a background thread, so switch_mode() returns with a live microphone and the mode can be left again - Vosk runs on a worker thread fed by a queue; final utterance is flushed when the stream ends; real Vosk is used whenever the model is installed - Fixed broken import in test_main_loop.py, offline sample WAV fallback Mute Mode v2 - Hands + upper body landmarks (MediaPipe hand and pose, run in parallel) - Body frame position + wrist relative handshape features, augmentation - Activity based sign segmentation for live use - SignClassifierV2 (182,245 params): 64.5% top-1 / 81.9% top-5 on the official WLASL test split (Phase 1: 33.3% on a random split) - Parallel WLASL fetcher respecting frame windows and official splits, landmark builder, training with ablations, live-path evaluation - v1 recogniser kept as a fallback Shared - TTS cooldown per phrase stamped at queue time, bounded queue flushed on mode switches, COM initialised for pyttsx3 on Windows, listener hook - Mode switches serialised with a lock Demo console - Rebuilt emulator: draggable obstacles and Blind scenarios, browser microphone to Vosk, webcam or held out test clips to the sign recogniser, landmark overlay, spoken output in the browser, live resource ownership Tests: 59 pytest tests (was 38), torch tests skip in the lightweight CI. Details in docs/PHASE2_IMPROVEMENTS.md; README updated.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A bug-fix and improvement pass across the whole project before hardware bring-up. Full write-up with tables:
docs/PHASE2_IMPROVEMENTS.md.Mute Mode v2: 33.3% -> 64.5% top-1
Root causes fixed: frames outside the WLASL sign window were extracted (wrong labels), the downloader stopped at 8 clips per word and YouTube/Signbank downloads failed silently, and features discarded hand location and movement. Live path (60 test videos streamed frame by frame): 61.7% top-1, and 90.6% correct when the confidence gate lets it speak. Ablation over 3 seeds is in the docs.
Blind Mode
Deaf Mode
switch_mode("DEAF")no longer blocks forever with a live microphoneShared / demo
Testing
pytest: 59 tests pass (was 38); the 3 torch tests skip in the lightweight CI environmentruff check .cleanblind_mode/synthetic_test.py,deaf_mode/test_main_loop.py(real Vosk),shared/test_mode_manager.py(v2 recogniser) pass