Skip to content

Phase 2: fix all three modes, Mute Mode v2 (33% -> 64.5%), live demo console - #1

Merged
Ryzuuuu merged 1 commit into
mainfrom
phase2-improvements
Nov 18, 2026
Merged

Ryzuuuu merged 1 commit into
mainfrom
phase2-improvements

Conversation

@Ryzuuuu

@Ryzuuuu Ryzuuuu commented Nov 18, 2026

Copy link
Copy Markdown
Owner

Summary

A bug-fix and improvement pass across the whole project before hardware bring-up. Full write-up with tables: docs/PHASE2_IMPROVEMENTS.md.

Mute Mode v2: 33.3% -> 64.5% top-1

Phase 1 Phase 2
Evaluation random split, same signers in train/val official WLASL test split, unseen signers
Clips 734 1,004
Top-1 / Top-5 33.3% / n.a. 64.5% / 81.9%

Root causes fixed: frames outside the WLASL sign window were extracted (wrong labels), the downloader stopped at 8 clips per word and YouTube/Signbank downloads failed silently, and features discarded hand location and movement. Live path (60 test videos streamed frame by frame): 61.7% top-1, and 90.6% correct when the confidence gate lets it speak. Ablation over 3 seeds is in the docs.

Blind Mode

  • Closing speed now uses measured time (was a hard coded 0.1 s), with median filtering and a jitter dead band
  • Proximity alerts: a static wall used to produce no warning at all
  • Non-blocking haptic driver: the ModeManager loop used to freeze for up to 1.5 s per alert

Deaf Mode

  • switch_mode("DEAF") no longer blocks forever with a live microphone
  • Vosk runs on a worker thread; the last sentence is flushed at stream end; real Vosk is used whenever the model is installed

Shared / demo

  • TTS cooldown per phrase at queue time, bounded queue flushed on mode switch, Windows COM fix
  • Rebuilt demo console: draggable obstacles and scenarios, browser microphone captions, webcam or test-clip sign recognition with landmark overlay

Testing

  • pytest: 59 tests pass (was 38); the 3 torch tests skip in the lightweight CI environment
  • ruff check . clean
  • blind_mode/synthetic_test.py, deaf_mode/test_main_loop.py (real Vosk), shared/test_mode_manager.py (v2 recogniser) pass
  • Demo console exercised in a browser: Blind scenarios, Deaf sample and streamed audio, Mute test clips
  • Not yet tested on a physical Raspberry Pi

…console

Blind Mode
- New decision engine (blind_mode/engine.py): closing speed from measured
  timestamps instead of a hard coded 0.1 s, median filter, range rejection,
  speed smoothing with a dead band, and proximity alerts for static obstacles
- Non blocking HapticDriver; the ModeManager adapter no longer freezes the
  loop while a motor buzzes
- Echo timing uses perf_counter; mock GPIO simulates each sensor separately
  and starts echoes on the trigger's falling edge

Deaf Mode
- ModeManager runs the app in a background thread, so switch_mode() returns
  with a live microphone and the mode can be left again
- Vosk runs on a worker thread fed by a queue; final utterance is flushed
  when the stream ends; real Vosk is used whenever the model is installed
- Fixed broken import in test_main_loop.py, offline sample WAV fallback

Mute Mode v2
- Hands + upper body landmarks (MediaPipe hand and pose, run in parallel)
- Body frame position + wrist relative handshape features, augmentation
- Activity based sign segmentation for live use
- SignClassifierV2 (182,245 params): 64.5% top-1 / 81.9% top-5 on the
  official WLASL test split (Phase 1: 33.3% on a random split)
- Parallel WLASL fetcher respecting frame windows and official splits,
  landmark builder, training with ablations, live-path evaluation
- v1 recogniser kept as a fallback

Shared
- TTS cooldown per phrase stamped at queue time, bounded queue flushed on
  mode switches, COM initialised for pyttsx3 on Windows, listener hook
- Mode switches serialised with a lock

Demo console
- Rebuilt emulator: draggable obstacles and Blind scenarios, browser
  microphone to Vosk, webcam or held out test clips to the sign recogniser,
  landmark overlay, spoken output in the browser, live resource ownership

Tests: 59 pytest tests (was 38), torch tests skip in the lightweight CI.
Details in docs/PHASE2_IMPROVEMENTS.md; README updated.
@Ryzuuuu
Ryzuuuu merged commit 011d1e0 into main Nov 18, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant