Every panel drives the real TriSense code: the ModeManager, the Blind Mode engine, the Vosk recogniser and the sign recogniser. Only the hardware is simulated.
+
+
Blind ModeDrag obstacles in front of three ultrasonic sensors and feel the haptic and spoken alerts.
+
Deaf ModeSpeak into your microphone and watch offline captions appear on the OLED.
+
Mute ModeSign to your webcam or play an unseen test clip and hear the word spoken.
+
+
+
+
+
+
+
Blind Mode · obstacle detection
+
Drag the obstacles (or use the sliders) to change what each HC-SR04 sensor hears. The real decision engine computes speed, time to collision and proximity, then pulses the motors and speaks.
+
+
+
+ Scenarios:
+
+
+
+
+
+
+
+
+
Pulse length encodes urgency: 150 ms warning · 300 ms urgent · 500 ms imminent. TTC = approaching fast, proximity = close even if nothing moves.
-
-
+
+
+
Deaf Mode · offline captions
+
Audio goes to the Vosk recogniser running locally, with no Internet. The OLED shows exactly what the 128×64 display on the device would show.
+
+
+
+
+
+
+
+
+
Device OLED
+
TriSense ASR [REC]
Waiting for speech...
+
-
-
Mode Switch
-
-
-
-
-
-
-
-
-
-
ResourceGuard Owners
-
-
-
Resource
Current Owner
-
-
-
SPEAKER
IDLE
-
CAMERA
IDLE
-
MICROPHONE
IDLE
-
DISPLAY
IDLE
-
GPIO_SENSORS
IDLE
-
GPIO_MOTORS
IDLE
-
-
-
+
Live transcript
+
+
+
+
-
-
-
Live Interface View
-
-
-
-
System Idle. Select a mode to begin.
-
-
-
-
-
Spatial Haptic Pulse Visualizer
-
-
-
-
LEFT
-
-- s
-
-
-
-
CENTER
-
-- s
-
-
-
-
RIGHT
-
-- s
-
-
-
-
-
-
-
OLED Caption Viewport (128x64)
-
TriSense ASR [REC] Waiting for speech...
-
-
-
-
-
-
⚠️ LOW-ACCURACY / EXPERIMENTAL
-
Predicted Sign:
-
...
-
Confidence: --%
-
-
+
+
+
Mute Mode · sign to speech
+
Frames go to the real recogniser: MediaPipe hand and pose landmarks, automatic sign segmentation, then the 100 word classifier. A word is spoken only when the model is confident.
+
+
+
+
+
+
+
+
+
+
+
+
waiting for hands
+
Choose the webcam or a test clip to start.
+
+
Tip: start with your hands down, sign one word, then lower your hands. Lowering the hands ends the sign.