Local robot-character audio and an operator view for a simulated mobile robot.
Two experiments share one concern: keeping useful feedback flowing while inputs arrive in small blocks, at different rates, or late.
A full orbit through the original kitchen scan: all 216,953 Gaussians, with a 12-second recorded simulator joint excerpt looped twice. Blue outlines show the stereo head; orange and pink hemispheres show the wrist fisheyes. Offline joint replay and camera-layout illustration; no physical robot or new timing result.
voice_demo.mp4
33 seconds: p226 original -> robot, then p237 original -> robot. Both speakers read the same sentence from CSTR VCTK 0.92. Source recordings and DSP/video adaptations retain CC BY 4.0: Yamagishi, Veaux & MacDonald (2019), University of Edinburgh CSTR, dataset. Subtitles / source and changes. This illustrates the default CPU DSP; it is not a new listening or latency result.
Twelve seconds from an original SAPIEN recording, sampled at 6 fps per stream. Each panel holds its latest delivered frame and displays source exposure time. These cameras render the fitted scene primitives and scan-derived tabletop texture; they do not render the full Gaussian scan shown above.
The circular 180-degree views retain the recorded arm occlusions. This small public sample supplies RGB only; wrist depth in the full simulator is simulated.
Left: the approved phone recording, original seconds 18.4-26.4 (8 seconds). Right: the same kitchen scan with the phone's image-derived position, orientation and camera frustum. The checkerboard's measured placement anchors PnP estimates to the scene. Valid samples leave a cyan trail; failed detections hide the current frustum, with no interpolation. 48/48 sampled frames have a pose. The silent input also passed a frame-by-frame check at 15 fps (120/120 valid); this is a selected presentation segment, not a full-recording detection rate. Silent comparison MP4. This is an offline RGB pose diagnostic, with video-fitted intrinsics, not independent world accuracy or real depth fusion. No USB camera footage was saved in the inspected source material.
Kitchen, phone and derived scene/camera media are copyright retained, with public display permission only and no reuse licence. See the exact asset list.
Voice. A causal, CPU-only pipeline turns microphone audio into a fixed robot character. The default uses DSP and a noise gate, with 240-sample blocks at 48 kHz. There is no sentence boundary or cloud service in that path. An optional neural adapter is retained for research; its timing and quality misses are documented.
Scene and robot. A holonomic base, lift, two 7-DOF arms, stereo head and wrist fisheye cameras feed a local operator view. A fixed Gaussian-splat prior is checked against timestamped live observations. Old, live, stale and unknown geometry have separate states. The public demo includes the owner-approved kitchen scan, a sampled historical simulator RGB/state replay and a separate silent phone excerpt.
Windows PowerShell, Git and Python 3.11 x64 are the tested setup. No Git LFS, models or GPU are needed for the default audio path, tests and replay.
git clone https://github.com/EveningStudy/relay-workbench.git
cd relay-workbench
.\setup.ps1 -Python python
.\run.ps1 smoke
.\run.ps1 testIf python points to a different version, pass the full Python 3.11 executable
path to -Python. Setup creates a local .venv and installs the hashed lock file.
It does not download checkpoints. Installation needs Internet access; subsequent
tests and the smoke replay run offline.
The smoke replay generates a harmonic audio probe and a tiny camera/state episode,
runs the default voice pipeline and the original scene replay publisher, and writes
its outputs under outputs/. These are functional checks, not listening results.
Use wired headphones and microphone/output endpoints on the same physical codec. List devices, then use their IDs:
.\.venv\Scripts\python.exe -m sounddevice
.\run.ps1 voice --input 1 --output 3 --same-clock --seconds 60Replace 1 and 3 with your own device IDs. Keys 1/2/3 select bypass, baseline
and default character; Space/Q/Esc stops. The launcher selects the model-free path.
Live audio requires Windows and working devices and was not rerun for this copy.
.\run.ps1 scene
.\run.ps1 operator
# In a second shell at the repository root:
.\run.ps1 cameras --loops 3Open localhost:8090. The first command restores the packed kitchen prior and generates robot assets. The replay publishes four recorded RGB streams, joints and odometry. See the scene guide. For the full SAPIEN simulator, see the optional installation and GPU requirements there.
These results were measured on the original build; this repository is a renamed copy with new packaging and short presentation replays. Original freeze hashes and receipts do not certify this code. The source revision and retained numerical evidence are described in provenance.
| Experiment | Recorded result | Meaning |
|---|---|---|
| Default voice latency | 48.02 / 48.60 / 51.67 ms median/p95/max estimate | Driver timestamp delta plus a declared 42 ms allowance; not measured end to end. |
| Default heldout WER | 7/828 = 0.845% | 12 speakers, five clips each; mostly read English. |
| Neural consistency WER | 43/828 = 5.193% | Separate optional mode, not default DSP. |
| Unfamiliar speaker stress WER | Default 17.391%; neural 43.478% | One speaker, prompt referenced; distinct from heldout results. |
| Scene nominal capture-to-DOM p95 | 120.19 ms (n=775); 136.86 ms (n=743) | Two recorded runs; host/browser proxy, not physical display timing. |
| Scene change to LIVE | Mug 1.60 s; moved box 13.36 s; chair 15.87 s | All miss the 300 ms response aim. |
| Arm command to camera / DOM | No observation in 20/20 trials | Nulls retained; no passing timing inferred. |
Full results and definitions include all 13 later timing runs, the neural compute misses, soak scope and the failed listening criterion.
- The default has weak fixed-target similarity and limited pitch preservation. The noise gate can miss quiet speech. A character effect is not proof that every speaker sounds like the same recognizable identity.
- Neural conversion remains noisy on a live microphone and misses the latency aim. It is not part of the quick start and no weights or fixed reference are bundled.
- The one-hour voice soak was mostly quiet/gated. Physical microphone-to-speaker end-to-end latency has not been measured as a distribution.
- Scene-change response can take seconds even when capture-to-DOM is fast. Fusion does not always improve changed-region F1 over live observations alone.
- Wrist depth and noisy odometry are simulated. There is no physical robot, no onboard compute validation and no physical depth/display timing result.
- The public scan retains the original renderer inputs, but its short sampled RGB replay omits wrist depth and cannot reproduce the historical experiments. Full raw episodes and some historical tooling remain excluded or unavailable.
- Architecture: audio blocks, camera timestamps and fusion boundaries
- Voice guide and scene guide
- Results and verification
- Data and media policy: clone size, privacy and optional assets
- Third-party attribution and licence status
- Contributing
Original code is licensed under Apache-2.0. Third-party code and data retain their own terms; the notices include the GPL-3.0 grant for the excluded Seed-VC comparison.
Personal kitchen/phone media and their derivatives are excluded from Apache-2.0: public display is permitted, with copyright retained and no media reuse licence.