Speech Runtime

Speech Runtime Lab v0 is a multilingual listening-quality benchmark lab for testing VAD and ASR/STT behavior across English, Hindi, and Hinglish.

Current status

Implemented so far:

• project definition
• multilingual ground-truth JSON
• metrics spec
• pure transcript metrics utilities
• VAD summary utility
• TypeScript validation harness
• browser recording UI skeleton
• local Silero browser VAD summary
• local OpenAI ASR transcription path
• server-side WER/CER metrics against selected ground truth

Not implemented yet:

• telephony

Setup

npm install

Environment

Copy the example environment file:

cp .env.example .env

Set:

OPENAI_API_KEY=your_api_key_here
OPENAI_TRANSCRIBE_MODEL=gpt-4o-transcribe
API_PORT=8787

Never commit .env.

Validation

npm run typecheck
npm run example:metrics
npm run test:metrics
npm test
npm run build

Browser UI

Run both the local API server and browser UI:

npm run dev

The UI runs on http://127.0.0.1:5173/.

The local API server runs on http://127.0.0.1:8787/.

The browser sends recorded audio to the local API server. The OpenAI API key stays server-side.

OpenAI ASR model selection

The dashboard supports selecting one of three OpenAI transcription models:

• gpt-4o-transcribe
• gpt-4o-mini-transcribe
• whisper-1

The selected model is sent to the local API server and validated against an allowlist before calling OpenAI.

Build the UI:

npm run build

Preview the production build:

npm run preview

The current UI supports ground-truth selection, local browser recording, local Silero browser VAD summary, and server-side OpenAI transcription with WER/CER metrics against the selected ground truth. It does not run telephony or streaming STT yet.

Final v0 demo flow

1. Install dependencies:

npm install

2. Configure environment:

cp .env.example .env

Set:

OPENAI_API_KEY=your_api_key_here
OPENAI_TRANSCRIBE_MODEL=gpt-4o-transcribe
API_PORT=8787

3. Run the lab:

npm run dev

4. Open:

http://127.0.0.1:5173/

5. In the UI:

• select a ground truth
• choose VAD mode: none or silero
• record audio
• stop recording
• review VAD summary if using silero
• click Transcribe recording
• review ASR transcript
• review WER, CER, word accuracy, and token diff

Current v0 capabilities

• browser recording
• ground-truth selection
• local Silero browser VAD summary
• server-side OpenAI ASR
• WER/CER/word accuracy metrics
• English, Hindi, and Hinglish ground truths

Still not included

• telephony
• LiveKit
• Twilio
• streaming STT
• database
• raw audio storage
• production deployment

Source of truth

• docs/speech_runtime_lab_v0.md
• docs/speech_runtime_lab_metrics.md
• research/speech-runtime-lab/ground_truths.v0.json