Whispers

Whispers routes native gRPC speech calls to separate speech-to-text (STT) and text-to-speech (TTS) model services. This repository includes the router and both Kyutai model services; no model weights belong in the repository. The router binary contains no inference engine, GPU runtime, or model weights.

The Rust router exposes four Google RPCs:

Service	RPC	Model selector
google.cloud.speech.v2.Speech	Recognize	config.model
google.cloud.speech.v2.Speech	StreamingRecognize	First request's streaming_config.config.model
google.cloud.texttospeech.v1.TextToSpeech	SynthesizeSpeech	voice.model_name
google.cloud.texttospeech.v1.TextToSpeech	StreamingSynthesize	First request's streaming_config.voice.model_name

The protobuf contracts preserve upstream fields; each backend validates its supported audio formats, voices, and options. See protocol provenance for upstream revisions, licenses, and generated API details. These four RPCs do not constitute full Google Cloud API compatibility. Whispers does not implement Google authentication, long-running recognition, model discovery, or voice discovery.

License

Original Whispers code uses Apache-2.0. Third-party sources, dependencies, model weights, and voice assets retain their own terms. See third-party notices for attribution, license texts, and redistribution requirements.

Quick start

Docker Engine and Docker Compose 2.24 or later provide the local container environment.

Start only the router:

docker compose up --build --wait
docker compose exec whispers whispers-server health --endpoint http://127.0.0.1:50051

The host endpoint is 127.0.0.1:50051. Router readiness does not depend on backend availability. The default routes reference fixtures, which require the test profile:

docker compose --profile test up --build --wait
docker compose --profile test run --build --rm smoke
docker compose --profile test down

The smoke command checks all four RPCs with test-stt and test-tts. Fixtures return deterministic test data, not transcripts or speech from real inference. The separate smoke profile keeps the one-shot service out of up --wait.

For an isolated test project with automatic cleanup, run bash scripts/test-compose.sh. The script uses a random host port and removes only its own containers and network.

Optional real CPU backends

License restriction: the selected Expresso Happy voice uses CC BY-NC 4.0, for non-commercial use only. The Kyutai STT and TTS models use CC BY 4.0. Review the STT provenance and TTS artifact licenses before use or redistribution.

compose.cpu.yaml defines a separate three-service stack with actual speech inference in private backend processes. All build contexts use this repository root; no sibling checkout supplies a build or runtime dependency. The default Compose file and fixture tests remain unchanged.

Do not start this stack on an 8 GB Docker VM. STT float32 weights alone need approximately 10.5 GB, before decoder, caches, conversion buffers, and runtime overhead. The stack permits 24 GiB for STT, 16 GiB for TTS, and 512 MiB for the router. These are configured caps, not measured requirements; size host memory from workload measurements with headroom. Model load and warmup can take several minutes; the backend health grace periods are 300 and 600 seconds.

First, explicitly prepare artifacts through the STT preparation instructions and TTS preparation instructions. The ignored directories models/kyutai-stt and models/kyutai-tts must exist and permit UID 65532 to read their contents. The stack mounts both directories read-only and refuses to create absent host paths. Images contain no model or voice assets; builds and normal startup do not download them. Preparation requires outbound network access for pinned downloads; the private Compose network serves only offline inference. For container-based TTS preparation, override both HF_HUB_OFFLINE=0 and TRANSFORMERS_OFFLINE=0. Use writable artifact and cache mounts for preparation, not the read-only Compose mounts.

After artifact preparation and resource checks, run from this repository:

docker compose -f compose.cpu.yaml config --quiet
docker compose -f compose.cpu.yaml build
docker compose -f compose.cpu.yaml up --wait --wait-timeout 900
docker compose -f compose.cpu.yaml down

The stack publishes only 127.0.0.1:50051; use WHISPERS_HOST_PORT if the default router already occupies that port. The internal backend network permits no external egress, and neither backend publishes a host port. The router owns no model mount or inference process. Router readiness remains independent; up --wait also checks each backend's actual model readiness.

config/cpu.yaml selects these explicit routes:

Backend	Selector	Supported audio
STT	config.model=kyutai-stt-2.6b-en	Explicit LINEAR16, 24 kHz mono PCM16 little-endian, no WAV header; English only
TTS	voice.model_name=kyutai-tts-1.6b-en-fr	Unary LINEAR16 (1) includes a WAV header; stream PCM (7) has no header

TTS uses voice.name=expresso-happy, voice.language_code=en-US or fr-FR, and native 24 kHz mono PCM16. Stream selectors belong in the first configuration request, as above. Both backends admit one session; the router's higher concurrency limit does not add backend capacity. See each backend's API contract for additional validation and lifecycle restrictions. Ordinary tests use fixtures, not this stack; the CPU stack provides no latency or real-time guarantee.

Optional CPU speaker diarization

compose.diarization-cpu.yaml extends only STT in the real CPU stack. It uses a separate kyutai-stt:diarization-cpu image and an explicit opt-in configuration. Plain requests retain the 60-second audio limit; default images and stacks retain their current behavior. The child runs Community-1 locally after audio input ends. Final word labels identify speakers within one recording, not people across recordings. Labels use approximate ASR emission alignment, not acoustic word alignment. The public response includes no word times, confidence scores, or multi-voice synthesis.

Read the STT integration contract before use. The worker guide owns dependency details, gated model preparation, artifact validation, and model attribution. Community-1 uses CC-BY-4.0; the CPU stack's separate voice restriction still applies. Prepare its pinned artifacts explicitly outside the repository before startup. No build stage or normal startup downloads model weights.

This overlay targets Linux x86-64 (linux/amd64), CPython 3.11, and CPU only. It does not support ARM64 CPU or the CUDA stack. See the worker guide for the current ARM64 TorchCodec dependency boundary. The four-hour input limit does not establish four-hour capacity. Stock Pyannote decodes complete audio and uses global clustering with potentially quadratic RAM growth. Native imports, real inference, four-hour capacity, and GPU execution require separate target validation. Production infrastructure does not enable diarization.

Provide these required environment variables through your operator environment:

Variable	Requirement
DIARIZATION_MODEL_DIR	Absolute path to the prepared, immutable Community-1 artifact tree; read-only mount at /diarization-models
DIARIZATION_SCRATCH_DIR	Absolute path to a dedicated quota-bounded disk directory; writable mount at /recordings
DIARIZATION_STT_CPUS	Explicit CPU budget for STT and its child
DIARIZATION_STT_MEMORY	Explicit combined STT and child RAM cap, based on workload measurements

Both directories must exist before startup; Compose refuses absent paths. UID/GID 65532 needs model read access and scratch write access. The overlay preserves the Kyutai model mount, private network, read-only root filesystem, PID cap, init, and cancellation grace period. It leaves the router and TTS unchanged. Two-thread OpenMP, BLAS, and Tokio limits reduce thread pressure; they do not establish capacity.

Use a filesystem quota or a dedicated filesystem with an operator-controlled free-space budget. A bind mount alone provides no disk quota. Four hours of mono PCM16 at 24 kHz consumes 691,200,000 bytes per admitted session, before other files. Reserve additional disk space for the 16 MiB child output, atomic output staging, native temporary files, and filesystem overhead. Reserve separate space for model snapshots and preparation caches. Compose sets TMPDIR=/recordings as the service-level disk path. The integration requires the parent to override the child's TMPDIR, TMP, and TEMP with its private session directory before spawn. This keeps native temporary files within the parent's cleanup boundary and off the inherited 16 MiB /tmp tmpfs. The parent removes its private session directory after success, failure, or cancellation. A process crash or power loss can leave stale owned session directories. After all sessions stop, inspect stale directories before operator cleanup. Do not delete unknown scratch entries. The demo also needs its own quota-bounded local scratch directory outside the repository.

After license review, artifact preparation, and resource checks:

docker compose -f compose.cpu.yaml -f compose.diarization-cpu.yaml config --quiet
docker compose -f compose.cpu.yaml -f compose.diarization-cpu.yaml build kyutai-stt
docker compose -f compose.cpu.yaml -f compose.diarization-cpu.yaml up --wait --wait-timeout 900
docker compose -f compose.cpu.yaml -f compose.diarization-cpu.yaml down

Use the live speech demo with the explicit local endpoint:

cargo run --manifest-path examples/live-speech/Cargo.toml --release -- \
  --endpoint http://127.0.0.1:50051 \
  --diarization --duration 14400 --scratch-dir /absolute/quota-bounded/demo-scratch

Choose a duration no greater than 14,400 seconds. The demo's default remote endpoint does not imply deployed diarization support. Space ends audio input and starts finalization; Q or Ctrl-C cancels. N cancels the current round before a new round. Interim text remains provisional until the final cumulative response supplies recording-local labels. The final cumulative message has a 4 MiB cap; overflow fails explicitly rather than truncates or emits partial success.

Optional ARM64 CUDA diarization candidate

compose.diarization-cuda.yaml extends only STT in the ARM64 CUDA stack. It selects kyutai-stt:diarization-cuda and the CUDA diarization configuration. Use it with compose.cuda.yaml, never the CPU stack. The native ARM64 build, native imports, and GPU inference remain unverified. This candidate provides no four-hour, latency, RAM, or shared-VRAM capacity guarantee.

The worker source-build guide defines the frozen environment and upstream source pin. TorchCodec 0.8.1 lacks a compatible ARM64 wheel; the candidate compiles unmodified upstream source instead. The build selects CPU-only TorchCodec with ENABLE_CUDA=0; Pyannote and Torch still select CUDA for diarization. The worker supplies waveform input and needs no NVDEC video decoder. Neither STT nor its child falls back to CPU.

The overlay requires the same model, scratch, CPU, and RAM variables as the CPU overlay. It also requires I_CONFIRM_THIS_IS_NOT_A_LICENSE_VIOLATION as an explicit build argument, without a default. Before affirmation, review the pinned TorchCodec source, the image's FFmpeg configuration, and the intended distribution license obligations. Only after that review, supply the value 1 through your operator environment. Neither the Dockerfile nor Compose automatically affirms this guard. Model gated-access acceptance remains a separate operator action.

After host checks, license review, explicit affirmation, artifact preparation, and resource checks:

docker compose -f compose.cuda.yaml -f compose.diarization-cuda.yaml config --quiet
docker compose -f compose.cuda.yaml -f compose.diarization-cuda.yaml build kyutai-stt
docker compose -f compose.cuda.yaml -f compose.diarization-cuda.yaml up --wait --wait-timeout 900
docker compose -f compose.cuda.yaml -f compose.diarization-cuda.yaml down

The overlay retains the base STT GPU reservation and STT_GPU_DEVICE_ID selection. The default physical device remains GPU 0; the service sees it as cuda:0. The STT process and child use that same GPU; the reservation provides no additional or exclusive VRAM. Router and TTS configurations remain unchanged. The CPU guide's disk bounds, private-session cleanup requirement, demo controls, and approximate final speaker-label semantics also apply here. Production infrastructure does not enable this candidate.

Optional ARM64 SBSA CUDA backends

License restriction: the selected Expresso Happy voice uses CC BY-NC 4.0, for non-commercial use only. The Kyutai STT and TTS models use CC BY 4.0; the artifact license links above also apply here.

compose.cuda.yaml defines a separate router, STT, and TTS stack for Linux ARM64/SBSA. The target is an Ampere Altra M128 with a discrete NVIDIA RTX A6000 48 GB, compute capability 8.6. This is not Jetson/L4T, AMD64 emulation, or a macOS CUDA execution path. RTX A6000 uses Ampere, not the RTX 6000 Ada architecture. The default stack, CPU stack, kyutai-stt:latest, and kyutai-tts:latest retain their current behavior. Use the CUDA file alone, not as an override for either other Compose file.

GPU host requirements

Read the STT CUDA requirements and TTS CUDA requirements before startup. The host requires:

• Linux ARM64/SBSA and an accessible RTX A6000.
• A native SBSA NVIDIA driver 570.124.06 or newer, the CUDA 12.8 Update 1 baseline.
• NVIDIA Container Toolkit configured for Docker GPU reservations.
• Docker Engine and Docker Compose 2.24 or later.
• Explicitly prepared artifacts, as described in the CPU preparation procedure.
• Enough host RAM and GPU VRAM for both float32 models, decoder state, caches, and runtime workspaces.

These prerequisites do not establish the target host's driver version or toolkit configuration. Both backends share physical GPU 0 by default; two reservations do not imply two GPUs or exclusive capacity. Set STT_GPU_DEVICE_ID and TTS_GPU_DEVICE_ID to select host device IDs or UUIDs independently. Each backend sees its selected physical GPU as cuda:0. Neither backend falls back to CPU if CUDA initialization fails.

The stack caps host RAM at 24 GiB for STT, 16 GiB for TTS, and 512 MiB for the router. These are configured caps, not measured requirements or GPU VRAM limits. CPU float32 RSS does not predict GPU VRAM use. Combined startup and inference capacity on the shared 48 GB GPU remains unmeasured. An image build establishes neither GPU readiness nor inference, VRAM use, latency, throughput, or audio quality. Those checks require the actual Altra/A6000 host; this stack makes no capacity or real-time guarantee.

Build and operate

The stack builds kyutai-stt:cuda and kyutai-tts:cuda through STT and TTS Dockerfiles, with root build contexts. All three services use linux/arm64; the router retains its ordinary CPU-only runtime with no GPU reservation or weights. STT targets sm_86 kernels with Candle CUDA. TTS uses Python 3.11.14, PyTorch 2.9.1+cu128, and its separate frozen CUDA lock. Its ARM64 wheel includes sm_80 kernels, which NVIDIA's Ampere compatibility guide permits on compute 8.6. See the TTS CUDA dependency details for exact wheel and library pins.

After host checks, license review, and explicit artifact preparation, run from this repository:

docker compose -f compose.cuda.yaml config --quiet
docker compose -f compose.cuda.yaml build
docker compose -f compose.cuda.yaml up --wait --wait-timeout 900
docker compose -f compose.cuda.yaml down

Only the router publishes 127.0.0.1:${WHISPERS_HOST_PORT:-50051}; neither backend publishes a host port. The internal backend network permits no external egress. Model mounts remain read-only and require prior host directories readable by UID 65532. Images contain no model or voice assets; builds and normal startup never download those assets. Each backend receives a bounded 256 MiB /tmp for CUDA cache and temporary files. All three services set TOKIO_WORKER_THREADS=2; health probes inherit the same bound. This avoids oversized async thread pools against PID caps on the 128-core M128; it does not limit model GPU kernels. Backend health checks require actual model readiness, with 300-second STT and 600-second TTS startup grace periods. Router readiness remains independent of backend readiness.

config/cuda.yaml preserves the CPU stack's explicit model routes, four native Google RPCs, and router limits. The selectors, audio formats, and voice settings above remain unchanged. Each backend admits one active session; the router concurrency limit does not add GPU capacity. Ordinary tests still use fixtures, not CUDA inference.

Configuration and behavior

config/local.yaml defines the listener, route maps, and explicit local test limits. Pass another file with whispers-server --config PATH. Compose mounts its configuration read-only; restart the router after a configuration change.

Configuration key	Local value	Purpose
bind	0.0.0.0:50051	Container listener address
routes.stt / routes.tts	Fixture route maps	Exact model ID to backend gRPC endpoint
limits.max_message_bytes	4194304	Maximum individual gRPC message size
limits.max_concurrent_calls	64	Total call bound, including STT queue wait
limits.stream_buffer_capacity	8	Bounded stream queue capacity
limits.connect_timeout_ms	3000	Backend connection timeout
limits.default_timeout_ms	300000	Default call timeout
limits.shutdown_timeout_ms	10000	Graceful shutdown timeout

There is no default model. Empty or malformed selectors return INVALID_ARGUMENT; unknown model IDs return NOT_FOUND. Unavailable backends return UNAVAILABLE. Stream clients send one configuration request before content requests. The router forwards streams incrementally, with no full-stream buffer, retries, or failover. It preserves message order, half-close, cancellation, deadlines, backend status, and trailers.

To connect a model service, replace the relevant route endpoint with its compatible gRPC endpoint. Clients select the map key, not a caller-supplied backend URL. The model service owns inference and backend-specific validation.

STT queue

Each router process permits one active STT call per canonical backend endpoint. Unary and stream calls share that slot, including model aliases with the same endpoint. The router never preempts an active call; TTS behavior remains unchanged. The slot lasts through the response's terminal status, EOF, cancellation, or drop, not just response headers.

When the slot becomes free, the router selects the shortest observed audio duration among queued calls. For explicit headerless LINEAR16, MULAW, and ALAW, it computes milliseconds as audio_bytes * 1000 / (sample_rate * channels * bytes_per_sample). It requires a declared sample rate of 8000–48000 Hz and 1–8 channels for this estimate. Unary content uses its complete byte count; streams use only audio bytes that the router already observes. A stream estimate is a lower bound, not a prediction of its eventual total. Compressed audio, automatic format detection, URI content, and streams without observed audio use a fixed 60-second priority score. Equal scores use arrival order. After 30 seconds of queue wait, a call takes priority over all younger calls; such older calls use arrival order. These fixed rules require no additional configuration.

An idle endpoint dispatches immediately after route selection, even before a stream sends audio. A queued stream also starts as soon as it wins the slot; it does not need to finish its upload. Responses flow while the client continues its upload. The input relay retains at most stream_buffer_capacity frames plus one frame of lookahead, apart from bounded transport and codec buffers. Each frame remains subject to max_message_bytes; a full relay queue applies backpressure to the client. The router neither buffers a whole recording nor spools audio to disk. Its estimate stops at that bounded lookahead until dispatch permits more input.

max_concurrent_calls bounds all admitted calls, including queue wait; calls above that limit return RESOURCE_EXHAUSTED. The original call deadline includes queue wait, and cancellation or expiry removes the queued call without backend dispatch. Age priority prevents younger calls from indefinitely overtaking older calls; it does not bound an active call's duration. This queue coordinates one router process only, not multiple replicas or direct backend clients. Gateway cancellation does not acknowledge backend cleanup completion. The native STT backend permits one active/reset session plus at most one bounded handoff call. That call waits for the actual inference lease under its original deadline; cancellation, expiry, or shutdown prevents inference. Reset acknowledgement precedes the next session; neither active work preemption nor two concurrent inference sessions occur. Direct clients and other router processes share this backend bound. Full HTTP admission returns RESOURCE_EXHAUSTED; the router forwards that safety response without retries or replay.

Telemetry and local exposure

The router and both model adapters emit content-free JSON logs and optional OTLP traces, metrics, and correlated logs. The telemetry contract defines resource attributes, span phases, privacy limits, collector configuration, and post-deployment checks. Every service uses service.namespace=whispers, a unique service.name, and deployment.environment.name. Compose sets environment local; Pulumi derives the environment from its validated stack name (prod for production). All base stacks accept optional .env.telemetry values; no local stack requires an external collector. Use OTEL_EXPORTER_OTLP_ENDPOINT for all signals, or the respective TRACES, METRICS, and LOGS endpoint variables. Only http/protobuf transport is supported; no service exposes a metrics scrape endpoint. Without OTLP endpoints, JSON logs remain local. The CPU/CUDA stacks require the explicit telemetry network overlay for external export. That overlay grants bridge egress; default private model networking remains unchanged. Keep credentials outside the repository.

Standard gRPC health and reflection share the native gRPC listener. Compose exposes only the router's loopback port; fixtures have no host ports. Containers use UID/GID 65532, a read-only root filesystem, no capabilities, and no privilege escalation. The default stack's bridge network permits outbound backend and telemetry connections. Whispers provides no authentication or TLS termination. This local setup provides no production access-control or TLS guarantees. The router Pulumi project declares native gRPC through an HTTPRoute and the cluster's TLS gateway. Gateway TLS does not add application authentication; backend connections use private plaintext gRPC.

Repository map

Path	Responsibility
crates/whispers-proto/	Committed Google protobuf types, gRPC clients/servers, and reflection descriptor
crates/whispers-server/	Router core, gRPC adapters, configuration, telemetry, and test fixture binary
crates/kyutai-stt-server/	STT model service with an in-process native owner thread
crates/whispers-transport/	Shared bounded H2 transport, terminal status, flow credit, and connection shutdown
crates/whispers-observability/	Shared resource identity, trace propagation, exporters, and JSON log correlation
crates/kyutai-tts-server/	TTS adapter and isolated Python worker lifecycle
services/kyutai-stt/	STT model manifest, pinned Moshi dependency, provenance, and preparation
services/kyutai-tts/	TTS Python worker, artifact pins, dependency locks, licenses, and tests
services/pyannote-diarization/	Optional Community-1 child, explicit preparation, CPU lock, and attribution
proto/	Vendored upstream contracts and provenance
config/local.yaml	Local routes and limits
Dockerfile	Separate router runtime and test fixture images
docker/	Separate CPU and CUDA model images with root build contexts
compose.yaml	Router-only default and optional fixture services
compose.cpu.yaml	Optional real CPU backends through config/cpu.yaml
compose.cuda.yaml	Optional ARM64 SBSA CUDA backends through config/cuda.yaml
compose.diarization-cpu.yaml	Opt-in Linux x86-64 Community-1 child overlay for the CPU stack
compose.diarization-cuda.yaml	Optional ARM64 CUDA Community-1 source-build candidate overlay
scripts/	Reproducible protocol generation and isolated container test
infra/pulumi/	Three independent projects: router, STT, and TTS

See CONTRIBUTING.md for source builds, protocol generation, and validation. See manual deployment for immutable ARM64 images, Pulumi phases, NFS artifacts, and gateway TLS checks.