ai-proxy — audio language-detection + model router (POC)

A small, config-driven router that, given an audio file:

1. Detects the spoken language with a dedicated LID model (SpeechBrain VoxLingua107).
2. Selects a transcription model from a registry based on the detected language and the caller's priority — cost, speed, or accuracy.
3. Forwards the file to the selected provider ("the Agent") for real transcription, walking a fallback chain if a provider fails.
4. Emits OpenTelemetry traces + metrics along the whole path (per-step timing, total time, LID confidence, cost estimate, fallbacks).

Naming: the project/repo is ai-proxy; the importable Python package is aiproxy (hyphens aren't valid in module names); the OTel service.name is ai-proxy.

This is the load-bearing core of the larger transcription platform. A single uploaded file is just the one-chunk case of the eventual chunked-stream pipeline, so the router drops straight into that later as the per-chunk decision step.

Architecture

flowchart TD
    CLI["CLI<br/>cli.py: main"] --> RUN["pipeline.run(file, priority)<br/><i>span: transcribe_request</i>"]
    RUN --> AUD["audio.load_and_prep<br/><i>span: audio.load</i><br/>→ mono 16k + 30s LID window"]
    AUD --> LID["lid.detect(window)<br/><i>span: lid.detect</i><br/>→ language, confidence"]
    LID --> CONF{"confidence ≥<br/>threshold?"}
    CONF -- "no (stage 1)" --> FB["effective language =<br/>fallback_language"]
    CONF -- yes --> EFF["effective language =<br/>detected language"]
    FB --> SEL
    EFF --> SEL["router.select(effective language, priority)<br/><i>span: router.select</i><br/>→ ordered candidates"]
    SEL -. "no model lists language (stage 2)" .-> FB
    SEL --> FWD["provider.transcribe(<b>original file</b>)<br/><i>span: transcribe.forward</i>"]
    FWD -- "ProviderError" --> NEXT["next candidate"]
    NEXT --> FWD
    FWD -- success --> OUT["transcript + timings + cost"]

Loading

One router, two inputs: detected language is a hard constraint, the priority is a soft ranking. Language is resolved in two stages — a low-confidence LID result is demoted to fallback_language before routing, and the selector independently falls back if no model lists the (effective) language.

The model registry is bundled with the package at src/aiproxy/resources/models.yaml (override with --config). Adding a model for an existing provider is a data change; a brand-new provider also needs an adapter in providers/ plus a factory.py entry.

Detection runs on a decoded mono-16k window; transcription sends the original file to the provider (they decode themselves). Same audio, two representations — intentional.

For the full component breakdown, telemetry/metric reference, and routing internals, see docs/architecture.md.

How the router works

The router (src/aiproxy/router/selector.py) turns (effective language, priority) into an ordered list of models — the head is the primary choice, the tail is the fallback chain the pipeline walks if a provider call fails. Two steps:

1. Filter by language (hard). Keep enabled models whose declared languages list includes the language (multilingual is metadata and does not widen this). If none match, fall back: models for fallback_language → any multilingual model → all models.
2. Rank by priority (soft). A stable multi-key sort orders the survivors: cost → cheapest cost_per_min first; speed → fastest latency_class first; accuracy → highest accuracy_tier first. Ties break by cost, then accuracy.

So with the packaged registry, Groq's LPU-hosted Whisper tends to win cost and speed, while gpt-4o-transcribe wins accuracy — and if the top pick's provider errors, the next candidate transcribes instead (fallback used: True).

Setup

Requires Python 3.11+. Core install is lightweight; the heavy LID stack (torch, torchaudio, speechbrain) and provider SDKs are optional extras.

pip install -e ".[dev]"     # routing + telemetry + tests (no torch/SDKs) — enough for `pytest`
pip install -e ".[all]"     # everything needed for a real run (large: pulls torch)
cp .env.example .env        # then fill in provider API keys

On first real run the VoxLingua107 model downloads into pretrained_models/voxlingua107-ecapa/ (git-ignored). The first lid.detect in a process also loads PyTorch and the model (~5s cold start). Set AIPROXY_LID_WARMUP=on in .env to preload the classifier at startup (recommended for long-lived workers). CUDA is auto-selected when available; override with AIPROXY_LID_DEVICE (auto, cpu, cuda, cuda:0, etc.).

Run

samples/ ships short openly-licensed clips, one per language (see samples/README.md for the list and attribution). Either entrypoint works:

python -m aiproxy.cli samples/es.mp3 --priority cost
aiproxy samples/es.mp3 --priority speed        # installed console script
aiproxy samples/es.mp3 --priority accuracy

Same audio, three priorities → with Groq in the registry, its LPU-hosted Whisper wins both cost and speed (groq-whisper-large-v3-turbo at ~$0.04/hr, 216× real-time), while OpenAI gpt-4o-transcribe still wins accuracy. The English-only groq-distil-whisper-en is the cheapest model but is only eligible for English. The CLI prints a run summary (language, model, timings) and the transcript. Pass --verbose / -v to also emit console telemetry and LID warmup details.

aiproxy samples/es.mp3 --priority cost -v   # verbose: OTel console export + warmup line

Registry providers: Deepgram, OpenAI, AssemblyAI, and Groq (whisper-large-v3, whisper-large-v3-turbo, distil-whisper-large-v3-en). Toggle any model with enabled in the registry, or point --config at your own YAML to test a different mix.

Sample run

A real run on the bundled French clip with --priority cost. LID detects fr confidently, the router ranks the fr-capable models by cost, and Groq's turbo Whisper wins as the primary — transcribing on the first try:

$ aiproxy samples/fr.mp3 --priority cost

========================================================
  TRANSCRIPTION ROUTER — RUN SUMMARY
========================================================
  detected language : fr (confidence 0.95)
  selected model    : groq-whisper-large-v3-turbo (groq/whisper-large-v3-turbo)
  audio duration    : 4.5s
  est. cost         : $0.00005
  fallback used     : False (retries: 0)
--------------------------------------------------------
  step timings (ms):
    audio.load                 2402.7
    lid.detect                 2168.4
    router.select                 0.2
    transcribe.forward         1253.1
    TOTAL                      5824.7
========================================================

  transcript:

 Je voudrais pouvoir vivre à un rythme plus relâché, au lieu d'avoir regardé l'heure tout le temps.

Switching the same clip to --priority accuracy shows the fallback chain in action: the router ranks gpt-4o-transcribe first (highest accuracy tier), but when that provider errors it falls back to the next candidate — same transcript, now fallback used: True:

$ aiproxy samples/fr.mp3 --priority accuracy
...
  detected language : fr (confidence 0.95)
  selected model    : groq-whisper-large-v3-turbo (groq/whisper-large-v3-turbo)
  fallback used     : True (retries: 1)
...

On Windows, transcripts in non-Latin scripts (Japanese, Chinese, Korean, …) can trip a UnicodeEncodeError from the console's legacy code page; run with PYTHONIOENCODING=utf-8 (or chcp 65001) to print them. The first lid.detect in a process also pays a one-time model-load cost — see AIPROXY_LID_WARMUP above to amortize it.

Telemetry

Spans and metrics are recorded on every run. Console export is opt-in via --verbose; otherwise output stays quiet unless OTEL_EXPORTER_OTLP_ENDPOINT is set (OTLP export always on when configured). Set AIPROXY_TELEMETRY=off to disable export entirely. Spans flush immediately (SimpleSpanProcessor); metrics use a periodic reader, so the CLI calls force_flush() on exit to ensure a short run's metrics are exported.

Optional hybrid tracing is available via agent-trace:

• AIPROXY_AGENT_TRACE=on enables provider/LLM-level tracing interception.
• AIPROXY_AGENT_TRACE_EXPORTER optionally forces the agent-trace exporter (otlp or console).
• AGENT_TRACE_OTLP_ENDPOINT optionally overrides the OTLP endpoint for agent-trace.

In hybrid mode, ai-proxy keeps its existing domain spans/metrics (audio.load, lid.detect, router.select, transcribe.forward) and agent-trace adds lower-level provider call visibility beneath the same request flow. Recommended rollout: enable in staging first, compare traces, then expand.

Test

pytest                      # offline: selector + packaged registry + mocked pipeline

test_selector.py / test_config.py run with no models or network; test_config.py loads the real packaged registry so YAML drift is caught. test_pipeline.py mocks audio, LID, and providers to verify orchestration, span emission, and fallback; test_cli_output.py checks the run-summary rendering (including the low-confidence line and the verbose warmup line), and test_lid.py covers the detector's warmup-env parsing, device resolution, classifier caching, and label normalization with the heavy stack mocked out — so the whole suite needs neither torch nor API keys.

Configuration

resources/models.yaml defines defaults (priority, LID confidence threshold, fallback language) and a list of models with capability metadata. languages is authoritative for routing (a model is only a candidate for languages it lists); multilingual is metadata that influences how the adapter passes the language argument and does not widen routing. latency_class and accuracy_tier are validated at load time.

Sources (pricing, verified June 2026)

• Deepgram pricing 2026
• OpenAI gpt-4o-transcribe guide
• AssemblyAI pricing
• Groq pricing 2026 · Whisper v3 Turbo on Groq