LLMProxy

LLMProxy is a FastAPI-based proxy that load balances across multiple Large Language Model (LLM) providers. It handles failover, retries, caching, and rate limits for you while remaining fully compatible with the OpenAI API surface.

Why LLMProxy?

• Provider-agnostic: Register OpenAI, Azure OpenAI, and any OpenAI-compatible endpoints in a single configuration.
• Graceful failover: Automatically detect failing upstreams, cool them down, and retry requests against healthy endpoints. A request every endpoint refuses (400, 413, 422) comes back to the caller as that refusal, not as a 503.
• Observability built in: Health and statistics endpoints expose live state; Redis-backed state tracking keeps multiple proxy instances in sync.
• Deterministic caching: Cache both regular and streaming responses in Redis with fine-grained controls and manual cache invalidation.
• Drop-in OpenAI compatibility: Reuse existing SDK clients (chat completions, responses, embeddings) by only changing the base URL.
• Battle-tested test suite: End-to-end pytest harness spins up mock upstreams, Redis, and the proxy itself for reliable CI.

Project Layout

llmproxy/
  api/          # OpenAI-compatible route handlers (chat, responses, embeddings)
  clients/      # Async HTTP client with streaming + retry helpers
  core/         # Caching, Redis, logging utilities
  managers/     # Load balancer + endpoint state management
  config/       # Pydantic config models & YAML loader
tests/          # Pytest suite with mock upstream servers

Quick Start

Requirements

• Python 3.12 or newer
• Redis 6+
• uv >= 0.13.4 (includes the certificate fix for GHSA-hq85-5cpj-mr62)
• API keys for your upstream providers (OpenAI, Azure OpenAI, etc.)

1. Clone and install

git clone https://github.com/yourusername/llmproxy.git
cd llmproxy
uv sync --locked --no-dev

For development tooling (black, mypy, etc.):

make install-dev

2. Configure credentials

cp llmproxy.yaml.example llmproxy.yaml
cp env.example .env

Fill in provider keys and Redis connection info in .env, then update llmproxy.yaml to point at your upstream endpoints. Configuration supports os.environ/VARNAME references so secrets can stay in the environment.

Key sections in llmproxy.yaml:

• model_groups: group endpoints that serve the same model (weights control routing bias).
• general_settings: bind address/port, retry/cooldown behavior, Redis connection.
• cache_params: optional cache override settings (namespace, TTL, custom Redis host).

3. Run the proxy

Start Redis if you do not have one running already (brew services start redis on macOS). Then launch the proxy:

uv run --locked --no-dev llmproxy --config llmproxy.yaml
# or
uv run --locked --no-dev python -m llmproxy.cli --log-level INFO

By default the proxy reads llmproxy.yaml in the working directory and binds to the address/port defined under general_settings (127.0.0.1:4243 in the sample file).

Environment-based config: set LLMPROXY_CONFIG=/path/to/config.yaml to point the proxy at another file.

Using LLMProxy

Any OpenAI SDK can target the proxy by swapping the base URL. Authentication to upstream providers is handled by the proxy.

Chat Completions

from openai import OpenAI

client = OpenAI(
    base_url="http://127.0.0.1:4243",
    api_key="dummy-key",  # ignored by the proxy
)

response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Hello from LLMProxy"}],
)

print(response.choices[0].message.content)

Responses API + Streaming Cache

response = client.responses.create(
    model="gpt-4.1",
    input=[{"role": "user", "content": "Tell me a story"}],
    stream=True,
)

for event in response:
    # Streaming events are cached; repeated requests replay immediately
    print(event)

Disable caching per-request by passing extra_body={"cache": {"no-cache": True}}.

Responses API Stateful Affinity (previous_response_id + encrypted reasoning)

Stateful Responses API flows require routing consistency. LLMProxy enforces endpoint affinity for:

• previous_response_id chains
• encrypted reasoning items (include: ["reasoning.encrypted_content"])

How it works:

• previous_response_id: LLMProxy stores response_id → endpoint_id for 6 hours and routes follow-up requests to the same upstream.
• Encrypted reasoning: LLMProxy hashes each encrypted_content blob from Responses API outputs and maps hash → endpoint_id. When a request includes encrypted reasoning items, all of them must resolve to the same endpoint or LLMProxy returns 409.

If an affinity mapping is missing or expired, LLMProxy returns 409 to avoid silently losing context. This is intentional: you should retry with a valid previous_response_id or a current encrypted reasoning payload.

Embeddings

embeddings = client.embeddings.create(
    model="text-embedding-3-large",
    input="Searchable content",
)

vector = embeddings.data[0].embedding

Administrative Endpoints

• GET /health: readiness info, upstream counts, Redis state.
• GET /stats: live per-endpoint statistics pulled from Redis.
• DELETE /cache: invalidate cached responses (useful for testing).

Usage Telemetry

/stats reports live endpoint health, which expires with the endpoint state TTL. For durable per-request accounting — token counts, cost inputs, reasoning effort, which endpoint actually served, and cache-hit rate — enable the usage_stream block to append one record per request to a Redis Stream:

general_settings:
  usage_stream:
    enabled: True
    stream_key: "llmproxy-telemetry:usage"
    caller_headers:
      - x-coral-agent
      - x-coral-run-id

Callers attribute their own traffic with the request headers listed in caller_headers; headers that carry credentials cannot be listed. Telemetry is disabled when the block is absent, and a Redis failure never affects the proxied request.

See docs/usage-telemetry.md for the record shape, how to size the stream, and how to consume it.

Configuration Deep Dive

llmproxy/config_model.py defines the schema enforced at load time. Highlights:

• Weighted round-robin routing per model_group with Azure/OpenAI parameters stored under params.
• Redis-backed endpoint state shared across processes via EndpointStateManager.
• Tunable retry/cooldown thresholds (allowed_fails, cooldown_time, num_retries).
• Optional dedicated cache namespace and TTL overrides.
• response_affinity_ttl: TTL (seconds) for response/encrypted reasoning affinity mappings (default 21600 = 6 hours).

The async loader in llmproxy/config/config_loader.py resolves os.environ/VAR references and merges missing cache fields from general Redis settings.

Architecture Overview

1. FastAPI app (llmproxy/main.py) wires the lifespan events, initializes Redis, cache, LLM client, and the load balancer.
2. LoadBalancer selects an endpoint per request, tracking health stats in Redis so multiple instances can share state.
3. LLMClient issues upstream HTTP/streaming requests with retries, respecting per-endpoint configuration.
4. CacheManager stores deterministic responses in Redis and replays streaming content from cached chunks.
5. Request handlers in llmproxy/api expose OpenAI-compatible endpoints (/chat/completions, /responses, /embeddings).

Development Workflow

make install-dev   # editable install + dev dependencies + pre-commit
make upgrade       # refresh locked dependencies with the configured package-age delay
make pre-commit    # format, lint, type-check
make test          # run full pytest suite (starts mock servers + Redis)

Pytest spins up mock upstream servers, a dedicated proxy instance, and handles Redis automatically. Tests cover caching behavior, failover logic, streaming, and CLI ergonomics.

Dependency Policy

Dependency resolution uses uv with a 7-day exclude-newer cooldown. Routine installs use the committed uv.lock file via uv sync --locked; use make upgrade when you intentionally want to refresh dependency versions under that cooldown.

Security minimum versions are recorded in pyproject.toml. Certifi 2026.10.19 and virtualenv 21.14.12 temporarily bypass the cooldown for the emSign CA-trust fixes. Virtualenv bundles pip 26.2.2 for the supported Python versions, including pre-commit environments. Remove the certifi exception after 2026-10-27 and the virtualenv exception after 2026-10-28, once their PyPI uploads have passed the normal cooldown. CI also uses patched uv to install pip 26.2.2 into its base Python environment.

Existing pre-commit environments retain their installed pip. After upgrading, rebuild them with uv run --locked --extra dev pre-commit clean followed by make pre-commit. The clean command clears the current pre-commit cache, which may be shared with other repositories; their hooks will rebuild on next use.

Useful scripts:

• clear_cache.sh: quick helper to hit the /cache endpoint locally.
• make test-cov: generate HTML coverage in htmlcov/.

Contributing

1. Install dev dependencies and enable pre-commit.
2. Make your changes with accompanying tests.
3. Run make pre-commit and make test before opening a PR.

We welcome improvements to additional providers, caching strategies, and observability tooling.

License

MIT License - see LICENSE for details.