Repository navigation
feat(anthropic): fail over to spare Claude API keys - #95
Conversation
ANTHROPIC_API_KEY_1, _2, ... join ANTHROPIC_API_KEY in the credential pool. Requests use the first key that is not resting, so the spare only carries traffic after the primary is refused. The Claude API reports an empty credit balance and an owner-set spend limit as 400 invalid_request_error, and a tier spend cap as a 429 with enforced_spend_limit_reached. Raw dispatch now reads that small JSON error body, rests the key for an hour and marks the response. Unified routes, native /anthropic/v1/* routes and the auto:intelligence chain then send the request once on the next key. Other 400 errors still go straight back to the caller. The Worker forwards ANTHROPIC_API_KEY_<n> to the Container. A body read for classification is replayed from the cache, so relays still forward Anthropic's error.
…wn flags Rerun the route failover tests with scoped cooldowns, reset scheduling and prompt-cache affinity enabled, as production runs them, and add a streamed chat whose first key ran out of credit. The shared refusal check is now public, since the native route imports it.
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars.
 |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Claude API keys now form a credential pool, so the intelligence transport reads them with get_api_keys. The fixture that marks the primary unconfigured patches that list too.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8ac554d3a5
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| POOLED_KEY_ENVS = { | ||
| **{pool.provider: pool.credential_env for pool in CODEX_EVERYWHERE_POOLS}, | ||
| "ce-image": "CODEX_EVERYWHERE_API_KEY_GPT_IMAGE", | ||
| "anthropic": "ANTHROPIC_API_KEY", |
There was a problem hiding this comment.
When keys belong to different organizations, a conversation started on the spare while the primary is resting will silently switch back to the primary once its cooldown expires. The advertised affinity protection does not cover native /anthropic/v1/messages or explicit unified Anthropic requests: request_scope() in services/prompt_cache_affinity.py only creates scopes for unified auto: selectors, and prefer_credential() returns nothing without an active scope and candidate. Consequently, even with production affinity enabled, replaying the spare organization's signed thinking blocks after the primary recovers produces an invalid-signature 400, which is not retried. Extend conversation-to-credential binding to these routes so recovery does not move an otherwise healthy conversation between organizations.
Useful? React with 👍 / 👎.
| CredentialPool.record_headers(api_provider, headers, status, | ||
| **(cooldown_context or dispatch_context(api_provider, data, url)), **observation) |
There was a problem hiding this comment.
For accounts with an administrator-owned data policy, such as official_only=true, services/provider_dispatch.py::_credential() replaces the selected X-Api-Key with operator_credential("anthropic"), which reads only ANTHROPIC_API_KEY and ignores the pool. Therefore, after the primary is refused, selecting a spare still sends the exhausted primary again. This recording call then attributes that refusal to the originally selected spare, cooling down a healthy credential that was never sent and potentially exhausting the pool for other users too. Update the trusted credential-resolution path to honor eligible operator-configured pool keys, and record the credential actually dispatched.
Useful? React with 👍 / 👎.
| POOLED_KEY_ENVS = { | ||
| **{pool.provider: pool.credential_env for pool in CODEX_EVERYWHERE_POOLS}, | ||
| "ce-image": "CODEX_EVERYWHERE_API_KEY_GPT_IMAGE", | ||
| "anthropic": "ANTHROPIC_API_KEY", |
There was a problem hiding this comment.
With production's POOL_RESET_SCHEDULING_ENABLED=true, exhausting the configured Claude keys now also makes /admin/auto-routes fail with 429. Adding Anthropic to this pool makes AuthService.get_api_key("anthropic") raise ModelCooldownExhausted when every key is resting, but routes/auto_routes.py::_provider_is_configured() calls that method without catching cooldown errors. _admin_payload() checks every registered provider, so both GET and PUT responses fail even when the administrator is managing routes unrelated to Anthropic. Use configured-key enumeration rather than dispatch admission for this configuration check, or handle cooldown exhaustion while retaining the configured status.
Useful? React with 👍 / 👎.
What
ANTHROPIC_API_KEY_1,_2, … joinANTHROPIC_API_KEYin the credential pool. Requests use the first key that is not resting, so the spare only carries traffic once the primary is refused. A refused request runs once more on the next key, on:/v1/chat/completions,/v1/responses,/v1/messages)/anthropic/v1/*auto:intelligencechain (Haiku 5.5)Why
Production has two Claude API keys, each in its own organization with its own credit. Until now the gateway read only
ANTHROPIC_API_KEY, so once that key ran dry every Anthropic route failed.How
invalid_request_error, not 402. A tier spend cap is a 429 witherror.details.error_code: enforced_spend_limit_reached.providers/anthropic.pyreads that small JSON error body, nothing else. Raw dispatch then rests the key for an hour as a credential-wide rejection and marks the response.ANTHROPIC_API_KEY_<n>to the Container._stream_contentwould otherwise read a drained socket.Trade-off
The two keys belong to different organizations, and signed thinking blocks only replay in the organization that created them. Prompt-cache affinity keeps each conversation on its key, so a conversation only moves when its key is refused. After a move, its next turn can fail with an invalid
signature. This is documented indocs/anthropic.md.Tests
tests/test_anthropic_key_failover.pycovers the classifier table, body relay after classification, and the pool order/rest. It also runs the unified chat (plain and streamed), native Messages and the intelligence chain through the real raw-dispatch path. Every route test runs twice: legacy settings, and production's scoped cooldown + reset scheduling + cache affinity.test_cloudflare_worker.mjs/test_anthropic_worker.mjspass.wrangler devboots cleanly.