Repository navigation
feat(lean): keep Haiku coding sessions cached and under the 5x long-prompt price - #97
Conversation
…r the long-prompt price /lean/v1/messages, /lean/v1/chat/completions and /lean/v1/responses serve the unified routes. For Anthropic API models the upstream body gains automatic prompt caching, server-side clearing of old tool results, deterministic shell-output condensing and head-and-tail clipping of oversized results. A prompt counted above 100,000 tokens is refused with each protocol's own context-overflow error, so the harness compacts instead of paying the long-prompt price. Every change depends only on the request, so later turns repeat earlier bytes and replayed thinking blocks stay valid.
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars.
 |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
…ault Anthropic's pricing page does not say whether a prompt shortened by server-side context editing is priced by its size before or after clearing. count_tokens reports both, so the guard now refuses on the larger size unless LEAN_PRICE_LIMIT_BASIS=cleared, and reports the size after clearing in its own header. The docs note that a harness stops after repeated failed compactions when one message alone is too large.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 288bd62e80
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if value.get("type") in {"image", "document"}: | ||
| media += 1 | ||
| return |
There was a problem hiding this comment.
For Messages requests containing a large document, this early return ignores all document contents and assigns only 1,600 tokens regardless of size. For example, a text-source document containing hello world repeated 100,000 times produces an estimate of 1,601 tokens and never calls count_claude_prompt_tokens, even though its actual prompt exceeds the 100,000-token price threshold. Large PDFs have the same unbounded underestimate. Count text/content sources recursively and require an exact count for documents whose token size cannot be bounded locally, so these requests cannot silently bypass the price guard.
Useful? React with 👍 / 👎.
| environ["PATH_INFO"] = path[len(LEAN_PREFIX):] | ||
| environ[LEAN_ENVIRON_KEY] = True |
There was a problem hiding this comment.
When a caller opts into response caching with X-MultiLLM-Cache: on and sends a cacheable request such as temperature: 0, rewriting the path makes /lean/v1/chat/completions share its cache identity with /v1/chat/completions. routes/chat_cache.py keys entries using request.path and the original payload, without the lean marker or style header. Consequently, an answer generated with lean's concise system instruction can be returned to an ordinary /v1 request, or an ordinary cached answer can bypass the requested concise instruction entirely. Preserve lean mode and the effective style in the response-cache identity, or bypass that cache for lean requests.
Useful? React with 👍 / 👎.
| count_body = {key: value for key, value in body.items() if key in CLAUDE_COUNT_FIELDS} | ||
| return _post_count("anthropic", url, CLAUDE_COUNT_PATH, credential, incoming, count_body) |
There was a problem hiding this comment.
For a lean prompt large enough to trigger exact counting, secrets are sent to Anthropic before the configured secret policy runs. _post_count submits this body directly through provider_http, which enforces credentials and data policy but does not apply the secret firewall; generation requests only reach protect_body later in ProxyService._make_base_request. With the default redaction policy, an embedded API key therefore leaves the gateway unredacted, and with secret_scan_mode: block, it is transmitted before generation is refused. Apply the secret policy before this outbound count and reuse the protected body for generation so both requests obey the same policy.
Useful? React with 👍 / 👎.
| def walk(value: Any) -> None: | ||
| nonlocal characters, media | ||
| if isinstance(value, dict): | ||
| if value.get("type") in {"image", "document"}: |
There was a problem hiding this comment.
For valid tool schemas containing a union such as "type": ["string", "null"], this set-membership check raises TypeError: unhashable type: 'list'. The estimator recursively visits every dictionary in tool schemas and tool inputs, so an ordinary tool argument named type with an array or object value also triggers the failure. Because estimation runs on every lean Anthropic request, these requests fail before generation even when the prompt is tiny. Check that the discriminator is a string before testing membership, while continuing to traverse arbitrary nested JSON.
Useful? React with 👍 / 👎.
…ompts and pooled keys - Text documents and base64 PDFs are estimated by their contents instead of a flat size, so a large document is always counted. - URL and file sources, which count_tokens rejects, are judged by the estimate, and so is any prompt whose count fails: an estimate above the limit is refused rather than sent. - Server tools are left out of the count and added back at a fixed size, so requests with web search can still be counted. - The count uses the key that will send the request. Thinking blocks from another account are dropped before counting, so any other key could undercount. - Only block-level cache_control counts as a caller breakpoint; a tool schema property of that name no longer turns caching off. - The default limit is 99,000 tokens, since count_tokens is an estimate.
Bandit read the shell= keyword of the condensing helpers as a subprocess call; the parameter is now terminal_output. Narrow the estimate's source and tools lookups for mypy.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b0a3397f49
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| elif isinstance(value, list): | ||
| for item in value: | ||
| walk(item) | ||
| elif isinstance(value, str): | ||
| characters += len(value) |
There was a problem hiding this comment.
For lean requests containing large numeric arrays in historical tool_use.input blocks, the estimator ignores every numeric element because the walker only accounts for strings. A Messages body with 120,000 instances of 123456789 in a tool input is approximately 1.3 MB but produces an estimate of just 34 tokens. This bypasses exact counting entirely, allowing a prompt well above the price limit through the guard. Account for numeric primitives and JSON structure, or require provider counting when their size cannot be bounded by the local estimate.
Useful? React with 👍 / 👎.
| count_body, allowance = _count_body(decision.body) | ||
| beta_header = merged_beta_value(_header(request_headers, "anthropic-beta"), decision.betas) | ||
| try: | ||
| after, before = count_claude_prompt_tokens(count_body, beta_header, base_urls, credential=credential) |
There was a problem hiding this comment.
For a near-limit lean request with X-MultiLLM-Deadline-Ms, this synchronous count can continue well beyond the caller's generation budget. The count transport in services/native_token_count.py uses fixed (5, 10) timeouts and never registers its response with the deadline's cancellation owner; provider_http does not enforce deadlines either. Consequently, a request with a 100 ms deadline can remain blocked on a stalled count for roughly 10 seconds before reaching the deadline check in _send_native_request. Bound the count transport by the remaining generation budget and attach its response to deadline cancellation so counting cannot hold the request open after expiry.
Useful? React with 👍 / 👎.
…le refused keys count_tokens runs the same thinking-block prefix check as generation, so a body that replays thinking blocks is now counted with its tools unchanged instead of without its server tools. The advisor tool, which count_tokens accepts, is no longer stripped. A count that refuses the key (401, 402, 403) no longer decides the request: it goes out so rotation can rest that key and count again with the next. Prompts with a PDF are always counted, since file size does not bound the text inside. The docs say plainly when the limit is an estimate rather than a guarantee.
GitHub Actions
Failure
Haiku 5.5 is cheap ($0.10 / $0.50 per million tokens) only while a request's prompt stays at or under 100,000 tokens. One token over, the whole request costs 5x, cached tokens included. Coding harnesses grow their prompts long, and Chat Completions and Responses traffic reaching the Anthropic API today gets no prompt caching at all, because the gateway never adds cache markers and translation drops them.
This adds a
/leanbase path. Point a harness at/lean/v1/messages,/lean/v1/chat/completionsor/lean/v1/responsesand, foranthropic:models, the upstream request gets:clear_tool_uses_20250919) from 80K tokens. The harness keeps its full history, and Anthropic documents that this never invalidates thinking blocks.count_tokensusing the key that will send it. Context editing is included and server tools are left out with an allowance. Prompts that can't be counted are judged by a content-aware estimate. Above 99K (a 1K margin, since the count is an estimate) the request is refused with the harness's own overflow error: Anthropic'sprompt is too long, OpenAI'scontext_length_exceeded, and a streamedresponse.failedfor Codex. The harness compacts with its own summarizer instead of paying 5x.Anthropic doesn't document whether a prompt shortened by clearing is priced by its size before or after clearing. So by default the limit applies to the size before;
LEAN_PRICE_LIMIT_BASIS=clearedswitches it once the bill confirms.LEAN_STYLE=conciseorX-MultiLLM-Lean-Style). Off by default: published trials found small output savings, with no measured quality loss.Model, effort, thinking and
max_tokensare never changed. Every transform depends only on the request, so turn N+1 repeats turn N's bytes. That matters on these accounts: they reject a replayed thinking block whose earlier conversation changed.Checked against the live Anthropic API (local app, a few Haiku calls):
applied_editsreported 4 tool uses / 3,839 tokens clearedcount_tokensreturned 5,042 tokens against a 5,000 test limit; Messages and Chat Completions refusals came back in the right shapes/leanwas accepted when turn 2 replayed it through/lean, and rejected ("bound to a different conversation") when the same turn went to/v1with raw shell output. So a conversation must stay on one base path, asdocs/lean.mdsaysDesign inputs: Anthropic's context-editing, compaction, caching, token-counting and preserved-thinking docs, and the compaction code of Codex (90% trigger,
context_length_exceededhandling), opencode, Cline, Roo, Aider and Gemini CLI. The research ruled out proxy-side summarizing and age-based pruning: both edit history behind the harness, which breaks caching and thinking replay.A second model reviewed the code. It found that documents, failed counts and pooled keys could slip past the guard, and that a schema property named
cache_controldisabled caching; all four are fixed with regression tests. Tests:tests/test_lean_mode.pycovers the transforms, the cross-turn byte-stability invariant, caller-owned cache and context settings, the count body, and the three refusal shapes. The related Python suites (445 tests) and the Worker tests (79) pass.