Repository navigation
fix(models): concurrency_limit auto is accepted before its engine is up - #284
Merged
Merged
Conversation
ethan-scitix
force-pushed
the
fix/auto-concurrency-before-launch
branch
from
November 7, 2026 15:28
3b585bd to
88af6b0
Compare
`--dry-run` and the planning `sieval run` does before launching bind each root to the deployment its launch will realize. That deployment has not reported a KV pool, so `_kv_token_budget(adaptive=True)` returned None and pool setup refused `auto`: `--dry-run` refused it on every SGLang engine `sieval run` launches, and a run refused it on a model started at a barrier, which is bound that way for its whole run. A model launched up front was not affected at run time, since its launch reads the pool first. #278's tests covered the budget helper on a deployment that already had facts, not these paths. A root bound before its engine is up now gets a PendingTokenBudget: the pool reads the engine's /get_server_info at the first request through it, once the engine is up, and admits by the budget that read gives. `auto` on an engine that reports no pool and window fails that request, naming the root, before anything is sent; a number or no limit runs without KV admission, as on a launched engine that reports none. An engine already up at binding keeps the budget its launch read. `auto` on an endpoint, vLLM, a derived model or the top level is refused before launch as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ethan-scitix
force-pushed
the
fix/auto-concurrency-before-launch
branch
from
November 7, 2026 15:35
88af6b0 to
1d0938e
Compare
PendingTokenBudget now keeps a failed engine read and raises it to every later request, so an engine reporting no pool is read once rather than per request, each up to a 10s timeout. A root started at a barrier with no concurrency_limit now gets KV admission, as one started up front does; the CHANGELOG says so and a test covers it. The `auto` refusal names its real cause when the engine is not one sieval run launched. The new integration test shares the staged-run stub engines through a plain generator instead of a fixture's `__wrapped__`, which ty rejects. Docs and comments trimmed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Nov 7, 2026
Merged
ethan-scitix
added a commit
that referenced
this pull request
Nov 8, 2026
* feat(models)!: KV admission only with concurrency_limit auto KV admission and `concurrency_limit: auto` are now on together or off together. Since #277 an SGLang engine sieval run launched admitted requests by a KV TokenBudget whenever its root set no concurrency_limit; `auto` added the engine's load on top (#278), and #284 deferred either to the first request for a root bound before its engine was up. A root with no concurrency_limit is back on the path it had before #277: no shared root limiter, no token budget, and the launch reads no /get_server_info, so it prints no `KV:` line and its request records carry no `kv_tokens`. A number keeps its fixed cap. `auto` is unchanged: an AdaptiveTokenBudget from the pool the launch read, or a PendingTokenBudget read at the first request when the engine is not up yet, and the same refusals when no pool is reported. No identity or plan field depended on the default: the KV facts are not in the deployment fingerprint, and admission was throughput only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(session): share the concurrency_limit lookup `sets_auto_concurrency` copied the "top level wins over args" rule from `_connection_options`; if the two drifted, the launch would skip the KV pool read while the session still saw `auto`. Both now call one helper, and a test pins the precedence. Docs, CHANGELOG and comments trimmed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
12 of 13 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Type
Summary
concurrency_limit: auto(#278) is refused by--dry-runon every SGLang enginesieval runlaunches, and at run time on a model started at a barrier, before any engine starts. Harmony's config check runs the dry-run, so 0.15.5 cannot useautothere:--dry-runand the planning a run does before launching bind each root to the deployment its launch will realize (unlaunched_deployment). That deployment has reported no KV pool yet, so_kv_token_budget(adaptive=True)returnedNoneand_create_owned_poolraised. A judge started at a barrier is bound that way for its whole run, so a run refused it too. A model launched up front was not affected at run time: its launch reads the pool before binding. feat(models): concurrency_limit auto reads a launched SGLang engine's load #278's tests covered the budget helper on a deployment that already had facts, not these paths. Seen in Harmony's config check, which runssieval run --dry-run: the throughput benchmark'sautorun was refused before it held a GPU.PendingTokenBudget. The pool reads the engine's/get_server_infoonce, at the first request through it, when the engine is up, and admits by the budget that read gives. Planning sends no request, so it never reads. An engine already up at binding keeps the budget its launch read, as before.autofails its requests, naming the root, before anything is sent; the failed read is kept, so the engine is read once, not per request. A run whose engine is up at binding is refused at setup. A number runs without KV admission, as on a launched engine that reports none (feat(models): admit requests to a launched engine by the KV they may hold #277).concurrency_limitnow gets KV admission, as a root started up front has since feat(models): admit requests to a launched engine by the KV they may hold #277. It ran without it before.autoon an external endpoint, vLLM, a derived model, or the top level.Related Issues
Follows #278. Found by the throughput comparison run for 0.15.5.
Test Plan
Automated
ruff check && ruff format --check)ty check: All checks passed; the first push failed CI on a pytest fixture's__wrapped__, now a shared plain generatorstub_engines)pytest -n 8 tests/unit tests/integration: 11286 passed, 4 failed. The same 4 fail onmainhere:test_wikisql×3 (babel) andtest_exec_quotebench_anchor(GNU coreutils on macOS).tests/integration/test_auto_concurrency.py(new), through_run_dry_run/_run_allwith the stub engines oftest_staged_runanswering/get_server_infoonce up:autoon a launched SGLang subject has no errors, launches nothing and reads no engine (refused onmain);AdaptiveTokenBudgetof that size;from: inferencewithautogets its budget after it starts (refused onmain);/get_server_info, with no request sent;TokenBudgetonce up;autoon vLLM and on an external endpoint is refused by the dry-run.tests/unit/core/models/test_admission.py: aPendingTokenBudgetis read once at the first request even with concurrent requests, a failed read fails every request and is read once (6 reads if the error is not kept), and it closes the adaptive budget it resolved.Manual
concurrency_limit: autorun (subject on 4 × H100, the general-1k-32k heavy lanes) on an image built from this branch.Checklist
Required (all PRs)
type(scope): description)AI-Generated Code - <model> (<provider>)in module docstringcore/🤖 Generated with Claude Code