Skip to content

fix(models): concurrency_limit auto is accepted before its engine is up - #284

Merged
ethan-scitix merged 2 commits into
mainfrom
fix/auto-concurrency-before-launch
Nov 7, 2026
Merged

ethan-scitix merged 2 commits into
mainfrom
fix/auto-concurrency-before-launch

Conversation

@ethan-scitix

@ethan-scitix ethan-scitix commented Nov 7, 2026 •

Copy link
Copy Markdown
Collaborator

Type

  • fix — bug fix

Summary

concurrency_limit: auto (#278) is refused by --dry-run on every SGLang engine sieval run launches, and at run time on a model started at a barrier, before any engine starts. Harmony's config check runs the dry-run, so 0.15.5 cannot use auto there:

resume_plan: Model root 'model:subject': concurrency_limit 'auto' needs an SGLang engine sieval run launched that reported its KV pool; set a number here
  • Cause: --dry-run and the planning a run does before launching bind each root to the deployment its launch will realize (unlaunched_deployment). That deployment has reported no KV pool yet, so _kv_token_budget(adaptive=True) returned None and _create_owned_pool raised. A judge started at a barrier is bound that way for its whole run, so a run refused it too. A model launched up front was not affected at run time: its launch reads the pool before binding. feat(models): concurrency_limit auto reads a launched SGLang engine's load #278's tests covered the budget helper on a deployment that already had facts, not these paths. Seen in Harmony's config check, which runs sieval run --dry-run: the throughput benchmark's auto run was refused before it held a GPU.
  • Fix: a root bound before its engine is up gets a PendingTokenBudget. The pool reads the engine's /get_server_info once, at the first request through it, when the engine is up, and admits by the budget that read gives. Planning sends no request, so it never reads. An engine already up at binding keeps the budget its launch read, as before.
  • An engine that reports no pool: auto fails its requests, naming the root, before anything is sent; the failed read is kept, so the engine is read once, not per request. A run whose engine is up at binding is refused at setup. A number runs without KV admission, as on a launched engine that reports none (feat(models): admit requests to a launched engine by the KV they may hold #277).
  • Behavior change: a root started at a barrier with no concurrency_limit now gets KV admission, as a root started up front has since feat(models): admit requests to a launched engine by the KV they may hold #277. It ran without it before.
  • Still refused before launch: auto on an external endpoint, vLLM, a derived model, or the top level.
  • Throughput only; no identity, schema or on-disk change.

Related Issues

Follows #278. Found by the throughput comparison run for 0.15.5.

Test Plan

Automated

  • Lint/format clean (ruff check && ruff format --check)
  • Type check clean (ty check: All checks passed; the first push failed CI on a pytest fixture's __wrapped__, now a shared plain generator stub_engines)
  • pytest -n 8 tests/unit tests/integration: 11286 passed, 4 failed. The same 4 fail on main here: test_wikisql ×3 (babel) and test_exec_quotebench_anchor (GNU coreutils on macOS).
  • tests/integration/test_auto_concurrency.py (new), through _run_dry_run / _run_all with the stub engines of test_staged_run answering /get_server_info once up:
    • a dry-run with auto on a launched SGLang subject has no errors, launches nothing and reads no engine (refused on main);
    • a run reads the pool once the engine is up and admits through an AdaptiveTokenBudget of that size;
    • a judge from: inference with auto gets its budget after it starts (refused on main);
    • an engine up at binding that reports no pool is refused naming /get_server_info, with no request sent;
    • a deferred judge that reports no pool fails its grading calls, with no request reaching it;
    • a deferred judge with no limit gets a plain TokenBudget once up;
    • auto on vLLM and on an external endpoint is refused by the dry-run.
  • tests/unit/core/models/test_admission.py: a PendingTokenBudget is read once at the first request even with concurrent requests, a failed read fails every request and is read once (6 reads if the error is not kept), and it closes the adaptive budget it resolved.

Manual

  • On aries, rerun the throughput comparison's concurrency_limit: auto run (subject on 4 × H100, the general-1k-32k heavy lanes) on an image built from this branch.

Checklist

Required (all PRs)

  • PR title follows conventional format (type(scope): description)
  • No internal paths, credentials, or personal info in committed files
  • AI-generated code has AI-Generated Code - <model> (<provider>) in module docstring
  • No new upper-layer dependencies added to core/
  • Deleted code verified — no remaining call sites depend on it

🤖 Generated with Claude Code

@ethan-scitix
ethan-scitix force-pushed the fix/auto-concurrency-before-launch branch from 3b585bd to 88af6b0 Compare November 7, 2026 15:28
`--dry-run` and the planning `sieval run` does before launching bind each
root to the deployment its launch will realize. That deployment has not
reported a KV pool, so `_kv_token_budget(adaptive=True)` returned None and
pool setup refused `auto`: `--dry-run` refused it on every SGLang engine
`sieval run` launches, and a run refused it on a model started at a barrier,
which is bound that way for its whole run. A model launched up front was not
affected at run time, since its launch reads the pool first. #278's tests
covered the budget helper on a deployment that already had facts, not
these paths.

A root bound before its engine is up now gets a PendingTokenBudget: the
pool reads the engine's /get_server_info at the first request through it,
once the engine is up, and admits by the budget that read gives. `auto`
on an engine that reports no pool and window fails that request, naming
the root, before anything is sent; a number or no limit runs without KV
admission, as on a launched engine that reports none. An engine already up
at binding keeps the budget its launch read. `auto` on an endpoint, vLLM,
a derived model or the top level is refused before launch as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ethan-scitix
ethan-scitix force-pushed the fix/auto-concurrency-before-launch branch from 88af6b0 to 1d0938e Compare November 7, 2026 15:35
PendingTokenBudget now keeps a failed engine read and raises it to every
later request, so an engine reporting no pool is read once rather than
per request, each up to a 10s timeout. A root started at a barrier with
no concurrency_limit now gets KV admission, as one started up front does;
the CHANGELOG says so and a test covers it. The `auto` refusal names its
real cause when the engine is not one sieval run launched.

The new integration test shares the staged-run stub engines through a
plain generator instead of a fixture's `__wrapped__`, which ty rejects.
Docs and comments trimmed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ethan-scitix
ethan-scitix merged commit 3b4799a into main Nov 7, 2026
9 checks passed
@ethan-scitix
ethan-scitix deleted the fix/auto-concurrency-before-launch branch November 7, 2026 16:18
ethan-scitix added a commit that referenced this pull request Nov 8, 2026
* feat(models)!: KV admission only with concurrency_limit auto

KV admission and `concurrency_limit: auto` are now on together or off
together. Since #277 an SGLang engine sieval run launched admitted
requests by a KV TokenBudget whenever its root set no concurrency_limit;
`auto` added the engine's load on top (#278), and #284 deferred either to
the first request for a root bound before its engine was up.

A root with no concurrency_limit is back on the path it had before #277:
no shared root limiter, no token budget, and the launch reads no
/get_server_info, so it prints no `KV:` line and its request records
carry no `kv_tokens`. A number keeps its fixed cap. `auto` is unchanged:
an AdaptiveTokenBudget from the pool the launch read, or a
PendingTokenBudget read at the first request when the engine is not up
yet, and the same refusals when no pool is reported.

No identity or plan field depended on the default: the KV facts are not
in the deployment fingerprint, and admission was throughput only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(session): share the concurrency_limit lookup

`sets_auto_concurrency` copied the "top level wins over args" rule from
`_connection_options`; if the two drifted, the launch would skip the KV
pool read while the session still saw `auto`. Both now call one helper,
and a test pins the precedence. Docs, CHANGELOG and comments trimmed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant