Skip to content

feat(models)!: KV admission only with concurrency_limit auto - #286

Merged
ethan-scitix merged 2 commits into
mainfrom
feat/kv-admission-with-auto
Nov 8, 2026
Merged

ethan-scitix merged 2 commits into
mainfrom
feat/kv-admission-with-auto

Conversation

@ethan-scitix

Copy link
Copy Markdown
Collaborator

Type

  • feature — new benchmark, task, or capability

Summary

  • KV admission and concurrency_limit: auto are now on together or off together (maintainer decision). Since feat(models): admit requests to a launched engine by the KV they may hold #277 an SGLang engine sieval run launches admitted requests by a KV TokenBudget whenever its root set no concurrency_limit.
  • A root with no concurrency_limit is back on the pre-feat(models): admit requests to a launched engine by the KV they may hold #277 path: no root limiter, no token budget, and _launch returns deployer.build_deployment(...) without reading /get_server_info. So no KV: launch line and no kv_tokens in request records. A number keeps its fixed cap.
  • auto is unchanged: an AdaptiveTokenBudget from the pool the launch read, or a PendingTokenBudget read at the first request when the engine is not up yet (fix(models): concurrency_limit auto is accepted before its engine is up #284), and the same refusals when no pool is reported. The KV: line prints only for auto roots.
  • No identity or plan field depended on the default. The KV facts are not in deployment_fingerprint, and admission was throughput only. RESULT_SCHEMA_VERSION and identity schema are unchanged.
  • docs/guide/concurrency.md: the KV Admission section is now part of concurrency_limit: auto. CHANGELOG has an Unreleased "Changed" entry.

Breaking change

A root with no concurrency_limit on a launched SGLang engine loses the KV admission 0.15.5/0.15.6 gave it. To keep it, set concurrency_limit: auto.

Related Issues

Refs #277, #278, #284

Test Plan

Automated

  • Lint/format clean (ruff check && ruff format --check, plus pre-commit on the changed files)
  • Type check: ty check reports the same 26 unresolved-import diagnostics on origin/main and on this branch. All of them come from optional task extras (nltk, sympy, ...) that this venv does not have.
  • Unit tests: pytest tests/unit -n 8 fails the same 190 tests on origin/main and on this branch, and the failing set is identical (diffed). All of them need task extras that are not installed. No new failures.
  • Integration: pytest tests/integration -n 8 gives 179 passed, 19 skipped.
  • New and updated tests:
    • TestAutoConcurrency.test_no_auto_gets_no_kv_admission (no limit / a number: no budget even when the engine reported its pool)
    • test_auto_builds_the_kv_budget
    • test_sets_auto_concurrency_reads_both_spellings
    • integration test_no_limit_gets_no_kv_admission and test_a_judge_with_no_limit_started_at_a_barrier_gets_no_kv_admission (no /get_server_info read, no reservation); the existing auto run test now also asserts that only the auto root reads the pool

Checklist

Required (all PRs)

  • PR title follows conventional format (type(scope): description)
  • No internal paths, credentials, or personal info in committed files
  • AI-generated code has AI-Generated Code - <model> (<provider>) in module docstring
  • No new upper-layer dependencies added to core/
  • Deleted code verified — no remaining call sites depend on it

If: Breaking Change

  • Described what breaks and migration path in Summary
  • Existing tests updated to reflect new behavior

🤖 Generated with Claude Code

ethan-scitix and others added 2 commits November 8, 2026 12:07
KV admission and `concurrency_limit: auto` are now on together or off
together. Since #277 an SGLang engine sieval run launched admitted
requests by a KV TokenBudget whenever its root set no concurrency_limit;
`auto` added the engine's load on top (#278), and #284 deferred either to
the first request for a root bound before its engine was up.

A root with no concurrency_limit is back on the path it had before #277:
no shared root limiter, no token budget, and the launch reads no
/get_server_info, so it prints no `KV:` line and its request records
carry no `kv_tokens`. A number keeps its fixed cap. `auto` is unchanged:
an AdaptiveTokenBudget from the pool the launch read, or a
PendingTokenBudget read at the first request when the engine is not up
yet, and the same refusals when no pool is reported.

No identity or plan field depended on the default: the KV facts are not
in the deployment fingerprint, and admission was throughput only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`sets_auto_concurrency` copied the "top level wins over args" rule from
`_connection_options`; if the two drifted, the launch would skip the KV
pool read while the session still saw `auto`. Both now call one helper,
and a test pins the precedence. Docs, CHANGELOG and comments trimmed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ethan-scitix
ethan-scitix merged commit b37cb62 into main Nov 8, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant