Repository navigation
feat(models)!: concurrency_limit auto adapts to measured throughput - #287
ethan-scitix wants to merge 1 commit into
Conversation
`concurrency_limit: auto` now bounds the sequences in flight and moves the
bound by the token throughput the engine reports, instead of reserving KV
per request and holding requests while the engine queues. Queueing means
the engine is saturated; KV usage stays low under DP attention and does not
show MoE throughput falling past its peak.
ThroughputController is a hill climb over 5 s windows: slow-start doubling,
then quarter steps while throughput gains 5%; back to the best limit on a
plateau or fall; periodic up/down probes; halving on retractions, timeouts
or 5xx. Tokens come from SGLang's /metrics (realtime_tokens_total,
prefill_compute + decode), with retractions from
num_retracted_requests_total and the ceiling from /v1/loads
max_running_requests; sieval run adds --enable-metrics to an auto root's
launch. Without counters it counts answered usage. A failed read changes
nothing and fails no request.
`concurrency_limit: {auto: {signal: kv | both, ...}}` keeps the 0.15.6 gate
or stacks it, and start/min/max/kv_ceiling/retraction_backoff/interval tune
the controller for ablation. Request records show the limit and last
decision under in_flight.auto; the run log prints each change.
Throughput only: identity and RESULT_SCHEMA_VERSION unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ca8b7f6 to
6bdc8b3
Compare
Live ablation (2026-11-08)Mixed slice of arck-general-1k (546 requests, 912 sequences, ~11M output tokens: mmlu_pro, gpqa, livecodebench, aime, hmmt, livebench data_analysis, ifeval), 4× H100 per run, identical engine settings (tp 4, context 44096 YaRN, mem 0.9,
Run-to-run noise ≈1–2%; scores agree across complete runs. Verdict: stays draft. The throughput controller beats 0.15.6's KV gate by 60–80% but is ~8% below a fixed cap of 256 and does not settle. Causes found:
Directions the data favour: a high cap with SGLang's scheduler absorbing overcommit, or a controller driven by KV usage + retraction rate read on a timer, instead of a hill climb on windowed throughput. 0.15.7 ships #286 alone; Artifacts: |
Type
Summary
Follows #286 (merged as b37cb62). 0.15.7 ships both after the live ablation below; draft until then.
autoused to gate on KV and engine queueing. Both are the wrong signal. A saturated engine should queue, sowaiting > 0stopped admission exactly when the engine was full. KV usage stays low under DP attention, and for MoE total throughput peaks and then falls while KV looks healthy. The staticn × min(context, max_tokens + 4096)reservation ignores the prompt.concurrency_limit: autois nowAdaptiveLimit: a FIFO bound on sequences in flight (n: 8counts 8).ThroughputControllermoves the bound on measured token throughput. Start 32. Each 5 s window counts only while requests wait at the limit; one window settles, two are averaged. Slow-start doubling, then quarter steps while throughput gains ≥5%. A plateau or a fall goes back to the best limit and holds for 6 measurements, then probes up or down (−⅕); a lower limit within 5% is kept. Retractions, timeouts and 5xx halve the limit and cap the climb below where they happened until the next probe. Floor 8; ceiling2 × Σ max_running_requests(a coarse KV bound), else 1024; a KV pool ≥95% full stops a climb./metrics:sglang:realtime_tokens_total{mode=prefill_compute|decode}summed over ranks, which counts every scheduler step and leaves out cache hits;prompt_tokens_total + generation_tokens_totalas a fallback (counted per finished request);num_retracted_requests_total./v1/loads?include=core:token_usage(max over ranks) andmax_running_requests(sum)./v1/loadshas no counters (itsgen_throughputis a gauge updated only under--enable-metrics), sosieval runappends--enable-metricsto anautoroot's SGLang argv. The flag is serving only and is visible in the dry-runlaunch.argv; no plan or identity field changes. With no counters it falls back to the usage of answered requests (source: client).auto <root>: limit 64 -> 128 (rising, 21,350 tok/s), plus a closing summary. Request records carryin_flight.auto = {borrowed, total, decision, source}, wheredecisionis the controller's last reason.kv/bothaddkv_tokensas before.concurrency_limit: {auto: {signal: throughput|kv|both, start, min, max, kv_ceiling, retraction_backoff, interval}}.signal: kvis exactly 0.15.6'sAdaptiveTokenBudget(KV reservation + queue/KV-full hold).both= controller + plain KV reservation. Plainauto= recommended defaults. I chose the nestedauto:mapping soconcurrency_limitkeeps one key that is a number,auto, orautowith options. Unknown or invalid options are refused before launch.ConnectionPooltakes anAdmissionGate(admit(n, max_tokens),completed(usage),failed(exc),record_limiters()).PendingTokenBudget→PendingGate. The defaultautono longer needs/get_server_info; theKV:launch line still prints.RESULT_SCHEMA_VERSIONunchanged; resumes across anyautooption or a number.Breaking change
Plain
autochanges behaviour (KV → throughput). To keep 0.15.6's gate setconcurrency_limit: {auto: {signal: kv}}. Records under defaultautoshowin_flight.autoinstead ofkv_tokens.Related Issues
Refs #277, #278, #284, #286
Ablation plan: arck-general-1k, 4×H100, SGLang self-hosted
One engine config for every run (same recipe, same
dp/tp, same seed). Run them in sequence on the same node, each a fresh result dir (no resume), and start every run with a cold prefix cache (fresh engine).concurrency_limit1664256auto{auto: {signal: both}}{auto: {signal: kv}}Optional component runs if A-tput wins:
{auto: {retraction_backoff: false}},{auto: {kv_ceiling: false}}.Record for each run:
sglang:realtime_tokens_totaldeltas (scrape/metricsat start and end), cross-checked againstrequests.jsonlusage. The F* runs must setinfer.overrides: {enable_metrics: true}on the subject so their/metricsexists. Checked by dry run and a stub run: the override reacheslaunch.argvas--enable-metrics, and on anautoroot the flag still appears once. It lands in the plan'sengine_params, but identity reads only the numerics and parser keys there, so no stage identity changes. Set it on theautoruns too, so all six plans match;profile.json) and end to end;meta.jsonruns[].requestsoutcomes;sglang:num_retracted_requests_totaldelta;token_usagefrom a 1 s/v1/loadsscrape;auto … limit a -> b (reason, tok/s)log lines andin_flight.auto.totaloveroffset_sinrequests.jsonl;For
auto(throughput) to become the default it has to show:If A-tput+kv beats A-tput, make
boththe default. If neither beats the best fixed cap, keepautoopt-in and document the measured cap.Test Plan
Automated
ruff check . && ruff format --check .; pre-commit hooks on the commit)ty checkshows the same 26unresolved-importdiagnostics as feat(models)!: KV admission only with concurrency_limit auto #286 (optional task extras not installed); no otherspytest tests/unit -n 8fails the same 190 tests on feat(models)!: KV admission only with concurrency_limit auto #286's head and on this branch, an identical set (diffed); all need uninstalled extras. 10387 passed.pytest tests/integration -n 8: 185 passed, 19 skippedsieval/core/models/admission.pycoverage 99% (coverage runover its unit + integration tests)TestThroughputControlleron synthetic curves: rising (climbs to the ceiling), plateau (settles at the knee, 64), MoE peak-then-fall (overshoots once, stays at 80–128 around the 96 peak, ≥95% of peak tok/s), retraction spike (halves, caps below the spike), retraction backoff off, timeouts/5xx, floor, unreadable windows (no reading / below limit / draining), a probe gap restarting the measurement, KV ceiling on/off, determinism, engine maxTestAdaptiveLimitwith an injected clock and probe: the limit moves and every change logs one line; an always-failing probe moves nothing and fails no request; an intermittent probe still climbs; engine retractions back off; client-side usage fallback; the engine bound caps the limit; overload classificationTestCombinedGate,TestPoolReportsToTheGate(failures reach the gate; the pool releases on error)TestSglangThroughputProbeagainst a 0.5.13-shaped/metrics(mode filter, rank sums, fallback, no metrics, bad loads, no answer);TestWithEngineMetricsTestAutoConcurrencyOptionsand session tests:auto= defaults, nested options, refusals (unknown option, bad signal,min > max, extra keys)sieval runwith the stub engines:autoadmits by throughput and reads no pool for the gate,--enable-metricsonly on theautoroot, records carryin_flight.autowithdecision, the probe is closed;kv/bothread the pool once up; a barrier-started judge;autoon an engine that reports no pool now runs; dry run accepts all three formsManual
Checklist
Required (all PRs)
type(scope): description)AI-Generated Code - <model> (<provider>)in module docstringcore/(core.utils,core.typesonly;request_logimported lazily to avoid a cycle)PendingTokenBudget,_kv_token_budget,_pending_kv_token_budget,pool.token_budgethave no remaining call sitesIf: Breaking Change
🤖 Generated with Claude Code