Skip to content

feat(models)!: concurrency_limit auto adapts to measured throughput - #287

Draft
ethan-scitix wants to merge 1 commit into
mainfrom
feat/auto-throughput-controller
Draft

ethan-scitix wants to merge 1 commit into
mainfrom
feat/auto-throughput-controller

Conversation

@ethan-scitix

@ethan-scitix ethan-scitix commented Nov 8, 2026 •

Copy link
Copy Markdown
Collaborator

Type

  • feature — new benchmark, task, or capability

Summary

Follows #286 (merged as b37cb62). 0.15.7 ships both after the live ablation below; draft until then.

  • Why. auto used to gate on KV and engine queueing. Both are the wrong signal. A saturated engine should queue, so waiting > 0 stopped admission exactly when the engine was full. KV usage stays low under DP attention, and for MoE total throughput peaks and then falls while KV looks healthy. The static n × min(context, max_tokens + 4096) reservation ignores the prompt.
  • What. concurrency_limit: auto is now AdaptiveLimit: a FIFO bound on sequences in flight (n: 8 counts 8). ThroughputController moves the bound on measured token throughput. Start 32. Each 5 s window counts only while requests wait at the limit; one window settles, two are averaged. Slow-start doubling, then quarter steps while throughput gains ≥5%. A plateau or a fall goes back to the best limit and holds for 6 measurements, then probes up or down (−⅕); a lower limit within 5% is kept. Retractions, timeouts and 5xx halve the limit and cap the climb below where they happened until the next probe. Floor 8; ceiling 2 × Σ max_running_requests (a coarse KV bound), else 1024; a KV pool ≥95% full stops a climb.
  • Signals (checked against SGLang v0.5.13.post1 source). /metrics: sglang:realtime_tokens_total{mode=prefill_compute|decode} summed over ranks, which counts every scheduler step and leaves out cache hits; prompt_tokens_total + generation_tokens_total as a fallback (counted per finished request); num_retracted_requests_total. /v1/loads?include=core: token_usage (max over ranks) and max_running_requests (sum). /v1/loads has no counters (its gen_throughput is a gauge updated only under --enable-metrics), so sieval run appends --enable-metrics to an auto root's SGLang argv. The flag is serving only and is visible in the dry-run launch.argv; no plan or identity field changes. With no counters it falls back to the usage of answered requests (source: client).
  • Safety. A probe that raises, returns nothing or hits a counter reset is a window with "no reading": nothing changes and no request fails. The probe has a 2 s timeout and is read by one waiter at a time. Clock and probe are injectable, and the controller is a pure windows→decisions function.
  • Observability. On each change the run log prints auto <root>: limit 64 -> 128 (rising, 21,350 tok/s), plus a closing summary. Request records carry in_flight.auto = {borrowed, total, decision, source}, where decision is the controller's last reason. kv / both add kv_tokens as before.
  • Ablation knobs. concurrency_limit: {auto: {signal: throughput|kv|both, start, min, max, kv_ceiling, retraction_backoff, interval}}. signal: kv is exactly 0.15.6's AdaptiveTokenBudget (KV reservation + queue/KV-full hold). both = controller + plain KV reservation. Plain auto = recommended defaults. I chose the nested auto: mapping so concurrency_limit keeps one key that is a number, auto, or auto with options. Unknown or invalid options are refused before launch.
  • Plumbing. ConnectionPool takes an AdmissionGate (admit(n, max_tokens), completed(usage), failed(exc), record_limiters()). PendingTokenBudget → PendingGate. The default auto no longer needs /get_server_info; the KV: launch line still prints.
  • Throughput only: identity schema and RESULT_SCHEMA_VERSION unchanged; resumes across any auto option or a number.

Breaking change

Plain auto changes behaviour (KV → throughput). To keep 0.15.6's gate set concurrency_limit: {auto: {signal: kv}}. Records under default auto show in_flight.auto instead of kv_tokens.

Related Issues

Refs #277, #278, #284, #286

Ablation plan: arck-general-1k, 4×H100, SGLang self-hosted

One engine config for every run (same recipe, same dp/tp, same seed). Run them in sequence on the same node, each a fresh result dir (no resume), and start every run with a cold prefix cache (fresh engine).

Run concurrency_limit
F16 16
F64 64
F256 256
A-tput auto
A-tput+kv {auto: {signal: both}}
A-kv (0.15.6) {auto: {signal: kv}}

Optional component runs if A-tput wins: {auto: {retraction_backoff: false}}, {auto: {kv_ceiling: false}}.

Record for each run:

  • total tok/s: (Σ prompt computed + Σ generated) / wall, from sglang:realtime_tokens_total deltas (scrape /metrics at start and end), cross-checked against requests.jsonl usage. The F* runs must set infer.overrides: {enable_metrics: true} on the subject so their /metrics exists. Checked by dry run and a stub run: the override reaches launch.argv as --enable-metrics, and on an auto root the flag still appears once. It lands in the plan's engine_params, but identity reads only the numerics and parser keys there, so no stage identity changes. Set it on the auto runs too, so all six plans match;
  • wall time of the infer phase (profile.json) and end to end;
  • timeouts / 5xx: meta.json runs[].requests outcomes;
  • preemptions/retractions: sglang:num_retracted_requests_total delta;
  • KV peak: max token_usage from a 1 s /v1/loads scrape;
  • limit trajectory: the auto … limit a -> b (reason, tok/s) log lines and in_flight.auto.total over offset_s in requests.jsonl;
  • the score, as a sanity check that every run graded the same samples.

For auto (throughput) to become the default it has to show:

  1. total tok/s ≥ the best fixed cap minus 5%, and > A-kv by more than run-to-run noise (repeat A-tput and the best fixed cap twice to measure that noise);
  2. no more timeouts/5xx than the best fixed cap, and retractions no higher than F64;
  3. a trajectory that settles: after the first ~2 min it stays within ±25% of one value and does not ratchet to the floor or the ceiling;
  4. on the 3 long-output math lanes (n=8, decode-bound), where every gate gave about 3200 tok/s, no regression beyond 5%.

If A-tput+kv beats A-tput, make both the default. If neither beats the best fixed cap, keep auto opt-in and document the measured cap.

Test Plan

Automated

  • Lint/format clean (ruff check . && ruff format --check .; pre-commit hooks on the commit)
  • Type check: ty check shows the same 26 unresolved-import diagnostics as feat(models)!: KV admission only with concurrency_limit auto #286 (optional task extras not installed); no others
  • Unit tests: pytest tests/unit -n 8 fails the same 190 tests on feat(models)!: KV admission only with concurrency_limit auto #286's head and on this branch, an identical set (diffed); all need uninstalled extras. 10387 passed.
  • Integration: pytest tests/integration -n 8: 185 passed, 19 skipped
  • sieval/core/models/admission.py coverage 99% (coverage run over its unit + integration tests)
  • New tests:
    • TestThroughputController on synthetic curves: rising (climbs to the ceiling), plateau (settles at the knee, 64), MoE peak-then-fall (overshoots once, stays at 80–128 around the 96 peak, ≥95% of peak tok/s), retraction spike (halves, caps below the spike), retraction backoff off, timeouts/5xx, floor, unreadable windows (no reading / below limit / draining), a probe gap restarting the measurement, KV ceiling on/off, determinism, engine max
    • TestAdaptiveLimit with an injected clock and probe: the limit moves and every change logs one line; an always-failing probe moves nothing and fails no request; an intermittent probe still climbs; engine retractions back off; client-side usage fallback; the engine bound caps the limit; overload classification
    • TestCombinedGate, TestPoolReportsToTheGate (failures reach the gate; the pool releases on error)
    • TestSglangThroughputProbe against a 0.5.13-shaped /metrics (mode filter, rank sums, fallback, no metrics, bad loads, no answer); TestWithEngineMetrics
    • TestAutoConcurrencyOptions and session tests: auto = defaults, nested options, refusals (unknown option, bad signal, min > max, extra keys)
    • Integration through sieval run with the stub engines: auto admits by throughput and reads no pool for the gate, --enable-metrics only on the auto root, records carry in_flight.auto with decision, the probe is closed; kv/both read the pool once up; a barrier-started judge; auto on an engine that reports no pool now runs; dry run accepts all three forms

Manual

  • Live ablation above on 4×H100 before 0.15.7

Checklist

Required (all PRs)

  • PR title follows conventional format (type(scope): description)
  • No internal paths, credentials, or personal info in committed files
  • AI-generated code has AI-Generated Code - <model> (<provider>) in module docstring
  • No new upper-layer dependencies added to core/ (core.utils, core.types only; request_log imported lazily to avoid a cycle)
  • Deleted code verified: PendingTokenBudget, _kv_token_budget, _pending_kv_token_budget, pool.token_budget have no remaining call sites

If: Breaking Change

  • Described what breaks and migration path in Summary
  • Existing tests updated to reflect new behavior

🤖 Generated with Claude Code

`concurrency_limit: auto` now bounds the sequences in flight and moves the
bound by the token throughput the engine reports, instead of reserving KV
per request and holding requests while the engine queues. Queueing means
the engine is saturated; KV usage stays low under DP attention and does not
show MoE throughput falling past its peak.

ThroughputController is a hill climb over 5 s windows: slow-start doubling,
then quarter steps while throughput gains 5%; back to the best limit on a
plateau or fall; periodic up/down probes; halving on retractions, timeouts
or 5xx. Tokens come from SGLang's /metrics (realtime_tokens_total,
prefill_compute + decode), with retractions from
num_retracted_requests_total and the ceiling from /v1/loads
max_running_requests; sieval run adds --enable-metrics to an auto root's
launch. Without counters it counts answered usage. A failed read changes
nothing and fails no request.

`concurrency_limit: {auto: {signal: kv | both, ...}}` keeps the 0.15.6 gate
or stacks it, and start/min/max/kv_ceiling/retraction_backoff/interval tune
the controller for ablation. Request records show the limit and last
decision under in_flight.auto; the run log prints each change.

Throughput only: identity and RESULT_SCHEMA_VERSION unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ethan-scitix
ethan-scitix force-pushed the feat/auto-throughput-controller branch from ca8b7f6 to 6bdc8b3 Compare November 8, 2026 05:52
@ethan-scitix
ethan-scitix changed the base branch from feat/kv-admission-with-auto to main November 8, 2026 05:52
@ethan-scitix

Copy link
Copy Markdown
Collaborator Author

Live ablation (2026-11-08)

Mixed slice of arck-general-1k (546 requests, 912 sequences, ~11M output tokens: mmlu_pro, gpqa, livecodebench, aime, hmmt, livebench data_analysis, ifeval), 4× H100 per run, identical engine settings (tp 4, context 44096 YaRN, mem 0.9, --enable-metrics), image built from this branch at 6bdc8b3. Useful tok/s = generated tokens of answered requests / wall time.

Run Wall (s) Useful tok/s Timeouts Retracted Limit trajectory
fixed 64 3063 3603 0 236 fixed
fixed 256 (×2) 2463 / 2499 4361 / 4389 18 / 14 1152 / 1036 fixed
auto throughput (×2) 2846 / 2680 3974 / 4057 2 / 0 456 / 444 32→256→128 (≈15 min)→312
auto throughput, 15 s windows 2827 3908 0 516 ended at 200
auto both 3300 (cap) ≈2350 0 0 42↔65, 33 changes
auto kv (= 0.15.6 KV gate) 3300 (cap) ≈2470 0 0 reservation held ~24 running

Run-to-run noise ≈1–2%; scores agree across complete runs.

Verdict: stays draft. The throughput controller beats 0.15.6's KV gate by 60–80% but is ~8% below a fixed cap of 256 and does not settle. Causes found:

  1. Workload drift confounds window comparisons: at a fixed limit throughput falls ~5× within 15 min as contexts grow, so comparing a window now against the best limit's window minutes ago mostly measures time (the 256→128 back-off compared minute 1 with minute 3).
  2. Retraction backoff never fired: it is only evaluated while requests wait at the limit; once demand drops below it nothing waits, though the engine retracted 100+ requests per 2 min at 312.
  3. The default ceiling (2 × max_running_requests = 8192 here) is not a ceiling.
  4. Many request records show no reading late in the run; cause unknown.

Directions the data favour: a high cap with SGLang's scheduler absorbing overcommit, or a controller driven by KV usage + retraction rate read on a timer, instead of a hill climb on windowed throughput. 0.15.7 ships #286 alone; auto stays opt-in.

Artifacts: /ai4s-data/sieval/ablation/pr287/ on aries (per-run summary.json, scrape.jsonl, run.log, configs, scripts); image SiFlow 208602.

@ethan-scitix ethan-scitix mentioned this pull request Nov 8, 2026
6 tasks done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant