Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/).

## [Unreleased]

### Changed

- **One rule decides whether a context window serves a task, whoever picked it (#289).** `min_context` and a declared need count the prompt (window ≥ need); `max_tokens` counts the output only (window > `max_tokens`). An endpoint with `context_limit: 40960` now runs `max_tokens: 38000`, as a launched engine already did, instead of refusing it as needing 42096. Under 4096 tokens of prompt room is a warning (`task 't' leaves 2960 tokens for the prompt at window 40960; ...`), on `--dry-run` and at run time.
- **A served window is a declared one, never sized to a need.** A scaled group serves and is keyed by `context.max` (`<model>@<max>`) for every task subset. A need between the recipe window and `native`, including a `max_tokens` left under 4096 tokens of prompt room by the recipe window, raises the default group to `native`. A pinned window is never raised. A scaled engine keeps the recipe's tp; `--dry-run` warns on low KV room at `context.max`.
- **Resume:** a directory recorded at a need-sized window (a scaled group, or a default group raised to a `min_context` below `native`) is refused on `inference`: start fresh, or pin the recorded window with `infer.overrides`. `IDENTITY_SCHEMA` and `RESULT_SCHEMA_VERSION` are unchanged; the window is already in `identity.inference`.
- `sieval.cli.serving`: new `fits(need, window)` and `prompt_room_warnings(...)`; `check_context_fits` returns the warnings; a `max_tokens` need is the raw `max_tokens`.

## [0.15.7] - 2026-11-08

A patch release: KV admission now needs `concurrency_limit: auto` (#286).
Expand Down
6 changes: 4 additions & 2 deletions docs/guide/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -601,9 +601,11 @@ An unknown mode, or `--mode` without `--dry-run`, exits 2.
`infer_plans.yaml` would record it, each process the launch starts (`argv`,
and `env` and `cwd` when set), and `needs_inference`, whether this run starts
its engine. `gpu` is what the plans were resolved for.
- `serving` has one entry per serving group (`default`, or `<model>@<window>`):
- `serving` has one entry per serving group (`default`, or `<model>@<max>`):
its `tasks`, their served windows, the `engines` it launches and whether any
starts. The `serving` check fails on a window shortfall. See
starts. The `serving` check fails on a window shortfall and warns on a
`max_tokens` task left under 4096 tokens of prompt room (an endpoint's
warning goes to `capability_reconcile`). See
[Serving plans](infer.md#serving-plans-context-window-and-dialect). An
engine serving input scoring has `headroom`
([Input-scoring headroom](infer.md#input-scoring-headroom)).
Expand Down
57 changes: 32 additions & 25 deletions docs/guide/infer.md
Original file line number Diff line number Diff line change
Expand Up @@ -225,12 +225,18 @@ SiEval (`api_base`, `sieval infer start`) is not reported.
window from what its tasks need, before any engine starts (`--dry-run`
included).

A task needs its `min_context` if set, else the larger of what its class
A task needs its `min_context` if set, else the stricter of what its class
declares (`TaskRequirements.min_context`; RULER: `max_seq_length` plus any
unreserved thinking budget) and its effective `max_tokens` plus 4096 for the
prompt. Both `min_context` forms already count prompt and output and are used
as they are. A task whose prompts are longer (or known to be shorter) sets
`min_context`. A root model states what it can serve:
unreserved thinking budget) and its effective `max_tokens`. One rule picks a
window and checks one SiEval did not pick:

- `min_context` (either form) counts prompt and output: window ≥ need.
- `max_tokens` counts output only: window > `max_tokens`. The rest is the
prompt's room; a longer prompt is that sample's
[context overflow](metrics.md#always-present). Room under 4096 tokens is
warned about, not refused.

A root model states what it can serve:

```yaml
models:
Expand All @@ -248,32 +254,33 @@ tasks:
min_context: 38000 # optional: override the derived need
```

- Within the recipe window: nothing changes.
- Within `native`: the plan's `context_length` / `max_model_len` is raised to
the largest such need, for every task on that model. A `max_tokens` below
`native` raises it to `native` itself, even when the 4096 would not fit:
such a task is never scaled, and its prompt gets `native - max_tokens`
tokens. A longer prompt is that sample's
[context overflow](metrics.md#always-present), as before.
- Within `max`: a plan of its own with `scaling`, in serving group
`<model>@<window>`. Scaling changes short-context outputs, so no other task
shares that engine. SGLang launches it with
`SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1`, so `max` is the only bound:
SGLang no longer checks the window against the scaling. `infer.env`
overrides it.
A served window is a declared one, never sized to a need:

- The recipe window serves every task with 4096 tokens of prompt room:
nothing changes.
- Otherwise, when `native` serves the task: the plan's `context_length` /
`max_model_len` is raised to `native`, for every task on that model.
- Only `max` serves it: a plan of its own at `max`, with `scaling`, in serving
group `<model>@<max>`, the same for every subset of tasks. Scaling changes
short-context outputs, so no other task shares that engine. SGLang launches
it with `SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1`, so `max` is the only
bound; `infer.env` overrides it. The engine keeps the recipe's tp;
`--dry-run` warns when one `max` sequence leaves little KV room. Raise tp in
`infer.overrides` then.
- Within `max` with `extrapolate: true` instead of `scaling`: the same
group and variable, with no scaling parameters. SGLang only; exclusive with
`scaling`.
- Otherwise, or beyond a window pinned by `infer.overrides` / `context_limit`:
refused before launch, naming the task, its need and the model's limit. An
unknown native window refuses any raise (set `context.native`).
- Otherwise, or beyond a window pinned by `infer.overrides` / `context_limit`
(never raised): refused before launch, naming the task, what was checked and
the model's limit. An unknown native window refuses any raise (set
`context.native`).

An `api_base` model is checked against its `context_limit`; without one, an
overflow shows per sample.
An `api_base` model is checked against its `context_limit` by the same rule;
without one, an overflow shows per sample.

**Resume.** The served window is in each task's inference identity, and a
resume never reruns inference. A run recorded at a window this release now
raises is refused: start fresh, or pin the recorded window with
resume never reruns inference. A run recorded at a window this release serves
differently is refused: start fresh, or pin the recorded window with
`infer.overrides`.

Groups run one after another, each group's engines stopped before the next
Expand Down
10 changes: 9 additions & 1 deletion sieval/cli/run.py
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,7 @@
ServingGroup,
SessionServing,
plan_serving_groups,
prompt_room_warnings,
served_contexts,
served_window,
)
Expand Down Expand Up @@ -1162,7 +1163,9 @@ def _serving_json(

Each engine's ``memory`` is how each role fits its GPUs, ``None`` when
unknown; a fit with little KV room appends the warning
:func:`~sieval.infer.topology.memory.size_tp` logs to *warnings*.
:func:`~sieval.infer.topology.memory.size_tp` logs to *warnings*, and a
``max_tokens`` task left little room for its prompt the one
:func:`~sieval.cli.serving.check_context_fits` logs.
"""
models = config.get("models") or {}
engines: dict[str, dict[str, object]] = {}
Expand Down Expand Up @@ -1191,6 +1194,11 @@ def _serving_json(
where = engine_name if len(fits or ()) == 1 else f"{engine_name}.{role}"
warnings.append(f"engine {where}: {warning}")
engines[engine_name] = engine
warnings.extend(
prompt_room_warnings(
config, group.plan_dicts, frozenset(group.group.tasks), launched=True
)
)
return {
"tasks": list(group.group.tasks),
"windows": {
Expand Down
Loading
Loading