Skip to content

fix(serving)!: one predicate chooses and checks a context window - #290

Merged
ethan-scitix merged 2 commits into
mainfrom
fix/one-window-predicate
Nov 10, 2026
Merged

ethan-scitix merged 2 commits into
mainfrom
fix/one-window-predicate

Conversation

@ethan-scitix

@ethan-scitix ethan-scitix commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator

Type

  • fix — bug fix or alignment correction

Summary

  • One predicate, fits(need, window), now both chooses and checks a window. min_context and a task's declared need count the prompt, so need <= window. A max_tokens need counts only the output, so max_tokens < window: at least one token is left for the prompt. The [Bug]: One context window, three verdicts: prompt headroom refuses an endpoint a launched engine serves #289 table now gives one verdict for max_tokens: 38000 at 40960. A launched engine at native and an endpoint with context_limit: 40960 both run the task and both warn task 't' leaves 2960 tokens for the prompt at window 40960; longer prompts overflow per sample. An endpoint without context_limit runs it unchecked, as before.
  • Choosing a window only picks a declared one. The recipe window stays when every task fits it with at least PROMPT_HEADROOM (4096) tokens of prompt room. Otherwise the window is native, then context.max for the scaled group, keyed <model>@<max> for every task subset. A pinned window (infer.overrides, context_limit) is never raised, and an unknown native window behaves as before.
  • Short prompt room still raises to native. If the recipe window is below native and leaves a max_tokens task under 4096 tokens of prompt room, the default group is raised to native, as before [Bug]: One context window, three verdicts: prompt headroom refuses an endpoint a launched engine serves #289. Keeping the recipe window would have shrunk these runs' window (e.g. max_tokens: 30000 at recipe 32768 / native 40960) and refused their resume. If every task has room, the recipe window stays, as test_tasks_that_fit_share_the_unchanged_recipe_plan requires.
  • Checking: check_context_fits refuses only a task that does not fit. It returns the prompt-room warnings: max_tokens tasks left under PROMPT_HEADROOM (4096) tokens of room. EvalSession logs them. sieval run --dry-run shows a launched engine's warnings on the serving check (the channel the low-KV warning uses) and an endpoint's on capability_reconcile. Each warning appears once.
  • Shortfall messages state what was checked, e.g. max_tokens 40960 leaves no room for the prompt at window 40960 or needs 40961 tokens (its min_context). _native_serves, _required and _fits are gone, replaced by fits.
  • Memory: a scaled group's with_window, reserve_scoring_headroom and _longest_prompt now see context.max. _longest_prompt ignores a max_tokens need: scoring calls send their own max_tokens, so a generation one from the model's args must not shrink the logits reserve. The scaled engine keeps the recipe's tp. Nothing re-runs size_tp for a serving group, either before this change or after it, so an explicit tp is untouched. The issue's "can raise the auto-chosen tp" does not happen. Instead, --dry-run reports the engine's memory at context.max and gives the low-KV warning, and a user raises tp in infer.overrides. The new test pins this behaviour.
  • RESULT_SCHEMA_VERSION stays 1 and IDENTITY_SCHEMA stays 5. No result directory layout changes. At a given window, no served token changes: the default group's rule and the scaling parameters are the same. A window that does change is in identity.inference.context_limit, which already refuses a resume, so the "nothing else catches it" condition in AGENTS.md does not apply.

Breaking change

A result directory recorded at a need-sized window refuses resume on inference and needs inference again. That covers a scaled group (e.g. m@44096, now m@131072) and a default group raised to a min_context between the recipe window and native. Migration: start fresh, or pin the recorded window with infer.overrides. The maintainer accepted this in #289. On small GPUs, a scaled engine at context.max may need a higher tp in infer.overrides; --dry-run warns about it.

Changed public API in sieval.cli.serving:

  • check_context_fits(...) returns list[str] (warnings); it used to return None.
  • task_context_needs(...): a max_tokens need is the raw max_tokens, without + PROMPT_HEADROOM. When both a declared need and max_tokens exist, the stricter one under fits wins.
  • New: fits(need, window) and prompt_room_warnings(config, infer_plans, tasks, *, launched: bool).
  • plan_serving_groups(...): the scaled group's key and window are context.max. The default group is raised to native (also on short prompt room), never to a need.

Related Issues

Closes #289

Test Plan

Automated

  • Lint/format clean (ruff check && ruff format --check)
  • Type check clean (ty check or mypy --strict)
  • Unit tests pass (pdm run pytest)
$ pdm run ruff check && pdm run ruff format --check
All checks passed!
704 files already formatted
$ pdm run ty check
All checks passed!
$ pdm run python scripts/sync_package_stubs.py --check   # clean
$ pdm run pre-commit run -a                              # all hooks Passed
$ pdm run python -m pytest tests/unit tests/integration -q -n auto --cov --cov-fail-under=95 -m "not stress and not benchmark"
1 failed, 11190 passed, 84 skipped in 143.81s
Required test coverage of 95% reached. Total coverage: 97.19%

After the review fixes (ef4107b), on Linux:

$ pytest tests/unit tests/integration -q -n 16 -m "not stress and not benchmark" --cov --cov-fail-under=95
11325 passed, 1 skipped in 95.83s
Required test coverage of 95% reached. Total coverage: 97.19%

The one failure is tests/unit/vendor/code_evaluator/test_exec_quotebench_anchor.py::test_replaying_stored_replies_reproduces_upstreams_gnu_verdicts. It also fails on main (41b47e1) on the macOS machine I ran on: it replays GNU verdicts. This PR does not touch it.

New and rewritten tests:

  • tests/unit/cli/test_serving.py
    • test_context_limit_refuses_only_max_tokens_with_no_prompt_room: at context_limit: 32768, 28672 passes without a warning, 28673 warns (4095 room), 32767 warns (1 room), 32768 is refused.
    • test_min_context_is_unchanged_and_never_warns: 32768 fits, 32769 is refused.
    • test_one_window_one_verdict_whoever_serves_it: the [Bug]: One context window, three verdicts: prompt headroom refuses an endpoint a launched engine serves #289 table, all three rows.
    • test_scaled_window_is_the_same_for_every_task_subset: needs 44096 and 42096 both give m@131072.
    • test_max_tokens_at_or_above_native_scales_to_max, test_min_context_is_a_hard_need_served_at_a_declared_window, test_pinned_window_is_never_raised, test_unknown_native_checks_the_recipe_window.
    • test_recipe_window_short_of_prompt_room_is_raised_to_native: at recipe 32768 / native 40960, 28672 stays at 32768; 28673 and 32767 are raised to 40960.
    • test_a_max_tokens_does_not_bound_the_scored_prompt: a ppl task beside a gen task keeps the 3072-token prefill reserve whatever max_tokens the model's args set.
    • test_warnings_split_by_who_serves_the_model.
  • tests/unit/cli/test_run.py
    • test_a_scaled_engine_is_checked_at_context_max, for sized tp and stated tp: the scaled engine launches at 131072 with the recipe's tp, its memory is reported at 131072, and the low-KV warning names m@131072.
    • test_little_prompt_room_warns_on_the_serving_check.
  • tests/unit/cli/test_validation.py::test_an_endpoint_with_little_prompt_room_warns (capability_reconcile warnings).

Checklist

Required (all PRs)

  • PR title follows conventional format (type(scope): description)
  • No internal paths, credentials, or personal info in committed files
  • AI-generated code has AI-Generated Code - <model> (<provider>) in module docstring
  • No new upper-layer dependencies added to core/
  • Deleted code verified — no remaining call sites depend on it (_native_serves, _required, _fits were private to serving.py)

If: Breaking Change

  • Described what breaks and migration path in Summary
  • Existing tests updated to reflect new behavior

🤖 Generated with Claude Code

ethan-scitix and others added 2 commits November 10, 2026 17:07
A window serves a task when fits(need, window): min_context and a declared
need count the prompt (need <= window); a max_tokens need leaves at least
one token for it (max_tokens < window). The same rule picks the window
sieval run launches and checks an endpoint's context_limit, so a launched
engine at native 40960 and an endpoint declaring 40960 now give one verdict
on max_tokens 38000: run, with a warning that the prompt has 2960 tokens.

Windows chosen are declared ones: the recipe window when it fits every
task, else native, else context.max for the scaled group, keyed
<model>@<max> for every task subset. PROMPT_HEADROOM is the warning
threshold only.

BREAKING CHANGE: a result directory recorded at a need-sized window (a
scaled group, or a default group raised to a min_context below native)
differs in identity.inference and refuses resume; it needs inference again.

Closes #289

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ort prompt room

Scoring calls send their own max_tokens, so a generation max_tokens from
the model's args no longer bounds the longest scored prompt; it had cut
the logits reserve (3072 to 1024 prefill tokens at 32768).

A recipe window below native that leaves a max_tokens task under
PROMPT_HEADROOM of prompt room is raised to native, as before #289; only
a pinned window stays and warns. prompt_room_warnings takes launched as a
required bool. Docs, CHANGELOG and docstrings trimmed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ethan-scitix
ethan-scitix merged commit 7fa099a into main Nov 10, 2026
9 checks passed
@ethan-scitix
ethan-scitix deleted the fix/one-window-predicate branch November 10, 2026 09:46
@ethan-scitix ethan-scitix mentioned this pull request Nov 10, 2026
4 of 5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: One context window, three verdicts: prompt headroom refuses an endpoint a launched engine serves

1 participant