Repository navigation
fix(serving)!: one predicate chooses and checks a context window - #290
Merged
Merged
Conversation
A window serves a task when fits(need, window): min_context and a declared need count the prompt (need <= window); a max_tokens need leaves at least one token for it (max_tokens < window). The same rule picks the window sieval run launches and checks an endpoint's context_limit, so a launched engine at native 40960 and an endpoint declaring 40960 now give one verdict on max_tokens 38000: run, with a warning that the prompt has 2960 tokens. Windows chosen are declared ones: the recipe window when it fits every task, else native, else context.max for the scaled group, keyed <model>@<max> for every task subset. PROMPT_HEADROOM is the warning threshold only. BREAKING CHANGE: a result directory recorded at a need-sized window (a scaled group, or a default group raised to a min_context below native) differs in identity.inference and refuses resume; it needs inference again. Closes #289 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ort prompt room Scoring calls send their own max_tokens, so a generation max_tokens from the model's args no longer bounds the longest scored prompt; it had cut the logits reserve (3072 to 1024 prefill tokens at 32768). A recipe window below native that leaves a max_tokens task under PROMPT_HEADROOM of prompt room is raised to native, as before #289; only a pinned window stays and warns. prompt_room_warnings takes launched as a required bool. Docs, CHANGELOG and docstrings trimmed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Merged
4 of 5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Type
Summary
fits(need, window), now both chooses and checks a window.min_contextand a task's declared need count the prompt, soneed <= window. Amax_tokensneed counts only the output, somax_tokens < window: at least one token is left for the prompt. The [Bug]: One context window, three verdicts: prompt headroom refuses an endpoint a launched engine serves #289 table now gives one verdict formax_tokens: 38000at 40960. A launched engine at native and an endpoint withcontext_limit: 40960both run the task and both warntask 't' leaves 2960 tokens for the prompt at window 40960; longer prompts overflow per sample. An endpoint withoutcontext_limitruns it unchecked, as before.PROMPT_HEADROOM(4096) tokens of prompt room. Otherwise the window is native, thencontext.maxfor the scaled group, keyed<model>@<max>for every task subset. A pinned window (infer.overrides,context_limit) is never raised, and an unknown native window behaves as before.max_tokenstask under 4096 tokens of prompt room, the default group is raised to native, as before [Bug]: One context window, three verdicts: prompt headroom refuses an endpoint a launched engine serves #289. Keeping the recipe window would have shrunk these runs' window (e.g.max_tokens: 30000at recipe 32768 / native 40960) and refused their resume. If every task has room, the recipe window stays, astest_tasks_that_fit_share_the_unchanged_recipe_planrequires.check_context_fitsrefuses only a task that does not fit. It returns the prompt-room warnings:max_tokenstasks left underPROMPT_HEADROOM(4096) tokens of room.EvalSessionlogs them.sieval run --dry-runshows a launched engine's warnings on theservingcheck (the channel the low-KV warning uses) and an endpoint's oncapability_reconcile. Each warning appears once.max_tokens 40960 leaves no room for the prompt at window 40960orneeds 40961 tokens (its min_context)._native_serves,_requiredand_fitsare gone, replaced byfits.with_window,reserve_scoring_headroomand_longest_promptnow seecontext.max._longest_promptignores amax_tokensneed: scoring calls send their ownmax_tokens, so a generation one from the model'sargsmust not shrink the logits reserve. The scaled engine keeps the recipe's tp. Nothing re-runssize_tpfor a serving group, either before this change or after it, so an explicit tp is untouched. The issue's "can raise the auto-chosen tp" does not happen. Instead,--dry-runreports the engine's memory atcontext.maxand gives the low-KV warning, and a user raises tp ininfer.overrides. The new test pins this behaviour.RESULT_SCHEMA_VERSIONstays 1 andIDENTITY_SCHEMAstays 5. No result directory layout changes. At a given window, no served token changes: the default group's rule and the scaling parameters are the same. A window that does change is inidentity.inference.context_limit, which already refuses a resume, so the "nothing else catches it" condition in AGENTS.md does not apply.Breaking change
A result directory recorded at a need-sized window refuses resume on
inferenceand needs inference again. That covers a scaled group (e.g.m@44096, nowm@131072) and a default group raised to amin_contextbetween the recipe window and native. Migration: start fresh, or pin the recorded window withinfer.overrides. The maintainer accepted this in #289. On small GPUs, a scaled engine atcontext.maxmay need a higher tp ininfer.overrides;--dry-runwarns about it.Changed public API in
sieval.cli.serving:check_context_fits(...)returnslist[str](warnings); it used to returnNone.task_context_needs(...): amax_tokensneed is the rawmax_tokens, without+ PROMPT_HEADROOM. When both a declared need andmax_tokensexist, the stricter one underfitswins.fits(need, window)andprompt_room_warnings(config, infer_plans, tasks, *, launched: bool).plan_serving_groups(...): the scaled group's key and window arecontext.max. The default group is raised to native (also on short prompt room), never to a need.Related Issues
Closes #289
Test Plan
Automated
ruff check && ruff format --check)ty checkormypy --strict)pdm run pytest)After the review fixes (ef4107b), on Linux:
The one failure is
tests/unit/vendor/code_evaluator/test_exec_quotebench_anchor.py::test_replaying_stored_replies_reproduces_upstreams_gnu_verdicts. It also fails onmain(41b47e1) on the macOS machine I ran on: it replays GNU verdicts. This PR does not touch it.New and rewritten tests:
tests/unit/cli/test_serving.pytest_context_limit_refuses_only_max_tokens_with_no_prompt_room: atcontext_limit: 32768, 28672 passes without a warning, 28673 warns (4095 room), 32767 warns (1 room), 32768 is refused.test_min_context_is_unchanged_and_never_warns: 32768 fits, 32769 is refused.test_one_window_one_verdict_whoever_serves_it: the [Bug]: One context window, three verdicts: prompt headroom refuses an endpoint a launched engine serves #289 table, all three rows.test_scaled_window_is_the_same_for_every_task_subset: needs 44096 and 42096 both givem@131072.test_max_tokens_at_or_above_native_scales_to_max,test_min_context_is_a_hard_need_served_at_a_declared_window,test_pinned_window_is_never_raised,test_unknown_native_checks_the_recipe_window.test_recipe_window_short_of_prompt_room_is_raised_to_native: at recipe 32768 / native 40960, 28672 stays at 32768; 28673 and 32767 are raised to 40960.test_a_max_tokens_does_not_bound_the_scored_prompt: a ppl task beside a gen task keeps the 3072-token prefill reserve whatevermax_tokensthe model'sargsset.test_warnings_split_by_who_serves_the_model.tests/unit/cli/test_run.pytest_a_scaled_engine_is_checked_at_context_max, for sized tp and stated tp: the scaled engine launches at 131072 with the recipe's tp, its memory is reported at 131072, and the low-KV warning namesm@131072.test_little_prompt_room_warns_on_the_serving_check.tests/unit/cli/test_validation.py::test_an_endpoint_with_little_prompt_room_warns(capability_reconcilewarnings).Checklist
Required (all PRs)
type(scope): description)AI-Generated Code - <model> (<provider>)in module docstringcore/_native_serves,_required,_fitswere private toserving.py)If: Breaking Change
🤖 Generated with Claude Code