Repository navigation
feat(core)!: record exact error statuses and provider error codes - #294
Merged
Merged
Conversation
Request outcomes now spell every error status as its code ("401", "402",
"502", ...) instead of folding them into 4xx/5xx, and each attempt records
the provider's own error code from its body (error.status, error.code,
error.type, or a vLLM/SGLang top-level type). The request summaries gain
error_codes beside outcomes. Readers group codes into actions; SiEval no
longer pre-collapses them.
Closes #293
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A gateway wrapping its upstream's status ({"error": {"code": 500}} on a 502)
recorded "500" as a provider code. Any number in the HTTP status range is now
skipped, so provider_error_code no longer takes the status. Also renames a
mislabelled test, notes the early error-body read on retried attempts, and
trims the docstrings and CHANGELOG entries.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Merged
8 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Type
outcomeskeys changeSummary
Record in detail; readers group codes into actions.
status_outcomereturns an error status as itself ("400","401","402","429","502","503", …) in place of4xx/5xx.2xx,context_overflow(still wins),timeout_<phase>,connection_error,cancelledandotherare unchanged;outcomesstill partitions attempts.requests.jsonlline gainserror_code; the request summaries (profile.json,report.jsonsieval.requests,meta.jsonruns[].requests) gainerror_codesbesideoutcomes. Attempts without a code are not counted there.RecordingAsyncClient._send_oncereads each error body once (aread()), so retried attempts get a code too. httpx caches the bytes, so the SDK and every dialect'sraise_for_provider_statusreuse them with no extra request, streamed or not. Success bodies are not touched.provider_error_codeinsieval/core/models/request_log.py):error.status(Google) →error.code(OpenAI, New API) →error.type(Anthropic); with no error object, top-levelcode→type(vLLM/SGLangBadRequestError). A Google stream error's one-item list is unwrapped. Skipped: any HTTP status number (including a gateway wrapping its upstream's, e.g.{"error":{"code":500}}on a 502), placeholders (null), and anything that is not a single token of ≤64 chars[A-Za-z0-9][A-Za-z0-9_.:-]*. Non-JSON bodies record nothing; the message is never read..idx, score or identity changes, and no resume readsrequestscounts.RequestCounts.error_codesisNotRequired, so 0.16meta.jsonstill type-checks and resumes.Breaking Change
Readers matching
"4xx"/"5xx"inoutcomesno longer find them in new runs. Migration: group numeric keys (400 <= int(k) < 500). The non-2xxtotal is unchanged. Older directories keep their keys.Related Issues
Closes #293
Test Plan
pdm run ruff check .,ruff format --check .,ty check→ cleanpytest tests/unit tests/integration -q -n 8→ 11254 passed, 84 skipped, 1 failed (test_exec_quotebench_anchor.py::test_replaying_stored_replies_reproduces_upstreams_gnu_verdicts, a GNU-toolchain anchor that fails on untouchedorigin/mainon macOS; unrelated)pytest tests/unit/core/models tests/unit/core/tasks tests/integration/test_request_records.py -n 8→ 4405 passed, 1 skippedpytest tests/acceptance -q -n 8→ 12 passed, 10 skippedpre-commit run -a,check_preflight.py --level quick,sync_package_stubs.py --check→ cleanrequest_log.pycoverage 96% (uncovered lines predate this PR)New tests cover: each status spelled exactly, 1xx/3xx →
other; the outcome partition; extraction per provider shape (OpenAI, New API, LiteLLM, vLLM/SGLang, Anthropic, Google incl. stream list, wrapped upstream status) and every rejected shape; on the wire throughRecordingAsyncClientwith the OpenAI SDK (chat, completions, responses × stream), per-attempt codes across retries, Anthropic / Google GenAI / SGLang dialects still building their messages from the body, a broken error body keeping its status and releasing the connection, success streams not read early,context_overflowkeeping the code;error_codesaggregation insummary()andcounts().Checklist
!for the breaking changecore/"4xx"/"5xx"literals grepped acrosssieval/,tests/,docs/,scripts/; the one doc line updated🤖 Generated with Claude Code