Skip to content

feat(core)!: record exact error statuses and provider error codes - #294

Merged
ethan-scitix merged 2 commits into
mainfrom
feat/detailed-request-outcomes
Nov 14, 2026
Merged

ethan-scitix merged 2 commits into
mainfrom
feat/detailed-request-outcomes

Conversation

@ethan-scitix

@ethan-scitix ethan-scitix commented Nov 14, 2026 •

Copy link
Copy Markdown
Collaborator

Type

  • breaking change — outcomes keys change
  • feature — new benchmark, task, or capability

Summary

Record in detail; readers group codes into actions.

  • Exact statuses. status_outcome returns an error status as itself ("400", "401", "402", "429", "502", "503", …) in place of 4xx/5xx. 2xx, context_overflow (still wins), timeout_<phase>, connection_error, cancelled and other are unchanged; outcomes still partitions attempts.
  • Provider error code. Each requests.jsonl line gains error_code; the request summaries (profile.json, report.json sieval.requests, meta.json runs[].requests) gain error_codes beside outcomes. Attempts without a code are not counted there.
  • One capture point. RecordingAsyncClient._send_once reads each error body once (aread()), so retried attempts get a code too. httpx caches the bytes, so the SDK and every dialect's raise_for_provider_status reuse them with no extra request, streamed or not. Success bodies are not touched.
  • Mapping (provider_error_code in sieval/core/models/request_log.py): error.status (Google) → error.code (OpenAI, New API) → error.type (Anthropic); with no error object, top-level code → type (vLLM/SGLang BadRequestError). A Google stream error's one-item list is unwrapped. Skipped: any HTTP status number (including a gateway wrapping its upstream's, e.g. {"error":{"code":500}} on a 502), placeholders (null), and anything that is not a single token of ≤64 chars [A-Za-z0-9][A-Za-z0-9_.:-]*. Non-JSON bodies record nothing; the message is never read.
  • No schema bump. No shard, .idx, score or identity changes, and no resume reads requests counts. RequestCounts.error_codes is NotRequired, so 0.16 meta.json still type-checks and resumes.

Breaking Change

Readers matching "4xx"/"5xx" in outcomes no longer find them in new runs. Migration: group numeric keys (400 <= int(k) < 500). The non-2xx total is unchanged. Older directories keep their keys.

Related Issues

Closes #293

Test Plan

  • pdm run ruff check ., ruff format --check ., ty check → clean
  • pytest tests/unit tests/integration -q -n 8 → 11254 passed, 84 skipped, 1 failed (test_exec_quotebench_anchor.py::test_replaying_stored_replies_reproduces_upstreams_gnu_verdicts, a GNU-toolchain anchor that fails on untouched origin/main on macOS; unrelated)
  • After review fixes: pytest tests/unit/core/models tests/unit/core/tasks tests/integration/test_request_records.py -n 8 → 4405 passed, 1 skipped
  • pytest tests/acceptance -q -n 8 → 12 passed, 10 skipped
  • pre-commit run -a, check_preflight.py --level quick, sync_package_stubs.py --check → clean
  • request_log.py coverage 96% (uncovered lines predate this PR)

New tests cover: each status spelled exactly, 1xx/3xx → other; the outcome partition; extraction per provider shape (OpenAI, New API, LiteLLM, vLLM/SGLang, Anthropic, Google incl. stream list, wrapped upstream status) and every rejected shape; on the wire through RecordingAsyncClient with the OpenAI SDK (chat, completions, responses × stream), per-attempt codes across retries, Anthropic / Google GenAI / SGLang dialects still building their messages from the body, a broken error body keeping its status and releasing the connection, success streams not read early, context_overflow keeping the code; error_codes aggregation in summary() and counts().

Checklist

  • PR title follows conventional format, ! for the breaking change
  • No internal paths, credentials, or personal info in committed files
  • AI-generated code marker present in touched modules
  • No new upper-layer dependencies in core/
  • "4xx"/"5xx" literals grepped across sieval/, tests/, docs/, scripts/; the one doc line updated
  • Breaking change and migration described; existing tests updated

🤖 Generated with Claude Code

ethan-scitix and others added 2 commits November 14, 2026 16:22
Request outcomes now spell every error status as its code ("401", "402",
"502", ...) instead of folding them into 4xx/5xx, and each attempt records
the provider's own error code from its body (error.status, error.code,
error.type, or a vLLM/SGLang top-level type). The request summaries gain
error_codes beside outcomes. Readers group codes into actions; SiEval no
longer pre-collapses them.

Closes #293

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A gateway wrapping its upstream's status ({"error": {"code": 500}} on a 502)
recorded "500" as a provider code. Any number in the HTTP status range is now
skipped, so provider_error_code no longer takes the status. Also renames a
mislabelled test, notes the early error-body read on retried attempts, and
trims the docstrings and CHANGELOG entries.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ethan-scitix ethan-scitix changed the title feat(core): record exact error statuses and provider error codes feat(core)!: record exact error statuses and provider error codes Nov 14, 2026
@ethan-scitix
ethan-scitix merged commit 76c675b into main Nov 14, 2026
9 checks passed
@ethan-scitix
ethan-scitix deleted the feat/detailed-request-outcomes branch November 14, 2026 08:58
@ethan-scitix ethan-scitix mentioned this pull request Nov 14, 2026
8 tasks done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Spell out 401, 403 and other actionable statuses in request outcomes

1 participant