Skip to content

exclude supplied substeps from inspect scoring - #68

Open
ZenAlexa wants to merge 1 commit into
scicode-bench:mainfrom
ZenAlexa:contrib/reference-step-scoring
Open

ZenAlexa wants to merge 1 commit into
scicode-bench:mainfrom
ZenAlexa:contrib/reference-step-scoring

Conversation

@ZenAlexa

@ZenAlexa ZenAlexa commented Nov 14, 2026 •

Copy link
Copy Markdown

Why

ScicodeEvaluator.test_code skips the supplied reference steps 13.6, 62.1 and 76.3 during execution, then includes them in Total Steps. Even when every generated substep passes, these three problems receive Problem Correctness=0, and their reference-only steps inflate the aggregate denominator.

These answers are part of the question design, as clarified in #1 and #12. The current published test split contains 291 entries including these three supplied steps, leaving 288 evaluated substeps. The deprecated evaluator already uses 288 for that split.

What changed

Count steps after the existing reference-step exclusion, including both freshly executed steps and cached verdicts. The same count now determines complete-problem correctness and the aggregate subproblem rate. Generated code, reference injection and cached-result behavior remain unchanged.

Add two pytest-discovered behavior scenarios through inspect_ai.eval, the actual scorer, generated Python subprocesses, HDF5 targets and Inspect's metric aggregation. The fixtures cover all three supplied-step positions and a problem without supplied steps, with all generated answers correct or one wrong answer per problem, then repeat against cached verdicts.

Verification

  • On main at e3158ea011d4235245a547460d3688d7ccbf9900, both new scenarios fail on the incorrect step counts. With the change, both pass through fresh and cached scoring.
  • pytest -q --cov passes all four tests, including the two existing CLI scenarios, on macOS arm64, Python 3.11.18 and inspect-ai 0.3.289.
  • The grading fixtures use independently specified numerical targets and no model generation calls. The public validation/test metadata was loaded to confirm the 50/288 evaluated-step counts.

Full model evaluations against the published numeric-reference artifact were not run. Python 3.10 remains for project CI.

Copilot AI balanced review requested due to automatic review settings November 14, 2026 04:54

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants