Repository navigation
Conversation
There was a problem hiding this comment.
Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.
An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
ScicodeEvaluator.test_codeskips the supplied reference steps 13.6, 62.1 and 76.3 during execution, then includes them inTotal Steps. Even when every generated substep passes, these three problems receiveProblem Correctness=0, and their reference-only steps inflate the aggregate denominator.These answers are part of the question design, as clarified in #1 and #12. The current published test split contains 291 entries including these three supplied steps, leaving 288 evaluated substeps. The deprecated evaluator already uses 288 for that split.
What changed
Count steps after the existing reference-step exclusion, including both freshly executed steps and cached verdicts. The same count now determines complete-problem correctness and the aggregate subproblem rate. Generated code, reference injection and cached-result behavior remain unchanged.
Add two pytest-discovered behavior scenarios through
inspect_ai.eval, the actual scorer, generated Python subprocesses, HDF5 targets and Inspect's metric aggregation. The fixtures cover all three supplied-step positions and a problem without supplied steps, with all generated answers correct or one wrong answer per problem, then repeat against cached verdicts.Verification
mainate3158ea011d4235245a547460d3688d7ccbf9900, both new scenarios fail on the incorrect step counts. With the change, both pass through fresh and cached scoring.pytest -q --covpasses all four tests, including the two existing CLI scenarios, on macOS arm64, Python 3.11.18 and inspect-ai 0.3.289.Full model evaluations against the published numeric-reference artifact were not run. Python 3.10 remains for project CI.