Repository navigation
FIX Drain nested scoring and batch tasks on failure - #2856
Merged
Roman Lutz (romanlutz) merged 8 commits intoOct 8, 2026
Merged
Roman Lutz (romanlutz) merged 8 commits into
Roman Lutz (romanlutz) merged 8 commits into
Conversation
This was referenced Sep 25, 2026
Roman Lutz (romanlutz)
approved these changes
Oct 3, 2026
Roman Lutz (romanlutz)
enabled auto-merge
October 3, 2026 14:01
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> # Conflicts: # pyrit/score/message_scorer.py # pyrit/score/true_false/true_false_composite_scorer.py
auto-merge was automatically disabled
October 7, 2026 04:14
Head branch was pushed to by a user without write access
Roman Lutz (romanlutz)
approved these changes
Oct 7, 2026
Update the branch for the required up-to-date check before merging PR microsoft#2856. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
A failed scoring call can return while nested scorers or other items in its batch still run. This allows old work to overlap a retry or persist scores after the caller has reported failure.
This change shares task cleanup across the existing response-level protection and the remaining concurrency boundaries:
Scorer.score_with_scorers_async()TrueFalseCompositeScorerMessageScorerscore_batch_async()/batch_task_async()The helper shields the initial gather and explicitly owns cancellation. It cancels unfinished children only if they are not already processing cancellation, then shields and drains both the original gather and every child. A slow asynchronous cleanup therefore receives only one cancellation request, even if the caller cancels repeatedly. Further caller cancellation propagates after cleanup; sibling cleanup errors do not replace the primary failure.
Waiting for the original gather also matters when children finish in the same event-loop turn as caller cancellation: their completion callbacks may still be queued. Reading the original gather's exception before those callbacks run raises
InvalidStateErrorand replaces the intendedCancelledError. The shared drain now waits for and consumes that result safely.Successful result ordering, completed score persistence, batch sizes, and rate-limit validation are preserved. Later batches do not start after a failure. Completed requests and scores are not rolled back.
The branch includes current main and the SQLite cancellation cleanup from #2982. That fix is required when cancelling a score-validation SELECT leaves a native cursor holding a writer lock and the attack subsequently tries to persist its error result.
Tests and Documentation
get_scores_async(score_type="true_false", include_intermediate=True)query.InvalidStateErrorbefore the fix and now preserveCancelledError.database table is lockedon the previous PR head,bb29c8e, in a separate checkout. It passes with the updated branch, preserving the original scorer failure and persisting the error result.make unit-testrun on Python 3.13.12 with all optional dependencies: 24,346 passed, 11 skipped, 0 failed.ty check pyritwith all optional dependencies installed.uv run --extra all pytest -q tests/unit/memory/test_sqlite_cancellation.py tests/unit/executor/attack/test_scoring_expectation_transport.py tests/unit/prompt_target/test_batch_helper.py tests/unit/score/test_concurrent_scoring_lifetime.py --tb=short make unit-test CMD='uv run --extra all -m'The common helper and root/batch docstrings document the cleanup contract. No notebooks changed, and validation used mocked targets and local SQLite rather than live model endpoints or SQL Server.