Repository navigation
Conversation
…dit export The Every Eval Ever converter names the model from studio_provenance.target_model. It is set to the target's frozen model id and left out when the run has none.
Author
|
The workflow runs on this fork PR are waiting for maintainer approval ( |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #50.
Adds a read-only export that returns a completed run in SimpleAudit's own results format (the document
AuditResults.to_dict()builds), so a converter such as the proposed Every Eval Ever one can consume it. It is built from the storedScenarioResultrows and needs no engine run.Changes
infra/ui.py:3567and:3600:RunExportViewgets?format=simpleaudit. It uses the view's existing project scoping, so a run in another workspace is a 404 and anonymous requests are redirected to login.format=jsonandformat=csvare unchanged.audits/services.py:422and:432:export_engine_results(run)reads the rows, fills in defaults for failed scenarios (severityERROR, summary from the stored error) and adds arunblock with the provenance frozen on the run (id, name,simpleaudit_version,git_commit, timestamps, and the target/auditor/judge model names; no URLs or credentials). Only completed runs are exported (409 otherwise).infra/engine.py:885:build_audit_results_exportbuilds the document through the engine'sAuditResults, so the summary block is computed by the engine and Studio-only keys (trace_ids,_language, ...) are dropped. It stays the only file that imports the engine.templates/partials/run_results.html:12: "Export (SimpleAudit JSON)" link on the run page, shown for completed runs.docs/architecture.md:101: URL table row.infra/tests/test_run_export_simpleaudit.py: round trip from storedAuditResult.to_dict()rows to the HTTP response, compared withAuditResults(...).to_dict(), plusAuditResults.loadon the exported file, therunblock, failed scenarios, unfinished runs, other workspace, login and the unchangedformat=json.Results
infra/tests/test_run_export_simpleaudit.py: failed before the change (the new format fell through to the Studio list), passed after.uv run pytest -m "not embedded_hatchet": passed.uv run ruff check .: passed.Not covered
n_repetitions > 1answer 422. Their stored results are repeated-experiment records, which I left out to keep this to the single-run shape.tests/data/from item 3 of the issue.Design questions
runblock is an addition to the engine's keys.AuditResults.loadignores it. Would you rather drop it, or put the provenance somewhere else?RepeatedExperimentResults(one label per model), or as one single-run export per repetition?AI disclosure: implemented with Claude Code.