Skip to content

feat(runs): export a completed run as the engine's results JSON - #56

Open
avalyset wants to merge 2 commits into
SimulaMet:mainfrom
avalyset:run-export-simpleaudit-json
Open

avalyset wants to merge 2 commits into
SimulaMet:mainfrom
avalyset:run-export-simpleaudit-json

Conversation

@avalyset

@avalyset avalyset commented Oct 16, 2026 •

Copy link
Copy Markdown

Part of #50.

Adds a read-only export that returns a completed run in SimpleAudit's own results format (the document AuditResults.to_dict() builds), so a converter such as the proposed Every Eval Ever one can consume it. It is built from the stored ScenarioResult rows and needs no engine run.

Changes

  • infra/ui.py:3567 and :3600: RunExportView gets ?format=simpleaudit. It uses the view's existing project scoping, so a run in another workspace is a 404 and anonymous requests are redirected to login. format=json and format=csv are unchanged.
  • audits/services.py:422 and :432: export_engine_results(run) reads the rows, fills in defaults for failed scenarios (severity ERROR, summary from the stored error) and adds a run block with the provenance frozen on the run (id, name, simpleaudit_version, git_commit, timestamps, and the target/auditor/judge model names; no URLs or credentials). Only completed runs are exported (409 otherwise).
  • infra/engine.py:885: build_audit_results_export builds the document through the engine's AuditResults, so the summary block is computed by the engine and Studio-only keys (trace_ids, _language, ...) are dropped. It stays the only file that imports the engine.
  • templates/partials/run_results.html:12: "Export (SimpleAudit JSON)" link on the run page, shown for completed runs.
  • docs/architecture.md:101: URL table row.
  • infra/tests/test_run_export_simpleaudit.py: round trip from stored AuditResult.to_dict() rows to the HTTP response, compared with AuditResults(...).to_dict(), plus AuditResults.load on the exported file, the run block, failed scenarios, unfinished runs, other workspace, login and the unchanged format=json.

Results

  • infra/tests/test_run_export_simpleaudit.py: failed before the change (the new format fell through to the Studio list), passed after.
  • uv run pytest -m "not embedded_hatchet": passed.
  • uv run ruff check .: passed.

Not covered

  • Runs with n_repetitions > 1 answer 422. Their stored results are repeated-experiment records, which I left out to keep this to the single-run shape.
  • The anonymized fixture under tests/data/ from item 3 of the issue.

Design questions

  1. The run block is an addition to the engine's keys. AuditResults.load ignores it. Would you rather drop it, or put the provenance somewhere else?
  2. Should repeated runs be exported as RepeatedExperimentResults (one label per model), or as one single-run export per repetition?

AI disclosure: implemented with Claude Code.

…dit export

The Every Eval Ever converter names the model from studio_provenance.target_model.
It is set to the target's frozen model id and left out when the run has none.
@avalyset

Copy link
Copy Markdown
Author

The workflow runs on this fork PR are waiting for maintainer approval (action_required), so no checks have run yet. The PR now also writes studio_provenance.target_model, so the converter in evaleval/every_eval_ever#312 picks up the target model.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant