Skip to content

fix(envs): confine benchmark task output paths - #309

Open
Adong / 阿东玩 AI (adongwanai) wants to merge 1 commit into
microsoft:mainfrom
adongwanai:fix/306-confine-benchmark-task-output
Open

Adong / 阿东玩 AI (adongwanai) wants to merge 1 commit into
microsoft:mainfrom
adongwanai:fix/306-confine-benchmark-task-output

Conversation

@adongwanai

Copy link
Copy Markdown

Summary

Extend the task-output confinement introduced in #264 to DocVQA, SearchQA, OfficeQA, and LiveMathematicianBench. The benchmark rollouts share one path resolver; safe identifiers keep their existing directory names, while LiveMath identifiers such as 202602:12 map to stable directory segments that work on Windows.

Slow-update trajectory reads use the same mapping and retain support for legacy POSIX LiveMath directories after checking containment. Regression tests cover rejected identifiers, symlink escapes, mapped rollout output, and both mapped and legacy trajectory reads.

Fixes #306.

Testing

  • python -m pytest -q tests/test_benchmark_task_output_paths.py tests/test_spreadsheetbench_task_id_confinement.py tests/test_slow_update_robustness.py — 34 passed.
  • python -m pytest -q — 1,723 passed, 11 skipped, 359 subtests passed.
  • uvx ruff check skillopt/envs/task_output.py skillopt/optimizer/slow_update.py tests/test_benchmark_task_output_paths.py — passed.
  • git diff --check — passed.

The test runs above were completed on October 7. Before submission, the October 8 check confirmed that upstream main remains at 343db229dbd5ddaf9df6b1d5540d8bcdb2604d5c and that all eight changed files match the tested patch exactly.

Existing Ruff findings in the full rollout files reproduce on the unchanged upstream baseline and are outside this change.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Same unconfined predictions/<item_id> path pattern remains in four benchmark rollouts

1 participant