Skip to content
Valen-TeamPublic

About

Turn agent runs into training data. Roller rolls out Claude Code, Codex, OpenClaw and four other Harnesses in disposable Docker containers, grades benchmark tasks in place, and exports SFT-ready trajectories that keep reasoning, images and compacted-context segments.

Topics

Resources

Code of conduct

Contributing

Security policy

Stars

105 stars

Watchers

0 watching

Forks

Roller

Roller

Collect, grade and publish agent trajectories

Roller runs Claude Code, Codex, OpenClaw and four other Harnesses on your tasks or on a benchmark, grades each sample where it ran, and imports the sessions already on your machine. Every run exports to SFT training data, ATIF and a Hugging Face dataset.

7 Harnesses 5 benchmark adapters PyPI version Roller-WildClawBench-Trajectories on Hugging Face Test status License: Apache 2.0 Python 3.10+

English · 简体中文
Documentation · Getting started · CLI reference · Cookbook · Examples

More from Valen Team: Valen-Decision, a multimodal System One decision model

Four uses of Roller. Training data: roller export --format sft writes SFT records that keep reasoning, images and compacted-context segments. Benchmarks: roller collect --bench runs seven Harnesses on five benchmarks and grades each sample in its container. Local sessions: roller import turns the Claude Code and Codex sessions on this machine into a run. Datasets: roller export --format hf writes a redacted Hugging Face dataset folder. Every path writes the same run directory, which roller view opens.

What Roller does · News · Quick start · Harnesses · Benchmarks · Outputs · Architecture · Docs · Contributing

What Roller does

uv tool install roller-agent    # or: python -m pip install roller-agent

1. Agent runs in, training data out

Each successful rollout becomes a messages + tools record in sft.jsonl. The record keeps the visible reasoning of the model and has an images field when the trace contains images. When the agent compacts its context, each verified context window is exported as its own record under sft_segments/. On a graded run, keep the samples above a score:

roller export runs/wildclaw --format sft --min-score 0.5 --output training

SFT format →

2. One command per benchmark

roller collect --bench converts the benchmark tasks, runs them on any of the seven Harnesses, executes each grader in the container where the task ran, and writes bench-report.json. Adapters cover WildClawBench, SkillsBench, PinchBench, QwenClawBench and SWE-Bench Pro.

roller collect --bench wildclawbench --bench-root ~/WildClawBench \
  --model <provider>/<model> --model-config <models.json> \
  --harness openclaw --openclaw-image ghcr.io/valen-team/roller-wildclaw-base:1.3-clean.1 \
  --parallel 4 --output runs --run-name wildclaw

Benchmarks ↓

3. Import the sessions you already have

roller import turns the Claude Code and Codex sessions on this machine into a run, one sample per session. No Docker, model endpoint or task list is needed.

roller import --dry-run                             # list the sessions by project
roller import --output runs --run-name my-sessions
roller view runs/my-sessions

Roller reads ~/.claude/projects, ~/.codex/sessions and ~/.codex/archived_sessions, including compressed Codex sessions. --harness, --since and --project narrow the selection, and running the command again imports the new and changed sessions.

Session import guide →

4. Publish a run as a dataset

roller export --format hf packages a collected or imported run as a Hugging Face dataset folder that the Hub opens in its trace viewer. The export replaces the model endpoints, host names, host paths, home directories, user names and credential values of the run, and writes what it could not classify to <output>.redaction-report.json.

roller export runs/my-sessions --format hf --output my-sessions-hf
roller export runs/my-sessions                      # ATIF, the trajectory format of Harbor

Read the report, add --redact-rules FILE for what remains, and scan the folder with a secret scanner before uploading. Roller-WildClawBench-Trajectories is a dataset published this way, and the cookbook follows the run behind it.

Hugging Face export → · ATIF export →

Browse every run in the Viewer

roller view RUN_DIR opens a read-only web interface for browsing tasks, comparing models and inspecting each step, with benchmark scores, score filters and user images.

Step-by-step inspection of one rollout in Roller Viewer: task search, model selection, and the reasoning, tool calls and tool results of the trajectory steps One synthetic task compared across two models in Roller Viewer, with benchmark scores Roller Viewer with the sample navigator on the left, a user image in the middle, and the Inspector on the right

Recordings of the built-in Viewer using synthetic demonstration data. Left: the reasoning, tool calls and tool results of one rollout. Middle: one task compared across models. Right: the complete layout. Select an image to view it at full size.

News

  • 2026-11-19: Roller 0.5.0. roller import is on PyPI. DeepSeek Harness can collect WildClawBench in an E2B sandbox with --sandbox-provider e2b, for collectors without a Docker daemon. Collection checks credentials up front and logs progress and ETA, roller clean removes the containers of dead collectors, and roller export --format sft --min-score filters training data by score.
  • 2026-11-18: Session import. roller import turns the Claude Code and Codex sessions on this machine into a Roller run that the Viewer and every export read like a collected one.
  • 2026-11-17: WildClawBench with OpenClaw and Codex: four self-hosted models, browsable trajectories and scores, public task images and runnable collection examples.
  • 2026-11-16: Hugging Face export and ATIF export. The first dataset is Roller-WildClawBench-Trajectories.
  • 2026-11-15: Roller is on PyPI as roller-agent. roller init writes the example tasks and configuration templates into the working directory.
Earlier
  • 2026-11-11: SFT export for compacted context. When the Task Agent compacts its context, each verified context window is exported as a separate SFT record under sft_segments/.
  • 2026-11-08: QwenClawBench adapter: 100 tasks in 8 domains, with automated grading, hybrid grading and repeated runs per task.
  • 2026-11-05: One-command evaluation for SkillsBench and PinchBench. SFT records support images.
  • 2026-11-01: Hermes, OpenCode, DeepSeek Harness and Pi Agent added, for a total of seven Harnesses.

The changelog lists every change of each release.

Quick start

The following procedure collects a first batch of trajectories with OpenClaw, the default Harness.

1. Installation

Requirements: Python 3.10 or later and a running Docker daemon. With Docker Desktop on macOS, enable Enable host networking under Settings → Resources → Network and restart Docker Desktop.

uv tool install roller-agent    # or: python -m pip install roller-agent
docker pull ghcr.io/openclaw/openclaw:2026.7.1-2
mkdir roller-quickstart && cd roller-quickstart
roller init                     # writes the example tasks and configuration templates

2. Model configuration

Copy the environment template, set MY_PROXY_BASE_URL and MY_PROXY_API_KEY, and export the variables to the current shell:

cp examples/env/api.env.example examples/env/api.env
# Edit examples/env/api.env, then export its variables:
set -a; source examples/env/api.env; set +a

In examples/config/openclaw/models.json, set the model id to the model name used by the service. Commands reference a model as <provider>/<id>; with the example file, the model reference is my-openai-proxy/my-model. For services that support only /chat/completions, change api from openai-responses to openai-completions.

3. Collection

roller collect \
  --tasks examples/tasks/tasks.jsonl \
  --model my-openai-proxy/my-model \
  --model-config examples/config/openclaw/models.json \
  --parallel 4 --retries 1 --timeout 600 \
  --output runs --run-name quickstart

Add --dry-run first to parse the tasks, validate the input files and print the planned rollouts without starting containers or calling the model API. To compare models, repeat --model; each task then runs once per model.

4. Viewer

roller view runs/quickstart

Open the URL printed in the terminal (default http://127.0.0.1:17865).

Long runs

  • Progress. Collection checks the required credentials up front, logs progress, ETA and heartbeats to collect.log, and stops dispatch after 10 consecutive errors (--max-consecutive-errors 0 disables this).
  • Resume. Rerun the same command with --resume to skip the completed samples. Tasks and models can be added; if another setting recorded in lock.json has changed, Roller reports the difference and exits.
  • Grading failures. Failed graders retry in the live container, and --resume --resume-unscored reruns the agents whose grading failed.
  • Leftovers. roller clean RUN_DIR --dry-run lists the containers left by dead collectors.

Getting started guide →

Harnesses

A Harness is the agent runtime inside the container. It starts the agent, provides its tools and calls the model API. Specify the Harness for all tasks with --harness, or for an individual task with the harness field. All seven read the same task format and write the same sample layout.

Harness Model API Description
OpenClaw (default) OpenAI Responses or Chat Completions Custom and open models; multimodal input
Codex Responses Codex CLI with GPT or Responses-compatible services
Claude Code Anthropic /v1/messages Claude Code with native thinking
Hermes OpenAI Chat Completions Hermes Agent with terminal and file tools
OpenCode OpenAI Chat Completions OpenCode CLI sessions with text and file tools
DeepSeek Harness DeepSeek Chat Completions Official SDK with a single persistent Bash tool
Pi Agent OpenAI Chat Completions Pi SDK with seven tools: read, bash, edit, write, grep, find, ls

OpenClaw reads model definitions from --model-config. The other six Harnesses use --api-base and --api-key-env; in runs that combine several Harnesses, each takes its own prefixed pair, for example --hermes-api-base and --hermes-api-key-env. Every Harness can serve as the Task Agent or as the User Simulator, a second model that plays the user and writes the next turn.

Rollouts run in Docker containers. DeepSeek Harness can also run WildClawBench in an E2B sandbox: install roller-agent[e2b] and pass --sandbox-provider e2b --sandbox-template NAME, as the DeepSeek guide describes.

Harness guides: OpenClaw · Codex · Claude Code · Hermes · OpenCode · DeepSeek · Pi Agent.

Benchmarks

Bench adapters convert benchmark tasks into Roller tasks and grade each sample in the container where it ran. Grading failures are recorded separately from zero scores, and bench-report.json aggregates each model with failed samples in the denominator.

Bench Default Harness Scope
WildClawBench openclaw 60 real-scenario tasks in 6 categories
SkillsBench opencode One built environment per task; with-skill and no-skill runs scored by official verifiers
PinchBench openclaw Automated, LLM-judge, hybrid and multi-session tasks
QwenClawBench openclaw 100 tasks in 8 domains, with repeated runs per task
SWE-Bench Pro openclaw 731 software-engineering tasks in 11 repositories, scored by hidden tests in official instance images

A bench run starts with roller collect --bench, as in One command per benchmark, or from a YAML file with roller bench run --config ...; see Benchmark run configuration. Each bench guide lists the settings that must match the upstream configuration and the environment variables required by its graders. To add a bench, see Bench integration and scoring.

Tasks

A task is a JSON object. A minimal task contains a schema version, an ID and a prompt:

{"schema_version":"roller.task/v1","id":"hello","prompt":"Compute 2+3. Reply with the number only."}

A task list can be a .jsonl file, a JSON array or an object of the form {"tasks": [...]}. Optional fields add predefined user turns (messages), input and Context files (files), an output directory that Roller collects in full (files.output_dir), environment and setup commands (runtime), a Harness for this task (harness) and a user_simulator. The task format reference specifies every field.

Outputs

Each run is stored in one directory. Run-level files are at the top level, and each sample has its own directory under samples/:

runs/quickstart/
├── info.json            settings, progress and total usage
├── lock.json            settings compared by --resume
├── sft.jsonl            SFT records of successful, uncompacted samples
├── sft_index.jsonl      task, model and score for each training record
└── samples/<model>/<index>_<task>/
    ├── trace.jsonl      Harness-native session
    ├── trace_ACTF.json  unified trace: reasoning, tool calls, results, tokens, timing
    ├── trace_SFT.json   SFT record of this sample
    ├── info.json        status, attempts, usage, versions and input digests
    └── results/         files written to the task's output directory

Use sft.jsonl for training and trace_ACTF.json for analysis. trace.jsonl retains the Harness-native events for debugging and re-conversion and is not cleaned for training.

The following figure shows how each file is produced. Dashed paths are enabled by the flag shown on the arrow.

Trace data pipeline: the Harness session, collected or imported, is copied to trace.jsonl and converted to trace_ACTF.json, which yields trace_SFT.json and sft.jsonl; roller export redacts the sessions into a Hugging Face dataset and converts the trace to ATIF; --capture-calls enables compaction segments and --wire-trace full keeps request bodies in wire_trace.jsonl

Three files are written only under specific conditions:

  • sft_segments/: written when the Task Agent compacts its context. The rollout is then left out of sft.jsonl, and each verified context window is a separate record whose target_start_index marks the first supervised message.
  • wire_trace.jsonl: written with --wire-trace full. It contains every request body that Roller's proxy forwarded to the model API.
  • conversation.json: written for User Simulator tasks. It contains the dialogue visible to both participants.

--keep-thinking is on by default and stores the visible reasoning that the model service returns; --thinking sets the requested reasoning level. The trajectory output guide specifies when each file is written, and the SFT format describes the training records.

Architecture

Roller runs as a single host process. The Collector schedules rollouts in parallel and starts one Docker container, or one E2B sandbox, per rollout. Inside it, the Harness runtime executes the agent loop in a workspace that contains the declared inputs and Context. Depending on the Harness and the --wire-trace setting, model requests either pass through Roller's request proxy, which records them before forwarding, or go directly to the model API. After the last turn, Roller collects the native session and the output directory, and the trace converters write the sample to the run directory. roller import feeds local sessions to the same converters without a container.

Roller architecture: the host process loads and snapshots tasks, the Collector starts one Docker container or E2B sandbox per rollout, model calls pass through the request proxy, and the trace converters write samples to the run directory; roller import feeds local sessions to the same converters, and the Viewer, training, bench reports and dataset exports read the run directory

Blue arrows: model calls. Dashed elements: optional components and conditional paths.

Each rollout follows the lifecycle below. The Collector first checks whether a valid sample already exists. Otherwise it starts the container, and the Harness executes the turns; with a User Simulator, each Task Agent reply goes to the Simulator, which returns the next user message or ends the conversation. For bench runs, the grader runs before teardown. Roller then converts the native session to the normalized trace, and the Collector either retries a failed attempt (up to --retries times) or finalizes the sample.

Lifecycle of one rollout: Resume check, container or sandbox start, turns with model calls and an optional User Simulator, optional bench grading, teardown, trace conversion, and retry or finalization

Roller snapshots and hashes all inputs before a container starts, and lock.json records the settings that determine sample content, so a resumed run produces the samples the first run would have.

Documentation

New users should start with Getting started and refer to Task format when writing tasks. The complete handbook is at valen-team.github.io/roller.

Guide Contents
Introduction Core concepts: task, Harness, rollout, run and trace
Getting started Installation, dry run, first collection and result verification
CLI reference All flags, their defaults and the Harnesses they apply to
Docker builds and package mirrors apt, npm and Python mirrors and proxies for image builds
Task format Prompts, multi-turn messages, inputs, Context and output directories
Files, Task Bundles and result storage Input snapshots, run layout, retries and Resume
User Simulator Dynamic conversations driven by a second model
Trajectory outputs Output files and their conditions; comparison of raw, normalized, SFT and wire traces
SFT format Record structure, images and compaction segments
Viewer Startup, ports and remote access
Session import Importing local Claude Code and Codex sessions, and redacting them for upload
Bench integration and scoring Bench runs, grading and new bench adapters
Examples (Chinese) Runnable commands for each Harness
Coverage suite 1,624 functional test tasks across all seven Harnesses

Security and privacy

  • Before a container starts, Roller rejects inputs and Context that contain symlinks or special files, escape their declared paths or exceed the size limits, and records a digest of every file.
  • API keys are read from the host environment or from external configuration and are not written to run metadata.
  • Only regular files and directories inside the task's output_dir are collected.
  • wire_trace.jsonl stores full request text, including prompts, reasoning and tool results. Treat it as sensitive data.
  • The Viewer listens on 127.0.0.1 by default. Use SSH port forwarding for remote access.

The complete rules are specified in Security and privacy boundaries.

Contributing

Report defects through Issues. New Harnesses, new benches and documentation improvements are accepted as Pull Requests. The code is organized as follows:

Area Location
Harness adapters src/roller/harnesses/, with trace converters in src/roller/*_actf.py
Bench adapters src/roller/benches/
SFT export src/roller/sft.py, src/roller/compaction_sft.py
ATIF export src/roller/atif.py
Viewer src/roller/viewer/

Run the checks before submitting a pull request:

python -m pip install -e ".[test]" ruff
python -m pytest
ruff check
ruff format --check

CONTRIBUTING.md describes the conventions for tests, documentation and the run lock, and CHANGELOG.md lists the changes in each release. Report a vulnerability as described in SECURITY.md. Participation is covered by the Code of Conduct.

Contributors

Liuziyu77
Liuziyu77
yph22
yph22
SYuan03
SYuan03
Dodojordi
Dodojordi
Nina-Hyn
Nina-Hyn
Credica
Credica
Orion-DH
Orion-DH
MqLeet
MqLeet
llreda
llreda
yuhangzang
yuhangzang

More from Valen Team

Roller is one of the open-source projects of Valen Team, which works on fast decision models, recursive self-improvement (RSI) and agents.

Valen-Decision is a multimodal System One decision model: text, images and video in; structured decision probabilities out. It comes in five sizes from 0.8B to 27B, answers bbox questions, and has a web demo and free API.

License and acknowledgements

Roller is licensed under the Apache License 2.0.

Roller invokes the following third-party agent runtimes and benchmarks: OpenClaw, Codex, Claude Code, Hermes Agent, OpenCode, DeepSeek Harness, Pi Agent, WildClawBench, SkillsBench, PinchBench, QwenClawBench and SWE-Bench Pro. Their images, code and data are subject to their respective licenses. We thank the authors of these projects.

Citation

If you use Roller in your work, please cite this repository:

@misc{roller,
  author       = {{Valen Team}},
  title        = {Roller},
  howpublished = {GitHub - Valen-Team/Roller: Agent trajectory collection across models and Harnesses.},
  note         = {GitHub repository},
  year         = {2026},
  url          = {https://github.com/Valen-Team/Roller}
}

About

Turn agent runs into training data. Roller rolls out Claude Code, Codex, OpenClaw and four other Harnesses in disposable Docker containers, grades benchmark tasks in place, and exports SFT-ready trajectories that keep reasoning, images and compacted-context segments.

Topics

Resources

Code of conduct

Contributing

Security policy

Stars

105 stars

Watchers

0 watching

Forks

Releases

Packages

Used by

Contributors

Languages