Collect, grade and publish agent trajectories
Roller runs Claude Code, Codex, OpenClaw and four other Harnesses on your tasks or on a benchmark, grades each sample where it ran, and imports the sessions already on your machine. Every run exports to SFT training data, ATIF and a Hugging Face dataset.
English · 简体中文
Documentation · Getting started · CLI reference · Cookbook · Examples
More from Valen Team: Valen-Decision, a multimodal System One decision model
What Roller does · News · Quick start · Harnesses · Benchmarks · Outputs · Architecture · Docs · Contributing
uv tool install roller-agent # or: python -m pip install roller-agentEach successful rollout becomes a messages + tools record in sft.jsonl. The record keeps the visible reasoning of the model and has an images field when the trace contains images. When the agent compacts its context, each verified context window is exported as its own record under sft_segments/. On a graded run, keep the samples above a score:
roller export runs/wildclaw --format sft --min-score 0.5 --output trainingroller collect --bench converts the benchmark tasks, runs them on any of the seven Harnesses, executes each grader in the container where the task ran, and writes bench-report.json. Adapters cover WildClawBench, SkillsBench, PinchBench, QwenClawBench and SWE-Bench Pro.
roller collect --bench wildclawbench --bench-root ~/WildClawBench \
--model <provider>/<model> --model-config <models.json> \
--harness openclaw --openclaw-image ghcr.io/valen-team/roller-wildclaw-base:1.3-clean.1 \
--parallel 4 --output runs --run-name wildclawroller import turns the Claude Code and Codex sessions on this machine into a run, one sample per session. No Docker, model endpoint or task list is needed.
roller import --dry-run # list the sessions by project
roller import --output runs --run-name my-sessions
roller view runs/my-sessionsRoller reads ~/.claude/projects, ~/.codex/sessions and ~/.codex/archived_sessions, including compressed Codex sessions. --harness, --since and --project narrow the selection, and running the command again imports the new and changed sessions.
roller export --format hf packages a collected or imported run as a Hugging Face dataset folder that the Hub opens in its trace viewer. The export replaces the model endpoints, host names, host paths, home directories, user names and credential values of the run, and writes what it could not classify to <output>.redaction-report.json.
roller export runs/my-sessions --format hf --output my-sessions-hf
roller export runs/my-sessions # ATIF, the trajectory format of HarborRead the report, add --redact-rules FILE for what remains, and scan the folder with a secret scanner before uploading. Roller-WildClawBench-Trajectories is a dataset published this way, and the cookbook follows the run behind it.
Hugging Face export → · ATIF export →
roller view RUN_DIR opens a read-only web interface for browsing tasks, comparing models and inspecting each step, with benchmark scores, score filters and user images.
Recordings of the built-in Viewer using synthetic demonstration data. Left: the reasoning, tool calls and tool results of one rollout. Middle: one task compared across models. Right: the complete layout. Select an image to view it at full size.
- 2026-11-19: Roller 0.5.0.
roller importis on PyPI. DeepSeek Harness can collect WildClawBench in an E2B sandbox with--sandbox-provider e2b, for collectors without a Docker daemon. Collection checks credentials up front and logs progress and ETA,roller cleanremoves the containers of dead collectors, androller export --format sft --min-scorefilters training data by score. - 2026-11-18: Session import.
roller importturns the Claude Code and Codex sessions on this machine into a Roller run that the Viewer and every export read like a collected one. - 2026-11-17: WildClawBench with OpenClaw and Codex: four self-hosted models, browsable trajectories and scores, public task images and runnable collection examples.
- 2026-11-16: Hugging Face export and ATIF export. The first dataset is Roller-WildClawBench-Trajectories.
- 2026-11-15: Roller is on PyPI as
roller-agent.roller initwrites the example tasks and configuration templates into the working directory.
Earlier
- 2026-11-11: SFT export for compacted context. When the Task Agent compacts its context, each verified context window is exported as a separate SFT record under
sft_segments/. - 2026-11-08: QwenClawBench adapter: 100 tasks in 8 domains, with automated grading, hybrid grading and repeated runs per task.
- 2026-11-05: One-command evaluation for SkillsBench and PinchBench. SFT records support images.
- 2026-11-01: Hermes, OpenCode, DeepSeek Harness and Pi Agent added, for a total of seven Harnesses.
The changelog lists every change of each release.
The following procedure collects a first batch of trajectories with OpenClaw, the default Harness.
Requirements: Python 3.10 or later and a running Docker daemon. With Docker Desktop on macOS, enable Enable host networking under Settings → Resources → Network and restart Docker Desktop.
uv tool install roller-agent # or: python -m pip install roller-agent
docker pull ghcr.io/openclaw/openclaw:2026.7.1-2
mkdir roller-quickstart && cd roller-quickstart
roller init # writes the example tasks and configuration templatesCopy the environment template, set MY_PROXY_BASE_URL and MY_PROXY_API_KEY, and export the variables to the current shell:
cp examples/env/api.env.example examples/env/api.env
# Edit examples/env/api.env, then export its variables:
set -a; source examples/env/api.env; set +aIn examples/config/openclaw/models.json, set the model id to the model name used by the service. Commands reference a model as <provider>/<id>; with the example file, the model reference is my-openai-proxy/my-model. For services that support only /chat/completions, change api from openai-responses to openai-completions.
roller collect \
--tasks examples/tasks/tasks.jsonl \
--model my-openai-proxy/my-model \
--model-config examples/config/openclaw/models.json \
--parallel 4 --retries 1 --timeout 600 \
--output runs --run-name quickstartAdd --dry-run first to parse the tasks, validate the input files and print the planned rollouts without starting containers or calling the model API. To compare models, repeat --model; each task then runs once per model.
roller view runs/quickstartOpen the URL printed in the terminal (default http://127.0.0.1:17865).
- Progress. Collection checks the required credentials up front, logs progress, ETA and heartbeats to
collect.log, and stops dispatch after 10 consecutive errors (--max-consecutive-errors 0disables this). - Resume. Rerun the same command with
--resumeto skip the completed samples. Tasks and models can be added; if another setting recorded inlock.jsonhas changed, Roller reports the difference and exits. - Grading failures. Failed graders retry in the live container, and
--resume --resume-unscoredreruns the agents whose grading failed. - Leftovers.
roller clean RUN_DIR --dry-runlists the containers left by dead collectors.
A Harness is the agent runtime inside the container. It starts the agent, provides its tools and calls the model API. Specify the Harness for all tasks with --harness, or for an individual task with the harness field. All seven read the same task format and write the same sample layout.
| Harness | Model API | Description |
|---|---|---|
| OpenClaw (default) | OpenAI Responses or Chat Completions | Custom and open models; multimodal input |
| Codex | Responses | Codex CLI with GPT or Responses-compatible services |
| Claude Code | Anthropic /v1/messages |
Claude Code with native thinking |
| Hermes | OpenAI Chat Completions | Hermes Agent with terminal and file tools |
| OpenCode | OpenAI Chat Completions | OpenCode CLI sessions with text and file tools |
| DeepSeek Harness | DeepSeek Chat Completions | Official SDK with a single persistent Bash tool |
| Pi Agent | OpenAI Chat Completions | Pi SDK with seven tools: read, bash, edit, write, grep, find, ls |
OpenClaw reads model definitions from --model-config. The other six Harnesses use --api-base and --api-key-env; in runs that combine several Harnesses, each takes its own prefixed pair, for example --hermes-api-base and --hermes-api-key-env. Every Harness can serve as the Task Agent or as the User Simulator, a second model that plays the user and writes the next turn.
Rollouts run in Docker containers. DeepSeek Harness can also run WildClawBench in an E2B sandbox: install roller-agent[e2b] and pass --sandbox-provider e2b --sandbox-template NAME, as the DeepSeek guide describes.
Harness guides: OpenClaw · Codex · Claude Code · Hermes · OpenCode · DeepSeek · Pi Agent.
Bench adapters convert benchmark tasks into Roller tasks and grade each sample in the container where it ran. Grading failures are recorded separately from zero scores, and bench-report.json aggregates each model with failed samples in the denominator.
| Bench | Default Harness | Scope |
|---|---|---|
| WildClawBench | openclaw |
60 real-scenario tasks in 6 categories |
| SkillsBench | opencode |
One built environment per task; with-skill and no-skill runs scored by official verifiers |
| PinchBench | openclaw |
Automated, LLM-judge, hybrid and multi-session tasks |
| QwenClawBench | openclaw |
100 tasks in 8 domains, with repeated runs per task |
| SWE-Bench Pro | openclaw |
731 software-engineering tasks in 11 repositories, scored by hidden tests in official instance images |
A bench run starts with roller collect --bench, as in One command per benchmark, or from a YAML file with roller bench run --config ...; see Benchmark run configuration. Each bench guide lists the settings that must match the upstream configuration and the environment variables required by its graders. To add a bench, see Bench integration and scoring.
A task is a JSON object. A minimal task contains a schema version, an ID and a prompt:
{"schema_version":"roller.task/v1","id":"hello","prompt":"Compute 2+3. Reply with the number only."}A task list can be a .jsonl file, a JSON array or an object of the form {"tasks": [...]}. Optional fields add predefined user turns (messages), input and Context files (files), an output directory that Roller collects in full (files.output_dir), environment and setup commands (runtime), a Harness for this task (harness) and a user_simulator. The task format reference specifies every field.
Each run is stored in one directory. Run-level files are at the top level, and each sample has its own directory under samples/:
runs/quickstart/
├── info.json settings, progress and total usage
├── lock.json settings compared by --resume
├── sft.jsonl SFT records of successful, uncompacted samples
├── sft_index.jsonl task, model and score for each training record
└── samples/<model>/<index>_<task>/
├── trace.jsonl Harness-native session
├── trace_ACTF.json unified trace: reasoning, tool calls, results, tokens, timing
├── trace_SFT.json SFT record of this sample
├── info.json status, attempts, usage, versions and input digests
└── results/ files written to the task's output directory
Use sft.jsonl for training and trace_ACTF.json for analysis. trace.jsonl retains the Harness-native events for debugging and re-conversion and is not cleaned for training.
The following figure shows how each file is produced. Dashed paths are enabled by the flag shown on the arrow.
Three files are written only under specific conditions:
sft_segments/: written when the Task Agent compacts its context. The rollout is then left out ofsft.jsonl, and each verified context window is a separate record whosetarget_start_indexmarks the first supervised message.wire_trace.jsonl: written with--wire-trace full. It contains every request body that Roller's proxy forwarded to the model API.conversation.json: written for User Simulator tasks. It contains the dialogue visible to both participants.
--keep-thinking is on by default and stores the visible reasoning that the model service returns; --thinking sets the requested reasoning level. The trajectory output guide specifies when each file is written, and the SFT format describes the training records.
Roller runs as a single host process. The Collector schedules rollouts in parallel and starts one Docker container, or one E2B sandbox, per rollout. Inside it, the Harness runtime executes the agent loop in a workspace that contains the declared inputs and Context. Depending on the Harness and the --wire-trace setting, model requests either pass through Roller's request proxy, which records them before forwarding, or go directly to the model API. After the last turn, Roller collects the native session and the output directory, and the trace converters write the sample to the run directory. roller import feeds local sessions to the same converters without a container.
Blue arrows: model calls. Dashed elements: optional components and conditional paths.
Each rollout follows the lifecycle below. The Collector first checks whether a valid sample already exists. Otherwise it starts the container, and the Harness executes the turns; with a User Simulator, each Task Agent reply goes to the Simulator, which returns the next user message or ends the conversation. For bench runs, the grader runs before teardown. Roller then converts the native session to the normalized trace, and the Collector either retries a failed attempt (up to --retries times) or finalizes the sample.
Roller snapshots and hashes all inputs before a container starts, and lock.json records the settings that determine sample content, so a resumed run produces the samples the first run would have.
New users should start with Getting started and refer to Task format when writing tasks. The complete handbook is at valen-team.github.io/roller.
| Guide | Contents |
|---|---|
| Introduction | Core concepts: task, Harness, rollout, run and trace |
| Getting started | Installation, dry run, first collection and result verification |
| CLI reference | All flags, their defaults and the Harnesses they apply to |
| Docker builds and package mirrors | apt, npm and Python mirrors and proxies for image builds |
| Task format | Prompts, multi-turn messages, inputs, Context and output directories |
| Files, Task Bundles and result storage | Input snapshots, run layout, retries and Resume |
| User Simulator | Dynamic conversations driven by a second model |
| Trajectory outputs | Output files and their conditions; comparison of raw, normalized, SFT and wire traces |
| SFT format | Record structure, images and compaction segments |
| Viewer | Startup, ports and remote access |
| Session import | Importing local Claude Code and Codex sessions, and redacting them for upload |
| Bench integration and scoring | Bench runs, grading and new bench adapters |
| Examples (Chinese) | Runnable commands for each Harness |
| Coverage suite | 1,624 functional test tasks across all seven Harnesses |
- Before a container starts, Roller rejects inputs and Context that contain symlinks or special files, escape their declared paths or exceed the size limits, and records a digest of every file.
- API keys are read from the host environment or from external configuration and are not written to run metadata.
- Only regular files and directories inside the task's
output_dirare collected. wire_trace.jsonlstores full request text, including prompts, reasoning and tool results. Treat it as sensitive data.- The Viewer listens on
127.0.0.1by default. Use SSH port forwarding for remote access.
The complete rules are specified in Security and privacy boundaries.
Report defects through Issues. New Harnesses, new benches and documentation improvements are accepted as Pull Requests. The code is organized as follows:
| Area | Location |
|---|---|
| Harness adapters | src/roller/harnesses/, with trace converters in src/roller/*_actf.py |
| Bench adapters | src/roller/benches/ |
| SFT export | src/roller/sft.py, src/roller/compaction_sft.py |
| ATIF export | src/roller/atif.py |
| Viewer | src/roller/viewer/ |
Run the checks before submitting a pull request:
python -m pip install -e ".[test]" ruff
python -m pytest
ruff check
ruff format --checkCONTRIBUTING.md describes the conventions for tests, documentation and the run lock, and CHANGELOG.md lists the changes in each release. Report a vulnerability as described in SECURITY.md. Participation is covered by the Code of Conduct.
![]() Liuziyu77 |
![]() yph22 |
![]() SYuan03 |
![]() Dodojordi |
![]() Nina-Hyn |
![]() Credica |
![]() Orion-DH |
![]() MqLeet |
![]() llreda |
![]() yuhangzang |
Roller is one of the open-source projects of Valen Team, which works on fast decision models, recursive self-improvement (RSI) and agents.
Valen-Decision is a multimodal System One decision model: text, images and video in; structured decision probabilities out. It comes in five sizes from 0.8B to 27B, answers bbox questions, and has a web demo and free API.
Roller is licensed under the Apache License 2.0.
Roller invokes the following third-party agent runtimes and benchmarks: OpenClaw, Codex, Claude Code, Hermes Agent, OpenCode, DeepSeek Harness, Pi Agent, WildClawBench, SkillsBench, PinchBench, QwenClawBench and SWE-Bench Pro. Their images, code and data are subject to their respective licenses. We thank the authors of these projects.
If you use Roller in your work, please cite this repository:
@misc{roller,
author = {{Valen Team}},
title = {Roller},
howpublished = {GitHub - Valen-Team/Roller: Agent trajectory collection across models and Harnesses.},
note = {GitHub repository},
year = {2026},
url = {https://github.com/Valen-Team/Roller}
}











