UFO: A General Unsupervised Reinforcement Learning Framework for Humanoid COntrol

English | 中文

      

UFO is an unsupervised reinforcement learning framework for humanoid control, built on mjlab. It supports Unitree G1 / RoboParty RP1.3, FB-CPR-Aux / TeCH, multi-GPU training, and tracking, reward and goal inference.

The RP1 TeCH policy has not been tested on the physical robot.

G1 / RP1 training data and the G1 FB policy tested on hardware are available.

News

2026-10-15 — Unified mjlab release

• Simpler framework, two robots: replaced the legacy humanoidverse layout with ufo, with unified G1 / RP1 and FB / TeCH training commands.
• More efficient training: evaluation across all GPUs, compiled network forwards, fused Adam, bucketed gradient synchronization, and separate model/replay save schedules.
• Updated FB defaults: 1024-wide networks with a shared critic trunk; independent critics remain available via --no-shared-critic.
• Correctness fixes: preserve true final observations across resets, correct contact-force direction, and align reference and simulator observation scaling.
• Deployment exports: ONNX exports include joint order, observation scales, history order, motor gains and action scales. RP1 training data and a G1 FB policy tested on hardware are now available through the links above.

See architecture and training/inference for details.

Install

Requires Linux and an NVIDIA driver compatible with CUDA 12.8. The lockfile selects PyTorch 2.7.1 / CUDA 12.8 and NCCL 2.26.5, including Blackwell support. Run these commands, then work from the repository root:

git clone https://github.com/Roboparty/UFO.git
cd UFO
python3 -m pip install --user uv
export PATH="$HOME/.local/bin:$PATH"
uv sync --locked

This guide requires the mjlab checkout containing ufo/ and run_inference.sh; it does not apply to the older humanoidverse/ release. Video export requires ffmpeg (sudo apt install ffmpeg on Ubuntu).

Prepare data

Download matching full motions and training clips; the script verifies SHA256:

bash scripts/download_data.sh g1_lafan
bash scripts/download_data.sh rp1_lafan

G1 contains 40 motions. RP1 contains 55 motions at 50 Hz, converted for its 24 controlled joints, with 1,166 training clips. Each robot requires matching data; G1 data cannot be used directly for RP1.

For local files, see data paths. To convert your own RP1 NPZ files, see RP1 data import.

Train

After preparing data, choose one command:

./run_train.sh --robot g1  --agent fb   --gpu-ids all
./run_train.sh --robot rp1 --agent fb   --gpu-ids all
./run_train.sh --robot g1  --agent tech --gpu-ids all
./run_train.sh --robot rp1 --agent tech --gpu-ids all

--gpu-ids all uses all visible GPUs; use CUDA_VISIBLE_DEVICES to select them. --gpu-ids single --smoke checks startup and sampling only; it does not update weights. See a short training check with updates.

Runs default to runs/<robot>_<agent>. Use a new --work-dir for each experiment; an existing checkpoint in that directory is resumed. To enable W&B:

uv run wandb login
WANDB_PROJECT=UFO ./run_train.sh --robot g1 --agent fb --gpu-ids all \
  --work-dir runs/g1_fb_exp --use-wandb --wandb-run-name g1_fb_exp

Inference

Inference reads robot settings from the saved run. Use a checkpoint trained for the selected robot and matching motion data:

./run_inference.sh tracking --model-folder runs/g1_fb --motion-list 0 --save-mp4
./run_inference.sh reward --model-folder runs/g1_fb --save-mp4
./run_inference.sh goal --model-folder runs/g1_fb \
  --data-path ufo/data/lafan_29dof.pkl --save-mp4

• Tracking: FB and TeCH, G1 and RP1. Add --data-path to select motion data; videos and z are saved under the run directory.
• Reward: FB only; requires saved rank-0 replay. RP1 currently supports root/locomotion tasks.
• Goal: FB and TeCH, G1 and RP1. Use full-motion data for the bundled G1 goals. For RP1 commands and a goal JSON example, see goal inference.

See ./run_inference.sh <tracking|reward|goal> --help for options.

The released G1 FB model includes the approximately 192M-step checkpoint, ONNX exports and matching tracking, goal and forward-walking z. Download it into model/:

uv run hf download xuewang/ufo-g1-policy --local-dir model
./run_inference.sh tracking --model-folder model/g1_policy \
  --data-path ufo/data/lafan_29dof.pkl --motion-list 7 --save-mp4

The download includes model weights, but no optimizer state or replay buffer. Computing new reward z requires your own saved replay; the provided reward z is ready for deployment.

Documentation

• Training and inference: defaults, evaluation, checkpoints and inference options.
• Robot and data adaptation: add robots with robot_setup and import retargeted motions.
• Architecture: code layout and component responsibilities.
• G1 real-robot deployment.
• RP1 UFO deployment: Roboparty/roboparty_deploy.

Citation and license

Cite UFO

@misc{ufo2026,
  author       = {{RoboParty Lab Team}},
  title        = {UFO: An Unsupervised Reinforcement Learning Framework for Humanoid Control},
  year         = {2026},
  howpublished = {\url{https://github.com/Roboparty/UFO}},
  note         = {Project page: \url{https://roboparty.github.io/UFO/}}
}

UFO is released under CC BY-NC 4.0 for non-commercial research use.