Ruixiang Wang1,2*, Yongyi Su3*†, Wenlve Zhou3, Bo Yue1,2, Hengyan Liu1,2, Dekun Lu2
Yuxin Tian1,2, Yihan Fang1,2, Zerui Wu2, Xing Hu2, Jietao Chen2, Yong Guo2
Ziyan He2, Junbin Yuan4, Guiliang Liu1, Xiaofen Xing5, Kui Jia1,2
1 CUHK-SZ 2 DexForce 3 FSU 4 SYSU 5 SCUT ·
* Equal contribution † Correspondence to <suyongyi@fosu.edu.cn>
UNITAS is a 3D-native world action model for embodied manipulation. It unifies observations, actions, and scene dynamics in a shared metric 3D frame within each interaction, using a common representation across robot embodiments and human hands.
- One physical space. Observations, actions, and scene dynamics share a metric 3D world-coordinate frame.
- A common motion representation. Point trajectories represent robot motion, human hand motion, and scene dynamics.
- Complementary capabilities. The framework supports robot-action generation and action-conditioned scene prediction.
- Paper results with 1.7B parameters. Success rates reach 99.8% on LIBERO, 89.1% on LIBERO-Plus, 56.2% on VLABench, and 92.94% / 93.04% on RoboTwin Clean / Randomized. Moving ADE is 35% lower than PointWorld on RoboTwin scene prediction.
- Model checkpoints
- Inference and evaluation code
- Training code
- Training dataset
Clone this repo and set up the environment with a few commands:
git clone https://github.com/DexForce/UNITAS.git
cd UNITAS
conda create -n unitas python=3.10 pip -y
conda activate unitas
python -m pip install -e .Optional FlashAttention 2 (requires a compatible CUDA toolkit):
python -m pip install "flash-attn>=2.6,<3" --no-build-isolationFlorence uses FlashAttention 2 when available, otherwise PyTorch SDPA.
Download Florence-2-large
to checkpoints/Florence-2-large; all UNITAS checkpoints share this encoder:
python -m pip install "huggingface_hub[cli]>=0.34,<1"
hf download microsoft/Florence-2-large \
--local-dir checkpoints/Florence-2-large \
--exclude "*.bin"Download the checkpoints you need to checkpoints/.
| Model | Use case | From scratch | Pretrained → Fine-tuned |
|---|---|---|---|
| UNITAS-RoboTwin | RoboTwin Clean / Randomized evaluation | Checkpoint | Checkpoint |
| UNITAS-LIBERO | LIBERO / LIBERO-Plus evaluation | Checkpoint | Checkpoint |
| UNITAS-VLABench | VLABench evaluation | Checkpoint | Checkpoint |
| UNITAS-SceneFlow | Action and scene flow prediction | Jointly trained | |
Run from the repository root after installation. The bundled RGB-D sample needs no simulator. inference.py saves predicted scene and gripper trajectories; visualize.py opens the saved result in Viser.
python src/unitas/scene_flow/inference.py \
--checkpoint checkpoints/unitas_scene_flow.pt \
--florence-dir checkpoints/Florence-2-large \
--input assets/scene_flow/observations/robotwin_7c59d8eb33_0088.npz \
--output outputs/scene_flow.npz
python src/unitas/scene_flow/visualize.py --input outputs/scene_flow.npz --port 8080Open http://127.0.0.1:8080. Drag the timeline, play/pause, or adjust point size.
See all six samples and custom inputs.
Run the model server in unitas. Install each simulator and its assets in a
separate environment following its official repository, then follow our evaluation guide.
The server and client communicate over HTTP.
| Benchmark | Official installation | UNITAS evaluation |
|---|---|---|
| LIBERO | LIBERO | Guide |
| LIBERO-Plus | LIBERO-Plus | Guide |
| RoboTwin | RoboTwin | Guide |
| VLABench | VLABench | Guide |
UNITAS is licensed under Apache-2.0.
We kindly encourage citation of our work if you find it useful.
@misc{wang2026unitas3dnativeworldaction,
title={UNITAS: A 3D-Native World Action Model for Embodied Manipulation},
author={Ruixiang Wang and Yongyi Su and Wenlve Zhou and Bo Yue and Hengyan Liu and Dekun Lu and Yuxin Tian and Yihan Fang and Zerui Wu and Xing Hu and Jietao Chen and Yong Guo and Ziyan He and Junbin Yuan and Guiliang Liu and Xiaofen Xing and Kui Jia},
year={2026},
eprint={2610.12099},
archivePrefix={arXiv},
primaryClass={cs.RO},
url={https://arxiv.org/abs/2610.12099},
}