diff --git a/docs/index.md b/docs/index.md index a8a14be..7b6c162 100644 --- a/docs/index.md +++ b/docs/index.md @@ -65,27 +65,74 @@ on the **CPU**: ten clients at once, 30 of 30 episodes successful, at Stepping is 10x slower in software, and most of that hides behind the queue of clients waiting on one policy. +It is lighter than the other systems that train through their channel, too. +`plugrl-env-client` installs 31 packages, 222 MB, with no torch, and the C++ +client below is an 86 KB binary. RLlib's external-env client and LeRobot's +HIL-SERL actor each install about 6.1 GB, torch and 15 CUDA packages among +it, because both run the policy on the environment side. A thin environment +side alone is not new: dm_env_rpc's and openpi's clients are smaller still, +but neither trains through its channel +([E45](https://github.com/PlugRL/plugrl-server/tree/main/experiments/e45-env-side-footprint)). + It does not have to be Python either. [The protocol](protocol/index.md) is written down, with a conformance checker. A C++ program with nothing beyond the standard library - no msgpack or WebSocket library - steps its own copy of Pendulum and trains a policy through it, as well as the Python env client does ([E44](https://github.com/PlugRL/plugrl-server/tree/main/experiments/e44-cpp-pendulum)). +## Other trainers train through it + +The protocol is not tied to PlugRL's own server. Small adapters present it as +the environment interface a trainer already has (`plugrl-bridges`, not public +yet). Through them, three trainers that were not written for PlugRL train +PlugRL env clients, and every seed of every pairing learns: +- RLinf's PPO, on the C++ Pendulum client; +- Stable-Baselines3's PPO, on the C++ Pendulum client and on HalfCheetah; +- CleanRL's PPO, on HalfCheetah + ([E49](https://github.com/PlugRL/plugrl-server/tree/main/experiments/e49-multi-trainer)). + +Episode return against environment steps, three seeds per trainer, 16 env clients each. Left, Pendulum run by the C++ env client: Stable-Baselines3 rises from about -1,250 to -200 within 50 thousand steps; RLinf stays near -1,150 until about 150 thousand steps, then climbs to between -200 and -800. Right, HalfCheetah: Stable-Baselines3 and CleanRL both rise from about -350 to between 800 and 1,300 over 400 thousand steps. + +Each line is one seed's mean return over its 16 env clients, from the +bridge's own episode log. RLinf learns Pendulum later than Stable-Baselines3 +at the same settings; E49 records that and does not explain it. + +RLinf also trained 16 HalfCheetah env clients that ran on a laptop with no +torch, Ray or RLinf, in a 276 MB environment +([E48](https://github.com/PlugRL/plugrl-server/tree/main/experiments/e48-rlinf-bridge)). + +Behind the protocol, training sees nothing different. Stable-Baselines3 with +its 16 environments in another process ends on the same weights, byte for +byte, as with them in its own process: on Pendulum and HalfCheetah +([E50](https://github.com/PlugRL/plugrl-server/tree/main/experiments/e50-boundary-transparency)), +on Atari frames with a CNN +([E53](https://github.com/PlugRL/plugrl-server/tree/main/experiments/e53-image-observations)), +and with every byte between the two sides held up to 25 ms each way +([E52](https://github.com/PlugRL/plugrl-server/tree/main/experiments/e52-latency-sweep)). +Only the time changes: 0.4-0.6 ms per step of 16 environments on one machine, +2.6-3.0 ms when each step carries 16 frames, and about two one-way delays per +step behind a slow link. + ## What the split costs On one machine, almost nothing: an exchange takes 0.1 ms with states only and under 1 ms with a 588 KiB observation. Between two machines it becomes two terms: -- **A fixed latency.** About 3 ms between a laptop on campus Wi-Fi and a wired +- **A fixed latency.** About 3 ms between a laptop on Wi-Fi and a workstation, over Tailscale. -- **Twice the observation's bytes over the link's bandwidth.** Each exchange - carries the observation twice, once asking for the action and once in the - feedback. - -On that link, at 18 MB/s, a state-only task pays about 3 ms per step, and a -184 KiB camera observation about 21 ms -([E43](https://github.com/PlugRL/plugrl-server/tree/main/experiments/e43-cross-machine-training)). -A faster link shrinks the second term in proportion. +- **The observation's bytes over the link's bandwidth, once.** The feedback + that ends a chunk carries the observation, and with `reuse-feedback-obs`, a + feature the server offers and the env client uses by default, the next + request for actions leaves it out. + +On a 23.7 MB/s link, a state-only task pays about 3 ms per step, and a +184 KiB camera observation 11.6 ms, against 18.5 ms when each observation +crossed twice. Training ends on the same weights, byte for byte, either way +([E46](https://github.com/PlugRL/plugrl-server/tree/main/experiments/e46-reuse-feedback-obs); +[E43](https://github.com/PlugRL/plugrl-server/tree/main/experiments/e43-cross-machine-training) +measured the first version). A faster link shrinks the second term in +proportion. ## A real VLA, end to end @@ -121,16 +168,17 @@ cd plugrl-server && uv sync && cd .. cd plugrl-env-client && uv sync --extra mujoco && cd .. ``` -Then, in two terminals: +Then, in two terminals, each inside its own repository (`uv run` uses that +repository's `.venv`): ```bash -# Terminal 1 - the training server -plugrl-run-server fpo-policy default fpo default \ +# Terminal 1, in plugrl-server - the training server +uv run plugrl-run-server fpo-policy default fpo default \ --port 8000 --policy.device cpu \ --algo.global-steps 500000 --algo.buffer-size 4096 -# Terminal 2 - the environment -plugrl-run-env-client mujoco-v1 \ +# Terminal 2, in plugrl-env-client - the environment +uv run plugrl-run-env-client mujoco-v1 \ --server-host 127.0.0.1 --server-port 8000 \ --num-envs 1 --num-episodes 600 --runner.replan-steps 1 --runner.seed 0 ``` diff --git a/docs/index.zh.md b/docs/index.zh.md index d19dcdf..491ee15 100644 --- a/docs/index.zh.md +++ b/docs/index.zh.md @@ -48,20 +48,61 @@ HalfCheetah，训练端在一台 Linux 工作站上。两边各六个种子落 [E13](https://github.com/PlugRL/plugrl-server/tree/main/experiments/e13-gpu-free-rendering)）。 软件渲染的单步慢 10 倍，其中大部分被"多个客户端排队等同一个策略"的等待掩盖了。 +和其他"通过通道训练"的系统比，它也更轻。`plugrl-env-client` 装 31 个包、222 MB， +不带 torch；下面那个 C++ 客户端是一个 86 KB 的可执行文件。RLlib 的 external env 客户端 +和 LeRobot 的 HIL-SERL actor 各装约 6.1 GB，其中有 torch 和 15 个 CUDA 包，因为它们都在 +环境端跑策略。光是环境端轻并不新鲜：dm_env_rpc 和 openpi 的客户端更小，但两者都不通过 +自己的通道训练（[E45](https://github.com/PlugRL/plugrl-server/tree/main/experiments/e45-env-side-footprint)）。 + 环境端也不必是 Python。[协议](protocol/index.zh.md)是写下来的，附带一致性检查器。一个 除了标准库之外什么都不用的 C++ 程序（没有 msgpack 库，也没有 WebSocket 库），自己实现了 Pendulum，通过 PlugRL 训练出了策略，学得和 Python 环境端一样好 （[E44](https://github.com/PlugRL/plugrl-server/tree/main/experiments/e44-cpp-pendulum)）。 +## 别的训练框架也能通过它训练 + +协议不绑定 PlugRL 自己的训练端。几个小适配器把它包装成训练框架本来就认的环境接口 +（`plugrl-bridges`，暂未公开）。通过它们，三个不是为 PlugRL 写的训练框架训练 PlugRL +的环境端，每种组合的每个种子都学会了： +- RLinf 的 PPO，训练 C++ Pendulum 客户端； +- Stable-Baselines3 的 PPO，训练 C++ Pendulum 客户端和 HalfCheetah； +- CleanRL 的 PPO，训练 HalfCheetah + （[E49](https://github.com/PlugRL/plugrl-server/tree/main/experiments/e49-multi-trainer)）。 + +三个训练框架各三个种子、各 16 个环境端的回报随环境步数变化。左：C++ 环境端跑的 Pendulum，Stable-Baselines3 在 5 万步内从约 -1250 升到 -200；RLinf 在约 15 万步前停在 -1150 附近，之后升到 -200 到 -800 之间。右：HalfCheetah，Stable-Baselines3 和 CleanRL 都在 40 万步内从约 -350 升到 800 到 1300 之间。 + +每条线是一个种子在 16 个环境端上的平均回报，数据来自 bridge 自己记的回合日志。同样的 +设置下，RLinf 学会 Pendulum 比 Stable-Baselines3 晚；E49 记录了这一点，但没有解释原因。 + +RLinf 还训练了跑在一台笔记本上的 16 个 HalfCheetah 环境端，那台笔记本没装 torch、Ray +和 RLinf，环境端只占 276 MB +（[E48](https://github.com/PlugRL/plugrl-server/tree/main/experiments/e48-rlinf-bridge)）。 + +隔着协议，训练看到的东西没有任何不同。Stable-Baselines3 把 16 个环境放到另一个进程里， +最后的权重和环境放在自己进程里时逐字节相同：Pendulum 和 HalfCheetah 上如此 +（[E50](https://github.com/PlugRL/plugrl-server/tree/main/experiments/e50-boundary-transparency)）， +换成 Atari 画面加 CNN 也如此 +（[E53](https://github.com/PlugRL/plugrl-server/tree/main/experiments/e53-image-observations)）， +两端之间每个字节每个方向都压住 25 毫秒还是如此 +（[E52](https://github.com/PlugRL/plugrl-server/tree/main/experiments/e52-latency-sweep)）。 +变的只有时间：同一台机器上 16 个环境每步多 0.4 到 0.6 毫秒，每步带 16 帧画面时多 2.6 到 +3.0 毫秒，链路慢时每步大约多两个单程延迟。 + ## 拆开的代价 同一台机器上几乎没有代价：只传状态时一次交换 0.1 毫秒，观测有 588 KiB 也不到 1 毫秒。 跨两台机器时，代价分成两项： -- **固定延迟**：校园 Wi-Fi 上的笔记本经 Tailscale 连有线工作站，大约 3 毫秒。 -- **两倍的观测数据量除以链路带宽**：每次交换里观测要传两次，一次在请求动作时，一次在反馈里。 +- **固定延迟**：Wi-Fi 上的笔记本经 Tailscale 连工作站，大约 3 毫秒。 +- **观测数据量除以链路带宽，只算一次**：结束一段动作的反馈里带着观测；有了 + `reuse-feedback-obs`（训练端提供、环境端默认使用的协议特性），下一次请求动作时不再 + 重复发送。 -在这条链路上（18 MB/s），只传状态的任务每步多约 3 毫秒，一个 184 KiB 的相机观测每步多约 -21 毫秒（[E43](https://github.com/PlugRL/plugrl-server/tree/main/experiments/e43-cross-machine-training)）。链路越快，第二项按比例越小。 +在一条 23.7 MB/s 的链路上，只传状态的任务每步多约 3 毫秒，一个 184 KiB 的相机观测每步多 +11.6 毫秒，观测传两次时是 18.5 毫秒。两种情况下训练出的权重逐字节相同 +（[E46](https://github.com/PlugRL/plugrl-server/tree/main/experiments/e46-reuse-feedback-obs)； +第一版的测量见 [E43](https://github.com/PlugRL/plugrl-server/tree/main/experiments/e43-cross-machine-training)）。 +链路越快，第二项按比例越小。 ## 真实 VLA，端到端 @@ -93,16 +134,16 @@ cd plugrl-server && uv sync && cd .. cd plugrl-env-client && uv sync --extra mujoco && cd .. ``` -然后开两个终端： +然后开两个终端，各自进到对应的仓库里运行（`uv run` 用的是那个仓库自己的 `.venv`）： ```bash -# 终端 1 —— 训练端 -plugrl-run-server fpo-policy default fpo default \ +# 终端 1，在 plugrl-server 里 —— 训练端 +uv run plugrl-run-server fpo-policy default fpo default \ --port 8000 --policy.device cpu \ --algo.global-steps 500000 --algo.buffer-size 4096 -# 终端 2 —— 环境端 -plugrl-run-env-client mujoco-v1 \ +# 终端 2，在 plugrl-env-client 里 —— 环境端 +uv run plugrl-run-env-client mujoco-v1 \ --server-host 127.0.0.1 --server-port 8000 \ --num-envs 1 --num-episodes 600 --runner.replan-steps 1 --runner.seed 0 ``` diff --git a/docs/media/other-trainers.png b/docs/media/other-trainers.png new file mode 100644 index 0000000..093db26 Binary files /dev/null and b/docs/media/other-trainers.png differ