Robot Learning: Domain Randomization with PPO on Custom Hopper

1. Project Overview

This project implements and compares domain randomization techniques for robust sim-to-real transfer in reinforcement learning using PPO (Proximal Policy Optimization) on a custom Hopper environment (based on MuJoCo and Gymnasium). The codebase supports training, evaluation, hyperparameter optimization, and policy rendering, with six domain randomization strategies.

The goal is to train agents on a "source" environment and evaluate their transfer performance on a "target" environment with different dynamics (the target has a heavier torso).

---

2. Directory Structure

codice/
├── dr/                       # Domain randomization controllers
│   ├── base.py               # Base DR controller superclass
│   ├── no_dr.py              # No Domain Randomization
│   ├── uniform_dr.py         # Uniform Domain Randomization (UDR)
│   ├── auto_dr.py            # Automatic Domain Randomization (ADR)
│   ├── gaus_adr.py           # Gaussian ADR
│   ├── mm_adr.py             # Bimodal (Multi-Modal) ADR
│   ├── mistake_adr.py        # Mistake-Driven ADR (MistakeADR)
│   └── __init__.py
├── env/                      # Custom Hopper environment and wrappers
│   ├── assets/
│   │   └── hopper.xml        # MuJoCo XML for Hopper
│   ├── custom_hopper.py      # Custom Hopper environment
│   └── wrappers.py           # Domain Randomization wrapper
├── training_hopper/          # Training scripts for each DR strategy
│   ├── train_noDR.py         # Train with no DR
│   ├── train_UDR.py          # Train with Uniform DR
│   ├── train_ADR.py          # Train with Automatic DR
│   ├── train_gADR.py         # Train with Gaussian ADR
│   ├── train_mmADR.py        # Train with Bimodal ADR
│   └── train_mistakeADR.py   # Train with Mistake-Driven ADR
├── eval_ppo.py               # Model evaluation script
├── optuna_search.py          # Hyperparameter optimization with Optuna
├── render_policy.py          # Render trained policy in the environment
├── utils/                    # Utility scripts (plotting, analysis)
├── results/                  # Saved models
├── training_results/         # Training logs and checkpoints
├── requirements.txt          # Python dependencies
└── readme.md                 # This file

---

3. Setup

1. Install Python 3.8+ and MuJoCo dependencies
2. Create and activate a virtual environment (recommended):

   python -m venv .venv
   .venv\Scripts\activate     # Windows
   # or
   source .venv/bin/activate  # macOS/Linux

3. Install required packages:

   pip install -r requirements.txt

---

4. Domain Randomization Strategies

The project implements six domain randomization strategies, all randomizing the masses of the Hopper's thigh, leg, and foot links:

Strategy	File	Description
No DR	dr/no_dr.py	Baseline with no randomization
Uniform DR (UDR)	dr/uniform_dr.py	Samples masses uniformly within fixed bounds
Automatic DR (ADR)	dr/auto_dr.py	Starts narrow, expands bounds when performance is good, shrinks when poor
Gaussian ADR	dr/gaus_adr.py	Like ADR, but samples from truncated Gaussian centered on nominal values
Bimodal ADR	dr/mm_adr.py	Samples from a bimodal distribution (two Gaussians at ±δ from nominal)
Mistake-Driven ADR	dr/mistake_adr.py	Samples parameters inversely proportional to past performance (focuses on weaknesses)

4.1 ADR Curriculum Mechanism

All ADR variants share a curriculum mechanism:

• Expansion: When average reward exceeds threshold_up, bounds expand by expansion_rate
• Shrinkage: When average reward falls below threshold_down, bounds contract
• Checkpointing: Model is saved before each expansion (captures best performance at current difficulty)

4.2 MistakeADR (Novel Contribution)

MistakeADR is a performance-weighted ADR strategy that:

1. Divides each parameter range into bins
2. Tracks average reward per bin
3. Samples harder bins (lower reward) more frequently via softmax
4. Resets statistics when bounds change

---

5. Training

Each DR strategy has its own training script in training_hopper/:

# No Domain Randomization
python training_hopper/train_noDR.py

# Uniform Domain Randomization
python training_hopper/train_UDR.py

# Automatic Domain Randomization
python training_hopper/train_ADR.py

# Gaussian ADR
python training_hopper/train_gADR.py

# Bimodal ADR
python training_hopper/train_mmADR.py

# Mistake-Driven ADR
python training_hopper/train_mistakeADR.py

Each script contains a Config dataclass with all hyperparameters. Key parameters:

Parameter	Default	Description
total_timesteps	2,000,000	Total training steps
learning_rate	3e-4	PPO learning rate
batch_size	128	Minibatch size
n_epochs	10	PPO epochs per update
auto_dr_expansion_rate	0.05	Bounds expansion rate (5%)
auto_dr_threshold_up	1550	Reward threshold for expansion
auto_dr_threshold_down	1250	Reward threshold for shrinkage

---

6. Evaluation

Evaluate a trained model on both source and target environments:

python eval_ppo.py --model_path results/models/your_model.zip --n_eval_episodes 200

Output:

Evaluation on CustomHopper-source-v0 - Return: 1850.23 +- 150.45
Evaluation on CustomHopper-target-v0 - Return: 1720.12 +- 180.32

---

7. Hyperparameter Optimization

Run Optuna to search for optimal PPO hyperparameters:

python optuna_search.py

Results are saved in training_results/optuna/.

---

8. Rendering

Visualize a trained policy:

python render_policy.py --model_path results/models/your_model.zip

---

9. Custom Hopper Environment

• Source: Torso mass reduced by 1kg (simulates model uncertainty)
• Target: True dynamics (for evaluation)
• Randomized parameters: Thigh, leg, and foot masses (indices 1, 2, 3)

Registered environments:

• CustomHopper-source-v0
• CustomHopper-target-v0

---

10. Results and Logs

Training produces:

• monitor.csv: Episode rewards and lengths
• adr_boundaries.txt: Bound expansion/shrinkage history (for ADR variants)
• adr_samples.txt: Sampled parameter values
• checkpoint_expand_*.zip: Model checkpoints before each expansion
• reward.png: Training reward curve

---

11. Requirements

gymnasium
imageio
matplotlib
mujoco
optuna
scipy
stable-baselines3[extra]>=2.0.0

Install with:

pip install -r requirements.txt

---

12. References

• OpenAI ADR Paper (Solving Rubik's Cube)
• Stable-Baselines3 Documentation
• Gymnasium Documentation