[shehio]

What This Adds

Experiment tracking through Weights & Biases (entity shehio, project rl-atari), sweep configs for the comparisons we care about, and a Modal GPU runner for the SB3 baseline trainer.

Hook Sites

All four trainers are instrumented:

Site	What is logged
atari/algorithms/dqn/scripts/dqn_trainer.py (train() episode loop)	per-episode episode_reward, episode_loss, epsilon, last_100_avg_reward, avg_max_q, episode_steps, total_steps, episode_duration_sec; final_episode / final_last_100_avg_reward as summary metrics
atari/algorithms/pg/scripts/pg_trainer.py (Trainer.train() / post_episode)	per-episode episode_reward, running_reward, points_scored, points_conceded; final episode and running reward as summary metrics
atari/baselines/atari_baseline_train.py	wandb.integration.sb3.WandbCallback + tensorboard_log + sync_tensorboard=True, so all rollout/* and train/* metrics stream through
atari/baselines/train_lunarlander.py	same SB3 callback path

Every run calls wandb.init(entity="shehio", project="rl-atari", config=<all hyperparams>, tags=[...]), and eval/final results land as summary metrics.

Running With and Without Wandb

cd atari/baselines
python atari_baseline_train.py --algorithm ppo --timesteps 100000    # tracked
python atari_baseline_train.py --algorithm ppo --no-wandb            # untracked
WANDB_MODE=disabled python train_lunarlander.py --algorithm dqn --timesteps 25000

Every touched script takes --no-wandb, honours WANDB_MODE=disabled/offline, and degrades gracefully with a printed notice if wandb is not installed. All three paths were exercised.

Sweeps

Under sweeps/, each with a wandb sweep usage comment at the top:

• sb3_algo_comparison.yaml: PPO vs DQN vs A2C at an equal timestep budget
• sb3_n_envs.yaml: n_envs 1 / 4 / 8
• sb3_seed_sensitivity.yaml: one config across 5 seeds
• dqn_exploration.yaml: custom DQN epsilon-decay x replay buffer size

The DQN sweep needed knobs that did not exist, so dqn_trainer.py gained --epsilon-decay and --memory-size, wired through dqn_game_configs.py into ExplorationConfig and TrainingConfig (with min_memory_len clamped below buffer capacity).

Modal

modal_app.py mirrors the atari_baseline_train.py CLI on an A10G worker: modal.App, debian_slim(python_version="3.12"), add_local_dir, @app.function(gpu=..., timeout=..., secrets=[modal.Secret.from_name("wandb")]), and a matching @app.local_entrypoint. Checkpoints persist in the rl-atari-models volume.

modal run modal_app.py --algorithm ppo --timesteps 1000000 --env ALE/Pong-v5 --n-envs 8

Code only, verified with python -m py_compile. It has never been executed, since there is no Modal token here.

Smoke A/B Results

PPO vs DQN on LunarLander-v3, identical config keys so the curves overlay, tagged smoke-ab. A pair had already been logged by an earlier pass at 20k timesteps; this pass re-ran at 25k before spotting them, so both pairs are listed. They agree.

Run	Algo	Timesteps	Final rollout/ep_rew_mean
mu7kbxdc	PPO	25k	-88.64
y4j8vh4h	DQN	25k	-149.81
hpyrf9u8	PPO	20k	-138.78
vqeu8ksn	DQN	20k	-269.14

PPO is ahead of DQN at an equal budget in both pairs, which is the expected ordering this early: DQN is still mostly filling its replay buffer at 25k steps.

Two further runs validate the other hook sites end to end:

• zxs59a7d: atari_baseline_train.py, PPO on ALE/Pong-v5, 3k timesteps, rollout/ep_rew_mean -20.44
• dal2xm0v: custom dqn_trainer.py on Pong, per-episode metrics confirmed live (episode_reward, last_100_avg_reward, epsilon, episode_loss)

Also Changed

train_lunarlander.py gained DQN as an algorithm choice plus --timesteps and --seed, so it can run finite, reproducible A/B jobs instead of only the previous infinite loop. .gitignore covers wandb/ and tb_logs/; wandb is pinned in requirements.txt.

Skipped

• Modal was never run. No Modal token available in this environment. The app is code-only.
• Fireworks / Harbor: no references anywhere in this repo, so nothing to document.
• The custom PG trainer hook site was not runtime-verified. Its code path mirrors the DQN one, which was verified live, but no PG training run was launched.

Test Status

pytest tests/ gives 38 passed, 1 failed. The failure, tests/test_atari_baselines.py::test_parse_arguments, is pre-existing and reproduces unchanged on master: the module does from atari_baseline_test import parse_arguments as test_parse_arguments, and pytest collects that alias as a test, calling it with no argv.

🤖 Generated with Claude Code

https://claude.ai/code/session_01P4b1Hy6RUgtNpNx2s72NHm