Skip to content

feat: Let each policy step on its own clock - #94

Merged
ApGa merged 1 commit into
mainfrom
apga/train-independent
Oct 29, 2026
Merged

ApGa merged 1 commit into
mainfrom
apga/train-independent

Conversation

@ApGa

@ApGa ApGa commented Oct 28, 2026

Copy link
Copy Markdown
Owner

Stacked on #91 ("feat: Train several policies in lockstep"); review that first.

This is RFC 0021's independent stepping (section 2.11, scenario S7 (m)): train.stepping: independent lets each
trained policy step, publish and save on its own clock, with its own batch, staleness bound and checkpoint cadence,
and no policy waits for another's training or saves. lockstep stays the default. With train.stepping unset nothing
a run does changes: the golden trace is byte-identical, every effect of its scenarios and their variants is the same,
in the same order, but for calls of pure functions, and the lockstep runs of two policies (the previous PR's tests and
its example) are unchanged too (below).

train:
  stepping: independent             # default: lockstep
  batch_size: 8                     # main's batch, and the plan's groups per main step
  max_staleness: 1                  # main's horizon: (1 + 1) x 8 = 16 groups
policies:
  worker:
    model_name: Qwen/Qwen3-4B-Instruct-2507
    renderer_name: qwen3_instruct
    batch_size: 2                   # its own batch: about 4 of its steps per main step
    max_staleness: 7                # its own bound: a horizon of (7 + 1) x 2 = 16
    checkpoint: {recovery_every: 4} # its own cadence, in its own steps
agents:
  roles:
    subagent: {policy: worker}

What it adds

Module What it holds
platoon/train/: config_defs.py, policy/spec.py, identity.py, trainer/api.py train.stepping (lockstep or independent); policies.<id>.batch_size and policies.<id>.checkpoint: {every, recovery_every} (PolicyCheckpointConfig; another key is refused by name), which reach the policy's spec; the refusal the lockstep PR gave them now applies to policies that step together only, saying to set train.stepping: independent; besides: a frozen policy's, and, stepping independently, a batch that the step's minibatches and microbatches do not divide and a group admission rule (stepping_problems, which loading asks of train.stepping and each start again of its trainer's needs); a warning for a trained policy that saves no full checkpoint before the run's end; steps_independently(config); identity.json records stepping: independent, and a resume that changes it stops with exit code 78; TrainerNeeds.stepping, which the rl trainer answers with train.stepping; under trainer: sft, which trains one policy, train.stepping is refused as the other train: keys only the rl trainer reads are
platoon/train/dispatch/shares.py (new) ShareLedger, which platoon.experimental exports: what a trainer of its own takes each policy's shares from, a protocol of five names: take(policy, n, stop=...) (the shares as settled groups, as the source's take gives them in lockstep; TrainInterrupted once stop is set), judge_version (the version to judge a share's staleness at: for a share a resume put back, the one at which it has the age it had when first consumed), consumed, done and lock. Shares, the run's ledger: each settled group, after credit, split into a share per trained policy with rows in it, and always one of the leader's (with none of its rows in a group without them, so that the plan counts every group; a failed group is the leader's share alone, so failures throttle dispatch as in lockstep); each policy's share queue for the rl trainer's steps (queue, PLAN_END once no share will come); consumed (each share's step, age and fate; a group's fate goes to the per-task statistics once, when its last share is consumed); durable and retirement (a group leaves the kept groups and the outstanding draws once every consuming policy has a row after the consuming step); snapshot, restore, revert (a resume puts back each share a policy consumed after its row, first, with its recorded age); stopped (a loop a stop request ended); the commit lock. The module imports neither Tinker nor torch
platoon/train/dispatch/: horizons.py (new), dispatcher.py, setup.py, source.py, runner.py one horizon per trained policy, (k + 1) x batch of its shares, counting the running groups and its shares queued and taken; a group starts only when every one has room, the leader's remaining steps bound the plan, and each applies the dispatcher's held-slot rule with its own k and batch; once a stop request ended a policy's loop and its horizon is full, no group starts again (held_for_good), and a step that still waits for shares closes short, as at the plan's end; train/dispatch/blocked_share/horizon:<id> (and horizon, their sum); the dispatcher hands settled groups to the ledger (SettledSink); the source builds the horizons and the ledger when the run's policies step independently, captures its draws for a commit (capture), and puts back shares on resume; the runner's settle section (_settle_group) takes the commit lock under independent stepping
platoon/train/checkpoints/: commits.py (new), participants.py, barrier.py, resume.py, saving.py, kept_groups.py, dispatch_state.py, pruning.py Commits: every full save of any policy a commit: capture under the commit lock, which the ledger takes once the draws open (attach), once no step's end runs its step hooks (hook_phase: no commit saves a hook's state part-way through its work at another policy's step end, rule 10), with commit_seq, the participants' capture or checkpoint_state, every other policy's newest durable row and published version, pinned; write (a writing that fails closes those after it, unrun); append to the policy's own checkpoints.jsonl; the launch and exit barriers carry commit_seq and every policy's entry (published_version, step); RunCheckpoints.commit; newest_commit (the highest commit_seq whose named rows hold: a row deleted, or whose state path another row of its file holds too, is missing; newer ones are skipped and recorded in lock.json; a run saved in lockstep is refused by name), resumed_version (the version rule), next_commit_seq; StartPoint.read(..., independent=True) and PolicyStart.step; kept groups record shares and recorded; each pruner skips the pinned rows, checked again before each delete
platoon/train/: step/independent.py (new), trainer/clocks.py (new), step/microbatch.py, step/staleness.py, policy/set.py, trainer/loops.py, trainer/rl.py, trainer/run.py, trainer/state.py, trainer/waiting.py the rl trainer's loop per policy (PolicyLoop), each step a PolicyStep: the lockstep step with the policy's one lane, its shares from the ledger in place of the result queue, its batch, its clock, reverted shares judged at their recorded age, its consumption recorded under the lock, its own cadence (a commit, else weights only), one row of metrics; a stop request ends each loop at its step's end, and a loop that stopped tells the ledger; OwnClocks (trainer/clocks.py): each policy's clock, its own checkpoint cadence and the names of its commits (another policy's carry its id, recovery-<id>-<step>, apart from the barriers' rows in its file, which one Tinker session saves too), what the exit save records of each; run.clocks, run.end_step(policy=...) and run.publish publish and commit one policy on its own cadence (the run's _save_due with the policy whose step ends, also for end_step(sampled=False); the leader's recovery commit goes to the trainer's _recovery_saved, as a lockstep step's end hands its recovery save; the rl trainer's steps commit through own_step_sampler and hand their saves to their own pruners); a start checks the stepping rules against its trainer's needs (_check_stepping) and that its run's rows were saved with the same stepping (_check_resumed_stepping: a commit resumed by policies that step together exits 78); a start that resumes with the leader at the plan's end (a stop or a crash after its last commit) trains the other policies' last shares and saves the final checkpoint (_save_at_exit, OwnClocks.others_stepped)
platoon/train/trainer/hooks.py, platoon/train/evaluation/, platoon/utils/stats_logger.py StepEnd.policy: hooks run at the end of each policy's own step and may update that policy only (end.learner of another is refused, and the run's learners send nothing from a hook, run.learner of any policy); StepHooks.running, held while a step's end runs its hooks (nothing in lockstep, the commits' hook phase when stepping independently); evaluations record versions_end when a policy published while they ran, and log a row of their own; StatsLogger.log_row: a row without a W&B step of its own, each metric prefix declared once against its row's axis (define_metric), a row on the run's axis with time/elapsed_seconds
examples/two-agent-independent (new) the subagents example's delegator (main, batch 8) and its workers (worker, batch 2, bound 7, a commit at every 4th of its steps), about four worker steps per main step; its CPU test validates the config, runs it on the fake service, kills it once at the worker's second commit, resumes, checks each policy's steps, rows and that no policy's weights trained a row twice, and checks that a resume in lockstep stops with exit code 78
docs the how-to's Step each policy on its own clock (with its interrupts, failures and step hooks, and what a trainer of your own does) and rows of its metric tables for the new metrics, the lockstep section's refusal brought up to date, the step hooks' guide (hooks of different policies' step ends run at the same time; a hook's state across restarts follows each policy's own step, with an example), the generated references, the CHANGELOG

How a policy's step goes

 settle    credit once over every agent; under the commit lock: the estimator's settle, the record, the group kept and
           split into shares (one per trained policy with rows, and the leader's always), each queued for its policy
           and counted on its horizon
 dispatch  a group starts when the plan (the leader's remaining steps, train.batch_size groups each) and every
           policy's horizon have room
 step      policy p takes batch_p shares (reverted ones first), judges each against its version and bound (a reverted
           one at its recorded age), trains them through its learner
 hooks     drain, update policy p only (with step hooks; they hear p's publication after it)
 publish   records the shares consumed under the lock; p's own cadence: a commit (capture under the lock once no step's
           end runs its hooks, write, save, append to p's file) or weights only; then p publishes; its clock advances
           and its horizon moves on
 stop      each loop stops at its step's end; once a stopped policy's horizon is full, a step still waiting for shares
           closes short; the exit save names every policy
 resume    the newest commit whose named rows hold; each policy from its row, at published + 1 when that is past the
           row's; each share consumed after its row goes back first, judged at its age then; with the leader at the
           plan's end, the others train their last shares and the start saves the final checkpoint

Where the code differs from the design, and why

  • A group admission rule is refused under independent stepping. The design keeps admission at settle and counts
    refill.max_groups_per_step per policy; a rule's markers, caps and short steps are counted for steps of every
    policy at once in this release's dispatcher. Loading says so; lockstep keeps every rule.
  • Every group is a share of the leader's, one that holds none of its rows when it has none, which carries the
    group's report and record; a failed group, or one with no trained policy's rows, is the leader's share alone. The
    design says a group without rows of p is no share of p's, and that the plan stays in draws, end_step x train.batch_size groups: both cannot hold for the leader, whose steps carry the plan. A group without its rows would
    count toward no step of the plan, and the run would draw more groups than planned (the verification measured 25
    draws for a plan of 20); with no share at all a failed group would leave every horizon at once, and an outage would
    start groups as fast as they fail. As the leader's share, each takes a place of the leader's step, as a lockstep step
    takes such a group. The group's fate in the per-task statistics goes by the shares that hold rows (stale when all of
    theirs went stale, as a lockstep step judges it). A trainer of its own sees such shares: an empty rollout (None
    for a failed group).
  • main's metrics keep their names (the lockstep PR's choice); every other policy's row is today's names under
    policies/<id>/, and the new per-policy metrics are policies/<id>/... for every policy, main included.
  • A step records its consumption when it publishes, under the commit lock: after its step hooks drain and update
    (when it has any), just before its policy's own save and publication, rather than in the step's synchronous commit
    stage (PolicyStep.commit does nothing), since taking the lock awaits, and the lockstep step's commit stays
    synchronous, its calls pinned by the comparison below. A commit of another policy captured in between records the
    shares as not consumed and names this policy's previous row, so a resume from it trains them again, once.
  • A kept share that a commit records as not consumed is judged against its policy's version when it is taken. As
    the design says: only shares consumed after a policy's row keep their recorded age. A resume that bumps the version
    (published + 1) and retrains several shares first can make such a share stale under a tight bound; the tests check
    every share trains once per lineage and every share put back trains again, and the docs ask for small
    recovery_every.
  • An atomic trainer whose step is open at exit skips the exit save of every policy, as lockstep does, rather than
    only that policy's; the rl trainer's steps are not atomic, so it saves every policy at exit.
  • policies.<id>.checkpoint.every counts steps only, not epochs: a policy other than the leader has no epochs of
    its own. Left out, it is the run's checkpoint.every in steps.
  • Cross-policy updates from step hooks stay refused under independent stepping, as decided (deferred).
  • A stop request can close a step short. The design has each loop finish a step that submitted training work. A
    step that applied an update and waits for more shares when another policy's loop has stopped would wait for good once
    the stopped policy's horizon is full (its shares are never taken); starting groups beyond that horizon instead would
    give the stopped policy shares older than its bound after the resume. So the step closes short with the shares it
    has, as at the plan's end, and counts as a step of the plan, which then draws that many fewer groups.
  • A commit waits until no step's end runs its step hooks, then captures. The design captures under the commit lock
    with no await; a hook's state (keys) is captured there too, so without the wait a commit could save it part-way
    through another policy's step end, which rule 10 forbids. A commit's own step end publishes between its hooks'
    phases, so it never waits for itself; it waits for the others' hooks.
  • Loading reads train.stepping; each start checks the same rules against its trainer's needs. The design lets a
    trainer of your own answer the stepping from its own options. Loading cannot know the trainer, so it reads
    train.stepping (the rl trainer's answer, the only trainer a config selects in this release); each start checks the
    rules again against needs().stepping before any service opens (a policy's own batch or cadence under a trainer
    whose policies step together, a group admission rule or an undividable batch under one whose policies step
    independently), and refuses to resume a run whose rows were saved with the other stepping (exit 78). A trainer of
    your own whose needs say independent sets train.stepping: independent to give a policy its own batch or cadence;
    identity.json, which platoon train writes for the rl trainer, records train.stepping.
  • platoon.experimental exports one name, ShareLedger, a protocol of what a trainer calls: take (which, as
    the design's, returns the shares as settled groups), judge_version, consumed, done and lock. A settled group
    carries no age, so judge_version gives the version a trainer that drops stale shares judges each at: a share a
    resume put back has the age it had when first consumed, as the rl trainer's steps judge it. The ledger's other methods
    (put_nowait for the dispatcher, durable, revert, snapshot, ...) belong to the run's implementation,
    Shares, which is not exported; take gains stop=, as the source's take has.
  • A step hook's state is saved as of each commit, for every policy at once. A commit of any policy captures a hook's
    state (keys) as it is then, while each other policy resumes from the older row the commit names and runs its step
    ends since that row again: state that moves at a policy's step ends would be ahead of that policy's weights. The
    step hooks' guide says so and shows the pattern, keeping such state by each policy's own step and taking it up at
    start.policies[<id>].step in resume(); a test runs that pattern across a kill. A per-policy hook state that the
    run keeps for each commit is left for later.
  • A policy other than the leader names its commits with its id (recovery-<id>-<step>, <id>-<step>): the
    barriers' rows in its file carry the leader's step, one Tinker session saves both, and a name of its own step alone
    could save over a barrier's row when it is behind the leader. A resume also refuses to name a row whose state path
    another row of its file holds.

With train.stepping unset

New piece What a run that does not set it does
train.stepping, TrainerNeeds.stepping lockstep: one clock (run.clocks maps every policy to it)
policies.<id>.batch_size, .checkpoint refused, as before (the message now names train.stepping: independent)
horizons, ledger, commits, StepClock.policy, StepEnd.policy not built; None
RunCheckpoints.save no commit_seq, the same entries
Dispatcher the same constructor call (no horizons argument), counters and checks; bind iterates a copy of the kept groups
the runner's settle section the same calls, in a function of its own (_settle_group), no lock
the step the same calls; its publication through TrainStep._publish
identity.json no stepping key
W&B rows logged with log, as before
the start's checks stepping_problems with independent=False and the resumed row's commit_seq looked up, both pure, at phases 1 and 3
step hooks (none in the golden) with hooks, StepHooks.running is contextlib.nullcontext, and the run's learners check the run's clock alone, as before
the keep_recovery check at loading reads train.stepping: every policy's bound against the run's recovery_every, as before

Tests

  • tests/train/policy/test_stepping.py: the key, a policy's own keys and every refusal (an unknown key of its cadence
    by name, a keep_recovery that its own recovery_every does not span past its bound), the warning for a policy that
    saves no full checkpoint before the run's end, the rl trainer's answer, identity.json and a resume that changes
    train.stepping (exit 78 through platoon train in the example's test).
  • tests/train/dispatch/test_shares.py: the split (a group without the leader's rows a share of the leader's with none,
    counted toward the plan, its fate its rows'), the fate recorded once, retirement, revert with recorded ages, the
    plan's end, a trainer's take stopped by a stop request (the shares it took stay unconsumed), judge_version (the
    version now for a fresh share, the recorded age's for one a resume put back), a queue that ends once a stopped
    policy's full horizon holds every group back, the horizons and their blocked shares.
  • tests/train/checkpoints/test_commits.py: the highest commit_seq whatever the order of the appends, the next number
    after it, a deleted named row skipped and recorded, a commit naming a state path that two rows of a file hold
    skipped, a run saved in lockstep refused, the version rule, pins while in flight and once newest, a commit whose
    writing fails (the writings after it closed, unrun; nothing pinned), and the ledger taking the commit lock that a
    commit waits on for the step hooks, also once a step's end ran its hooks before the draws opened.
  • tests/train/trainer/test_independent_run.py (two fake services, tests/train/fakes/independent.py): stepping 1:4,
    each policy's own rows and cadence (the worker's commits named with its id), each share trained once in each policy's
    weights, groups without main's rows counted toward the plan (the run draws its 12 groups, each row trained once into
    its policy), the metrics' rows and axes, a resume from the newest commit, an interrupt with one policy's step open,
    and with a step of two updates whose shares the stopped policy's horizon holds back (it ends, closing the step short;
    the resume trains each share once, none stale), a stop in main's last step with a slow worker (the resume, main at
    the plan's end, trains the worker's last shares and saves final), step hooks on a policy's own step and the refusal
    of another, through its end or through run.learner (under the rl trainer and a trainer of its own, whose own steps
    send through run.learner meanwhile), no commit saving a hook's state part-way through another policy's step end, a
    hook whose state follows each policy's own step across a kill after a commit that captured it past the worker's row
    (each of its examples trains once, in order; also keeping only the cursors a commit can name, at most its cadence's
    and one in every row), a trainer of its own through the ledger and stopped through take(stop=...), one that drops
    stale shares at judge_version across a resume whose commit names the worker's row four or more versions old (no
    share put back is dropped), a lockstep trainer's refusal of a policy's own batch, a start that keeps its run's
    stepping (a commit resumed by policies that step together, and a lockstep run by independent ones, exit 78), and a
    group admission rule refused against a trainer's needs.
  • tests/train/checkpoints/test_commits_run.py (two fake services, whose clients remember the lineage of each saved
    state): a kill at every point between consume, capture, write, save and append, of either policy (each share once per
    lineage, every put-back share trained again), also of a worker slower than main after a resume (its commits never save
    over its launch row); a commit that names the worker's row four of its steps old (every share put back trains again
    first, none dropped as stale); a kill right after main's last commit with a slow worker (the resume saves final,
    every share once, and a start after it trains nothing); a pruner that runs while a commit is in flight (the named row
    survives; a resume skips a commit whose named row was deleted, and records it); commits appended out of commit_seq
    order (the resume takes the highest; the next commit takes the number after it); a stateful credit estimator killed
    between a group's credit and its settling (each group settled once).
  • tests/examples/test_two_agent_independent_example.py, tests/train/evaluation/test_eval_versions.py
    (versions_end), tests/utils/test_stats_logger_rows.py (the run's rows carry the elapsed time); the lockstep, step
    hook, lifecycle, layers and credit workflow tests follow the new fields (StepEnd.policy, run.clocks, the
    source's ledger, trainer/clocks.py) and the lockstep refusal's new message, and the sft trainer's refusal of the
    train: keys only the rl trainer reads includes train.stepping.

Checks

  • The golden trace: tests/train/parity passes, its 52 golden files byte-identical.
  • The comparison against the base, with no new key set, over the same 6,669 variants of the golden scenarios as the
    earlier refactors (faults and stops included), run twice on the base: every eqtrace event (the trainer's calls, the
    dispatcher's, the statistics', the trackers', each await chain), logger names included, the golden-style files and
    the refactored code's external calls are the same, in the same order. What differs is calls of one pure function,
    getattr: the steps and the run asking whether the run has a clock per policy (run.clocks, the source's ledger
    and horizons). The await chains leave out the frames of the modules this PR changes (the dispatcher, waiting.py,
    the step hooks) and of its new ones, as the earlier comparisons did for theirs: their line numbers moved.
  • The lockstep runs of two policies, the previous PR's run tests (a resume of both, a hook that updates the worker, a
    worker's late datum, a worker without rows, two starts) and its example under both configs, run on the base and
    here: their checkpoint rows, metrics, publications and what each policy trained are the same, but for timings and
    the checksums of snapshot files.
  • The CI-equivalent runner: lint, core on Python 3.12, 3.13 and 3.14 (8,515 passed in each, and the serial tests), the
    extensions, the recipes, the docs (the generated references and the public API's snapshot up to date) and the
    release build. The full suite passes (8,716) but for one test that needs a git checkout, which fails on the base
    the same way in that runner.
  • ty reports nothing new, the dependency rule holds, every patch a test makes still reaches the code it patches
    (359), every name of the 0.4.0 module paths still imports through them, and a patch made through an old path right
    after its import reaches the policies' loops too (platoon.train.step.independent is imported with the trainer).
    Every function touched stays within 60 lines but Dispatcher.__init__, 95 lines at the base and 100 here (its
    horizons argument).
  • The new test cases (77): under tests/train/, policy/test_stepping.py 14, dispatch/test_shares.py 10,
    checkpoints/test_commits.py 8, checkpoints/test_commits_run.py 20, trainer/test_independent_run.py 19 and
    evaluation/test_eval_versions.py 1; tests/utils/test_stats_logger_rows.py 2; the example's 3 (and its config in
    the examples' no-defaults check). The lockstep refusal's test takes its new name. Each test module this PR touches
    also passes alone, from the repository's root and from tests/. The whole-start modules (the commits', the
    independent runs', the example's, the ledger's) passed 16 times in a row with 16 at once on one node, and 4 times
    under pytest -n 8.
  • The review's findings, each with a test that fails on the code before the fix (16 of the new cases fail there; two
    variants of the slow worker's kill, at its consumption and before its save, reach no collision and pass there too),
    and the reviewers' own probes run again here: a worker slower than main whose own commit saved over its launch row
    (12 shares trained twice after a resume) now names its commits apart; an interrupt in a step of two updates whose
    shares the stopped policy's horizon held back no longer hangs (32 stop points, each resumed: none hangs, none trains
    a share twice, none drops one as stale); no commit saves a step hook's state part-way through another policy's step
    end; a trainer's needs that say independent with train.stepping at lockstep are held to the same rules, and a
    lockstep resume of its run stops with exit code 78; a trainer of its own that takes shares with stop=run.shutdown
    stops on a stop request; the run's rows carry time/elapsed_seconds; an unknown key of a policy's cadence is refused
    by name.
  • The first verification's findings, each with a test that fails on the code before its fix (6 of the 7 new cases fail
    there; the hook test passes there, since its fix is the guide's pattern, and fails with a hook that takes its state up
    as the commit saved it), and the verification's probes run again here, on two and three fake services, without the
    probe-only patch it used:
    • A resume with main at the plan's end, after a kill, a stop or a failure that came after main's last commit
      with another policy behind, now trains that policy's last shares and saves final: 1,114 kills (32 leave no
      final row, each a resume with nothing left to train, as the base's run of one policy), 357 stops, 155 failures,
      21 runs of a trainer of its own and the 3 leader-at-the-end cases each train every share exactly once.
    • A trainer of its own that drops stale shares at judge_version trains every share a resume put back (7 of 7,
      each of which it dropped before; judged at the version now, the new test fails).
    • A hook that keeps its state by each policy's own step trains each of its examples once across 7 kills of three
      policies; one that takes its cursor up as saved loses examples in each, the hazard the guide now describes.
    • The metric names the docs give are those logged (train/dispatch/blocked_share/horizon:<id>,
      train/workload/*), and the how-to's metric tables list the new ones.
    • Cheaper fixes: take's docstring (fewer shares after a stop too), the policy spec's comments on batch_size and
      checkpoint (the reference had none for checkpoint), one commit lock that the ledger takes (Commits.attach;
      the condition a commit waits on was bound to the lock of the moment), and a commit whose writing fails closes the
      writings after it, unrun. Left: PolicyStep.commit does nothing (above), and the private names
      step/independent.py uses from other modules, which the open cleanup PR renames or moves.
  • The second verification's findings, each with a test that fails on the code before this fix (5 of the 7 new cases fail
    there; the two cases of the hook-state test pass there, since that fix is the guide's example, which the new case's
    hook follows), and the verification's probes that found them run again here:
    • checkpoint.keep_recovery was checked against the run's recovery_every for every policy, while each policy's
      pruner keeps its own bound at its own cadence: a worker saving every 3 of its steps with bound 7 and keep 3 was
      refused, and one saving every step with bound 3 and keep 1 under the run's cadence of 4 was accepted, its pruner
      then keeping more than keep_recovery. Stepping independently, the check now counts a policy's own
      recovery_every, against its own bound (else the run's largest), and names
      policies.<id>.checkpoint.recovery_every.
    • A group without main's rows counted toward no step of the plan, so the run drew more groups than planned (17 for a
      plan of 12 in the new test, 25 for 20 in the probes). Every group is now a share of main's (above): in the probes'
      runs whose every fifth group has no rows of main, killed or not, the run draws its plan exactly (20 of 20, 16 of
      16) and every drawn group with a policy's rows trains into that policy once; a step of main whose shares hold none
      of its rows takes no update, as a lockstep step of such groups.
    • A step hook at one policy's step end could still reach another policy through run.learner, whose guard looked at
      that policy's clock only: the run's learners now send nothing from any step end's hooks (RuntimeError), under the
      rl trainer and a trainer of its own, whose own steps send through run.learner meanwhile as before.
    • The step hooks' guide kept a cursor of every step of the policy forever, which every commit wrote: its example now
      drops those no commit can name any more, older than the policy's full-checkpoint cadence.
    • Cheaper: the age rule of a share put back has one home (judge_version and Share.judged_at used it twice), and
      the policy's view of the loops' state says why its leader is the policy whose steps run. Left: PolicyStep.commit
      does nothing (above).
    • The probes again, on two and three fake services: 897 chains (1,745 starts, 784 kills at a point and 82 at a
      random time) of step hooks, trainers of their own, a change of stepping, groups without main's rows, the bypass,
      and pruners that delete as commits run (keep_recovery: 2): none trains a share twice or loses one, none judges a
      share put back stale, no commit captures a hook part-way, and the final per-task statistics count each planned
      group once. What the checks flag the base does too: launch rows written after final by a start planned after the
      run ended (35), states a pruner deleted after a start opened them (568), and runs killed after main's last commit
      with nothing left to train (5). The verification's prune probes with keep_recovery: 1 stay refused at loading, by
      the run's own rule for main.
  • After the rebase onto the replayed lockstep commit, whose step's end saves through the run's _save_due (a trainer
    of its own commits there now), the verification's probes of the save and commit paths ran again: the commits'
    crafted run directories, trainers of their own, out-of-order appends, a change of stepping, and samples of the kills
    and the stops, 1,167 chains (2,257 starts, 823 kills at a point and 20 at a random time). None trains a share twice
    or loses one, none judges a share put back stale, and the final per-task statistics count each planned group once;
    what the checks flag the base does too (13 launch rows after final, 262 states a pruner deleted after a start
    opened them, 7 runs killed after main's last commit with nothing left to train).

@ApGa
ApGa force-pushed the apga/train-lockstep branch from 8cf1fa1 to 09feb74 Compare October 28, 2026 23:06
Several trained policies stepped together: each step trained every
policy on its rows of the same groups, and each checkpoint saved them
all, so a policy that trains on few rows, or saves fast, waited for the
slowest (RFC 0021's lockstep). train.stepping: independent lets each
trained policy step, publish and save on its own clock (RFC 0021,
section 2.11, and RFC 0003's milestone M2), with its own batch
(policies.<id>.batch_size, else train.batch_size), staleness bound and
checkpoint cadence in its own steps (policies.<id>.checkpoint: {every,
recovery_every}, else checkpoint.*). lockstep stays the default, and
with one trained policy both are the same. A trainer's needs say which
clocks the run builds (TrainerNeeds.stepping; the rl trainer answers
with train.stepping), and run.clocks holds each policy's.

Each group is credited once, over every agent, and then splits into a
share per trained policy with rows in it, and always one of the
leader's: that policy's rows of the group (none of the leader's in a
group without them, so that the leader's steps take every group and the
plan stays in draws, train.batch_size groups per leader step; a group
that failed is the leader's share alone). The run's share ledger
(dispatch/shares.py) takes the result queue's place: each policy's steps
take its shares in the order they settled, judge them against its own
version and bound, train them with its own learner, record them consumed
with their age and fate, and publish that policy alone, after each of
its steps. The rl trainer runs one loop per policy, each step the
lockstep step with the policy's one lane (step/independent.py); a
trainer of its own takes shares through platoon.experimental.ShareLedger
(take, which stops on run.shutdown; judge_version, the version a share
that a resume put back is judged at; consumed under its lock; done) and
ends each step with run.end_step(policy=...). The dispatcher keeps a
horizon per policy, (k + 1) x batch of its shares, and starts a group
only when every one has room (dispatch/horizons.py); the leader's
remaining steps bound the plan, as in lockstep.

Every full save of any policy is a commit (checkpoints/commits.py): it
captures, under a commit lock that each group's settle section and each
step's consumption take too, and once no step's end runs its step hooks
(so that it saves no hook's state part-way), the next commit_seq, the
run's draws (the groups not yet retired, with what each policy did with
its share) and every other policy's newest durable row and published
version, which stay pinned against the pruners; then writes them and the
policy's weights; then appends the policy's row to its own
checkpoints.jsonl, naming every policy's row by its state_path. Another
policy than the leader names its commits with its id
(recovery-<id>-<step>), apart from the barriers' rows in its file, and a
resume refuses a state path that two rows of a file hold
(trainer/clocks.py keeps each policy's clock and cadence). A group stays
kept until each policy that consumed its share has saved after that
step. A resume takes the commit with the highest commit_seq whose named
rows hold (a deleted one is skipped and recorded in lock.json), each
policy going on from the row it names, at one version more than the
commit records it published when that is past the row's; each share it
consumed after that row trains again first, judged at the age it had.
commit_seq goes on after the highest in any file, and the launch and
exit saves are barriers of every policy whose leader row is a commit.

Each step logs a row of its own without a W&B step (StatsLogger.log_row,
which declares each prefix's axis): main's with its names and run/step,
every other policy's under policies/<id>/ with policies/<id>/step;
policies/<id>/share_wait_s, retrained_shares and peer_lag/<q>,
run/retained_groups and train/dispatch/blocked_share/horizon:<id> are
new. Evaluations count main's steps and record versions_end when a
policy published while they ran. Step hooks run at the end of each
policy's own step (StepEnd.policy) and may update that policy only,
through their end (the run's learners send nothing from a hook); a
commit saves a hook's state as of its capture while the other policies
resume from older rows, so a hook keeps what moves at a policy's step
ends by that policy's step (docs/extending/step-hooks.md shows how). A
stop request ends each loop at its step's end; once a stopped policy's
horizon is full, a step that still waits for shares closes short, and
the exit save names every policy. A start that resumes with the leader
at the plan's end trains the other policies' last shares and saves the
final checkpoint.

Loading refuses a policy's batch_size or checkpoint in lockstep (the
refusal named this milestone; it now says to set train.stepping:
independent), on a frozen policy, a batch that the step's minibatches
and microbatches do not divide, a group admission rule under independent
stepping, and a checkpoint.keep_recovery that a policy's own
recovery_every (else the run's), in its own steps, does not span past
its bound, and warns of a trained policy that saves no full checkpoint
before the run's end; each start checks the same against its trainer's
needs. identity.json records stepping: independent, and a resume that
changes it, or whose rows were saved with the other stepping, stops with
exit code 78. examples/two-agent-independent steps the subagents
example's worker about four times for each of its delegator's steps,
with a CPU test that kills it once and resumes. With train.stepping
unset every run is unchanged: the golden trace is byte-identical.
@ApGa
ApGa force-pushed the apga/train-independent branch from d3b3cd9 to c0598db Compare October 29, 2026 01:24
@ApGa
ApGa changed the base branch from apga/train-lockstep to main October 29, 2026 01:24
@ApGa
ApGa merged commit 4709779 into main Oct 29, 2026
12 checks passed
@ApGa ApGa mentioned this pull request Oct 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant