Skip to content

fix(durable): keep timers across recovery and skip idle ticks - #236

Draft
reka18 wants to merge 1 commit into
fix/durable-effect-dispatchfrom
fix/durable-timers
Draft

reka18 wants to merge 1 commit into
fix/durable-effect-dispatchfrom
fix/durable-timers

Conversation

@reka18

@reka18 reka18 commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

What this change does

Timers survive recovery, and idle ticks stop growing the journal. Stacked on #235.

  • Start arms the initial state's timers and carries them in the baseline. An instance started before this change keeps its initial timer unarmed.
  • Recovery re-arms checkpointed timers before the tail replays. A held replay clock keeps that arming from consuming a recorded reading.
  • RunService records the clock read that arms a timer, and replay arms it, in both Recover and StateAt.
  • The Handle's timer table follows the scheduler: after a tick it keeps only what the scheduler still holds, and a chained timer takes the deadline the scheduler armed, after the transition's actions ran.
  • A tick that fires nothing records nothing. This stays in durable, so state is untouched.
  • Replay reads an older journal's idle tick Record as nothing fired, so a re-armed timer can't fire there. An older journal left a settle's clock read at the front of the next Record, so replay aligns on each Record: a step's own reads are the last ones in it. Two kinds of older journal can't be repaired, and can recover differently or fail Recover: an instance that continued after an older recovery missed a fired timer, and a checkpoint taken after a tick that followed a settle or whose clock moved mid-tick, which can hold a wrong timer table. No reader can tell these journals apart. An idle Tick doesn't touch the Store, so a stale owner finds out on its next real step, not on the idle tick.

StateAt also re-arms the baseline's timers, so a timer armed by Start or carried by a checkpoint fires in the replayed window.

Known and left open: a timer whose transition fails is consumed and reported only in its FireResult.Err, not in Tick's error (#214 question 8), which the Tick godoc now says. Checkpoints still grow with trace history, and the size metric #214 question 9 proposes has no home yet. DeliverToActor passes no effects to the scheduler, so a state entered by delivery never arms its timer. Tick doesn't absorb service or actor effects, so a timer entering an invoke state doesn't start the service live, and a timer that leaves and re-enters one can make Recover fail. Both exist on the base.

Why

Start never armed the initial state's timers, even live, so an instance starting in a timed state now fires a timer it didn't before. A timer fired after a checkpoint, or armed by a service result, was missed on replay. Every idle tick appended a Record.

Relates to: #214

Evidence

On GOTOOLCHAIN=go1.27.3, in durable:

Command Result
go test ./... Pass, including TestClock_StartInTimedState_ArmsTimer, TestClock_TickWithNothingDue_AppendsNoRecord, TestClock_TickWithNoTimer_AppendsNoRecord, TestService_TimerArmedBySettle_ReplaysOnRecover and TestTimerCheckpoint_FiredAfterCheckpoint_ByteIdentical
go test -race ./... pass, 0 race warnings
Pinned golangci-lint 0 issues

Checklist

  • Commits are signed off (git commit -s) per the DCO
  • Conventional commit messages (type: subject)
  • mage check passes locally (lint + race tests + govulncheck): not run as a whole
  • Tests added/updated for the change: TestStateAt_TimerArmedAtStart_Fires, the older-journal TestRecover_OlderJournal_* tests, and the timer tests above
  • Public API changes documented (godoc) and, if applicable, noted for the changelog

@reka18 reka18 added not-ready On every new PR until all review dimensions are confirmed wait:correctness ACR pending: correctness wait:regression-safety ACR pending: regression-safety wait:concurrency-discipline ACR pending: concurrency-discipline wait:replay-determinism ACR pending: replay-determinism wait:api-compatibility ACR pending: api-compatibility wait:missed-context ACR pending: missed-context wait:human-readability ACR pending: human-readability labels Nov 5, 2026
@reka18
reka18 force-pushed the fix/durable-timers branch from 29736b8 to c6e5aa6 Compare November 5, 2026 16:38
@reka18
reka18 force-pushed the fix/durable-timers branch from c6e5aa6 to f4697dc Compare November 5, 2026 16:40
@reka18
reka18 force-pushed the fix/durable-timers branch from f4697dc to e653227 Compare November 5, 2026 16:53
@reka18
reka18 force-pushed the fix/durable-timers branch from e653227 to 8db4d1e Compare November 5, 2026 17:09
@reka18
reka18 force-pushed the fix/durable-timers branch from 8db4d1e to cb8ca81 Compare November 5, 2026 17:14
@reka18
reka18 force-pushed the fix/durable-timers branch from cb8ca81 to 6e0908c Compare November 5, 2026 17:22
@reka18
reka18 force-pushed the fix/durable-timers branch from 6e0908c to bd0e889 Compare November 5, 2026 17:34
@reka18
reka18 added this pull request to stack #238 November 5, 2026 17:34
@reka18
reka18 force-pushed the fix/durable-timers branch from bd0e889 to 189e001 Compare November 5, 2026 17:42
@reka18
reka18 force-pushed the fix/durable-timers branch from 189e001 to c02481a Compare November 5, 2026 18:04
@reka18
reka18 force-pushed the fix/durable-timers branch from c02481a to a03ba52 Compare November 5, 2026 18:20
@reka18
reka18 force-pushed the fix/durable-timers branch from 7c4e075 to a9f4db1 Compare November 5, 2026 19:19
@reka18
reka18 force-pushed the fix/durable-timers branch 2 times, most recently from fe18571 to ac8edcd Compare November 5, 2026 19:39
@reka18
reka18 force-pushed the fix/durable-timers branch from ac8edcd to e509532 Compare November 5, 2026 19:48
@reka18
reka18 force-pushed the fix/durable-timers branch from e509532 to 39d8e39 Compare November 5, 2026 20:10
@reka18
reka18 force-pushed the fix/durable-timers branch from 39d8e39 to 1738b1a Compare November 5, 2026 20:23
@reka18
reka18 removed this pull request from stack #238 November 5, 2026 20:26
@reka18
reka18 added this pull request to stack #239 November 5, 2026 20:26
@reka18
reka18 force-pushed the fix/durable-timers branch from 1738b1a to 372a253 Compare November 5, 2026 20:52
@reka18
reka18 force-pushed the fix/durable-timers branch from 372a253 to b33de98 Compare November 5, 2026 21:02
@reka18 reka18 added ready All ACR dimensions confirmed - clear to merge confirmed:correctness ACR clean: correctness confirmed:regression-safety ACR clean: regression-safety confirmed:concurrency-discipline ACR clean: concurrency-discipline confirmed:replay-determinism ACR clean: replay-determinism and removed not-ready On every new PR until all review dimensions are confirmed wait:correctness ACR pending: correctness wait:regression-safety ACR pending: regression-safety wait:concurrency-discipline ACR pending: concurrency-discipline wait:replay-determinism ACR pending: replay-determinism wait:api-compatibility ACR pending: api-compatibility labels Nov 5, 2026
@reka18

reka18 commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor Author

Review results. One reviewer per dimension, each reading the others' findings. All seven dimensions have zero blockers on the latest round.

Fixed along the way: the Handle's timer table now follows the scheduler, and each chained timer takes the deadline the scheduler armed. Replay aligns on each Record, so older journals with a misplaced settle read recover, including across a restart. Every fix has a test confirmed to fail without it.

Fuzzing journals this PR writes found no mismatch in about 80,000 operations per run.

Known limits, in the changelog: two kinds of older journal can't be repaired and can fail Recover. A failed timer transition is reported only in its FireResult (#214 question 8).

Start now arms the initial state's timers and carries them in the baseline.
Recovery re-arms checkpointed timers before the tail replays, so one fired in
the tail isn't missed. RunService records the clock read that arms a timer, and
replay arms it. A tick that fires nothing records nothing, so a timer loop no
longer grows the journal.

Relates to #214.

Signed-off-by: Ray Karnes <ray.karnes@stablekernel.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

confirmed:api-compatibility ACR clean: api-compatibility confirmed:concurrency-discipline ACR clean: concurrency-discipline confirmed:correctness ACR clean: correctness confirmed:human-readability ACR clean: human-readability confirmed:missed-context ACR clean: missed-context confirmed:regression-safety ACR clean: regression-safety confirmed:replay-determinism ACR clean: replay-determinism ready All ACR dimensions confirmed - clear to merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant