Skip to content

fix(durable): carry the next step through a checkpoint - #232

Merged
reka18 merged 1 commit into
mainfrom
fix/durable-step-numbering
Nov 9, 2026
Merged

reka18 merged 1 commit into
mainfrom
fix/durable-step-numbering

Conversation

@reka18

@reka18 reka18 commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

What this change does

The next step ordinal survives a checkpoint that compacts the whole tail. The checkpoint envelope now records the step that follows it, and recovery uses the last tail Record, then the envelope, then the old Trace count. Stacked on #230.

A checkpoint written before this change keeps the old fallback, which can still reuse a step. durable has no release tags, despite the [0.1.0] changelog heading, so this should only reach unreleased checkpoints.

Why

Recovery derived the next step from the Trace count, which doesn't match the ordinals past a tick barrier. The next Fire reused a recorded step. This also removes the hard failure #229 gave recovery from a compacted checkpoint.

Relates to: #214

Evidence

On GOTOOLCHAIN=go1.27.3, in durable:

Command Result
go test ./... Pass, including TestRecover_StepAfterTickCheckpoint_IsPersisted
go test -race -skip TestRunner_FireDuringTimerLoop_LosesNoAcknowledgedSteps ./... 0 race warnings
Pinned golangci-lint 0 issues

Checklist

  • Commits are signed off (git commit -s) per the DCO
  • Conventional commit messages (type: subject)
  • mage check passes locally (lint + race tests + govulncheck): not run as a whole, and it cannot pass until the rest of the chain lands
  • Tests added/updated for the change
  • Public API changes documented (godoc) and, if applicable, noted for the changelog

@reka18 reka18 added not-ready On every new PR until all review dimensions are confirmed wait:correctness ACR pending: correctness wait:regression-safety ACR pending: regression-safety wait:concurrency-discipline ACR pending: concurrency-discipline wait:replay-determinism ACR pending: replay-determinism wait:api-compatibility ACR pending: api-compatibility wait:missed-context ACR pending: missed-context wait:human-readability ACR pending: human-readability labels Nov 5, 2026
@reka18

reka18 commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor Author

ACR round 1: correctness

Reviewed the step-numbering commit. 0 blockers, so correctness is confirmed.

Checked clean: 1,550 randomized Fire/Tick/Recover runs on MemStore and FileStore across checkpoint intervals, with exact step continuity. Service and single-Trace actor recovery, append-then-failed-checkpoint recovery, FileStore reopen, StateAt across compaction, and envelope byte determinism.

Non-blockers, folded in:

  • The comments and changelog said the Trace count undercounts. It can also overcount, on a multi-Trace actor delivery. Reworded.
  • The changelog overstated ErrStepConflict: a reused step with an identical record is still dropped as a retry. Reworded.

Not fixed here:

  • A checkpoint written before this change can still lose a step through the old fallback. durable has no release tags, so this is limited to unreleased checkpoints.
  • checkpointDue can be skipped forever when tick barriers keep stepping over the interval (every 8, barriers on even steps: 50 ticks, no checkpoint). This predates the change and isn't in durable: defects found in the platform-redesign review #214.
  • After a failed Checkpoint, the same Handle's next step could conflict. Fixed by the lock PR, which is stacked on this one.

@reka18 reka18 added confirmed:correctness ACR clean: correctness and removed wait:correctness ACR pending: correctness labels Nov 5, 2026
@reka18
reka18 force-pushed the fix/durable-step-numbering branch from fa0fe3f to 69ef146 Compare November 5, 2026 16:38
@reka18
reka18 force-pushed the fix/durable-step-numbering branch from 69ef146 to aeac33f Compare November 5, 2026 16:40
@reka18
reka18 force-pushed the fix/durable-step-numbering branch from aeac33f to 04b5e9d Compare November 5, 2026 16:53
@reka18
reka18 force-pushed the fix/durable-step-numbering branch from 04b5e9d to b1c7167 Compare November 5, 2026 17:09
@reka18
reka18 force-pushed the fix/durable-step-numbering branch from b1c7167 to 5d7a713 Compare November 5, 2026 17:14
@reka18
reka18 force-pushed the fix/durable-step-numbering branch from 5d7a713 to 3d338cd Compare November 5, 2026 17:22
@reka18
reka18 force-pushed the fix/durable-step-numbering branch from 3d338cd to 8b74a6b Compare November 5, 2026 17:33
@reka18
reka18 added this pull request to stack #238 November 5, 2026 17:34
@reka18
reka18 force-pushed the fix/durable-step-numbering branch from 8b74a6b to de02c29 Compare November 5, 2026 17:42
@reka18
reka18 force-pushed the fix/durable-step-numbering branch from b72d6ea to 595af41 Compare November 5, 2026 18:56
@reka18
reka18 force-pushed the fix/durable-step-numbering branch from 595af41 to 4a47e7a Compare November 5, 2026 19:04
@reka18
reka18 force-pushed the fix/durable-step-numbering branch from 4a47e7a to 208860a Compare November 5, 2026 19:05
@reka18
reka18 force-pushed the fix/durable-step-numbering branch from 208860a to cf71918 Compare November 5, 2026 19:19
@reka18
reka18 force-pushed the fix/durable-step-numbering branch from cf71918 to d760291 Compare November 5, 2026 20:23
@reka18
reka18 removed this pull request from stack #238 November 5, 2026 20:26
@reka18
reka18 added this pull request to stack #239 November 5, 2026 20:26
@reka18
reka18 force-pushed the fix/durable-step-numbering branch from d760291 to 02aa8da Compare November 5, 2026 21:02
@reka18

reka18 commented Nov 5, 2026

Copy link
Copy Markdown
Contributor Author

Update since review. #213 is closed. This PR now carries the tests it turns green, and it's stacked on #231 in stack #239. The code under review didn't change, and durable passes with -race and lint at this head.

@reka18
reka18 force-pushed the fix/durable-step-numbering branch from 02aa8da to 9f0db01 Compare November 5, 2026 21:25
@reka18
reka18 force-pushed the fix/durable-step-numbering branch from 9f0db01 to ad2b6c6 Compare November 5, 2026 22:00
@reka18
reka18 force-pushed the fix/durable-step-numbering branch 2 times, most recently from 831afc6 to aef9065 Compare November 5, 2026 22:17
@reka18
reka18 force-pushed the fix/durable-step-numbering branch from aef9065 to 4ac4984 Compare November 6, 2026 15:41
Base automatically changed from fix/durable-event-data to main November 9, 2026 15:52
Recovery derived the next step from the restored Trace count when a checkpoint
had compacted the tail. A tick barrier consumes an ordinal but leaves no Trace,
so the count fell short and the next Fire reused a recorded step. The checkpoint
envelope now records the step that follows it. A checkpoint written before this
change keeps the old fallback.

Relates to #214.

Signed-off-by: Ray Karnes <ray.karnes@stablekernel.com>
@reka18
reka18 force-pushed the fix/durable-step-numbering branch from 4ac4984 to 8aa8edf Compare November 9, 2026 15:52
@reka18
reka18 marked this pull request as ready for review November 9, 2026 16:03
@reka18
reka18 requested a review from joshua-temple November 9, 2026 16:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

confirmed:api-compatibility ACR clean: api-compatibility confirmed:concurrency-discipline ACR clean: concurrency-discipline confirmed:correctness ACR clean: correctness confirmed:human-readability ACR clean: human-readability confirmed:missed-context ACR clean: missed-context confirmed:regression-safety ACR clean: regression-safety confirmed:replay-determinism ACR clean: replay-determinism ready All ACR dimensions confirmed - clear to merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants