Skip to content

test(durable): add failing tests for recovery, timer and effect defects - #213

Closed
reka18 wants to merge 1 commit into
chore/bump-go-and-deps-for-vulnsfrom
test/durable-defect-repros
Closed

reka18 wants to merge 1 commit into
chore/bump-go-and-deps-for-vulnsfrom
test/durable-defect-repros

Conversation

@reka18

@reka18 reka18 commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Durable hardening: failing tests ahead of the fix checkpoints

Failing tests for #214 (defects from the platform-redesign review of durable; root causes and open questions are on the issue). Rows follow your ruled fix order. Each test asserts what any correct fix must produce, so it fails on main and moves into the checkpoint that fixes it.

Order Defect Test
Fencing A stale Handle and the Handle that recovered after it both get acknowledged Fires; recovery yields a third state TestRecover_TwoHandlesOnOneInstance_JournalMatchesAnAcknowledgedHistory
Fencing A Fire whose Append failed stays in the live instance, so the next acknowledged Fire diverges from the journal TestRunner_FireAfterFailedAppend_MatchesJournal
Concurrency Fire during the "Tick from its own timer loop" pattern loses acknowledged steps (2+ CPUs) TestRunner_FireDuringTimerLoop_LosesNoAcknowledgedSteps
Replay WithEventData is dropped on recovery TestRecover_FireWithEventData_MatchesLive
Replay A result from a canceled turn is committed on recovery TestService_ResultFromCanceledTurn_NotCommittedOnRecover
Effects Effects from Start, timer transitions and service results never reach the handler (DeliverToActor has the same gap) TestRunner_EffectDispatch_{Start_AppliesEntryEffect, TimerTransition_AppliesEffect, ServiceSettle_AppliesEffect}
Effects A failed dispatch is compacted away by the next checkpoint TestRunner_EffectDispatch_FailedThenCheckpointed_AppliedOnce
Timers Start doesn't arm the initial state's timer TestClock_StartInTimedState_ArmsTimer
Timers A checkpointed timer that fires in the tail is missed on replay TestTimerCheckpoint_FiredAfterCheckpoint_ByteIdentical
Timers A timer armed by a service result is lost on replay TestService_TimerArmedBySettle_ReplaysOnRecover
Step reuse After a checkpoint compacts tick barriers, the next Fire reuses a recorded step and is dropped with a nil error TestRecover_StepAfterTickCheckpoint_IsPersisted
Growth Ticks before a timer is due append a record each TestClock_TickWithNothingDue_AppendsNoRecord

timerNotifyMachine's timer edge now also emits notify. The existing test using it never ticks.

Decisions the tests leave open

  1. Fencing: takeover on Recover or optimistic concurrency on the step (both pass). A Recover that refuses while a Handle is held would need a release API.
  2. Append first-writer-wins: keep, or reject conflicting content for a recorded step.
  3. Store without a dispatch set: keep the no-op, or refuse the configuration.
  4. Recover while the effect handler is down: fail as today, or recover and redispatch later.
  5. Effect ids repeat across instances (step#ordinal#kind): add the instance (needs a dispatched-set migration), or pass the instance to the handler.
  6. Machine version on recovery: expose RejectMachineVersionMismatch to Recover, or make it strict.
  7. Middleware on replay: skip it, or mark replayed steps.
  8. A timer whose transition fails is consumed, with the error only in results[i].Err: re-arm, surface from Tick, or leave.
  9. Checkpoints carry the full trace history (8 KB at step 50, 157 KB at 1000): bound it or not.
  10. Handle concurrency: the timer-loop test assumes concurrent Fire/Tick is supported, per drainClock; the alternative is a documented single-goroutine contract.
  11. Journaling event data needs a state API (fireConfig is unexported); refusing the option on a Handle also passes the test.

Evidence (local, this branch)

Check Result
14 new tests all FAIL on main, each for its stated defect
existing durable tests, -race ok
golangci-lint v2.15.0 (GOTOOLCHAIN=go1.27.3), vet, gofmt 0 issues

Regressions vs baseline: none. API diff: none (tests only).

@reka18
reka18 force-pushed the test/durable-defect-repros branch from c7c0a2c to a2b3062 Compare October 22, 2026 21:07
@reka18 reka18 changed the title test(durable): add failing repro tests for eight durable defects test(durable): add failing tests for recovery, timer and effect defects Oct 22, 2026
@reka18
reka18 force-pushed the test/durable-defect-repros branch from a2b3062 to dc82df2 Compare October 22, 2026 21:37
@reka18 reka18 added not-ready On every new PR until all review dimensions are confirmed wait:correctness ACR pending: correctness wait:regression-safety ACR pending: regression-safety wait:concurrency-discipline ACR pending: concurrency-discipline wait:replay-determinism ACR pending: replay-determinism wait:api-compatibility ACR pending: api-compatibility wait:missed-context ACR pending: missed-context wait:human-readability ACR pending: human-readability labels Nov 5, 2026
@reka18
reka18 force-pushed the test/durable-defect-repros branch from dc82df2 to 40066e6 Compare November 5, 2026 17:33
@reka18
reka18 changed the base branch from main to chore/bump-go-and-deps-for-vulns November 5, 2026 17:34
@reka18
reka18 added this pull request to stack #238 November 5, 2026 17:34
@reka18
reka18 force-pushed the chore/bump-go-and-deps-for-vulns branch from d3b57d7 to 12706d8 Compare November 5, 2026 18:03
@reka18
reka18 force-pushed the test/durable-defect-repros branch from 40066e6 to cedc692 Compare November 5, 2026 18:04
@reka18
reka18 force-pushed the chore/bump-go-and-deps-for-vulns branch from 12706d8 to ca7a2dc Compare November 5, 2026 18:55
@reka18
reka18 force-pushed the test/durable-defect-repros branch from cedc692 to b4e2c73 Compare November 5, 2026 18:55
@reka18
reka18 force-pushed the chore/bump-go-and-deps-for-vulns branch from ca7a2dc to 3c49a70 Compare November 5, 2026 19:03
@reka18
reka18 force-pushed the test/durable-defect-repros branch from b4e2c73 to 5182e1a Compare November 5, 2026 19:04
Each test asserts what any correct fix must produce, so it fails on main
today and passes once its defect is fixed. Contract choices the fixes
depend on are listed in the PR, not tested.

- runner: two Handles on one instance split the journal; Fire during the
  timer loop loses steps; event data is dropped on recovery; the step
  after a tick-barrier checkpoint is dropped; a failed Append leaves the
  live instance ahead of the journal
- clock: Start doesn't arm the initial state's timer; ticks before a
  timer is due append records
- service: a timer armed by a service result is lost on replay; a result
  from a canceled turn is committed on recovery
- timer_checkpoint: a checkpointed timer that fires in the tail is missed
- effect: effects from Start, timer and service transitions never reach
  the handler; a failed dispatch is compacted away by a checkpoint

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U2eeCbrr5qRKgUAioqVKmc
Signed-off-by: Ray Karnes <ray.karnes@stablekernel.com>
@reka18
reka18 force-pushed the chore/bump-go-and-deps-for-vulns branch from 3c49a70 to 9e2d5f3 Compare November 5, 2026 19:19
@reka18
reka18 force-pushed the test/durable-defect-repros branch from 5182e1a to 6b6d3c3 Compare November 5, 2026 19:19
@reka18
reka18 removed this pull request from stack #238 November 5, 2026 20:26
@reka18

reka18 commented Nov 5, 2026

Copy link
Copy Markdown
Contributor Author

Superseded: each failing test moved into the PR in stack #239 that fixes it. The stack starts at #229.

@reka18 reka18 closed this Nov 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

not-ready On every new PR until all review dimensions are confirmed wait:api-compatibility ACR pending: api-compatibility wait:concurrency-discipline ACR pending: concurrency-discipline wait:correctness ACR pending: correctness wait:human-readability ACR pending: human-readability wait:missed-context ACR pending: missed-context wait:regression-safety ACR pending: regression-safety wait:replay-determinism ACR pending: replay-determinism

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant