Skip to content

fix(durable): add write fencing and reject conflicting steps - #229

Merged
reka18 merged 6 commits into
mainfrom
fix/durable-fence-and-event-data
Nov 6, 2026
Merged

reka18 merged 6 commits into
mainfrom
fix/durable-fence-and-event-data

Conversation

@reka18

@reka18 reka18 commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

What this change does

First of the #214 fixes, at the bottom of stack #239, on main. It also limits ci.yml to PRs into main, so a stacked PR runs CI only once it reaches the bottom of the stack; that's a separate commit, to save GitHub Actions minutes. Each PR in this chain carries the tests for what it fixes, so it passes on its own.

  • Fencing. Store writes can carry a fence token. A write below the highest token seen returns ErrFenced and writes nothing. Adds WithFence, WithCheckpointFence, AppendFence, CheckpointFence and the runner's WithOwnerFence. FileStore keeps the token in a fence file.
  • Conflicting steps. Append returns ErrStepConflict for a different record at a recorded step, instead of acknowledging a step the journal doesn't hold. Adds AppendIdempotencyKey, so an out-of-tree Store can read the key.
  • Docs. The Stores sections in durable/README.md and the docs site now cover fencing and the fence file.

After Tick's append succeeds and its checkpoint fails, nextStep doesn't advance, so the next write on that Handle conflicts until it's recovered. RunService and DeliverToActor do the same. #233, further up this stack, fixes that.

Not included: event data is in #230, the failed-Append fix and the Handle lock are in #233, and the #228 FileStore crash bugs get a separate PR.

Please check these:

  • WithCheckpointFence has its own name, because Append and Checkpoint take different option types.
  • Recover writes no fence token, so a stale write landing before the new owner's first write is accepted, and the owner then gets ErrStepConflict.
  • An owner here is the process holding the host's lease on an instance. The host issues the tokens.
  • An explicit WithIdempotencyKey skips the record comparison.
  • Risks: step reuse fails hard until the step-numbering fix in fix(durable): carry the next step through a checkpoint #232 lands; FileStore fencing only holds within one open store; MarkDispatched isn't fenced.

Why

Without a fence, a stale owner can keep writing to a session it has lost, and Append can acknowledge a step the journal doesn't hold. #214 has the defects.

Relates to: #214

Evidence

Run locally on Go 1.27.2, on durable and state only:

Command Result
go test ./... in durable Pass, including TestRecover_TwoHandlesOnOneInstance_JournalMatchesAnAcknowledgedHistory
go test -race -skip TestRunner_FireDuringTimerLoop_LosesNoAcknowledgedSteps ./... in durable 0 race warnings. The skipped test is the known Handle lock defect, and it races on the base too
go test ./... in state pass
GOWORK=off go build ./... in durable pass
go run golang.org/x/vuln/cmd/govulncheck@v1.3.0 ./... in durable, state no vulnerable code called

On CI's Go (GOTOOLCHAIN=go1.27.3), the pinned golangci-lint v2.15.0 reports 0 issues in durable and state.

#231 has merged, which fixed the vulnerability findings.

Checklist

  • Commits are signed off (git commit -s) per the DCO
  • Conventional commit messages (type: subject)
  • mage check passes locally (lint + race tests + govulncheck): not run as a whole, and it can't pass while the tests above fail
  • Tests added/updated for the change: durable/fence_test.go
  • Public API changes documented (godoc) and, if applicable, noted for the changelog

@reka18
reka18 force-pushed the fix/durable-fence-and-event-data branch 2 times, most recently from d09df8d to 456039d Compare November 5, 2026 14:57
@reka18 reka18 changed the title fix(durable): add write fencing, reject conflicting steps, journal event data fix(durable): add write fencing and reject conflicting steps Nov 5, 2026
@reka18 reka18 added not-ready On every new PR until all review dimensions are confirmed wait:correctness ACR pending: correctness wait:regression-safety ACR pending: regression-safety wait:concurrency-discipline ACR pending: concurrency-discipline wait:replay-determinism ACR pending: replay-determinism wait:api-compatibility ACR pending: api-compatibility wait:missed-context ACR pending: missed-context wait:human-readability ACR pending: human-readability labels Nov 5, 2026
@reka18

reka18 commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor Author

ACR round 1 (one reviewer per dimension)

Seven reviewers read this PR, then read each other's findings to settle conflicts before any code changed.

Dimension Blockers Status
correctness 0 confirmed
regression-safety 0 confirmed
concurrency-discipline 0 confirmed
replay-determinism 0 confirmed
api-compatibility 0 confirmed
missed-context 0 confirmed
human-readability 1 folded in, round 2 pending

Blocker, folded in: the README and docs site said an owner that lost an instance "can't keep writing". Both now state when the fence applies and the one-open-store limit.

Non-blockers, folded in:

  • The contract says the token must be as durable as the Records, and the check, raise and write must be atomic.
  • The contract states that step: keys are reserved and how mixed keyed appends behave.
  • MemStore treats a refused first write as never written, matching FileStore. A FileStore directory with no fence keeps its old meaning.
  • New tests: racing tokens, refused first write, golden digest. Record warns that new fields need omitempty.
  • Stale comments, doc.go and changelog entries, including what Store implementers must now do.

Reviewers disagreed, and the maintainer decided:

Dropped: comparing digests on keyed appends, which would make an honest retry conflict.

Deferred: hashing outside the MemStore lock, because the cost is unmeasured.

Known and left open: a keyed step: collision stays possible for a caller who ignores the contract.

@reka18 reka18 added confirmed:correctness ACR clean: correctness confirmed:regression-safety ACR clean: regression-safety confirmed:concurrency-discipline ACR clean: concurrency-discipline confirmed:replay-determinism ACR clean: replay-determinism confirmed:api-compatibility ACR clean: api-compatibility confirmed:missed-context ACR clean: missed-context and removed wait:correctness ACR pending: correctness wait:regression-safety ACR pending: regression-safety wait:concurrency-discipline ACR pending: concurrency-discipline wait:replay-determinism ACR pending: replay-determinism wait:api-compatibility ACR pending: api-compatibility wait:missed-context ACR pending: missed-context labels Nov 5, 2026
@reka18
reka18 force-pushed the fix/durable-fence-and-event-data branch 2 times, most recently from 3177871 to a2ade84 Compare November 5, 2026 16:53
@reka18

reka18 commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor Author

ACR rounds 2 to 4: human-readability

Round Blockers Folded in
2 1 The description said the failed-Append fix was on hold, but #233 has it. Plus 4 wording fixes.
3 1 "Recover writes nothing" was false, because replay can mark dispatched effects. Now "writes no fence token". Plus 3 wording fixes.
4 0 Confirmed.

Round 3 hit the ACR limit of three rounds. The maintainer chose to keep going, so round 4 ran as a normal round.

Round 4's non-blockers are small godoc additions, and they land with the next rebase of this stack.

All seven dimensions are now confirmed. ready means review is complete. This PR still needs #231 and #213 to land, and CI won't be green until the rest of the stack does.

@reka18
reka18 force-pushed the fix/durable-fence-and-event-data branch from 08e3dcb to 3950c55 Compare November 5, 2026 19:04
@reka18
reka18 force-pushed the fix/durable-fence-and-event-data branch 2 times, most recently from 1d93173 to 6e33531 Compare November 5, 2026 20:23
@reka18
reka18 removed this pull request from stack #238 November 5, 2026 20:26
@reka18
reka18 changed the base branch from test/durable-defect-repros to chore/bump-go-and-deps-for-vulns November 5, 2026 20:26
@reka18
reka18 added this pull request to stack #239 November 5, 2026 20:26
@reka18
reka18 force-pushed the fix/durable-fence-and-event-data branch from 6e33531 to b207bf1 Compare November 5, 2026 21:02
@reka18

reka18 commented Nov 5, 2026

Copy link
Copy Markdown
Contributor Author

Update since review. #213 is closed. This PR now carries the tests it turns green, and it's stacked on #231 in stack #239. The code under review didn't change, and durable passes with -race and lint at this head.

Base automatically changed from chore/bump-go-and-deps-for-vulns to main November 5, 2026 21:25
@reka18
reka18 force-pushed the fix/durable-fence-and-event-data branch 2 times, most recently from 52f82b1 to af3597e Compare November 5, 2026 21:59
Store writes can carry a fence token, and a write below the highest token
seen returns ErrFenced and writes nothing. Append now rejects a different
record at a recorded step with ErrStepConflict instead of acknowledging a
step the journal doesn't hold.

Relates to #214. The failed-Append fix and the Handle lock are not included.

Signed-off-by: Ray Karnes <ray.karnes@stablekernel.com>
…ions

Signed-off-by: Ray Karnes <ray.karnes@stablekernel.com>
Folds in the review of the fencing change. The docs no longer overclaim what a
fence guarantees. The Store contract now says the token must be as durable as
the Records and the check atomic with the write, and states how explicit keys
behave. MemStore treats an instance with a refused first write as never
written, matching FileStore, and a directory with no fence keeps its old
meaning. Adds a golden digest test and a racing-token test.

Relates to #214.

Signed-off-by: Ray Karnes <ray.karnes@stablekernel.com>
@reka18
reka18 force-pushed the fix/durable-fence-and-event-data branch from af3597e to a92f50a Compare November 5, 2026 22:12
A stacked PR re-ran the full suite on every push, though only the PR based on main can merge. Each PR runs CI once the stack retargets it to main, which saves GitHub Actions minutes.

Signed-off-by: Ray Karnes <ray.karnes@stablekernel.com>
@reka18
reka18 marked this pull request as ready for review November 5, 2026 23:43
@reka18
reka18 requested a review from joshua-temple November 5, 2026 23:43
joshua-temple
joshua-temple previously approved these changes Nov 6, 2026

@joshua-temple joshua-temple left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved pending relayed GB message

ErrInstanceNotFound = errors.New("crucible/durable: instance not found")

For all of the sentinel errors in this durable package, lets swap the value contents to private constants and drop the crucible prefix

Consider this a 'while you're in there' patch - it can be a part of this PR

eg

const (
    errorPrefix        = "durable: "
    errorNotFound      = "not found"
    errorInvalidStatus = "invalid status"
    errorUnauthorized  = "unauthorized"

var (
    ErrNotFound      = errors.New(errorPrefix + errorNotFound
    ErrInvalidStatus = errors.New(errorPrefix + errorInvalidStatus)
    ErrUnauthorized  = errors.New(errorPrefix + errorUnauthorized)
)

…e prefix

Sentinel error text now lives in private constants built on errorPrefix.
Every error in the package now starts with "durable: " rather than
"crucible/durable: ". Exported names are unchanged, and errors.Is
matching is unaffected.

Signed-off-by: Ray Karnes <ray.karnes@stablekernel.com>
The advisory, published after this branch last passed, covers a panic on
two-block frames and fails govulncheck on every PR into main. Four modules
reach the vulnerable code; all 19 that carry the dependency move together.

Signed-off-by: Ray Karnes <ray.karnes@stablekernel.com>
@reka18
reka18 merged commit b1611bb into main Nov 6, 2026
112 checks passed
@reka18
reka18 deleted the fix/durable-fence-and-event-data branch November 6, 2026 15:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

confirmed:api-compatibility ACR clean: api-compatibility confirmed:concurrency-discipline ACR clean: concurrency-discipline confirmed:correctness ACR clean: correctness confirmed:human-readability ACR clean: human-readability confirmed:missed-context ACR clean: missed-context confirmed:regression-safety ACR clean: regression-safety confirmed:replay-determinism ACR clean: replay-determinism ready All ACR dimensions confirmed - clear to merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants