durable: FileStore can lose or double-apply steps after a failed write or crash #228

Summary

FileStore can lose an acknowledged step or apply steps twice after a failed write or a crash, which contradicts its docs. Three separate gaps below, all reproduced on main @ 8770dcf. Line references are to durable/filestore.go. These come from FileStore's multi-file writes, so a backend that commits atomically isn't affected. I'll follow up with failing tests and a fix in one PR, after the first #214 fixes land.

Findings (in fix order)

• A torn journal line loses acknowledged steps. A failed write can leave a partial line, and nothing truncates it, so the next Append merges into it and returns success. On reopen the merged line fails to parse and is skipped (:254). No crash is needed: on an HFS+ volume, Handle.Fire hit a real disk-full error, the next Fire succeeded, and the instance recovered at 1 where live was 3. Because the skip continues rather than stops, as :43 says, recovered steps can have gaps such as [0 1 3]. A retry of a lost step is reported as applied (:333). A later Checkpoint in the same process heals it, because it rewrites the journal from memory. TestFileStore_TornTrailingRecord_RecoversCleanly never reopens after the merge, so it misses this.
• checkpoint.snap and checkpoint.meta aren't atomic together. The snapshot is renamed at :420 and the meta at :423, so a process kill or failed meta write between them pairs the new snapshot with the old step. Recovery then replays steps the snapshot already holds: a self-loop machine recovered 14 where live was 7, and a machine with a real initial transition failed Recover with "invalid transition". The Atomicity contract in store.go and :45-48 say this can't happen. TestFileStore_MissingCheckpointMeta_TreatedAsNoCheckpoint covers only a missing meta file.
• The journal and the applied ledger can disagree. Append writes the journal at :349 and the ledger at :358, and reopen doesn't seed the ledger from the journal.
  • After a kill between the two, a retry of the same record returns ErrStepOutOfOrder rather than the idempotent no-op the Store contract promises. Only direct Store callers see this: Recover replays the step and carries on.
  • A failed ledger write, with no crash, leaves the step out of memory too, so a retry writes it twice. Through stateless Runner.Fire this gave a count of 3 after 2 acknowledgements.
  • A torn ledger line also loses the next step's key.

Notes

• A fix on load, truncating a torn tail and reconciling the ledger with the journal, may cover the first and third findings together.
• These stay FileStore tests: the shared conformance suite proposed in durable: export a Store conformance suite for out-of-tree backends #223 has no crash injection.