Repository navigation
fix: atomic metadata writes and fsync of in-place writes; record crash-safety findings (known issues 14-20) - #522
Open
CastleSoft wants to merge 2 commits into
Conversation
…cord crash-safety findings (known issues 14-20) - meta.dat and .salt are written atomically (temp file, fsync, rename); an unchanged metadata save is skipped; an empty or truncated meta.dat is reported as an interrupted save instead of a wrong password (known issue 16). - Columnar in-place UPDATEs and DELETE tombstone markers are fsynced once per file at the transaction commit, at the end of an autocommit statement, on Flush() and before a cached handle closes - under WalDurabilityMode.FullSync, the default; Async leaves them to the OS (known issue 17). The write-path gate's raw UPDATE arm pays this fsync (1.54x its capability baseline, recorded without it); re-recording the baseline is an owner decision. - Known issues 14-20 record the crash-safety investigation: PageBased data stored unencrypted (14), PageBased autocommit/batch not durable until a flush (15), uncommitted PageBased pages reaching disk via eviction (18), torn pages losing rows silently (19), directory-mode WAL never written or replayed (20). - CrashDurabilityTests: child-process crash cases that hold (columnar; PageBased transactions), the PageBased gaps skipped with their issue, and the metadata error cases.
Collaborator
Author
|
Second gate measurement with this PR's commit fsync included: on #523's build (which contains #522), the write-path gate passed, raw UPDATE 1.25× its capability baseline (best rep 381,253 ops/s), against the 1.54× measured on this PR's own run. The raw UPDATE arm is documented as bimodal (~215k or ~420k ops/s), so the earlier failure may have been the slow mode rather than the fsync. Both runs are recorded as measured; a run on a quieted machine would settle whether the baseline decision is needed at all. |
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



Summary
Fixes findings 4 and 7 from the crash-safety investigation and records all seven findings as known issues §14–§20.
Finding 4 (§16): metadata written in place → database unopenable after a crash — fixed
meta.datwas written withFile.WriteAllBytes: truncate, then write, with no fsync. Cutting it to half or to zero bytes made both columnar and PageBased databases refuse to open with "the master password may be incorrect", even though every table file was intact.meta.datand.saltare now written byStorage.WriteFileAtomically: writepath.tmp, fsync it, rename it over the file. A crash leaves the old or the new content, never a mix.Flush(): a metadata save whose text hasn't changed is skipped, so an explicitFlush()with no schema change pays no file replace.meta.datthat is empty or shorter than an encrypted empty document is reported as an interrupted save, with the remedy (restore it from a backup), instead of blaming the password.Finding 7 (§17): columnar in-place UPDATE and DELETE weren't forced to disk — fixed
Appends are written through to disk. In-place UPDATE patches, DELETE tombstone markers and the commit's batched overwrite pages went through a cached handle opened with
FileOptions.None, and nothing flushed it. A power loss, unlike a process crash, could lose committed UPDATEs and DELETEs while keeping INSERTs.IStorage.SyncPendingWritesfsyncs each such file once at the next durability boundary:ExecuteSQL;Flush();WalDurabilityMode:FullSync, the default, syncs. That setting is documented as surviving power failures.Asyncleaves the bytes to the operating system and tracks nothing.Findings recorded (§14, §15, §18, §19, §20)
storage: nullandPageManageruses a plainFileStream; a test value appears verbatim in the.pagesfileExecuteBatchSQL, is lost by a crash until something flushesGroupCommitWALis never written or replayed, although presets enable it withFullSync§13's crash-atomicity note now points to §18. All seven findings are in the summary table.
Tests
CrashDurabilityTestsruns each crash case in a child test process that kills itself with no Dispose and no flush, then reopens the database in the parent.Crash cases that hold today (regression guards):
Skipped, each citing §15: PageBased autocommit and PageBased batch.
Metadata cases: empty file; truncated file; no leftover temp file, and a stale temp file doesn't affect open.
The two "interrupted save" error tests fail on #521's code, where the message blamed the password. The crash cases already held there; they guard against regressions.
SharpCoreDB.CI.slnfbuilds with 0 errors, and the documentation consistency tests pass.Performance — owner decision needed
Write-path gate: GATE FAILED on raw UPDATE, at 1.54× the capability baseline.
ExecuteBatchSQLof 10,000 in-place updates. Its commit now pays one fsync of the data file.To make the gate measure the durable default, re-record the baseline (an owner action). Alternatively, configure the arm with
WalDurabilityMode.Asyncto keep measuring the non-durable mode. I haven't done either.