Skip to content

fix: atomic metadata writes and fsync of in-place writes; record crash-safety findings (known issues 14-20) - #522

Open
CastleSoft wants to merge 2 commits into
fix/batch-rollback-and-sql-gapsfrom
fix/durability-metadata-and-fsync
Open

CastleSoft wants to merge 2 commits into
fix/batch-rollback-and-sql-gapsfrom
fix/durability-metadata-and-fsync

Conversation

@CastleSoft

Copy link
Copy Markdown
Collaborator

Stacked on #521. This PR targets fix/batch-rollback-and-sql-gaps, so its diff shows only this change. Merge in order: #518, #519, #520, #521, then this one, retargeting each to master in turn.

⚠️ The write-path gate fails on this PR, by design. It needs an owner decision; see Performance.

Summary

Fixes findings 4 and 7 from the crash-safety investigation and records all seven findings as known issues §14–§20.

Finding 4 (§16): metadata written in place → database unopenable after a crash — fixed

meta.dat was written with File.WriteAllBytes: truncate, then write, with no fsync. Cutting it to half or to zero bytes made both columnar and PageBased databases refuse to open with "the master password may be incorrect", even though every table file was intact.

  • Atomic writes: meta.dat and .salt are now written by Storage.WriteFileAtomically: write path.tmp, fsync it, rename it over the file. A crash leaves the old or the new content, never a mix.
  • Fallback: a whole-file write buffered in a transaction is replaced atomically too, except when a cached read or write handle is open on the file. There it stays a direct write, because a rename would strand that handle on the old file.
  • No extra cost on Flush(): a metadata save whose text hasn't changed is skipped, so an explicit Flush() with no schema change pays no file replace.
  • Clearer error: a meta.dat that is empty or shorter than an encrypted empty document is reported as an interrupted save, with the remedy (restore it from a backup), instead of blaming the password.

Finding 7 (§17): columnar in-place UPDATE and DELETE weren't forced to disk — fixed

Appends are written through to disk. In-place UPDATE patches, DELETE tombstone markers and the commit's batched overwrite pages went through a cached handle opened with FileOptions.None, and nothing flushed it. A power loss, unlike a process crash, could lose committed UPDATEs and DELETEs while keeping INSERTs.

  • Every write through that handle marks its file unsynced. IStorage.SyncPendingWrites fsyncs each such file once at the next durability boundary:
    • a transaction commit;
    • the end of an autocommit ExecuteSQL;
    • Flush();
    • before the handle closes.
  • As you chose, this follows WalDurabilityMode:
    • FullSync, the default, syncs. That setting is documented as surviving power failures.
    • Async leaves the bytes to the operating system and tracks nothing.
  • A process-kill test can't observe an fsync, so this was verified by reading the code.

Findings recorded (§14, §15, §18, §19, §20)

§ Finding Status
14 🔴 PageBased table data is stored unencrypted. The engine is created with storage: null and PageManager uses a plain FileStream; a test value appears verbatim in the .pages file Open, owner decision (page format change)
15 🔴 A PageBased autocommit statement, or a returned ExecuteBatchSQL, is lost by a crash until something flushes Open
18 🟠 An uncommitted PageBased transaction can reach disk when a dirty page is evicted; measured: the uncommitted UPDATE and DELETE survived a crash Open (needs a page WAL)
19 🟠 A torn PageBased page loses rows silently: no checksum, and the slot directory sits at the page end Open (needs page checksums)
20 🟡 The directory-mode GroupCommitWAL is never written or replayed, although presets enable it with FullSync Open, owner decision

§13's crash-atomicity note now points to §18. All seven findings are in the summary table.

Tests

CrashDurabilityTests runs each crash case in a child test process that kills itself with no Dispose and no flush, then reopens the database in the parent.

Crash cases that hold today (regression guards):

  • Columnar: autocommit UPDATE; INSERT + DELETE; batch; committed transaction; uncommitted transaction.
  • PageBased: committed transaction; uncommitted transaction.

Skipped, each citing §15: PageBased autocommit and PageBased batch.

Metadata cases: empty file; truncated file; no leftover temp file, and a stale temp file doesn't affect open.

The two "interrupted save" error tests fail on #521's code, where the message blamed the password. The crash cases already held there; they guard against regressions.

Suite Total Failed Skipped
Core 2536 0 8 (6 single-file, 2 PageBased crash cases of §15)
VectorSearch 255 0 0
EF Core 116 0 0
linq2DB 96 0 0
Regression canaries 51 0 0

SharpCoreDB.CI.slnf builds with 0 errors, and the documentation consistency tests pass.

Performance — owner decision needed

Write-path gate: GATE FAILED on raw UPDATE, at 1.54× the capability baseline.

Metric Capability ratio Verdict
raw INSERT 1.06× ok
raw READ 1.29× watch
raw UPDATE 1.54× REGRESSED
raw DELETE 1.21× ok
default INSERT 1.05× ok
default READ 1.32× watch
default UPDATE 1.33× watch
default DELETE 1.35× watch
  • What the arm does: the raw UPDATE arm is one ExecuteBatchSQL of 10,000 in-place updates. Its commit now pays one fsync of the data file.
  • The numbers:
    • this run's best rep: 310,529 ops/s;
    • capability baseline: 477,824 ops/s;
    • this session's earlier runs, without the fsync: 384–394k ops/s.
  • Why: the baseline was recorded while these writes were not power-loss durable.

To make the gate measure the durable default, re-record the baseline (an owner action). Alternatively, configure the arm with WalDurabilityMode.Async to keep measuring the non-durable mode. I haven't done either.

…cord crash-safety findings (known issues 14-20)

- meta.dat and .salt are written atomically (temp file, fsync, rename); an unchanged metadata save is skipped; an
  empty or truncated meta.dat is reported as an interrupted save instead of a wrong password (known issue 16).
- Columnar in-place UPDATEs and DELETE tombstone markers are fsynced once per file at the transaction commit, at the
  end of an autocommit statement, on Flush() and before a cached handle closes - under WalDurabilityMode.FullSync,
  the default; Async leaves them to the OS (known issue 17). The write-path gate's raw UPDATE arm pays this fsync
  (1.54x its capability baseline, recorded without it); re-recording the baseline is an owner decision.
- Known issues 14-20 record the crash-safety investigation: PageBased data stored unencrypted (14), PageBased
  autocommit/batch not durable until a flush (15), uncommitted PageBased pages reaching disk via eviction (18),
  torn pages losing rows silently (19), directory-mode WAL never written or replayed (20).
- CrashDurabilityTests: child-process crash cases that hold (columnar; PageBased transactions), the PageBased gaps
  skipped with their issue, and the metadata error cases.
@CastleSoft

Copy link
Copy Markdown
Collaborator Author

Second gate measurement with this PR's commit fsync included: on #523's build (which contains #522), the write-path gate passed, raw UPDATE 1.25× its capability baseline (best rep 381,253 ops/s), against the 1.54× measured on this PR's own run. The raw UPDATE arm is documented as bimodal (~215k or ~420k ops/s), so the earlier failure may have been the slow mode rather than the fsync. Both runs are recorded as measured; a run on a quieted machine would settle whether the baseline decision is needed at all.

@sonarqubecloud

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant