Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions docs/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- **Design note for PageBased crash safety and encryption.** `docs/storage/PAGEBASED_DURABILITY_DESIGN.md` proposes page checksums in the unused header bytes (no format change, §19), a per-table page redo log with recovery and checkpoints (§18), AES-GCM page encryption with a format flag (§14) and the future of the directory-mode WAL (§20), with the owner decisions each needs.
- **Investigated: the READ ratios in the write-path gate's watch band.** An interleaved A/B of `master` against this stack (point lookups by key and by name, full scans; three rounds each) shows no read regression — key lookups equal, name lookups ~25 % faster, scans within ±3 % — so the watch-band READ ratios reflect this machine against the recorded baseline. Found on the way and recorded: a PageBased lookup on a non-key column scans the table (~32 ms for 20,000 rows on `master` and here), because hash indexes exist for columnar tables only. Coverage of the stack: 48.47 % overall (floor 18 %), 84.9 % of the new lines. Suites: core **2546/0 failed (6 skipped)**, VectorSearch 255/0, EF Core 116/0, linq2DB 96/0, regression canaries 51/0; write-path gate: GATE PASSED (raw UPDATE 1.25× with #522's commit fsync included, where #522's run measured 1.54×; READ 1.43×/1.30× in the watch band, shown by the A/B to be this machine, not the code).

- **PageBased pages carry a checksum (known issue §19).** A torn page — the tail of an 8 KB page not written when a crash interrupted it — read as a page with empty slots: its rows vanished without an error and the table kept accepting writes. Every page written now carries a CRC32C in header bytes 9–12 with a format flag in byte 13 (bytes no version wrote, so the layout is unchanged and older pages read unverified); a damaged page raises `InvalidDataException` naming it. The PageBased scan's catch-all, which returned the rows read so far on any error, lets that error through. Cost ~2.9 µs per page. Pinned by `PageChecksumTests`.
- **An uncommitted PageBased transaction could reach the disk (known issue §18, interim fix).** A flush in the middle of a transaction — `ExecuteQuery` flushes pending writes inside `BeginBatchUpdate` — or a cache eviction wrote the transaction's dirty pages, so a crash before its commit kept them. Pages changed by the open transaction are now pinned until the commit; the cache may exceed its capacity meanwhile. The eviction loop also no longer spins when the cache is over capacity and no page is in its LRU list. A crash during the commit's own multi-page write still needs the page redo log. Pinned by `CrashDurabilityTests`.
- **PageBased non-key lookups (known issue §21): by design.** PageBased tables do not auto-create column indexes (columnar tables do; SQLite does not); `CREATE INDEX` takes a lookup on 20,000 rows from ~80 ms to 0.02–0.1 ms, and `PageBasedSecondaryIndexTests` pins its correctness through writes and a reopen.
- **SonarCloud quality gate on #521.** Three copies of the same quote- and parenthesis-aware scanner (UPDATE SET/WHERE split, ON CONFLICT finder, ORDER BY item end) are one helper now (`SqlLiteralScanner.FindTopLevel`), and the flagged complexity smells in the upsert code are split up; no behaviour change. Suites: core **2551/0 failed (6 skipped)**, VectorSearch 255/0, EF Core 116/0, linq2DB 96/0, regression canaries 51/0; write-path gate: GATE FAILED on raw READ (1.58×) in a run whose own rep spread was 1.33× READ / 1.76× UPDATE / 1.96× DELETE; an interleaved A/B of the gate's exact read statement (parameterized SELECT * … WHERE name = @name, NoEncryptMode) against master measured 7.45–9.05 µs vs 7.98–8.21 µs — no code regression.

## [2.1.4] - 2026-11-13

> The issue #497 follow-up: a quoted identifier in `CREATE TABLE` is catalogued without its quoting, so a schema
Expand Down
37 changes: 27 additions & 10 deletions docs/sharpcoredb-known-issues.md
Original file line number Diff line number Diff line change
Expand Up @@ -660,9 +660,15 @@ Before: appends were opened with `FileOptions.WriteThrough`, but the cached in-p
and nothing called `FlushToDisk`, so a power loss (not a process crash) could lose a committed UPDATE or DELETE while
keeping committed INSERTs.

## 18. 🟠 OPEN — An uncommitted PageBased transaction can reach the disk before a crash
## 18. ✅ FIXED (interim) — An uncommitted PageBased transaction could reach the disk before a crash

> ⚠️ **STILL TRUE (verified 2026-11-17, `fix/durability-metadata-and-fsync`).**
> ✅ **FIXED (2026-11-17, `fix/pagebased-checksums-pinning-indexes`) for pages written before the commit.** A dirty page
> the open transaction changed is pinned: neither a cache eviction nor a flush in the middle of the transaction writes it
> (the main path was not eviction but `ExecuteQuery`, which flushes pending writes inside `BeginBatchUpdate`); it stays in
> memory, where the transaction's own reads see it, and is written at the commit. The cache may exceed its capacity until
> the transaction ends. Pinned by `CrashDurabilityTests.Crash_AfterAnUncommittedPageBasedTransactionWhosePagesWereEvicted_LeavesTheCommittedState`
> (failed before). **Still open:** a crash *during* the commit's multi-page write can leave some of its pages written and
> others not; that needs the page redo log of `docs/storage/PAGEBASED_DURABILITY_DESIGN.md` §4.

Measured: a transaction updated row 1 and deleted row 2 of a PageBased table of ~1,400 pages, then read the table twice,
and the process was killed before the commit. After reopening, row 1 had the uncommitted value and row 2 was gone.
Expand All @@ -676,9 +682,16 @@ images are restored and written back); a crash between the eviction and the comm
open transaction from eviction (bounded memory then needs a spill file). Engine work; designed in
`docs/storage/PAGEBASED_DURABILITY_DESIGN.md` §4.

## 19. 🟠 OPEN — A torn PageBased page loses rows silently
## 19. ✅ FIXED (detection) — A torn PageBased page lost rows silently

> ⚠️ **STILL TRUE (verified 2026-11-17, `fix/durability-metadata-and-fsync`).**
> ✅ **FIXED (2026-11-17, `fix/pagebased-checksums-pinning-indexes`).** Every page written carries a CRC32C checksum in
> header bytes 9–12 and a format flag in byte 13 — header bytes no version ever wrote, so the page layout does not change
> and a page written before (flag 0) is read unverified, as it always was. A page whose checksum fails raises
> `InvalidDataException` naming the page and the file when its records are read; the page-index scan at open does not
> verify, so one damaged page does not stop the table from opening. The PageBased scan's catch-all no longer swallows
> that error (it returned the rows read so far, which is how the damage stayed hidden). Cost: ~2.9 µs per 8 KB page
> written or read from disk (`BitOperations.Crc32C`). Pinned by `PageChecksumTests` (the two damage cases failed before).
> **Still open:** repairing a damaged page needs the page redo log (design note §4).

Measured: zeroing the second half of a page's 8 KB — what a torn write leaves when a crash interrupts it — made 25 of 40
rows disappear without an error, and the table kept accepting writes; cutting the page file mid-page lost that page's
Expand All @@ -701,10 +714,14 @@ the configuration suggests a guarantee that does not exist. The single-file (`.s
**Decision needed:** wire the WAL in (log each committed change, replay on open — the natural home for §15/§18's fix), or
remove it and the WAL settings from directory mode so the configuration is honest.

## 21. 🟡 OPEN — A PageBased lookup on a non-key column scans the whole table
## 21. ✅ BY DESIGN — A PageBased lookup on a non-key column scans the table unless the column is indexed

> ⚠️ **STILL TRUE (verified 2026-11-17, `fix/pagebased-durability-case-and-coverage`).** Found in the READ A/B of the
> write-path gate's watch band; present on `master` as well.
> ✅ **RESOLVED (2026-11-17, `fix/pagebased-checksums-pinning-indexes`): not a defect.** PageBased tables deliberately do
> not create a column index on first use (`Table.AutoHashIndexesEnabled` — columnar tables do; SQLite does not either).
> An explicit `CREATE INDEX ix ON t (col)` serves the lookup: measured ~80 ms → 0.02–0.1 ms per lookup on 20,000 rows,
> and the index stays correct through INSERT, UPDATE, DELETE and a reopen (`PageBasedSecondaryIndexTests`). Turning on
> automatic indexes for PageBased tables would reverse that documented choice — an owner decision. The cause analysis
> below was wrong in one point (`CREATE INDEX` does give the scan an index route) and is kept as the record.

`SELECT … WHERE name = 'User42'` on a 20,000-row PageBased table takes ~32 ms (a full scan per lookup), against ~40 µs
on a columnar table, where the column gets an automatic hash index. **Cause:** hash indexes exist for columnar tables only
Expand Down Expand Up @@ -735,7 +752,7 @@ references, which in-place updates keep stable), or a B-tree secondary index; me
| 15 | PageBased autocommit / batch not durable until a flush | 🔴 High | — | **Fixed** — pages flushed at each autocommit statement and commit (§15) |
| 16 | Non-atomic metadata write → database unopenable after a crash | 🔴 High | — | **Fixed** — temp file + fsync + rename; clear error for a damaged file (§16) |
| 17 | Columnar in-place UPDATE/DELETE not fsynced | 🟠 Medium | — | **Fixed** — synced at commit, autocommit end and flush (§17) |
| 18 | Uncommitted PageBased pages reach disk via eviction | 🟠 Medium | Keep transactions within the page cache | **Open** — needs a page WAL (§18) |
| 19 | Torn PageBased page loses rows silently | 🟠 Medium | — | **Open** — needs page checksums (§19) |
| 18 | Uncommitted PageBased pages reach disk via eviction | 🟠 Medium | — | **Fixed (interim)** — pages of an open transaction are pinned; a crash during the commit's flush still needs the page WAL (§18) |
| 19 | Torn PageBased page loses rows silently | 🟠 Medium | — | **Fixed (detection)** — CRC32C per page, damage raises an error; repair needs the page WAL (§19) |
| 20 | Directory-mode WAL never written or replayed | 🟡 Low | — | **Open, owner decision** (§20) |
| 21 | PageBased non-key lookup scans the table | 🟡 Low | Look up by primary key, or use columnar storage | **Open** (§21) |
| 21 | PageBased non-key lookup scans the table | 🟡 Low | `CREATE INDEX` on the column | **By design** — explicit index serves it (§21) |
19 changes: 9 additions & 10 deletions docs/storage/PAGEBASED_DURABILITY_DESIGN.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,13 +40,13 @@ readable by the new one (all-zero fields mean "written before checksums").

## 3. Torn pages — a checksum per page (§19)

**Proposal.** Header bytes 9–16: `uint64` XxHash64 (System.IO.Hashing) of the page with these 8 bytes zeroed; byte 17:
format flag `0x01` = checksummed. `WritePage` computes it; `ReadPage` verifies it when the flag is set and throws a
`PageCorruptedException` naming the table and page id when it does not match. A page with flag `0x00` (written before)
is read unverified, exactly as today.
**Implemented (`fix/pagebased-checksums-pinning-indexes`).** Header bytes 9–12: CRC32C of the page with these four
bytes left out (`BitOperations.Crc32C`, the CPU's CRC instruction — no new dependency); byte 13: format flag `0x01` =
checksummed. `WritePage` computes it; `ReadPage` verifies it when the flag is set and throws `InvalidDataException`
naming the page and file. A page with flag `0x00` (written before) is read unverified, exactly as before. The page-index
scan at open skips verification, so one damaged page does not stop the table from opening.

- *Cost:* one 8 KB hash per page write and read — XxHash64 runs at several GB/s, i.e. around 1 µs per page; measure on
the PageBased benchmark (`--engine=pagebased`) and the gate.
- *Cost (measured):* ~2.9 µs per 8 KB page written or read from disk.
- *What it buys:* a torn page becomes a loud error instead of silently missing rows. It does not repair the page —
that needs §4's log.
- *Compatibility:* none broken; downgrade (unsupported by policy) would read checksummed pages fine, since an old reader
Expand Down Expand Up @@ -114,10 +114,9 @@ it changes what a public configuration flag means.

## 7. Suggested order and test plan

1. §3 checksums (no format break, independent) — tests: damage a page's tail → `PageCorruptedException` names it; a
page written before (flag 0) reads unverified; reopen round-trip matrix stays green.
2. Option A pinning (§18 interim) — test: the eviction crash case of the investigation (`tx-uncommitted-then-big-read`)
leaves the committed state.
1. ✅ §3 checksums — done (`PageChecksumTests`).
2. ✅ Option A pinning (§18 interim) — done; it covers mid-transaction flushes as well as evictions
(`CrashDurabilityTests`).
3. §4 log + recovery — tests: kill the child process at each step of the commit (after the log sync, between page
writes) via a test hook; replay restores exactly the committed state; torn page repaired from the log; checkpoint
truncates.
Expand Down
6 changes: 4 additions & 2 deletions src/SharpCoreDB/DataStructures/Table.PageBasedScan.cs
Original file line number Diff line number Diff line change
Expand Up @@ -95,10 +95,12 @@ private List<Dictionary<string, object>> ScanPageBasedTable(string? where)
Console.WriteLine($"[ScanPageBasedTable] ✅ Total records found: {recordCount}, deserialization failures: {deserializeFailureCount}, after filtering: {results.Count}");
#endif
}
catch
catch (Exception ex) when (ex is not InvalidDataException)
{
// Exception during scan initialization - return empty results
// The WHERE clause evaluation or engine access may have failed
// The WHERE clause evaluation or engine access may have failed. A damaged page (InvalidDataException: its
// checksum failed) is NOT one of these: returning the rows read so far would hide the damage, which is
// exactly how a torn page used to lose rows silently (known issue §19).
#if DEBUG
Console.WriteLine($"[ScanPageBasedTable] Exception during scan");
#endif
Expand Down
Loading
Loading