Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions docs/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- **Columnar in-place UPDATEs and DELETE markers were not forced to disk (known issue §17).** Appends were written through, but an UPDATE patched into its record and a DELETE's tombstone marker went through a buffered handle that nothing flushed, so a power loss could lose a committed UPDATE or DELETE. Each file written in place is now fsynced once at the next durability boundary — the transaction commit, the end of an autocommit statement, `Flush()`, or the closing of its handle — under `WalDurabilityMode.FullSync` (the default); `Async` leaves them to the operating system. Cost: the gate's raw UPDATE arm (10,000 in-place updates per commit) measured 1.54× its capability baseline, which was recorded without this fsync; re-recording the baseline is an owner decision.
- **Crash-safety investigation recorded (known issues §14–§20).** Measured with a killed child process and with damaged files: PageBased table data is stored unencrypted (§14, owner decision); a PageBased autocommit statement or `ExecuteBatchSQL` is lost by a crash until something flushes (§15); an uncommitted PageBased transaction can reach the disk through page eviction (§18); a torn page loses rows silently (§19); the directory-mode WAL is never written or replayed (§20, owner decision). Columnar storage kept the committed state in every crash case. `CrashDurabilityTests` pins the cases that hold and skips the PageBased ones with their issue. Suites: core **2536/0 failed (8 skipped: 6 single-file, 2 PageBased crash cases of §15)**, VectorSearch 255/0, EF Core 116/0, linq2DB 96/0, regression canaries 51/0; write-path gate: GATE FAILED on raw UPDATE (1.54× the capability baseline) — the cost of the §17 fsync under the default FullSync; baseline re-record is an owner decision.

- **A PageBased statement that returned was lost by a crash (known issue §15).** The PageBased engine kept changed pages in its cache until an explicit `Flush()`, a transaction commit, an eviction or `Dispose`, so a crash after an autocommit statement or a returned `ExecuteBatchSQL` lost it. Dirty pages are now written at the end of every autocommit statement and every commit, and synced under `WalDurabilityMode.FullSync` (the default); a flush with nothing dirty syncs nothing. Cost measured: an autocommit PageBased write ~50 → ~750 µs per statement, the order of a columnar autocommit write. Pinned by `CrashDurabilityTests`.
- **`CASE … WHEN … END`, and conditions used as values.** The scalar-expression evaluator — UPDATE SET values, CHECK constraints, ORDER BY keys and the SELECT list — did not parse CASE: `SET grade = CASE WHEN n >= 5 THEN 'high' ELSE 'low' END` stored the statement text, and a CHECK written with CASE rejected every row. Simple and searched CASE, nested CASE and a comparison as a value (`THEN amount < 0`, `SET active = length(name) > 0` — 1, 0 or NULL, as in SQLite) are evaluated now. Pinned by `CaseExpressionTests` and `UpdateSetExpressionTests.Update_ComputedValues_AreConvertedToEachColumnsType`.
- **Design note for PageBased crash safety and encryption.** `docs/storage/PAGEBASED_DURABILITY_DESIGN.md` proposes page checksums in the unused header bytes (no format change, §19), a per-table page redo log with recovery and checkpoints (§18), AES-GCM page encryption with a format flag (§14) and the future of the directory-mode WAL (§20), with the owner decisions each needs.
- **Investigated: the READ ratios in the write-path gate's watch band.** An interleaved A/B of `master` against this stack (point lookups by key and by name, full scans; three rounds each) shows no read regression — key lookups equal, name lookups ~25 % faster, scans within ±3 % — so the watch-band READ ratios reflect this machine against the recorded baseline. Found on the way and recorded: a PageBased lookup on a non-key column scans the table (~32 ms for 20,000 rows on `master` and here), because hash indexes exist for columnar tables only. Coverage of the stack: 48.47 % overall (floor 18 %), 84.9 % of the new lines. Suites: core **2546/0 failed (6 skipped)**, VectorSearch 255/0, EF Core 116/0, linq2DB 96/0, regression canaries 51/0; write-path gate: GATE PASSED (raw UPDATE 1.25× with #522's commit fsync included, where #522's run measured 1.54×; READ 1.43×/1.30× in the watch band, shown by the A/B to be this machine, not the code).

## [2.1.4] - 2026-11-13

> The issue #497 follow-up: a quoted identifier in `CREATE TABLE` is catalogued without its quoting, so a schema
Expand Down
1 change: 1 addition & 0 deletions docs/INDEX.md
Original file line number Diff line number Diff line change
Expand Up @@ -117,6 +117,7 @@ guide explaining when SharpCoreDB is fastest.
- `serialization/SERIALIZATION_AND_STORAGE_GUIDE.md`
- `storage/STORAGE_MODE_GUIDANCE.md` — which storage mode to pick, and why
- `storage/SINGLE_FILE_SQL_LIMITATIONS.md` — what `.scdb` single-file SQL does and does not support
- `storage/PAGEBASED_DURABILITY_DESIGN.md` — PageBased crash safety, page checksums, page WAL and encryption: design proposal (known issues §14–§20)
- `serialization/SERIALIZATION_FAQ.md` — common serialization questions

## 6. Engine Implementation Notes
Expand Down
36 changes: 29 additions & 7 deletions docs/sharpcoredb-known-issues.md
Original file line number Diff line number Diff line change
Expand Up @@ -596,9 +596,17 @@ uses.
usable page and changes the page file format) is an on-disk format change; existing page files would stay readable as
plaintext and be rewritten encrypted. Until then, PageBased storage should not hold data that relies on encryption.

## 15. 🔴 OPEN — A PageBased statement that returned is not durable until a flush

> ⚠️ **STILL TRUE (verified 2026-11-17, `fix/durability-metadata-and-fsync`).**
## 15. ✅ FIXED — A PageBased statement that returned was not durable until a flush

> ✅ **FIXED (2026-11-17, `fix/pagebased-durability-case-and-coverage`).** The PageBased engine's dirty pages are written
> to the page file at the end of every autocommit statement and at every transaction or batch commit (and synced under
> `WalDurabilityMode.FullSync`, the default; `Async` writes them without the sync — process-crash safe, not power-loss
> safe). `FlushDirtyPages` returns without a sync when nothing is dirty, so a SELECT pays nothing. **Cost:** an autocommit
> PageBased INSERT/UPDATE went from ~50 µs to ~750 µs per statement (2,000 statements, measured on this box) — the same
> order as a columnar autocommit write, which has always been written through; batches and transactions pay once per
> commit. The page redo log of `docs/storage/PAGEBASED_DURABILITY_DESIGN.md` §4 would make this a sequential log append.
> Pinned by `CrashDurabilityTests` (PageBased autocommit and batch cases, which failed before). The description below is
> kept as the record.

Measured with a child process killed (no Dispose, no flush) right after the statements returned, then reopened:

Expand Down Expand Up @@ -665,7 +673,8 @@ and an evicted dirty page is written to the file — committed or not. Rollback
images are restored and written back); a crash between the eviction and the commit or rollback is not.

**Fix outline:** a page WAL (redo or undo log with commit records, replayed on open), or withholding dirty pages of an
open transaction from eviction (bounded memory then needs a spill file). Engine work; also closes §15 cheaply.
open transaction from eviction (bounded memory then needs a spill file). Engine work; designed in
`docs/storage/PAGEBASED_DURABILITY_DESIGN.md` §4.

## 19. 🟠 OPEN — A torn PageBased page loses rows silently

Expand All @@ -675,7 +684,8 @@ Measured: zeroing the second half of a page's 8 KB — what a torn write leaves
rows disappear without an error, and the table kept accepting writes; cutting the page file mid-page lost that page's
rows the same way. **Cause:** a page has no checksum, and its slot directory sits at the end of the page, so a torn tail
reads as empty slots. **Fix outline:** a checksum per page verified on read (a damaged page then raises an error instead
of reading as empty), plus the page WAL of §18 to repair it. Columnar files are append-only and lose only an incomplete
of reading as empty), plus the page WAL of §18 to repair it (designed in `docs/storage/PAGEBASED_DURABILITY_DESIGN.md`
§3 — the 23 unused page-header bytes hold the checksum, so the page format does not change). Columnar files are append-only and lose only an incomplete
trailing record (measured: 39 of 40 rows, table writable).

## 20. 🟡 OPEN (owner decision) — The directory-mode write-ahead log is never written or replayed
Expand All @@ -691,6 +701,17 @@ the configuration suggests a guarantee that does not exist. The single-file (`.s
**Decision needed:** wire the WAL in (log each committed change, replay on open — the natural home for §15/§18's fix), or
remove it and the WAL settings from directory mode so the configuration is honest.

## 21. 🟡 OPEN — A PageBased lookup on a non-key column scans the whole table

> ⚠️ **STILL TRUE (verified 2026-11-17, `fix/pagebased-durability-case-and-coverage`).** Found in the READ A/B of the
> write-path gate's watch band; present on `master` as well.

`SELECT … WHERE name = 'User42'` on a 20,000-row PageBased table takes ~32 ms (a full scan per lookup), against ~40 µs
on a columnar table, where the column gets an automatic hash index. **Cause:** hash indexes exist for columnar tables only
(`Table.RebuildIndex`: "Hash indexes only exist in columnar mode"), and `CREATE INDEX` on a PageBased table does not give
the scan an index route. **Fix outline:** maintain hash indexes on PageBased tables (positions are the engine's storage
references, which in-place updates keep stable), or a B-tree secondary index; measure with `--engine=pagebased`.

---

## Priority summary
Expand All @@ -711,9 +732,10 @@ remove it and the WAL settings from directory mode so the configuration is hones
| 12 | Columnar scan inside a transaction misses its own UPDATE/DELETE | 🟠 Medium | — | **Fixed** — buffered overwrites and tombstones are laid over the scan (§12) |
| 13 | PageBased transaction rollback | 🔴 High | — | **Fixed** for `CancelBatchUpdate` (ADO.NET, linq2db, EF Core) and `ExecuteBatchSQL` via page undo images; crash atomicity without a page WAL tracked in §18 |
| 14 | PageBased data stored unencrypted | 🔴 High | Columnar storage for sensitive data | **Open, owner decision** — page format change (§14) |
| 15 | PageBased autocommit / batch not durable until a flush | 🔴 High | `Flush()` after writes, or explicit transactions | **Open** (§15) |
| 15 | PageBased autocommit / batch not durable until a flush | 🔴 High | — | **Fixed** — pages flushed at each autocommit statement and commit (§15) |
| 16 | Non-atomic metadata write → database unopenable after a crash | 🔴 High | — | **Fixed** — temp file + fsync + rename; clear error for a damaged file (§16) |
| 17 | Columnar in-place UPDATE/DELETE not fsynced | 🟠 Medium | — | **Fixed** — synced at commit, autocommit end and flush (§17) |
| 18 | Uncommitted PageBased pages reach disk via eviction | 🟠 Medium | Keep transactions within the page cache | **Open** — needs a page WAL (§18) |
| 19 | Torn PageBased page loses rows silently | 🟠 Medium | — | **Open** — needs page checksums (§19) |
| 20 | Directory-mode WAL never written or replayed | 🟡 Low | — | **Open, owner decision** (§20) |
| 20 | Directory-mode WAL never written or replayed | 🟡 Low | — | **Open, owner decision** (§20) |
| 21 | PageBased non-key lookup scans the table | 🟡 Low | Look up by primary key, or use columnar storage | **Open** (§21) |
Loading
Loading