diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index 3ffb05cc..66997e32 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -81,6 +81,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - **Columnar in-place UPDATEs and DELETE markers were not forced to disk (known issue §17).** Appends were written through, but an UPDATE patched into its record and a DELETE's tombstone marker went through a buffered handle that nothing flushed, so a power loss could lose a committed UPDATE or DELETE. Each file written in place is now fsynced once at the next durability boundary — the transaction commit, the end of an autocommit statement, `Flush()`, or the closing of its handle — under `WalDurabilityMode.FullSync` (the default); `Async` leaves them to the operating system. Cost: the gate's raw UPDATE arm (10,000 in-place updates per commit) measured 1.54× its capability baseline, which was recorded without this fsync; re-recording the baseline is an owner decision. - **Crash-safety investigation recorded (known issues §14–§20).** Measured with a killed child process and with damaged files: PageBased table data is stored unencrypted (§14, owner decision); a PageBased autocommit statement or `ExecuteBatchSQL` is lost by a crash until something flushes (§15); an uncommitted PageBased transaction can reach the disk through page eviction (§18); a torn page loses rows silently (§19); the directory-mode WAL is never written or replayed (§20, owner decision). Columnar storage kept the committed state in every crash case. `CrashDurabilityTests` pins the cases that hold and skips the PageBased ones with their issue. Suites: core **2536/0 failed (8 skipped: 6 single-file, 2 PageBased crash cases of §15)**, VectorSearch 255/0, EF Core 116/0, linq2DB 96/0, regression canaries 51/0; write-path gate: GATE FAILED on raw UPDATE (1.54× the capability baseline) — the cost of the §17 fsync under the default FullSync; baseline re-record is an owner decision. +- **A PageBased statement that returned was lost by a crash (known issue §15).** The PageBased engine kept changed pages in its cache until an explicit `Flush()`, a transaction commit, an eviction or `Dispose`, so a crash after an autocommit statement or a returned `ExecuteBatchSQL` lost it. Dirty pages are now written at the end of every autocommit statement and every commit, and synced under `WalDurabilityMode.FullSync` (the default); a flush with nothing dirty syncs nothing. Cost measured: an autocommit PageBased write ~50 → ~750 µs per statement, the order of a columnar autocommit write. Pinned by `CrashDurabilityTests`. +- **`CASE … WHEN … END`, and conditions used as values.** The scalar-expression evaluator — UPDATE SET values, CHECK constraints, ORDER BY keys and the SELECT list — did not parse CASE: `SET grade = CASE WHEN n >= 5 THEN 'high' ELSE 'low' END` stored the statement text, and a CHECK written with CASE rejected every row. Simple and searched CASE, nested CASE and a comparison as a value (`THEN amount < 0`, `SET active = length(name) > 0` — 1, 0 or NULL, as in SQLite) are evaluated now. Pinned by `CaseExpressionTests` and `UpdateSetExpressionTests.Update_ComputedValues_AreConvertedToEachColumnsType`. +- **Design note for PageBased crash safety and encryption.** `docs/storage/PAGEBASED_DURABILITY_DESIGN.md` proposes page checksums in the unused header bytes (no format change, §19), a per-table page redo log with recovery and checkpoints (§18), AES-GCM page encryption with a format flag (§14) and the future of the directory-mode WAL (§20), with the owner decisions each needs. +- **Investigated: the READ ratios in the write-path gate's watch band.** An interleaved A/B of `master` against this stack (point lookups by key and by name, full scans; three rounds each) shows no read regression — key lookups equal, name lookups ~25 % faster, scans within ±3 % — so the watch-band READ ratios reflect this machine against the recorded baseline. Found on the way and recorded: a PageBased lookup on a non-key column scans the table (~32 ms for 20,000 rows on `master` and here), because hash indexes exist for columnar tables only. Coverage of the stack: 48.47 % overall (floor 18 %), 84.9 % of the new lines. Suites: core **2546/0 failed (6 skipped)**, VectorSearch 255/0, EF Core 116/0, linq2DB 96/0, regression canaries 51/0; write-path gate: GATE PASSED (raw UPDATE 1.25× with #522's commit fsync included, where #522's run measured 1.54×; READ 1.43×/1.30× in the watch band, shown by the A/B to be this machine, not the code). + ## [2.1.4] - 2026-11-13 > The issue #497 follow-up: a quoted identifier in `CREATE TABLE` is catalogued without its quoting, so a schema diff --git a/docs/INDEX.md b/docs/INDEX.md index 615a0147..a645dc53 100644 --- a/docs/INDEX.md +++ b/docs/INDEX.md @@ -117,6 +117,7 @@ guide explaining when SharpCoreDB is fastest. - `serialization/SERIALIZATION_AND_STORAGE_GUIDE.md` - `storage/STORAGE_MODE_GUIDANCE.md` — which storage mode to pick, and why - `storage/SINGLE_FILE_SQL_LIMITATIONS.md` — what `.scdb` single-file SQL does and does not support +- `storage/PAGEBASED_DURABILITY_DESIGN.md` — PageBased crash safety, page checksums, page WAL and encryption: design proposal (known issues §14–§20) - `serialization/SERIALIZATION_FAQ.md` — common serialization questions ## 6. Engine Implementation Notes diff --git a/docs/sharpcoredb-known-issues.md b/docs/sharpcoredb-known-issues.md index 35f3f8cb..7d628647 100644 --- a/docs/sharpcoredb-known-issues.md +++ b/docs/sharpcoredb-known-issues.md @@ -596,9 +596,17 @@ uses. usable page and changes the page file format) is an on-disk format change; existing page files would stay readable as plaintext and be rewritten encrypted. Until then, PageBased storage should not hold data that relies on encryption. -## 15. 🔴 OPEN — A PageBased statement that returned is not durable until a flush - -> ⚠️ **STILL TRUE (verified 2026-11-17, `fix/durability-metadata-and-fsync`).** +## 15. ✅ FIXED — A PageBased statement that returned was not durable until a flush + +> ✅ **FIXED (2026-11-17, `fix/pagebased-durability-case-and-coverage`).** The PageBased engine's dirty pages are written +> to the page file at the end of every autocommit statement and at every transaction or batch commit (and synced under +> `WalDurabilityMode.FullSync`, the default; `Async` writes them without the sync — process-crash safe, not power-loss +> safe). `FlushDirtyPages` returns without a sync when nothing is dirty, so a SELECT pays nothing. **Cost:** an autocommit +> PageBased INSERT/UPDATE went from ~50 µs to ~750 µs per statement (2,000 statements, measured on this box) — the same +> order as a columnar autocommit write, which has always been written through; batches and transactions pay once per +> commit. The page redo log of `docs/storage/PAGEBASED_DURABILITY_DESIGN.md` §4 would make this a sequential log append. +> Pinned by `CrashDurabilityTests` (PageBased autocommit and batch cases, which failed before). The description below is +> kept as the record. Measured with a child process killed (no Dispose, no flush) right after the statements returned, then reopened: @@ -665,7 +673,8 @@ and an evicted dirty page is written to the file — committed or not. Rollback images are restored and written back); a crash between the eviction and the commit or rollback is not. **Fix outline:** a page WAL (redo or undo log with commit records, replayed on open), or withholding dirty pages of an -open transaction from eviction (bounded memory then needs a spill file). Engine work; also closes §15 cheaply. +open transaction from eviction (bounded memory then needs a spill file). Engine work; designed in +`docs/storage/PAGEBASED_DURABILITY_DESIGN.md` §4. ## 19. 🟠 OPEN — A torn PageBased page loses rows silently @@ -675,7 +684,8 @@ Measured: zeroing the second half of a page's 8 KB — what a torn write leaves rows disappear without an error, and the table kept accepting writes; cutting the page file mid-page lost that page's rows the same way. **Cause:** a page has no checksum, and its slot directory sits at the end of the page, so a torn tail reads as empty slots. **Fix outline:** a checksum per page verified on read (a damaged page then raises an error instead -of reading as empty), plus the page WAL of §18 to repair it. Columnar files are append-only and lose only an incomplete +of reading as empty), plus the page WAL of §18 to repair it (designed in `docs/storage/PAGEBASED_DURABILITY_DESIGN.md` +§3 — the 23 unused page-header bytes hold the checksum, so the page format does not change). Columnar files are append-only and lose only an incomplete trailing record (measured: 39 of 40 rows, table writable). ## 20. 🟡 OPEN (owner decision) — The directory-mode write-ahead log is never written or replayed @@ -691,6 +701,17 @@ the configuration suggests a guarantee that does not exist. The single-file (`.s **Decision needed:** wire the WAL in (log each committed change, replay on open — the natural home for §15/§18's fix), or remove it and the WAL settings from directory mode so the configuration is honest. +## 21. 🟡 OPEN — A PageBased lookup on a non-key column scans the whole table + +> ⚠️ **STILL TRUE (verified 2026-11-17, `fix/pagebased-durability-case-and-coverage`).** Found in the READ A/B of the +> write-path gate's watch band; present on `master` as well. + +`SELECT … WHERE name = 'User42'` on a 20,000-row PageBased table takes ~32 ms (a full scan per lookup), against ~40 µs +on a columnar table, where the column gets an automatic hash index. **Cause:** hash indexes exist for columnar tables only +(`Table.RebuildIndex`: "Hash indexes only exist in columnar mode"), and `CREATE INDEX` on a PageBased table does not give +the scan an index route. **Fix outline:** maintain hash indexes on PageBased tables (positions are the engine's storage +references, which in-place updates keep stable), or a B-tree secondary index; measure with `--engine=pagebased`. + --- ## Priority summary @@ -711,9 +732,10 @@ remove it and the WAL settings from directory mode so the configuration is hones | 12 | Columnar scan inside a transaction misses its own UPDATE/DELETE | 🟠 Medium | — | **Fixed** — buffered overwrites and tombstones are laid over the scan (§12) | | 13 | PageBased transaction rollback | 🔴 High | — | **Fixed** for `CancelBatchUpdate` (ADO.NET, linq2db, EF Core) and `ExecuteBatchSQL` via page undo images; crash atomicity without a page WAL tracked in §18 | | 14 | PageBased data stored unencrypted | 🔴 High | Columnar storage for sensitive data | **Open, owner decision** — page format change (§14) | -| 15 | PageBased autocommit / batch not durable until a flush | 🔴 High | `Flush()` after writes, or explicit transactions | **Open** (§15) | +| 15 | PageBased autocommit / batch not durable until a flush | 🔴 High | — | **Fixed** — pages flushed at each autocommit statement and commit (§15) | | 16 | Non-atomic metadata write → database unopenable after a crash | 🔴 High | — | **Fixed** — temp file + fsync + rename; clear error for a damaged file (§16) | | 17 | Columnar in-place UPDATE/DELETE not fsynced | 🟠 Medium | — | **Fixed** — synced at commit, autocommit end and flush (§17) | | 18 | Uncommitted PageBased pages reach disk via eviction | 🟠 Medium | Keep transactions within the page cache | **Open** — needs a page WAL (§18) | | 19 | Torn PageBased page loses rows silently | 🟠 Medium | — | **Open** — needs page checksums (§19) | -| 20 | Directory-mode WAL never written or replayed | 🟡 Low | — | **Open, owner decision** (§20) | \ No newline at end of file +| 20 | Directory-mode WAL never written or replayed | 🟡 Low | — | **Open, owner decision** (§20) | +| 21 | PageBased non-key lookup scans the table | 🟡 Low | Look up by primary key, or use columnar storage | **Open** (§21) | \ No newline at end of file diff --git a/docs/storage/PAGEBASED_DURABILITY_DESIGN.md b/docs/storage/PAGEBASED_DURABILITY_DESIGN.md new file mode 100644 index 00000000..168810d5 --- /dev/null +++ b/docs/storage/PAGEBASED_DURABILITY_DESIGN.md @@ -0,0 +1,134 @@ +# PageBased storage — crash safety, checksums and encryption: design note + +**Status:** proposal for owner decision (2026-11-17). No code in this note is implemented yet. It covers known issues +§14 (unencrypted pages), §18 (uncommitted pages reaching disk), §19 (torn pages undetected) and §20 (the directory-mode +WAL), and records why §15 (autocommit durability) was fixed without them. + +## 1. What holds today, and what does not + +Measured by killing a child process (no Dispose, no flush) and reopening, and by damaging files the way a torn write +would (`CrashDurabilityTests`, the investigation recorded in known issues §14–§20): + +| Situation | Columnar | PageBased | +|---|---|---| +| Autocommit statement / `ExecuteBatchSQL` that returned, then crash | kept | kept (since §15 was fixed: the page cache is flushed at the statement's end) | +| Committed transaction, then crash | kept | kept | +| Uncommitted transaction, then crash | not applied | not applied — **unless** a dirty page was evicted first (§18) | +| Torn write inside a file | the incomplete trailing record is lost, the rest reads | **rows vanish silently**, writes continue on the damaged page (§19) | +| Power loss after a commit | kept (appends written through, in-place writes synced since §17) | kept under `FullSync` (page file synced at flush) | +| Data at rest | AES-256-GCM per record | **plaintext** (§14) | + +The PageBased engine (`PageBasedEngine` → one `PageManager` per table → `table_.pages`, 8 KB pages) writes pages in +place. Its only atomicity mechanism is the in-memory undo images of §13 (`PageManager.BeginUndoScope`), which survive a +rollback but not a crash. + +## 2. The page as it is on disk + +``` +offset 0 TableId uint32 +offset 4 Type byte +offset 5 RecordCount uint16 +offset 7 FreeSpace uint16 +offset 9 — unused (23 bytes, always written as zero) — +offset 32 data area, growing upwards +... slot directory at the end: 4 bytes per slot (offset uint16, length uint16; length 0 = deleted since §15's fix) +``` + +The 23 unused header bytes are the design's main lever: a checksum and a log sequence number fit **without moving the +data area or the slot directory**, so the page format stays readable by the current reader and an old page stays +readable by the new one (all-zero fields mean "written before checksums"). + +## 3. Torn pages — a checksum per page (§19) + +**Proposal.** Header bytes 9–16: `uint64` XxHash64 (System.IO.Hashing) of the page with these 8 bytes zeroed; byte 17: +format flag `0x01` = checksummed. `WritePage` computes it; `ReadPage` verifies it when the flag is set and throws a +`PageCorruptedException` naming the table and page id when it does not match. A page with flag `0x00` (written before) +is read unverified, exactly as today. + +- *Cost:* one 8 KB hash per page write and read — XxHash64 runs at several GB/s, i.e. around 1 µs per page; measure on + the PageBased benchmark (`--engine=pagebased`) and the gate. +- *What it buys:* a torn page becomes a loud error instead of silently missing rows. It does not repair the page — + that needs §4's log. +- *Compatibility:* none broken; downgrade (unsupported by policy) would read checksummed pages fine, since an old reader + ignores bytes 9–31. + +This piece is independent of the others and could land first. + +## 4. Crash atomicity — a page redo log (§18, and the real fix for §15) + +**Options considered.** + +| Option | How | Pros | Cons | +|---|---|---|---| +| A. Pin dirty pages of an open transaction | Never evict a page that has an undo image | Small change; closes §18 for transactions that fit in the cache | A transaction larger than the cache (1,024 pages = 8 MB by default) must spill or fail; a crash *during* the commit's multi-page flush still tears the transaction | +| B. Page redo log (WAL) | Before a commit's pages are written in place, append their after-images plus a commit record to `table_.wal` and sync it; replay committed records on open; truncate after a checkpoint | Atomic and durable commits, torn-page repair (the log holds a full image), autocommit becomes one sequential append + one sync instead of N random page writes | New file, recovery code, checkpointing; the most work | +| C. Shadow paging | Write changed pages to new locations, then switch a page table atomically | No log replay | Needs a page-table indirection the engine does not have; fragmentation; largest change | + +**Recommendation: B, with A as an interim step if wanted.** + +Design of B: + +1. **Log file** per table: `table_.wal`, records `[LSN uint64][kind byte][page id uint64][length uint32][payload][XxHash64 of record]`, kinds `PageImage` (8 KB after-image) and `Commit` (transaction id). Encrypted with the same key as the pages once §14 lands. +2. **Commit** (`EndBatchUpdate`, `ExecuteBatchSQL`, the autocommit boundary): append the after-image of every dirty page and a `Commit` record, sync the log once — this is the durability point — then write the pages in place lazily (or at once) and stamp each page's LSN (header bytes 18–25). +3. **Eviction during a transaction** may write a dirty page in place only after its after-image is in the log (the + "write-ahead" rule); with the undo images of §13 the in-memory rollback stays as it is. On a crash, replay restores + only committed images; an uncommitted page written early is overwritten by the last committed image of that page + (the log keeps the before-image's committed predecessor) — or, simpler, Option A's pinning keeps uncommitted pages + out of the file entirely and the log needs no undo. +4. **Recovery on open**: scan the log, collect page images of transactions that have a `Commit` record, and write each + page whose header LSN is older than the logged one. Torn pages (§3 checksum fails) are repaired from the log when it + holds their image. +5. **Checkpoint**: after a flush has written and synced all pages up to an LSN, truncate the log. Trigger by size + (e.g. 16 MB) and on `Dispose`. + +*Cost to measure:* autocommit PageBased writes today pay a random 8 KB page write plus an fsync per touched page file +(~750 µs per statement, measured with §15's fix); a log append + one sync is typically cheaper than that, so B may be +faster than today's §15 fix. Batches pay one sequential log write per commit. + +## 5. Encryption at rest (§14) + +AES-256-GCM per page needs a 12-byte nonce and a 16-byte tag per page — 28 bytes, more than the 23 free header bytes. +Options: + +1. **Take 28 bytes from the data area** (usable page 8,160 → 8,132 bytes): a format change — existing pages keep the + current layout (flag `0x00`), new or rewritten pages use the encrypted layout (flag `0x02`); a background or + on-open rewrite converts old pages. Same key derivation as the columnar records. +2. **XTS-style length-preserving encryption** (no tag) using the 23 free bytes for a per-page tweak/counter: no layout + change, but no integrity — the §3 checksum would have to be computed over the ciphertext. + +Recommendation: option 1 (authenticated, matches the columnar records' AES-GCM), landed together with the checksum flag +so one format bump covers both. Needs the owner's decision: it is an on-disk format change, and a mixed-format file must +stay readable until it is rewritten. + +## 6. The directory-mode `GroupCommitWAL` (§20) + +It is created when `UseGroupCommitWal` is set, but nothing writes to it or replays it. Two honest outcomes: + +- **Remove it from directory mode** and the WAL settings' claims, since the columnar engine's durability comes from + written-through appends and §17's syncs, and the PageBased engine would get its own per-table log (§4). +- **Or** make §4's log the implementation behind `UseGroupCommitWal` for directory mode (group commit = several + transactions' records synced together). + +Recommendation: the second — one documented WAL, used by the engine that needs it — but it is the owner's call because +it changes what a public configuration flag means. + +## 7. Suggested order and test plan + +1. §3 checksums (no format break, independent) — tests: damage a page's tail → `PageCorruptedException` names it; a + page written before (flag 0) reads unverified; reopen round-trip matrix stays green. +2. Option A pinning (§18 interim) — test: the eviction crash case of the investigation (`tx-uncommitted-then-big-read`) + leaves the committed state. +3. §4 log + recovery — tests: kill the child process at each step of the commit (after the log sync, between page + writes) via a test hook; replay restores exactly the committed state; torn page repaired from the log; checkpoint + truncates. +4. §5 encryption with the format flag — tests: no plaintext in the page file; mixed old/new pages read; rewrite converts. + +Every step runs the write-path gate and the PageBased benchmark (`--engine=pagebased`), and adds its crash cases to +`CrashDurabilityTests`. + +## 8. Decisions needed from the owner + +- §14: approve the page format bump (encrypted layout, flag byte) and the conversion strategy for existing files. +- §20: remove the directory-mode `GroupCommitWAL`, or make it the PageBased log. +- Whether PageBased should be documented as not production-ready (the configuration already says "NOT yet OLTP-ready") + until §14 and §18 land. diff --git a/src/SharpCoreDB/DataStructures/Table.TransactionUndo.cs b/src/SharpCoreDB/DataStructures/Table.TransactionUndo.cs index 3a4d2733..e313fd17 100644 --- a/src/SharpCoreDB/DataStructures/Table.TransactionUndo.cs +++ b/src/SharpCoreDB/DataStructures/Table.TransactionUndo.cs @@ -35,6 +35,23 @@ internal void CommitTransactionUndo() if (_storageEngine is PageBasedEngine engine) { engine.CommitUndoScope(); + + // The commit is the durability boundary: the transaction's pages go to the file now. EndBatchUpdate flushed + // already; an ExecuteBatchSQL commit did not, so up to 2.1.4 a crash after a returned batch lost it (§15). + engine.Flush(); + } + } + + /// + /// After an autocommit statement: writes the PageBased engine's dirty pages to the file (and syncs it), so a crash + /// after the statement returned keeps it. Up to 2.1.4 the pages stayed in the cache until an explicit Flush(), a + /// commit, an eviction or Dispose (known issue §15). Free when the statement changed no page. + /// + internal void FlushAutocommitPages() + { + if (_storageEngine is PageBasedEngine engine) + { + engine.Flush(); } } diff --git a/src/SharpCoreDB/Database/Core/Database.Core.cs b/src/SharpCoreDB/Database/Core/Database.Core.cs index a906be56..84f918df 100644 --- a/src/SharpCoreDB/Database/Core/Database.Core.cs +++ b/src/SharpCoreDB/Database/Core/Database.Core.cs @@ -582,6 +582,13 @@ private void SyncAutocommitWrites() if (storage is { IsInTransaction: false } current) { current.SyncPendingWrites(); + foreach (var table in tables.Values) + { + if (table is Table { StorageMode: Storage.Hybrid.StorageMode.PageBased } pageTable) + { + pageTable.FlushAutocommitPages(); + } + } } } diff --git a/src/SharpCoreDB/Services/SqlParser.SetExpressions.cs b/src/SharpCoreDB/Services/SqlParser.SetExpressions.cs index 6098e3a4..072f3b87 100644 --- a/src/SharpCoreDB/Services/SqlParser.SetExpressions.cs +++ b/src/SharpCoreDB/Services/SqlParser.SetExpressions.cs @@ -30,7 +30,7 @@ public partial class SqlParser internal static bool IsComputedSetValue(string valueText, IReadOnlyList columns, out SqlScalarExpression.Node node) { node = null!; - if (IsPlainSetLiteral(valueText) || !SqlScalarExpression.TryParse(valueText.Trim(), out var parsed)) + if (IsPlainSetLiteral(valueText) || !SqlScalarExpression.TryParseValue(valueText.Trim(), out var parsed)) { return false; } diff --git a/src/SharpCoreDB/Services/SqlScalarExpression.cs b/src/SharpCoreDB/Services/SqlScalarExpression.cs index 87b985b2..47d330d0 100644 --- a/src/SharpCoreDB/Services/SqlScalarExpression.cs +++ b/src/SharpCoreDB/Services/SqlScalarExpression.cs @@ -56,6 +56,26 @@ internal static bool TryParse(string text, out Node node) return node is not null; } + /// + /// Parses a value: a scalar expression, or — as SQLite allows — a condition, whose value is 1, 0 or NULL + /// (SET active = length(name) > 0, THEN amount < 0). + /// + internal static bool TryParseValue(string text, out Node node) + { + if (TryParse(text, out node)) + { + return true; + } + + if (text.Length > 0 && ConditionCompiler.CompileAny(text) is { } condition) + { + node = new ConditionValue(condition); + return true; + } + + return false; + } + /// Returns true when the node is computed — a function, CAST, operator — not a literal or column. internal static bool IsComputed(Node node) => node is not (Literal or Column); @@ -129,7 +149,7 @@ internal static bool TryEvaluateFilter(string condition, Dictionary isColumn) => node switch { Column column => isColumn(column.Name), - Call or Cast => true, + Call or Cast or CaseExpression or ConditionValue => true, Binary binary => ReferencesColumnOrFunction(binary.Left, isColumn) || ReferencesColumnOrFunction(binary.Right, isColumn), Negate negate => ReferencesColumnOrFunction(negate.Operand, isColumn), _ => false, @@ -191,6 +211,25 @@ private sealed class Negate(Node operand) : Node public Node Operand { get; } = operand; } + /// A condition used as a value: 1 when true, 0 when false, NULL when unknown (SQLite's boolean values). + private sealed class ConditionValue(Condition condition) : Node + { + public Condition Condition { get; } = condition; + } + + /// + /// CASE [operand] WHEN … THEN … [ELSE …] END. A simple CASE compares the operand with each WHEN value (NULL + /// never matches); a searched CASE takes the first WHEN condition that is true. No match and no ELSE is NULL. + /// + private sealed class CaseExpression(Node? operand, (Condition? Condition, Node? Value, Node Result)[] arms, Node? otherwise) : Node + { + public Node? Operand { get; } = operand; + + public (Condition? Condition, Node? Value, Node Result)[] Arms { get; } = arms; + + public Node? Otherwise { get; } = otherwise; + } + // ── Parser ────────────────────────────────────────────────────────────────────────────────── private sealed class ParseFailure : Exception; @@ -309,6 +348,11 @@ private Node ParsePrimary() { int start = _pos; var word = ReadWord(); + if (word.Equals("CASE", StringComparison.OrdinalIgnoreCase)) + { + return ReadCase(); + } + SkipWhitespace(); if (_pos < text.Length && text[_pos] == '(') { @@ -338,6 +382,170 @@ private Node ParsePrimary() throw new ParseFailure(); } + /// + /// Reads a CASE expression after its CASE keyword, up to the matching END. Up to 2.1.4 CASE was not parsed: a SET + /// value CASE WHEN … END fell back to the literal reading and stored the text, and a CHECK written with CASE + /// was left to an evaluator that rejected the row. + /// + private Node ReadCase() + { + var keywords = FindCaseKeywords(text, _pos, out int afterEnd); + if (keywords.Count == 0 || keywords[^1].Keyword != "END" || keywords[0].Keyword != "WHEN") + { + throw new ParseFailure(); + } + + var operandText = text[_pos..keywords[0].At].Trim(); + Node? operand = operandText.Length == 0 ? null : ParseSegment(operandText); + + var arms = new List<(Condition?, Node?, Node)>(); + Node? otherwise = null; + for (int k = 0; k < keywords.Count - 1; k++) + { + var (keyword, at) = keywords[k]; + int segmentStart = at + keyword.Length; + int segmentEnd = keywords[k + 1].At; + var segment = text[segmentStart..segmentEnd].Trim(); + if (keyword == "WHEN") + { + if (keywords[k + 1].Keyword != "THEN" || k + 2 >= keywords.Count) + { + throw new ParseFailure(); + } + + var thenEnd = keywords[k + 2].At; + var result = ParseSegment(text[(keywords[k + 1].At + 4)..thenEnd].Trim()); + if (operand is null) + { + var condition = ConditionCompiler.CompileAny(segment) ?? throw new ParseFailure(); + arms.Add((condition, null, result)); + } + else + { + arms.Add((null, ParseSegment(segment), result)); + } + + k++; // the THEN was consumed with its WHEN + } + else if (keyword == "ELSE") + { + otherwise = ParseSegment(segment); + } + else + { + throw new ParseFailure(); + } + } + + if (arms.Count == 0) + { + throw new ParseFailure(); + } + + _pos = afterEnd; + return new CaseExpression(operand, [.. arms], otherwise); + } + + /// + /// Parses a CASE operand, WHEN value or result. A comparison or a boolean condition is a value in SQLite + /// (THEN amount < 0 is 1, 0 or NULL), so text the expression parser does not read is compiled as a + /// condition. + /// + private static Node ParseSegment(string segment) + { + if (segment.Length == 0) + { + throw new ParseFailure(); + } + + return TryParseValue(segment, out var node) ? node : throw new ParseFailure(); + } + + /// + /// The top-level WHEN/THEN/ELSE/END keywords of the CASE whose body starts at (nested CASE + /// expressions, parentheses and quoted text are skipped), ending with its END. + /// + internal static List<(string Keyword, int At)> FindCaseKeywords(string text, int start, out int afterEnd) + { + var found = new List<(string, int)>(); + afterEnd = -1; + int depth = 0; + int nested = 0; + for (int i = start; i < text.Length; i++) + { + char c = text[i]; + if (c is '\'' or '"' or '`' or '[') + { + char close = c == '[' ? ']' : c; + for (i++; i < text.Length; i++) + { + if (text[i] != close) + { + continue; + } + + if (close != ']' && i + 1 < text.Length && text[i + 1] == close) + { + i++; + continue; + } + + break; + } + + continue; + } + + if (c == '(') + { + depth++; + continue; + } + + if (c == ')') + { + depth--; + continue; + } + + if (depth != 0 || !char.IsLetter(c) || (i > 0 && (char.IsLetterOrDigit(text[i - 1]) || text[i - 1] == '_'))) + { + continue; + } + + int wordEnd = i; + while (wordEnd < text.Length && (char.IsLetterOrDigit(text[wordEnd]) || text[wordEnd] == '_')) + { + wordEnd++; + } + + var word = text[i..wordEnd].ToUpperInvariant(); + if (word == "CASE") + { + nested++; + } + else if (word == "END") + { + if (nested == 0) + { + found.Add(("END", i)); + afterEnd = wordEnd; + return found; + } + + nested--; + } + else if (nested == 0 && word is "WHEN" or "THEN" or "ELSE") + { + found.Add((word, i)); + } + + i = wordEnd - 1; + } + + return []; + } + private Node ReadCall(string word) { var name = word.ToUpperInvariant(); @@ -559,9 +767,28 @@ private void SkipWhitespace() Cast cast => CastValue(Eval(cast.Operand, row), cast.TypeName), Binary binary => EvalBinary(binary.Op, Eval(binary.Left, row), Eval(binary.Right, row)), Negate negate => EvalBinary("-", 0L, Eval(negate.Operand, row)), + CaseExpression caseExpression => EvalCase(caseExpression, row), + ConditionValue conditionValue => conditionValue.Condition.Evaluate(row) switch { true => 1L, false => 0L, null => null }, _ => null, }; + private static object? EvalCase(CaseExpression expression, Dictionary row) + { + var operand = expression.Operand is null ? null : Eval(expression.Operand, row); + foreach (var (condition, value, result) in expression.Arms) + { + bool matched = expression.Operand is null + ? condition!.Evaluate(row) == true + : operand is not null && TryCompare(operand, Eval(value!, row), out var c) && c == 0; + if (matched) + { + return Eval(result, row); + } + } + + return expression.Otherwise is null ? null : Eval(expression.Otherwise, row); + } + private static object? ResolveColumn(Dictionary row, Column column) { if (column.Qualifier is not null && row.TryGetValue($"{column.Qualifier}.{column.Name}", out var qualified)) @@ -1376,6 +1603,13 @@ private sealed class ConditionCompiler return null; } + // A condition that starts with a CASE expression is read around it: the CASE's own WHEN conditions hold + // comparisons and AND/OR that a split of the whole text would cut through. + if (StartsWithKeyword(text, "CASE") && CompileLeadingCase(text) is { } leadingCase) + { + return leadingCase; + } + var orParts = SqlInPredicate.SplitTopLevelLogical(text, "OR"); if (orParts.Count > 1) { @@ -1414,6 +1648,50 @@ private sealed class ConditionCompiler return combine(compiled); } + /// + /// CASE … END on its own (its truth value), or followed by a comparison or IS [NOT]. Returns + /// null for any other shape, which then takes the general path. + /// + private Condition? CompileLeadingCase(string text) + { + Parser.FindCaseKeywords(text, 4, out int afterEnd); + if (afterEnd <= 0) + { + return null; + } + + var left = ParseOperand(text[..afterEnd]); + var rest = text[afterEnd..].Trim(); + if (left is null) + { + return null; + } + + if (rest.Length == 0) + { + return new TruthCondition(left); + } + + if (StartsWithKeyword(rest, "IS")) + { + var isRest = rest[2..].TrimStart(); + bool negated = StartsWithKeyword(isRest, "NOT"); + var right = ParseOperand(negated ? isRest[3..] : isRest); + return right is null ? null : new IsCondition(left, right, negated); + } + + foreach (var op in SymbolOperators) + { + if (rest.StartsWith(op, StringComparison.Ordinal)) + { + var right = ParseOperand(rest[op.Length..]); + return right is null ? null : new CompareCondition(op, left, right); + } + } + + return null; + } + /// /// Puts a BETWEEN's own AND back together after a split on AND: n BETWEEN 1 AND 10 AND m > 0 splits into /// three parts, of which the first two are one BETWEEN. diff --git a/src/SharpCoreDB/Storage/PageManager.cs b/src/SharpCoreDB/Storage/PageManager.cs index d8258257..785b5332 100644 --- a/src/SharpCoreDB/Storage/PageManager.cs +++ b/src/SharpCoreDB/Storage/PageManager.cs @@ -123,6 +123,9 @@ public enum RecordFlags : byte private Dictionary>? undoTablePages; private ulong[]? undoFreePages; + /// Whether syncs the page file to disk (WalDurabilityMode.FullSync). + private readonly bool syncOnFlush = true; + // Cache statistics private long cacheHits = 0; private long cacheMisses = 0; @@ -180,6 +183,10 @@ public PageManager(string databasePath, uint tableId) public PageManager(string databasePath, uint tableId, DatabaseConfig? config) { cacheCapacity = config?.PageCacheCapacity ?? 1024; + + // FullSync (the default) forces flushed pages to the physical disk; Async writes them to the file and leaves the + // sync to the operating system — the same split as the columnar in-place writes (known issue §17). + syncOnFlush = (config?.WalDurabilityMode ?? Services.DurabilityMode.FullSync) == Services.DurabilityMode.FullSync; var pagesFilePath = Path.Combine(databasePath, $"table_{tableId}.pages"); pagesFile = new FileStream(pagesFilePath, FileMode.OpenOrCreate, FileAccess.ReadWrite, FileShare.ReadWrite); @@ -777,6 +784,13 @@ public virtual void FlushDirtyPages() { lock (writeLock) { + // Nothing to write: no fsync either. The flush now runs after every autocommit statement (known issue §15), + // and a SELECT must not pay a disk sync for it. + if (dirtyPages.Count == 0) + { + return; + } + foreach (var pageId in dirtyPages.ToList()) { if (pageCache.TryGetValue(pageId, out var page)) @@ -788,8 +802,8 @@ public virtual void FlushDirtyPages() } dirtyPages.Clear(); - // PERF: Single OS-level flush after all pages written - pagesFile?.Flush(flushToDisk: true); + // PERF: Single OS-level flush after all pages written (a disk sync under FullSync only) + pagesFile?.Flush(flushToDisk: syncOnFlush); } } diff --git a/tests/SharpCoreDB.Tests/CaseExpressionTests.cs b/tests/SharpCoreDB.Tests/CaseExpressionTests.cs new file mode 100644 index 00000000..6d0f26a8 --- /dev/null +++ b/tests/SharpCoreDB.Tests/CaseExpressionTests.cs @@ -0,0 +1,124 @@ +// +// Copyright (c) 2026 MPCoreDeveloper. All rights reserved. +// Licensed under the MIT License. See LICENSE file in the project root for full license information. +// + +namespace SharpCoreDB.Tests; + +using System; +using System.Globalization; +using System.IO; +using System.Linq; +using Microsoft.Extensions.DependencyInjection; +using SharpCoreDB.Interfaces; +using Xunit; + +/// +/// SQLite's CASE [operand] WHEN … THEN … [ELSE …] END in the scalar-expression evaluator — UPDATE SET values, +/// CHECK constraints, ORDER BY keys and the SELECT list. +/// +/// Up to 2.1.4 CASE was not parsed: SET grade = CASE WHEN n >= 5 THEN 'high' ELSE 'low' END fell back to the +/// literal reading and stored the statement text, and a CHECK written with CASE was left to an evaluator that rejected +/// every row. +/// +/// +public sealed class CaseExpressionTests : IDisposable +{ + private readonly DatabaseFactory _factory; + private readonly string _root = Path.Combine(Path.GetTempPath(), $"SCDB_Case_{Guid.NewGuid():N}"); + + public CaseExpressionTests() + { + var services = new ServiceCollection(); + services.AddSharpCoreDB(); + _factory = services.BuildServiceProvider().GetRequiredService(); + Directory.CreateDirectory(_root); + } + + public void Dispose() + { + try + { + if (Directory.Exists(_root)) + { + Directory.Delete(_root, true); + } + } + catch (IOException) + { + // Best-effort cleanup of a temp directory. + } + } + + private IDatabase Seed() + { + var db = _factory.Create(Path.Combine(_root, $"db_{Guid.NewGuid():N}"), "pw"); + db.ExecuteSQL("CREATE TABLE t (id INTEGER PRIMARY KEY, n INTEGER, kind TEXT, grade TEXT)"); + db.ExecuteSQL("INSERT INTO t (id, n, kind, grade) VALUES (1, 2, 'a', NULL), (2, 7, 'b', NULL), (3, NULL, 'c', NULL)"); + return db; + } + + private static string[] Rows(IDatabase db, string sql) => + [.. db.ExecuteQuery(sql).Select(r => string.Join(",", r.Values.Select(v => v is null or DBNull ? "NULL" : Convert.ToString(v, CultureInfo.InvariantCulture))))]; + + [Fact] + public void Update_SetToASearchedCase_TakesTheFirstTrueBranch() + { + var db = Seed(); + + db.ExecuteSQL("UPDATE t SET grade = CASE WHEN n >= 5 THEN 'high' WHEN n IS NULL THEN 'none' ELSE 'low' END"); + + Assert.Equal(["1,low", "2,high", "3,none"], Rows(db, "SELECT id, grade FROM t").Order()); + } + + [Fact] + public void Update_SetToASimpleCase_ComparesTheOperand() + { + var db = Seed(); + + db.ExecuteSQL("UPDATE t SET grade = CASE kind WHEN 'a' THEN 'first' WHEN 'b' THEN 'second' END"); + + Assert.Equal(["1,first", "2,second", "3,NULL"], Rows(db, "SELECT id, grade FROM t").Order()); + } + + [Fact] + public void Update_SetToANestedCaseWithArithmetic_IsComputedPerRow() + { + var db = Seed(); + + db.ExecuteSQL("UPDATE t SET n = CASE WHEN n IS NULL THEN 0 ELSE CASE WHEN n > 5 THEN n * 10 ELSE n + 1 END END"); + + Assert.Equal(["1,3", "2,70", "3,0"], Rows(db, "SELECT id, n FROM t").Order()); + } + + [Fact] + public void Check_WrittenWithCase_IsEnforced() + { + var db = _factory.Create(Path.Combine(_root, $"db_{Guid.NewGuid():N}"), "pw"); + db.ExecuteSQL("CREATE TABLE p (id INTEGER PRIMARY KEY, kind TEXT, amount INTEGER, " + + "CHECK (CASE WHEN kind = 'refund' THEN amount < 0 ELSE amount > 0 END))"); + + db.ExecuteSQL("INSERT INTO p (id, kind, amount) VALUES (1, 'sale', 10)"); + db.ExecuteSQL("INSERT INTO p (id, kind, amount) VALUES (2, 'refund', -5)"); + Assert.ThrowsAny(() => db.ExecuteSQL("INSERT INTO p (id, kind, amount) VALUES (3, 'refund', 5)")); + + Assert.Equal(["1,10", "2,-5"], Rows(db, "SELECT id, amount FROM p").Order()); + } + + [Fact] + public void OrderBy_ACaseKey_SortsByIt() + { + var db = Seed(); + + Assert.Equal(["2", "1", "3"], Rows(db, "SELECT id FROM t ORDER BY CASE kind WHEN 'b' THEN 0 WHEN 'a' THEN 1 ELSE 2 END")); + } + + [Fact] + public void Select_ACaseColumn_IsEvaluatedPerRow() + { + var db = Seed(); + + Assert.Equal(["1,small", "2,big", "3,unknown"], + Rows(db, "SELECT id, CASE WHEN n > 5 THEN 'big' WHEN n IS NULL THEN 'unknown' ELSE 'small' END AS size FROM t").Order()); + } +} diff --git a/tests/SharpCoreDB.Tests/CrashDurabilityTests.cs b/tests/SharpCoreDB.Tests/CrashDurabilityTests.cs index ace50721..6ab2cc08 100644 --- a/tests/SharpCoreDB.Tests/CrashDurabilityTests.cs +++ b/tests/SharpCoreDB.Tests/CrashDurabilityTests.cs @@ -20,8 +20,8 @@ namespace SharpCoreDB.Tests; /// The crash cases run the scenario in a child test process that kills itself (no Dispose, no finalizer, no flush) and /// reopen the database in this process. A process kill keeps whatever reached the operating system, so these pin /// crash consistency, not power-loss durability: the fsync of in-place writes and the atomic metadata replace -/// (known issues §17 and §16) are verified by reading the code, not by these tests. The PageBased cases that do not hold -/// yet are skipped with the known issue that tracks them (§15). +/// (known issues §17 and §16) are verified by reading the code, not by these tests. The PageBased autocommit and batch +/// cases failed up to 2.1.4 — the statement stayed in the page cache until a flush (known issue §15). /// /// public sealed class CrashDurabilityTests : IDisposable @@ -103,6 +103,9 @@ public void CrashChild_RunsTheScenarioAndKillsItsProcess() { "columnar", "batch", "1:999 2:2 3:998" }, { "columnar", "transaction-committed", "1:999 3:3 100:100" }, { "columnar", "transaction-uncommitted", "1:1 2:2 3:3" }, + { "page-based", "autocommit-update", "1:999 2:2 3:3" }, + { "page-based", "autocommit-insert-delete", "1:1 3:3 100:100" }, + { "page-based", "batch", "1:999 2:2 3:998" }, { "page-based", "transaction-committed", "1:999 3:3 100:100" }, { "page-based", "transaction-uncommitted", "1:1 2:2 3:3" }, }; @@ -119,13 +122,7 @@ public void Crash_AfterTheScenario_LeavesTheCommittedState(string storage, strin Assert.Equal(expected, RunCrashScenario(storage, scenario)); } - [Fact(Skip = "Known issue §15: a PageBased autocommit statement stays in the page cache until a flush, so a crash loses it.")] - public void Crash_AfterAPageBasedAutocommitUpdate_KeepsTheUpdate() => - Assert.Equal("1:999 2:2 3:3", RunCrashScenario("page-based", "autocommit-update")); - [Fact(Skip = "Known issue §15: ExecuteBatchSQL does not flush the PageBased page cache, so a crash loses a completed batch.")] - public void Crash_AfterAPageBasedBatch_KeepsTheBatch() => - Assert.Equal("1:999 2:2 3:998", RunCrashScenario("page-based", "batch")); [Fact] public void Open_WithAnEmptyMetadataFile_ReportsAnInterruptedSave_NotTheWrongPassword() diff --git a/tests/SharpCoreDB.Tests/SubqueryOrderByCollationTests.cs b/tests/SharpCoreDB.Tests/SubqueryOrderByCollationTests.cs index 3269aaa6..87ae0724 100644 --- a/tests/SharpCoreDB.Tests/SubqueryOrderByCollationTests.cs +++ b/tests/SharpCoreDB.Tests/SubqueryOrderByCollationTests.cs @@ -81,6 +81,16 @@ public void DerivedTable_OrderByTextWithoutCollate_SortsAsBinary() Assert.Equal(["A", "C", "a", "b"], Names(db, "SELECT name FROM (SELECT name FROM p) ORDER BY name")); } + [Fact] + public void InSubqueryWithOrderByAndLimit_SelectsTheRowsItsOwnOrderPicks() + { + // The subquery's own ORDER BY decides which rows its LIMIT keeps: the two highest scores, ties by name DESC. + var db = Seed(); + db.ExecuteSQL("INSERT INTO p (id, name, score) VALUES (5, 'd', 9)"); + + Assert.Equal(["a", "d"], Names(db, "SELECT name FROM p WHERE id IN (SELECT id FROM p ORDER BY score DESC, name DESC LIMIT 2) ORDER BY name")); + } + [Fact] public void DerivedTable_OrderByCollateNoCase_SortsCaseInsensitively() { diff --git a/tests/SharpCoreDB.Tests/UpdateSetExpressionTests.cs b/tests/SharpCoreDB.Tests/UpdateSetExpressionTests.cs index b19d74ad..c78b8e82 100644 --- a/tests/SharpCoreDB.Tests/UpdateSetExpressionTests.cs +++ b/tests/SharpCoreDB.Tests/UpdateSetExpressionTests.cs @@ -104,6 +104,34 @@ public void Update_ConstantExpression_IsEvaluatedForANumericColumn(string mode) Assert.Equal(["1,6"], Rows(db, "SELECT id, n FROM t")); } + [Fact] + public void Update_ComputedValues_AreConvertedToEachColumnsType() + { + var (db, _) = Open("columnar", "CREATE TABLE t (id INTEGER PRIMARY KEY, r REAL, big BIGINT, flag BOOLEAN, d DATETIME, label TEXT)"); + db.ExecuteSQL("INSERT INTO t (id, r, big, flag, d, label) VALUES (1, 1.5, 10, 0, '2026-01-01 00:00:00', 'x')"); + + db.ExecuteSQL("UPDATE t SET r = r * 2, big = big * 1000000000, flag = length(label) > 0, " + + "d = datetime(d, '+1 day'), label = id || '-' || r WHERE id = 1"); + + var row = db.ExecuteQuery("SELECT r, big, flag, d, label FROM t WHERE id = 1").Single(); + Assert.Equal(3.0, Convert.ToDouble(row["r"], CultureInfo.InvariantCulture)); + Assert.Equal(10_000_000_000L, Convert.ToInt64(row["big"], CultureInfo.InvariantCulture)); + Assert.True(Convert.ToBoolean(row["flag"], CultureInfo.InvariantCulture)); + Assert.Equal(new DateTime(2026, 1, 2), Convert.ToDateTime(row["d"], CultureInfo.InvariantCulture).Date); + Assert.Equal("1-1.5", row["label"]); + } + + [Fact] + public void Update_ComputedValueTooLargeForAnIntegerColumn_IsRejectedAndChangesNothing() + { + var (db, _) = Open("columnar", "CREATE TABLE t (id INTEGER PRIMARY KEY, n INTEGER)"); + db.ExecuteSQL("INSERT INTO t (id, n) VALUES (1, 2000000000)"); + + Assert.ThrowsAny(() => db.ExecuteSQL("UPDATE t SET n = n * 4 WHERE id = 1")); + + Assert.Equal(["1,2000000000"], Rows(db, "SELECT id, n FROM t")); + } + [Theory] [MemberData(nameof(Modes))] public void BatchUpdate_SetValueReadsTheRow_SeesTheEarlierStatement(string mode)