diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index 66997e32..cd354c0e 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -86,6 +86,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - **Design note for PageBased crash safety and encryption.** `docs/storage/PAGEBASED_DURABILITY_DESIGN.md` proposes page checksums in the unused header bytes (no format change, §19), a per-table page redo log with recovery and checkpoints (§18), AES-GCM page encryption with a format flag (§14) and the future of the directory-mode WAL (§20), with the owner decisions each needs. - **Investigated: the READ ratios in the write-path gate's watch band.** An interleaved A/B of `master` against this stack (point lookups by key and by name, full scans; three rounds each) shows no read regression — key lookups equal, name lookups ~25 % faster, scans within ±3 % — so the watch-band READ ratios reflect this machine against the recorded baseline. Found on the way and recorded: a PageBased lookup on a non-key column scans the table (~32 ms for 20,000 rows on `master` and here), because hash indexes exist for columnar tables only. Coverage of the stack: 48.47 % overall (floor 18 %), 84.9 % of the new lines. Suites: core **2546/0 failed (6 skipped)**, VectorSearch 255/0, EF Core 116/0, linq2DB 96/0, regression canaries 51/0; write-path gate: GATE PASSED (raw UPDATE 1.25× with #522's commit fsync included, where #522's run measured 1.54×; READ 1.43×/1.30× in the watch band, shown by the A/B to be this machine, not the code). +- **PageBased pages carry a checksum (known issue §19).** A torn page — the tail of an 8 KB page not written when a crash interrupted it — read as a page with empty slots: its rows vanished without an error and the table kept accepting writes. Every page written now carries a CRC32C in header bytes 9–12 with a format flag in byte 13 (bytes no version wrote, so the layout is unchanged and older pages read unverified); a damaged page raises `InvalidDataException` naming it. The PageBased scan's catch-all, which returned the rows read so far on any error, lets that error through. Cost ~2.9 µs per page. Pinned by `PageChecksumTests`. +- **An uncommitted PageBased transaction could reach the disk (known issue §18, interim fix).** A flush in the middle of a transaction — `ExecuteQuery` flushes pending writes inside `BeginBatchUpdate` — or a cache eviction wrote the transaction's dirty pages, so a crash before its commit kept them. Pages changed by the open transaction are now pinned until the commit; the cache may exceed its capacity meanwhile. The eviction loop also no longer spins when the cache is over capacity and no page is in its LRU list. A crash during the commit's own multi-page write still needs the page redo log. Pinned by `CrashDurabilityTests`. +- **PageBased non-key lookups (known issue §21): by design.** PageBased tables do not auto-create column indexes (columnar tables do; SQLite does not); `CREATE INDEX` takes a lookup on 20,000 rows from ~80 ms to 0.02–0.1 ms, and `PageBasedSecondaryIndexTests` pins its correctness through writes and a reopen. +- **SonarCloud quality gate on #521.** Three copies of the same quote- and parenthesis-aware scanner (UPDATE SET/WHERE split, ON CONFLICT finder, ORDER BY item end) are one helper now (`SqlLiteralScanner.FindTopLevel`), and the flagged complexity smells in the upsert code are split up; no behaviour change. Suites: core **2551/0 failed (6 skipped)**, VectorSearch 255/0, EF Core 116/0, linq2DB 96/0, regression canaries 51/0; write-path gate: GATE FAILED on raw READ (1.58×) in a run whose own rep spread was 1.33× READ / 1.76× UPDATE / 1.96× DELETE; an interleaved A/B of the gate's exact read statement (parameterized SELECT * … WHERE name = @name, NoEncryptMode) against master measured 7.45–9.05 µs vs 7.98–8.21 µs — no code regression. + ## [2.1.4] - 2026-11-13 > The issue #497 follow-up: a quoted identifier in `CREATE TABLE` is catalogued without its quoting, so a schema diff --git a/docs/sharpcoredb-known-issues.md b/docs/sharpcoredb-known-issues.md index 7d628647..87ddb2fc 100644 --- a/docs/sharpcoredb-known-issues.md +++ b/docs/sharpcoredb-known-issues.md @@ -660,9 +660,15 @@ Before: appends were opened with `FileOptions.WriteThrough`, but the cached in-p and nothing called `FlushToDisk`, so a power loss (not a process crash) could lose a committed UPDATE or DELETE while keeping committed INSERTs. -## 18. 🟠 OPEN — An uncommitted PageBased transaction can reach the disk before a crash +## 18. ✅ FIXED (interim) — An uncommitted PageBased transaction could reach the disk before a crash -> ⚠️ **STILL TRUE (verified 2026-11-17, `fix/durability-metadata-and-fsync`).** +> ✅ **FIXED (2026-11-17, `fix/pagebased-checksums-pinning-indexes`) for pages written before the commit.** A dirty page +> the open transaction changed is pinned: neither a cache eviction nor a flush in the middle of the transaction writes it +> (the main path was not eviction but `ExecuteQuery`, which flushes pending writes inside `BeginBatchUpdate`); it stays in +> memory, where the transaction's own reads see it, and is written at the commit. The cache may exceed its capacity until +> the transaction ends. Pinned by `CrashDurabilityTests.Crash_AfterAnUncommittedPageBasedTransactionWhosePagesWereEvicted_LeavesTheCommittedState` +> (failed before). **Still open:** a crash *during* the commit's multi-page write can leave some of its pages written and +> others not; that needs the page redo log of `docs/storage/PAGEBASED_DURABILITY_DESIGN.md` §4. Measured: a transaction updated row 1 and deleted row 2 of a PageBased table of ~1,400 pages, then read the table twice, and the process was killed before the commit. After reopening, row 1 had the uncommitted value and row 2 was gone. @@ -676,9 +682,16 @@ images are restored and written back); a crash between the eviction and the comm open transaction from eviction (bounded memory then needs a spill file). Engine work; designed in `docs/storage/PAGEBASED_DURABILITY_DESIGN.md` §4. -## 19. 🟠 OPEN — A torn PageBased page loses rows silently +## 19. ✅ FIXED (detection) — A torn PageBased page lost rows silently -> ⚠️ **STILL TRUE (verified 2026-11-17, `fix/durability-metadata-and-fsync`).** +> ✅ **FIXED (2026-11-17, `fix/pagebased-checksums-pinning-indexes`).** Every page written carries a CRC32C checksum in +> header bytes 9–12 and a format flag in byte 13 — header bytes no version ever wrote, so the page layout does not change +> and a page written before (flag 0) is read unverified, as it always was. A page whose checksum fails raises +> `InvalidDataException` naming the page and the file when its records are read; the page-index scan at open does not +> verify, so one damaged page does not stop the table from opening. The PageBased scan's catch-all no longer swallows +> that error (it returned the rows read so far, which is how the damage stayed hidden). Cost: ~2.9 µs per 8 KB page +> written or read from disk (`BitOperations.Crc32C`). Pinned by `PageChecksumTests` (the two damage cases failed before). +> **Still open:** repairing a damaged page needs the page redo log (design note §4). Measured: zeroing the second half of a page's 8 KB — what a torn write leaves when a crash interrupts it — made 25 of 40 rows disappear without an error, and the table kept accepting writes; cutting the page file mid-page lost that page's @@ -701,10 +714,14 @@ the configuration suggests a guarantee that does not exist. The single-file (`.s **Decision needed:** wire the WAL in (log each committed change, replay on open — the natural home for §15/§18's fix), or remove it and the WAL settings from directory mode so the configuration is honest. -## 21. 🟡 OPEN — A PageBased lookup on a non-key column scans the whole table +## 21. ✅ BY DESIGN — A PageBased lookup on a non-key column scans the table unless the column is indexed -> ⚠️ **STILL TRUE (verified 2026-11-17, `fix/pagebased-durability-case-and-coverage`).** Found in the READ A/B of the -> write-path gate's watch band; present on `master` as well. +> ✅ **RESOLVED (2026-11-17, `fix/pagebased-checksums-pinning-indexes`): not a defect.** PageBased tables deliberately do +> not create a column index on first use (`Table.AutoHashIndexesEnabled` — columnar tables do; SQLite does not either). +> An explicit `CREATE INDEX ix ON t (col)` serves the lookup: measured ~80 ms → 0.02–0.1 ms per lookup on 20,000 rows, +> and the index stays correct through INSERT, UPDATE, DELETE and a reopen (`PageBasedSecondaryIndexTests`). Turning on +> automatic indexes for PageBased tables would reverse that documented choice — an owner decision. The cause analysis +> below was wrong in one point (`CREATE INDEX` does give the scan an index route) and is kept as the record. `SELECT … WHERE name = 'User42'` on a 20,000-row PageBased table takes ~32 ms (a full scan per lookup), against ~40 µs on a columnar table, where the column gets an automatic hash index. **Cause:** hash indexes exist for columnar tables only @@ -735,7 +752,7 @@ references, which in-place updates keep stable), or a B-tree secondary index; me | 15 | PageBased autocommit / batch not durable until a flush | 🔴 High | — | **Fixed** — pages flushed at each autocommit statement and commit (§15) | | 16 | Non-atomic metadata write → database unopenable after a crash | 🔴 High | — | **Fixed** — temp file + fsync + rename; clear error for a damaged file (§16) | | 17 | Columnar in-place UPDATE/DELETE not fsynced | 🟠 Medium | — | **Fixed** — synced at commit, autocommit end and flush (§17) | -| 18 | Uncommitted PageBased pages reach disk via eviction | 🟠 Medium | Keep transactions within the page cache | **Open** — needs a page WAL (§18) | -| 19 | Torn PageBased page loses rows silently | 🟠 Medium | — | **Open** — needs page checksums (§19) | +| 18 | Uncommitted PageBased pages reach disk via eviction | 🟠 Medium | — | **Fixed (interim)** — pages of an open transaction are pinned; a crash during the commit's flush still needs the page WAL (§18) | +| 19 | Torn PageBased page loses rows silently | 🟠 Medium | — | **Fixed (detection)** — CRC32C per page, damage raises an error; repair needs the page WAL (§19) | | 20 | Directory-mode WAL never written or replayed | 🟡 Low | — | **Open, owner decision** (§20) | -| 21 | PageBased non-key lookup scans the table | 🟡 Low | Look up by primary key, or use columnar storage | **Open** (§21) | \ No newline at end of file +| 21 | PageBased non-key lookup scans the table | 🟡 Low | `CREATE INDEX` on the column | **By design** — explicit index serves it (§21) | \ No newline at end of file diff --git a/docs/storage/PAGEBASED_DURABILITY_DESIGN.md b/docs/storage/PAGEBASED_DURABILITY_DESIGN.md index 168810d5..609d2aaa 100644 --- a/docs/storage/PAGEBASED_DURABILITY_DESIGN.md +++ b/docs/storage/PAGEBASED_DURABILITY_DESIGN.md @@ -40,13 +40,13 @@ readable by the new one (all-zero fields mean "written before checksums"). ## 3. Torn pages — a checksum per page (§19) -**Proposal.** Header bytes 9–16: `uint64` XxHash64 (System.IO.Hashing) of the page with these 8 bytes zeroed; byte 17: -format flag `0x01` = checksummed. `WritePage` computes it; `ReadPage` verifies it when the flag is set and throws a -`PageCorruptedException` naming the table and page id when it does not match. A page with flag `0x00` (written before) -is read unverified, exactly as today. +**Implemented (`fix/pagebased-checksums-pinning-indexes`).** Header bytes 9–12: CRC32C of the page with these four +bytes left out (`BitOperations.Crc32C`, the CPU's CRC instruction — no new dependency); byte 13: format flag `0x01` = +checksummed. `WritePage` computes it; `ReadPage` verifies it when the flag is set and throws `InvalidDataException` +naming the page and file. A page with flag `0x00` (written before) is read unverified, exactly as before. The page-index +scan at open skips verification, so one damaged page does not stop the table from opening. -- *Cost:* one 8 KB hash per page write and read — XxHash64 runs at several GB/s, i.e. around 1 µs per page; measure on - the PageBased benchmark (`--engine=pagebased`) and the gate. +- *Cost (measured):* ~2.9 µs per 8 KB page written or read from disk. - *What it buys:* a torn page becomes a loud error instead of silently missing rows. It does not repair the page — that needs §4's log. - *Compatibility:* none broken; downgrade (unsupported by policy) would read checksummed pages fine, since an old reader @@ -114,10 +114,9 @@ it changes what a public configuration flag means. ## 7. Suggested order and test plan -1. §3 checksums (no format break, independent) — tests: damage a page's tail → `PageCorruptedException` names it; a - page written before (flag 0) reads unverified; reopen round-trip matrix stays green. -2. Option A pinning (§18 interim) — test: the eviction crash case of the investigation (`tx-uncommitted-then-big-read`) - leaves the committed state. +1. ✅ §3 checksums — done (`PageChecksumTests`). +2. ✅ Option A pinning (§18 interim) — done; it covers mid-transaction flushes as well as evictions + (`CrashDurabilityTests`). 3. §4 log + recovery — tests: kill the child process at each step of the commit (after the log sync, between page writes) via a test hook; replay restores exactly the committed state; torn page repaired from the log; checkpoint truncates. diff --git a/src/SharpCoreDB/DataStructures/Table.PageBasedScan.cs b/src/SharpCoreDB/DataStructures/Table.PageBasedScan.cs index c3c7b062..19f871a3 100644 --- a/src/SharpCoreDB/DataStructures/Table.PageBasedScan.cs +++ b/src/SharpCoreDB/DataStructures/Table.PageBasedScan.cs @@ -95,10 +95,12 @@ private List> ScanPageBasedTable(string? where) Console.WriteLine($"[ScanPageBasedTable] ✅ Total records found: {recordCount}, deserialization failures: {deserializeFailureCount}, after filtering: {results.Count}"); #endif } - catch + catch (Exception ex) when (ex is not InvalidDataException) { // Exception during scan initialization - return empty results - // The WHERE clause evaluation or engine access may have failed + // The WHERE clause evaluation or engine access may have failed. A damaged page (InvalidDataException: its + // checksum failed) is NOT one of these: returning the rows read so far would hide the damage, which is + // exactly how a torn page used to lose rows silently (known issue §19). #if DEBUG Console.WriteLine($"[ScanPageBasedTable] Exception during scan"); #endif diff --git a/src/SharpCoreDB/Storage/PageManager.cs b/src/SharpCoreDB/Storage/PageManager.cs index 785b5332..ce915c65 100644 --- a/src/SharpCoreDB/Storage/PageManager.cs +++ b/src/SharpCoreDB/Storage/PageManager.cs @@ -95,6 +95,13 @@ public enum RecordFlags : byte protected const int PAGE_SIZE = 8192; protected const int PAGE_HEADER_SIZE = 32; protected const int SLOT_SIZE = 4; + + // Page checksum (known issue §19): CRC32C of the page — computed with these four bytes left out — in header bytes 9–12, + // and a format flag in byte 13. The header is 32 bytes and only bytes 0–8 were ever written, so a page written before + // has flag 0 and is read unverified, as it always was; nothing else in the page moves. + private const int ChecksumOffset = 9; + private const int FormatFlagsOffset = 13; + private const byte ChecksummedFlag = 0x01; protected const int MIN_RECORD_SIZE = 16; protected const int MAX_RECORD_SIZE = PAGE_SIZE - PAGE_HEADER_SIZE - SLOT_SIZE; @@ -116,6 +123,10 @@ public enum RecordFlags : byte // O(1) Free list implementation private readonly Queue freePageList = new(); + // True while InitializeFromDisk scans every page header to rebuild the page index at open: a damaged page must not stop + // the whole table from opening, so the checksum is verified when the page's records are read, not here. + private bool skipChecksumVerification; + // Transaction undo scope (see BeginUndoScope). Null outside a scope, so a write outside a transaction pays one null // check. Maps each page changed inside the scope to its image before the first change, or to null for a page the // scope allocated past the end of the file. @@ -222,6 +233,19 @@ private void InitializeFromDisk() tablePagesIndex.Clear(); freePageList.Clear(); + skipChecksumVerification = true; + try + { + ScanPageHeaders(totalPages); + } + finally + { + skipChecksumVerification = false; + } + } + + private void ScanPageHeaders(ulong totalPages) + { for (ulong pageNumber = 1; pageNumber <= totalPages; pageNumber++) { var page = ReadPage(new PageId(pageNumber)); @@ -268,6 +292,17 @@ protected virtual Page ReadPage(PageId pageId) return new(); var bufferSpan = buffer.AsSpan(); + if (!skipChecksumVerification && bytesRead == PAGE_SIZE && + (bufferSpan[FormatFlagsOffset] & ChecksummedFlag) != 0 && + BinaryPrimitives.ReadUInt32LittleEndian(bufferSpan.Slice(ChecksumOffset)) != ComputePageChecksum(bufferSpan[..PAGE_SIZE])) + { + // Up to 2.1.4 a torn page read as a page whose slots are empty: its rows vanished without an error and + // the table went on accepting writes on top of it (known issue §19). + throw new InvalidDataException( + $"PageBased page {pageId.Value} of '{pagesFile.Name}' failed its checksum: the page is damaged " + + "(for example by a write a crash or power loss interrupted). Restore the table from a backup."); + } + var page = new Page { TableId = BinaryPrimitives.ReadUInt32LittleEndian(bufferSpan), @@ -345,6 +380,9 @@ protected virtual void WritePage(PageId pageId, Page page) BinaryPrimitives.WriteUInt16LittleEndian(bufferSpan.Slice(slotOffset + 2), deleted ? (ushort)0 : slot.length); } + bufferSpan[FormatFlagsOffset] = ChecksummedFlag; + BinaryPrimitives.WriteUInt32LittleEndian(bufferSpan.Slice(ChecksumOffset), ComputePageChecksum(bufferSpan)); + pagesFile.Seek(offset, SeekOrigin.Begin); pagesFile.Write(buffer, 0, PAGE_SIZE); // PERF: Defer flushToDisk to FlushDirtyPages/Commit for batch write efficiency @@ -791,16 +829,25 @@ public virtual void FlushDirtyPages() return; } + // A page the open transaction changed stays dirty and in memory: a flush in the middle of a transaction (an + // ExecuteQuery inside BeginBatchUpdate flushes, to make batched writes visible) wrote it to the file, and a + // crash before the commit or rollback kept the uncommitted change (known issue §18). The commit flushes it. foreach (var pageId in dirtyPages.ToList()) { + if (IsPinned(pageId)) + { + continue; + } + if (pageCache.TryGetValue(pageId, out var page)) { WritePage(new PageId(pageId), page); page.IsDirty = false; pageCache[pageId] = page; } + + dirtyPages.Remove(pageId); } - dirtyPages.Clear(); // PERF: Single OS-level flush after all pages written (a disk sync under FullSync only) pagesFile?.Flush(flushToDisk: syncOnFlush); @@ -874,30 +921,63 @@ private void UpdateLRU(ulong pageId) private void EvictIfNeeded() { - while (pageCache.Count > cacheCapacity) + // Least recently used first. A dirty page the open transaction changed is pinned: evicting it would write an + // uncommitted change to the file, and a crash before the commit or rollback kept it (known issue §18 — measured: + // an uncommitted UPDATE and DELETE survived a crash after a large read). The cache may then exceed its capacity + // until the transaction ends. The walk also ends when the list is exhausted; the previous loop spun forever when + // the cache held more pages than its capacity and none of them was in the LRU list. + var node = lruList.Last; + while (pageCache.Count > cacheCapacity && node is not null) { - // Evict least recently used - if (lruList.Last != null) + var previous = node.Previous; + var candidate = node.Value; + if (!IsPinned(candidate)) { - var evictPageId = lruList.Last.Value; - lruList.RemoveLast(); - lruNodeMap.Remove(evictPageId); - - // Flush if dirty - if (dirtyPages.Contains(evictPageId)) + lruList.Remove(node); + lruNodeMap.Remove(candidate); + if (dirtyPages.Remove(candidate) && pageCache.TryGetValue(candidate, out var page)) { - if (pageCache.TryGetValue(evictPageId, out var page)) - { - WritePage(new PageId(evictPageId), page); - } - dirtyPages.Remove(evictPageId); + WritePage(new PageId(candidate), page); } - - pageCache.Remove(evictPageId); + + pageCache.Remove(candidate); } + + node = previous; } } + /// True for a dirty page changed inside the open undo scope: it must not reach the file before the commit. + private bool IsPinned(ulong pageId) => + undoImages is not null && undoImages.ContainsKey(pageId) && dirtyPages.Contains(pageId); + + /// + /// CRC32C of an 8 KB page with its checksum bytes (header 9–12) left out, using the CPU's CRC32 instruction where there + /// is one ( falls back to software elsewhere). + /// + internal static uint ComputePageChecksum(ReadOnlySpan page) + { + uint crc = Crc32C(uint.MaxValue, page[..ChecksumOffset]); + crc = Crc32C(crc, page[(ChecksumOffset + 4)..]); + return ~crc; + } + + private static uint Crc32C(uint crc, ReadOnlySpan data) + { + int i = 0; + for (; i + 8 <= data.Length; i += 8) + { + crc = System.Numerics.BitOperations.Crc32C(crc, BinaryPrimitives.ReadUInt64LittleEndian(data.Slice(i))); + } + + for (; i < data.Length; i++) + { + crc = System.Numerics.BitOperations.Crc32C(crc, data[i]); + } + + return crc; + } + /// /// Starts a transaction undo scope: from here on, the first change to each page keeps a copy of the page as it was, /// so can put it back. Up to 2.1.4 the PageBased engine had no rollback at all — it diff --git a/tests/SharpCoreDB.Tests/CrashDurabilityTests.cs b/tests/SharpCoreDB.Tests/CrashDurabilityTests.cs index 6ab2cc08..7aa42a8a 100644 --- a/tests/SharpCoreDB.Tests/CrashDurabilityTests.cs +++ b/tests/SharpCoreDB.Tests/CrashDurabilityTests.cs @@ -28,6 +28,7 @@ public sealed class CrashDurabilityTests : IDisposable { private const string ScenarioVariable = "SCDB_CRASH_SCENARIO"; private const string PathVariable = "SCDB_CRASH_PATH"; + private const string CacheVariable = "SCDB_CRASH_PAGE_CACHE"; private readonly string _root = Path.Combine(Path.GetTempPath(), $"SCDB_Crash_{Guid.NewGuid():N}"); @@ -48,11 +49,11 @@ public void Dispose() } } - private static IDatabase Open(string path) + private static IDatabase Open(string path, DatabaseConfig? config = null) { var services = new ServiceCollection(); services.AddSharpCoreDB(); - return services.BuildServiceProvider().GetRequiredService().Create(path, "pw"); + return services.BuildServiceProvider().GetRequiredService().Create(path, "pw", false, config); } /// The child half: runs only when spawned by , then kills its process. @@ -65,9 +66,19 @@ public void CrashChild_RunsTheScenarioAndKillsItsProcess() return; } - var db = Open(Environment.GetEnvironmentVariable(PathVariable)!); + var cache = Environment.GetEnvironmentVariable(CacheVariable); + var db = cache is null + ? Open(Environment.GetEnvironmentVariable(PathVariable)!) + : Open(Environment.GetEnvironmentVariable(PathVariable)!, new DatabaseConfig { PageCacheCapacity = int.Parse(cache, CultureInfo.InvariantCulture) }); switch (scenario) { + case "transaction-uncommitted-then-full-reads": + db.BeginBatchUpdate(); + db.ExecuteSQL("UPDATE t SET n = 999 WHERE id = 1"); + db.ExecuteSQL("DELETE FROM t WHERE id = 2"); + _ = db.ExecuteQuery("SELECT id, n FROM t").Count; + _ = db.ExecuteQuery("SELECT id, n FROM t").Count; + break; case "autocommit-update": db.ExecuteSQL("UPDATE t SET n = 999 WHERE id = 1"); break; @@ -124,6 +135,38 @@ public void Crash_AfterTheScenario_LeavesTheCommittedState(string storage, strin + [Fact] + public void Crash_AfterAnUncommittedPageBasedTransactionWhosePagesWereEvicted_LeavesTheCommittedState() + { + if (!string.IsNullOrEmpty(Environment.GetEnvironmentVariable(ScenarioVariable))) + { + return; // inside a crash child + } + + // A 4-page cache and a table of ~30 pages: the reads evict, and an evicted dirty page of the open transaction was + // written to the file, so a crash kept the uncommitted UPDATE and DELETE (known issue §18). Such pages are pinned. + var path = Path.Combine(_root, $"db_{Guid.NewGuid():N}"); + var seed = Open(path); + seed.ExecuteSQL("CREATE TABLE t (id INTEGER PRIMARY KEY, n INTEGER, pad TEXT) STORAGE = PAGE_BASED"); + var pad = new string('p', 700); + seed.ExecuteBatchSQL([.. Enumerable.Range(1, 300).Select(i => $"INSERT INTO t (id, n, pad) VALUES ({i}, {i}, '{pad}')")]); + (seed as IDisposable)?.Dispose(); + + RunChild("transaction-uncommitted-then-full-reads", path, pageCache: 4); + + var reopened = Open(path); + try + { + Assert.Equal(["1"], reopened.ExecuteQuery("SELECT n FROM t WHERE id = 1").Select(r => Convert.ToString(r["n"], CultureInfo.InvariantCulture))); + Assert.Single(reopened.ExecuteQuery("SELECT n FROM t WHERE id = 2")); + Assert.Equal(300, reopened.ExecuteQuery("SELECT id FROM t").Count); + } + finally + { + (reopened as IDisposable)?.Dispose(); + } + } + [Fact] public void Open_WithAnEmptyMetadataFile_ReportsAnInterruptedSave_NotTheWrongPassword() { @@ -179,21 +222,7 @@ private string CreateSeeded(string storage) private string RunCrashScenario(string storage, string scenario) { var path = CreateSeeded(storage); - var start = new ProcessStartInfo( - Environment.ProcessPath!, - $"-filterVSTest \"FullyQualifiedName~{nameof(CrashDurabilityTests)}.{nameof(CrashChild_RunsTheScenarioAndKillsItsProcess)}\"") - { - UseShellExecute = false, - RedirectStandardOutput = true, - RedirectStandardError = true, - }; - start.Environment[ScenarioVariable] = scenario; - start.Environment[PathVariable] = path; - using (var child = Process.Start(start)!) - { - child.StandardOutput.ReadToEnd(); - child.WaitForExit(); - } + RunChild(scenario, path, pageCache: null); var reopened = Open(path); try @@ -208,4 +237,26 @@ private string RunCrashScenario(string storage, string scenario) (reopened as IDisposable)?.Dispose(); } } + + private static void RunChild(string scenario, string path, int? pageCache) + { + var start = new ProcessStartInfo( + Environment.ProcessPath!, + $"-filterVSTest \"FullyQualifiedName~{nameof(CrashDurabilityTests)}.{nameof(CrashChild_RunsTheScenarioAndKillsItsProcess)}\"") + { + UseShellExecute = false, + RedirectStandardOutput = true, + RedirectStandardError = true, + }; + start.Environment[ScenarioVariable] = scenario; + start.Environment[PathVariable] = path; + if (pageCache is not null) + { + start.Environment[CacheVariable] = pageCache.Value.ToString(CultureInfo.InvariantCulture); + } + + using var child = Process.Start(start)!; + child.StandardOutput.ReadToEnd(); + child.WaitForExit(); + } } diff --git a/tests/SharpCoreDB.Tests/PageBasedSecondaryIndexTests.cs b/tests/SharpCoreDB.Tests/PageBasedSecondaryIndexTests.cs new file mode 100644 index 00000000..806763c0 --- /dev/null +++ b/tests/SharpCoreDB.Tests/PageBasedSecondaryIndexTests.cs @@ -0,0 +1,80 @@ +// +// Copyright (c) 2026 MPCoreDeveloper. All rights reserved. +// Licensed under the MIT License. See LICENSE file in the project root for full license information. +// + +namespace SharpCoreDB.Tests; + +using System; +using System.IO; +using System.Linq; +using Microsoft.Extensions.DependencyInjection; +using SharpCoreDB.Interfaces; +using Xunit; + +/// +/// An explicit CREATE INDEX on a PageBased table serves equality lookups and stays correct across INSERT, UPDATE, +/// DELETE and a reopen (known issue §21). +/// +/// PageBased tables do not create a column index on first use, as columnar tables do (Table.AutoHashIndexesEnabled: +/// a deliberate choice, and SQLite's behaviour too), so a lookup on a non-key column without an index scans the table — +/// measured ~80 ms for 20,000 rows, against ~0.02–0.1 ms once CREATE INDEX has been run. These tests pin that the +/// index route returns exactly the rows a scan would. +/// +/// +public sealed class PageBasedSecondaryIndexTests : IDisposable +{ + private readonly DatabaseFactory _factory; + private readonly string _root = Path.Combine(Path.GetTempPath(), $"SCDB_PageIdx_{Guid.NewGuid():N}"); + + public PageBasedSecondaryIndexTests() + { + var services = new ServiceCollection(); + services.AddSharpCoreDB(); + _factory = services.BuildServiceProvider().GetRequiredService(); + Directory.CreateDirectory(_root); + } + + public void Dispose() + { + try + { + if (Directory.Exists(_root)) + { + Directory.Delete(_root, true); + } + } + catch (IOException) + { + // Best-effort cleanup of a temp directory. + } + } + + private static int[] Ids(IDatabase db, string sql) => + [.. db.ExecuteQuery(sql).Select(r => Convert.ToInt32(r["id"], System.Globalization.CultureInfo.InvariantCulture)).Order()]; + + [Fact] + public void ExplicitIndex_OnAPageBasedTable_ServesLookupsThroughWritesAndAReopen() + { + var path = Path.Combine(_root, $"db_{Guid.NewGuid():N}"); + var db = _factory.Create(path, "pw"); + db.ExecuteSQL("CREATE TABLE docs (id INTEGER PRIMARY KEY, name TEXT, score REAL) STORAGE = PAGE_BASED"); + db.ExecuteBatchSQL([.. Enumerable.Range(1, 500).Select(i => $"INSERT INTO docs (id, name, score) VALUES ({i}, 'User{i % 50}', {i}.5)")]); + db.ExecuteSQL("CREATE INDEX ix_name ON docs (name)"); + + Assert.Equal(Enumerable.Range(1, 500).Where(i => i % 50 == 7).ToArray(), Ids(db, "SELECT id FROM docs WHERE name = 'User7'")); + + db.ExecuteSQL("UPDATE docs SET name = 'Renamed' WHERE id = 7"); + db.ExecuteSQL("INSERT INTO docs (id, name, score) VALUES (1000, 'User7', 1.0)"); + db.ExecuteSQL("DELETE FROM docs WHERE id = 57"); + var expected = Enumerable.Range(1, 500).Where(i => i % 50 == 7 && i != 7 && i != 57).Append(1000).ToArray(); + Assert.Equal(expected, Ids(db, "SELECT id FROM docs WHERE name = 'User7'")); + Assert.Equal([7], Ids(db, "SELECT id FROM docs WHERE name = 'Renamed'")); + (db as IDisposable)?.Dispose(); + + var reopened = _factory.Create(path, "pw"); + Assert.Equal(expected, Ids(reopened, "SELECT id FROM docs WHERE name = 'User7'")); + Assert.Equal([7], Ids(reopened, "SELECT id FROM docs WHERE name = 'Renamed'")); + (reopened as IDisposable)?.Dispose(); + } +} diff --git a/tests/SharpCoreDB.Tests/PageChecksumTests.cs b/tests/SharpCoreDB.Tests/PageChecksumTests.cs new file mode 100644 index 00000000..ec0dbd90 --- /dev/null +++ b/tests/SharpCoreDB.Tests/PageChecksumTests.cs @@ -0,0 +1,120 @@ +// +// Copyright (c) 2026 MPCoreDeveloper. All rights reserved. +// Licensed under the MIT License. See LICENSE file in the project root for full license information. +// + +namespace SharpCoreDB.Tests; + +using System; +using System.Globalization; +using System.IO; +using System.Linq; +using Microsoft.Extensions.DependencyInjection; +using Xunit; + +/// +/// PageBased pages carry a CRC32C checksum in their header (known issue §19). +/// +/// Up to 2.1.4 a page had no checksum and its slot directory sits at its end, so a torn write — the tail of a page not +/// written — read as a page whose slots were empty: the rows vanished without an error and the table went on accepting +/// writes on top of the damage. A damaged page now raises an error naming it. Pages written before carry no checksum +/// (flag byte 0) and are read as they always were. +/// +/// +public sealed class PageChecksumTests : IDisposable +{ + private const int PageSize = 8192; + private readonly DatabaseFactory _factory; + private readonly string _root = Path.Combine(Path.GetTempPath(), $"SCDB_PageCrc_{Guid.NewGuid():N}"); + + public PageChecksumTests() + { + var services = new ServiceCollection(); + services.AddSharpCoreDB(); + _factory = services.BuildServiceProvider().GetRequiredService(); + Directory.CreateDirectory(_root); + } + + public void Dispose() + { + try + { + if (Directory.Exists(_root)) + { + Directory.Delete(_root, true); + } + } + catch (IOException) + { + // Best-effort cleanup of a temp directory. + } + } + + private string CreateTable(int rows) + { + var path = Path.Combine(_root, $"db_{Guid.NewGuid():N}"); + var db = _factory.Create(path, "pw"); + db.ExecuteSQL("CREATE TABLE t (id INTEGER PRIMARY KEY, n INTEGER, pad TEXT) STORAGE = PAGE_BASED"); + var pad = new string('z', 300); + db.ExecuteBatchSQL([.. Enumerable.Range(1, rows).Select(i => $"INSERT INTO t (id, n, pad) VALUES ({i}, {i}, '{pad}')")]); + (db as IDisposable)?.Dispose(); + return path; + } + + private static string PageFile(string path) => + Directory.GetFiles(path, "table_*.pages").OrderByDescending(f => new FileInfo(f).Length).First(); + + [Fact] + public void Read_OfATornPage_RaisesAChecksumError_InsteadOfLosingRowsSilently() + { + var path = CreateTable(40); + var file = PageFile(path); + var bytes = File.ReadAllBytes(file); + Array.Clear(bytes, PageSize / 2, PageSize / 2); // page 1's second half never written + File.WriteAllBytes(file, bytes); + + var db = _factory.Create(path, "pw"); + var error = Assert.ThrowsAny(() => db.ExecuteQuery("SELECT id FROM t")); + Assert.Contains("checksum", error.ToString(), StringComparison.OrdinalIgnoreCase); + (db as IDisposable)?.Dispose(); + } + + [Fact] + public void Read_OfAPageWithOneFlippedDataByte_RaisesAChecksumError() + { + var path = CreateTable(10); + var file = PageFile(path); + var bytes = File.ReadAllBytes(file); + bytes[200] ^= 0x40; + File.WriteAllBytes(file, bytes); + + var db = _factory.Create(path, "pw"); + var error = Assert.ThrowsAny(() => db.ExecuteQuery("SELECT id, pad FROM t")); + Assert.Contains("checksum", error.ToString(), StringComparison.OrdinalIgnoreCase); + (db as IDisposable)?.Dispose(); + } + + [Fact] + public void Read_OfPagesWrittenWithoutAChecksum_WorksAsBefore() + { + var path = CreateTable(40); + var file = PageFile(path); + var bytes = File.ReadAllBytes(file); + for (int page = 0; page + PageSize <= bytes.Length; page += PageSize) + { + Array.Clear(bytes, page + 9, 5); // checksum and flag byte: the layout of a page written before checksums + } + + File.WriteAllBytes(file, bytes); + + var db = _factory.Create(path, "pw"); + Assert.Equal(40, db.ExecuteQuery("SELECT id FROM t").Count); + db.ExecuteSQL("UPDATE t SET n = 0 WHERE id = 1"); + (db as IDisposable)?.Dispose(); + + var reopened = _factory.Create(path, "pw"); + Assert.Equal(40, reopened.ExecuteQuery("SELECT id FROM t").Count); + Assert.Equal("0", Convert.ToString(reopened.ExecuteQuery("SELECT n FROM t WHERE id = 1").Single()["n"], CultureInfo.InvariantCulture)); + (reopened as IDisposable)?.Dispose(); + } +}