diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index ecd292c2..48c5bb7e 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -63,6 +63,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - **Table-level CHECK constraints rejected every row.** A table-level `CHECK (lo <= hi)` was stored with its keyword, failed to evaluate and so failed every INSERT. CHECK constraints are now evaluated with SQLite's semantics: a NULL result passes (`CHECK (price > 0)` accepts a row without a price — it failed before), `<>` is an operator, and functions and expressions work; syntax that evaluator does not cover keeps the previous evaluator. Pinned by `TableCheckConstraintTests`. - **ORDER BY on JOIN and subquery queries.** The JOIN path read only the first token of the first ORDER BY key — `ORDER BY LENGTH(p.name) DESC` sorted by nothing, `ORDER BY c.score * -1` sorted ascending by score — and ignored OFFSET; LIMIT 0 returned every row. The AST path (derived tables, IN-subqueries) read each key as a bare name, so `ORDER BY UPPER(name)` left "(name)" unparsed and sorted nothing, and its subquery executor read the first key only. Both now read the whole list — every key with its direction, expression keys — the JOIN path through the single-table path's `SqlOrderBy`, and the JOIN path honours `LIMIT n OFFSET m` and `LIMIT m, n`. Pinned by `JoinAndSubqueryOrderByTests`. Suites: core **2449/0 failed (6 skipped)**, VectorSearch 255/0, EF Core 116/0, linq2DB 96/0, regression canaries 51/0; write-path gate: GATE PASSED (worst best-of-reps ratio 1.24× on raw UPDATE, limit 1.50×). +- **`EndBatchUpdate()` — every ADO.NET, linq2db and EF Core transaction commit — failed or lost its result.** Two defects. (1) Committing a transaction that had buffered a whole-file write (the database metadata among them) threw "Collection was modified": the flush sent each buffered write back through `Storage.WriteBytes`, which — the transaction still being open — buffered it again into the list being flushed; the write was lost, and had the loop not thrown, the already-encrypted bytes would have been encrypted a second time. Buffered writes now go straight to the file in their stored form. (2) Every commit rebuilt each table's primary-key index by searching the index it had just cleared, so after `EndBatchUpdate()` a columnar table's scan returned **no rows** until the database was reopened; the live rows' positions are read before the index is cleared now. Pinned by `BatchUpdateTransactionTests`. Two related gaps stay open, with cause and fix outline: a columnar scan inside a transaction does not see the transaction's own UPDATEs and DELETEs (known issue §12), and PageBased storage cannot roll a transaction back (§13). +- **A PageBased DELETE was undone by every reopen.** A page slot stores only offset and length, so the Deleted flag was never written and a reopen read every slot as active: deleted rows came back, in a transaction or not. A deleted slot is now stored with length 0 — a length no record has — and read back as deleted. Files written before read as they always did (their deleted slots kept real lengths, so rows deleted before this version still come back once). Pinned by `BatchUpdateTransactionTests.PageBasedDelete_IsKeptAfterReopen_AndLaterInsertsCoexist`. +- **`UPDATE … RETURNING` returned no rows when the SET changed a column the WHERE tests.** The returned rows were found by running the WHERE clause again after the update, so `SET status = 'done' WHERE status = 'open' RETURNING id` and `SET n = n + 10 WHERE n < 5 RETURNING n` came back empty. The matched rows are read first and the SET list is applied to them as the update applies it. Pinned by `UpdateReturningTests`. +- **CHECK constraints written with IN, NOT IN, BETWEEN, NOT BETWEEN or GLOB accepted every row.** They were left to an older evaluator that passed them: `CHECK (status IN ('a', 'b'))` let `'z'` in. The SQLite-semantics evaluator reads them now (three-valued: a NULL still passes; GLOB case-sensitive with `*`, `?` and character classes), and the same operators over a computed value in a WHERE clause (`length(name) BETWEEN 3 AND 6`, `upper(name) IN (…)`) use it too. Pinned by `CheckConstraintOperatorTests`. Suites: core **2473/0 failed (6 skipped)**, VectorSearch 255/0, EF Core 116/0, linq2DB 96/0, regression canaries 51/0; write-path gate: GATE PASSED (nothing slower than baseline × 1.50; raw UPDATE 1.31×, default READ 1.33×, default DELETE 1.40× in the watch band on an un-quieted box). + ## [2.1.4] - 2026-11-13 > The issue #497 follow-up: a quoted identifier in `CREATE TABLE` is catalogued without its quoting, so a schema diff --git a/docs/sharpcoredb-known-issues.md b/docs/sharpcoredb-known-issues.md index eb48754e..3f1d8ad3 100644 --- a/docs/sharpcoredb-known-issues.md +++ b/docs/sharpcoredb-known-issues.md @@ -526,6 +526,47 @@ on the tuple — for files written before, there is no way to recover the lost k unenforced), and only then enforce single-column uniqueness on single-file storage. Pinned by the skipped single-file cases in `PrimaryKeyUniquenessTests`. +## 12. 🟠 OPEN — Inside a transaction, a columnar full scan does not see the transaction's own UPDATEs and DELETEs + +> ⚠️ **STILL TRUE (verified 2026-11-17, `fix/returning-check-in-tx-reads`).** Found while investigating +> read-your-own-writes; the two commit defects found with it are fixed (see the CHANGELOG), this is not. + +Inside `BeginBatchUpdate()` / `EndBatchUpdate()` — the transaction an ADO.NET `DbTransaction` opens, and so linq2db's +`TransactionAsync` and EF Core's transactions — a columnar table answers reads inconsistently until the commit: + +| Read after `UPDATE t SET n = 50 WHERE id = 1` (then `DELETE FROM t WHERE id = 1`) | Columnar | PageBased | +|---|---|---| +| `SELECT n FROM t WHERE id = 1` (primary-key lookup) | 50 (then still 50) | 50 (then none) | +| `SELECT id, n FROM t` (full scan) | 1 — the old value (then still listed) | 50 (then gone) | +| `SELECT id FROM t WHERE n = 50` (scan with a filter) | no row | the row | + +**Cause:** the columnar full scan reads the data file through `Storage.ReadBytesWithRecordOffsets`, which returns what +is on disk; the transaction's in-place overwrites and its deletes are held in the storage layer's buffers until the +commit, and only the primary-key path (`engine.Read`) consults them. INSERTs are visible because appends are placed +where the scan finds them. After the commit every read is correct (`BatchUpdateTransactionTests`). + +**Fix outline:** overlay the transaction's buffered overwrites and deletes on the scanned records (keyed by physical +record offset, which the scan already carries for its stale-version filter). It is a read-path change on the hottest +scan, so it needs the write-path gate and a read benchmark; not attempted in this change. + +## 13. 🔴 OPEN — PageBased storage cannot roll a transaction back + +> ⚠️ **STILL TRUE (verified 2026-11-17, `fix/returning-check-in-tx-reads`).** + +`CancelBatchUpdate()` (an ADO.NET `DbTransaction.Rollback()`) does not undo a PageBased table's changes: a rolled-back +UPDATE or INSERT is still there — also after a reopen — and a rolled-back DELETE stays deleted. Columnar tables roll +back correctly (`BatchUpdateTransactionTests.CancelBatchUpdate_Columnar_DiscardsTheUpdate`). + +**Cause:** the PageBased engine writes into its page cache and page files directly, outside the storage transaction +buffer, and `PageBasedEngine.Rollback()` is a stub ("For simplicity, we don't support rollback in this POC — a +production implementation would use WAL or shadow paging"). `Database.CancelBatchUpdate` rolls back only the storage +buffer and removes rows it recorded as inserted. + +**Fix outline:** an undo image per page touched inside the transaction (copy the page on its first write, restore the +copies and drop the cache entries on rollback), with dirty pages withheld from disk until commit. That is a change to +the PageBased engine's durability path and needs the reopen round-trip matrix and the write-path gate; not attempted +in this change. + --- ## Priority summary @@ -542,4 +583,6 @@ single-file cases in `PrimaryKeyUniquenessTests`. | 8 | PK-changing UPDATE loses/duplicates a row | 🔴 High | — | **Fixed on directory storage** — every UPDATE path validates the new key first; single-file part tracked in §11 | | 9 | Column `UNIQUE` not enforced | 🟠 Medium | — | **Fixed on directory storage** — automatic unique index, composite `UNIQUE (a, b)` included; single-file still open (format decision, §10/§11) | | 10 | Single-file ignores `COLLATE` | 🟡 Low | — | **Open, owner decision** — needs a column-entry format extension (§10) | -| 11 | PK uniqueness / composite keys | 🔴 High | — | **Fixed on directory storage for new tables** (composite key = UNIQUE over its columns); **open, owner decision** for single-file storage, which enforces no PK uniqueness (§11) | \ No newline at end of file +| 11 | PK uniqueness / composite keys | 🔴 High | — | **Fixed on directory storage for new tables** (composite key = UNIQUE over its columns); **open, owner decision** for single-file storage, which enforces no PK uniqueness (§11) | +| 12 | Columnar scan inside a transaction misses its own UPDATE/DELETE | 🟠 Medium | Read by primary key, or commit first | **Open** — buffered overwrites/deletes are not overlaid on the scan (§12) | +| 13 | PageBased transaction rollback | 🔴 High | Columnar storage for tables that need rollback | **Open** — `PageBasedEngine.Rollback()` is a stub (§13) | \ No newline at end of file diff --git a/src/SharpCoreDB/Core/File/TransactionBuffer.cs b/src/SharpCoreDB/Core/File/TransactionBuffer.cs index 3658bd69..80b291dc 100644 --- a/src/SharpCoreDB/Core/File/TransactionBuffer.cs +++ b/src/SharpCoreDB/Core/File/TransactionBuffer.cs @@ -365,9 +365,21 @@ public void Flush() } else { + // The buffered bytes are already in their on-disk form (encrypted when the store is), so they go + // straight to the file. Up to 2.1.4 they went back through storage.WriteBytes, which — the transaction + // still being open — buffered them again into this very list: the commit threw "Collection was + // modified" and every whole-file write of the transaction (the database metadata among them) was lost. + // Had it not thrown, WriteBytes would have encrypted the bytes a second time. foreach (var write in pendingWrites) { - storage.WriteBytes(write.FilePath, write.Data); + if (storage is Services.Storage bufferedStorage) + { + bufferedStorage.WriteStoredBytes(write.FilePath, write.Data); + } + else + { + System.IO.File.WriteAllBytes(write.FilePath, write.Data); + } } pendingWrites.Clear(); diff --git a/src/SharpCoreDB/DataStructures/Table.BatchUpdateMode.cs b/src/SharpCoreDB/DataStructures/Table.BatchUpdateMode.cs index 6e38c426..420ae99c 100644 --- a/src/SharpCoreDB/DataStructures/Table.BatchUpdateMode.cs +++ b/src/SharpCoreDB/DataStructures/Table.BatchUpdateMode.cs @@ -290,31 +290,30 @@ private void RebuildPrimaryKeyIndexInternal() if (PrimaryKeyIndex < 0) return; // No primary key - // ✅ CRITICAL: Clear old index (important for correctness!) - // Old index may have stale or outdated references - Index.Clear(); - - // ✅ Get all current rows - var allRows = this.SelectInternal(null, null, true, false); - - // ✅ Re-insert each row's primary key - foreach (var row in allRows) + // The live rows' positions are read from the index BEFORE it is cleared, then re-inserted: the rebuild keeps + // the entry of every row the table still returns and drops any other. Up to 2.1.4 the index was cleared first + // and each position was then searched for in the empty index, so nothing was re-inserted: after every + // EndBatchUpdate() — every ADO.NET, linq2db or EF Core transaction — a columnar table's scan returned no rows + // until the database was reopened. + var pkCol = Columns[PrimaryKeyIndex]; + var entries = new List<(string Key, long Position)>(); + foreach (var row in this.SelectInternal(null, null, true, false)) { - var pkCol = Columns[PrimaryKeyIndex]; if (row.TryGetValue(pkCol, out var pkValue) && pkValue != null) { var pkStr = pkValue.ToString() ?? string.Empty; - - // Get storage position from primary key search - // For both columnar and page-based, positions are maintained in the B-tree var searchResult = Index.Search(pkStr); if (searchResult.Found) { - // Position exists from old index, reuse it - // (row was included in SelectInternal which means it's current version) - Index.Insert(pkStr, searchResult.Value); + entries.Add((pkStr, searchResult.Value)); } } } + + Index.Clear(); + foreach (var (key, position) in entries) + { + Index.Insert(key, position); + } } } diff --git a/src/SharpCoreDB/Services/SqlParser.DML.cs b/src/SharpCoreDB/Services/SqlParser.DML.cs index b2852fe1..4bfc131f 100644 --- a/src/SharpCoreDB/Services/SqlParser.DML.cs +++ b/src/SharpCoreDB/Services/SqlParser.DML.cs @@ -2495,22 +2495,57 @@ private void ExecuteUpdate(string sql, IWAL? wal) // Issue #8: single-pass — UpdateAffectedCount applies the update AND returns the affected // count, so change-tracking no longer needs a separate full Select pass (the old code // materialized every matching row just to count them). + // RETURNING reports the rows as the UPDATE left them. Up to 2.1.4 they were found by running the WHERE clause + // again after the update, so a SET that changes a column the WHERE tests (`SET status = 'done' WHERE status = + // 'open'`, `SET n = n + 10 WHERE n < 5`) returned no rows. The matched rows are read first and the SET list is + // applied to them exactly as the update applies it: constants, and expressions over the row before the statement. + var returningSource = returningColumns is null + ? null + : table.Select(whereClause, orderBy: null, asc: true, noEncrypt: false); + var affectedCount = computed is null ? table.UpdateAffectedCount(whereClause, updates) : table.UpdateAffectedCount(whereClause, updates!, computed); _lastChanges = affectedCount; _totalChanges += affectedCount; - // RETURNING: query updated rows after update - if (returningColumns is not null) + if (returningSource is not null) { - var updatedRows = table.Select(whereClause, orderBy: null, asc: true, noEncrypt: false); - _pendingQueryResults = ProjectReturningRows(updatedRows, returningColumns); + _pendingQueryResults = ProjectReturningRows(ApplySetList(returningSource, updates, computed), returningColumns!); } wal?.Log(sqlWithoutReturning); } + /// The rows as an UPDATE's SET list leaves them, for RETURNING. + private static List> ApplySetList( + List> rows, + Dictionary updates, + List, object?>>>? computed) + { + var result = new List>(rows.Count); + foreach (var row in rows) + { + var updated = new Dictionary(row, row.Comparer); + foreach (var (column, value) in updates) + { + updated[column] = value!; + } + + if (computed is not null) + { + foreach (var (column, compute) in computed) + { + updated[column] = compute(row)!; + } + } + + result.Add(updated); + } + + return result; + } + /// /// Splits the text after SET into the SET list and the WHERE clause (empty when there is none), at the first /// WHERE keyword outside a quoted literal, a quoted identifier or parentheses. diff --git a/src/SharpCoreDB/Services/SqlScalarExpression.cs b/src/SharpCoreDB/Services/SqlScalarExpression.cs index 4492d10f..516757cb 100644 --- a/src/SharpCoreDB/Services/SqlScalarExpression.cs +++ b/src/SharpCoreDB/Services/SqlScalarExpression.cs @@ -1161,6 +1161,181 @@ private sealed class LikeCondition(Node value, Node pattern, Node? escape, bool } } + private sealed class BetweenCondition(Node value, Node low, Node high, bool negated) : Condition + { + public override bool? Evaluate(Dictionary row) + { + var v = Eval(value, row); + bool? atLeastLow = TryCompare(v, Eval(low, row), out var c1) ? c1 >= 0 : null; + bool? atMostHigh = TryCompare(v, Eval(high, row), out var c2) ? c2 <= 0 : null; + + // Three-valued AND of the two bounds. + bool? inside = atLeastLow == false || atMostHigh == false + ? false + : atLeastLow is null || atMostHigh is null ? null : true; + return inside is null ? null : negated ? !inside : inside; + } + } + + private sealed class InCondition(Node value, Node[] items, bool negated) : Condition + { + public override bool? Evaluate(Dictionary row) + { + var v = Eval(value, row); + if (v is null) + { + return items.Length == 0 ? negated : null; + } + + bool sawNull = false; + foreach (var item in items) + { + var candidate = Eval(item, row); + if (candidate is null) + { + sawNull = true; + continue; + } + + if (TryCompare(v, candidate, out var c) && c == 0) + { + return !negated; + } + } + + // SQLite: no match against a list holding a NULL is unknown, not false. + return sawNull ? null : negated; + } + } + + private sealed class GlobCondition(Node value, Node pattern, bool negated) : Condition + { + public override bool? Evaluate(Dictionary row) + { + var v = Eval(value, row); + var p = Eval(pattern, row); + if (v is null || p is null) + { + return null; + } + + bool matched = GlobMatch(ToText(v), ToText(p)); + return negated ? !matched : matched; + } + } + + /// + /// SQLite's GLOB: case-sensitive, * any run, ? one character, [abc], [a-z] and + /// [^…] character classes (a ] first in a class is literal). + /// + internal static bool GlobMatch(string text, string pattern) + { + int t = 0; + int p = 0; + int starP = -1; + int starT = 0; + while (t < text.Length) + { + if (p < pattern.Length) + { + char pc = pattern[p]; + if (pc == '*') + { + starP = p++; + starT = t; + continue; + } + + if (pc == '?') + { + p++; + t++; + continue; + } + + if (pc == '[') + { + if (TryMatchGlobClass(pattern, p, text[t], out int next, out bool inClass)) + { + if (inClass) + { + p = next; + t++; + continue; + } + } + else if (text[t] == '[') + { + p++; // an unterminated class is a literal '[' + t++; + continue; + } + } + else if (pc == text[t]) + { + p++; + t++; + continue; + } + } + + if (starP < 0) + { + return false; + } + + p = starP + 1; + t = ++starT; + } + + while (p < pattern.Length && pattern[p] == '*') + { + p++; + } + + return p == pattern.Length; + } + + private static bool TryMatchGlobClass(string pattern, int start, char c, out int next, out bool matched) + { + int i = start + 1; + bool invert = i < pattern.Length && pattern[i] == '^'; + if (invert) + { + i++; + } + + bool found = false; + bool first = true; + while (i < pattern.Length) + { + char ch = pattern[i]; + if (ch == ']' && !first) + { + next = i + 1; + matched = found != invert; + return true; + } + + if (i + 2 < pattern.Length && pattern[i + 1] == '-' && pattern[i + 2] != ']') + { + found |= c >= ch && c <= pattern[i + 2]; + i += 3; + } + else + { + found |= c == ch; + i++; + } + + first = false; + } + + next = start; + matched = false; + return false; + } + private sealed class TruthCondition(Node expression) : Condition { public override bool? Evaluate(Dictionary row) => IsTrue(Eval(expression, row)); @@ -1185,9 +1360,9 @@ private sealed class ConditionCompiler private Condition? CompileCondition(string text) { text = SqlInPredicate.StripOuterParentheses(text.Trim()); - if (text.Length == 0 || FindTopLevelKeyword(text, "BETWEEN") >= 0) + if (text.Length == 0) { - return null; // BETWEEN's own AND would be split as a conjunction; leave it to the caller + return null; } var orParts = SqlInPredicate.SplitTopLevelLogical(text, "OR"); @@ -1196,7 +1371,7 @@ private sealed class ConditionCompiler return CompileAll(orParts, static p => new OrCondition(p)); } - var andParts = SqlInPredicate.SplitTopLevelLogical(text, "AND"); + var andParts = RejoinBetween(SqlInPredicate.SplitTopLevelLogical(text, "AND")); if (andParts.Count > 1) { return CompileAll(andParts, static p => new AndCondition(p)); @@ -1228,11 +1403,136 @@ private sealed class ConditionCompiler return combine(compiled); } + /// + /// Puts a BETWEEN's own AND back together after a split on AND: n BETWEEN 1 AND 10 AND m > 0 splits into + /// three parts, of which the first two are one BETWEEN. + /// + private static List RejoinBetween(List parts) + { + if (parts.Count < 2) + { + return parts; + } + + var joined = new List(parts.Count); + for (int i = 0; i < parts.Count; i++) + { + var part = parts[i]; + if (i + 1 < parts.Count && FindTopLevelKeyword(part, "BETWEEN") >= 0) + { + part = part + " AND " + parts[++i]; + } + + joined.Add(part); + } + + return joined; + } + + /// Reads x [NOT] BETWEEN lo AND hi; null when the text is something else. + private Condition? CompileBetween(string text, int betweenAt) + { + var left = text[..betweenAt].TrimEnd(); + bool negated = EndsWithKeyword(left, "NOT"); + if (negated) + { + left = left[..^3].TrimEnd(); + } + + var bounds = text[(betweenAt + 7)..]; + int andAt = FindTopLevelKeyword(bounds, "AND"); + if (andAt < 0) + { + return null; + } + + var value = ParseOperand(left); + var low = ParseOperand(bounds[..andAt]); + var high = ParseOperand(bounds[(andAt + 3)..]); + return value is null || low is null || high is null ? null : new BetweenCondition(value, low, high, negated); + } + + /// Reads x [NOT] IN (a, b, …) with a list of expressions; a subquery is left to the caller. + private Condition? CompileIn(string text, int inAt) + { + var left = text[..inAt].TrimEnd(); + bool negated = EndsWithKeyword(left, "NOT"); + if (negated) + { + left = left[..^3].TrimEnd(); + } + + var list = text[(inAt + 2)..].Trim(); + if (list.Length < 2 || list[0] != '(' || list[^1] != ')') + { + return null; + } + + var inner = list[1..^1].Trim(); + if (StartsWithKeyword(inner + " ", "SELECT") || StartsWithKeyword(inner + " ", "WITH")) + { + return null; + } + + var value = ParseOperand(left); + if (value is null) + { + return null; + } + + var items = new List(); + if (inner.Length > 0) + { + foreach (var itemText in SqlInPredicate.SplitTopLevelComma(inner)) + { + var item = ParseOperand(itemText); + if (item is null) + { + return null; + } + + items.Add(item); + } + } + + return new InCondition(value, [.. items], negated); + } + private Condition? CompileLeaf(string text) { + // BETWEEN and IN with a value list are read here; up to 2.1.4 they were left to the caller, and a CHECK + // constraint written with them (`CHECK (status IN ('a', 'b'))`, `CHECK (n BETWEEN 1 AND 10)`) accepted every + // row. + int betweenAt = FindTopLevelKeyword(text, "BETWEEN"); + if (betweenAt >= 0) + { + return CompileBetween(text, betweenAt); + } + + int inAt = FindTopLevelKeyword(text, "IN"); + if (inAt >= 0) + { + return CompileIn(text, inAt); + } + + int globAt = FindTopLevelKeyword(text, "GLOB"); + if (globAt >= 0) + { + var globLeft = text[..globAt].TrimEnd(); + bool globNegated = EndsWithKeyword(globLeft, "NOT"); + if (globNegated) + { + globLeft = globLeft[..^3].TrimEnd(); + } + + var globValue = ParseOperand(globLeft); + var globPattern = ParseOperand(text[(globAt + 4)..]); + return globValue is null || globPattern is null ? null : new GlobCondition(globValue, globPattern, globNegated); + } + // Keyword operators first: "a NOT LIKE b" also contains no symbol operator, but "x IS NOT NULL" must not // be read as a comparison. - foreach (var keyword in (string[])["IN", "GLOB", "REGEXP", "MATCH", "EXISTS"]) + foreach (var keyword in (string[])["REGEXP", "MATCH", "EXISTS"]) { if (FindTopLevelKeyword(text, keyword) >= 0) { diff --git a/src/SharpCoreDB/Services/Storage.ReadWrite.cs b/src/SharpCoreDB/Services/Storage.ReadWrite.cs index e6cfb8b1..1654afcb 100644 --- a/src/SharpCoreDB/Services/Storage.ReadWrite.cs +++ b/src/SharpCoreDB/Services/Storage.ReadWrite.cs @@ -206,6 +206,20 @@ public void Write(string path, string data) return DecryptTableFileToPlaintext(path, out physicalOffsets); } + /// + /// Writes bytes that are already in their on-disk form (encrypted when the store is) — a committed transaction's + /// buffered whole-file write — and drops what the read caches hold for the file. + /// + internal void WriteStoredBytes(string path, byte[] storedData) + { + File.WriteAllBytes(path, storedData); + InvalidateReadEncryption(path); + if (this.pageCache != null) + { + this.pageCache.Clear(flushDirty: false); + } + } + /// public void WriteBytes(string path, byte[] data) { diff --git a/src/SharpCoreDB/Storage/PageManager.cs b/src/SharpCoreDB/Storage/PageManager.cs index 2bbac2f4..e558f19a 100644 --- a/src/SharpCoreDB/Storage/PageManager.cs +++ b/src/SharpCoreDB/Storage/PageManager.cs @@ -276,7 +276,9 @@ protected virtual Page ReadPage(PageId pageId) var slotOffset = slotAreaStart + (i * SLOT_SIZE); var offset16 = BinaryPrimitives.ReadUInt16LittleEndian(bufferSpan.Slice(slotOffset)); var length = BinaryPrimitives.ReadUInt16LittleEndian(bufferSpan.Slice(slotOffset + 2)); - page.Slots.Add((offset16, length, RecordFlags.Active)); + + // A slot stored with length 0 is a deleted record (see WritePage). + page.Slots.Add((offset16, length, length == 0 ? RecordFlags.Deleted : RecordFlags.Active)); } return page; @@ -317,8 +319,16 @@ protected virtual void WritePage(PageId pageId, Page page) for (int i = 0; i < page.Slots.Count && i < page.RecordCount; i++) { var slotOffset = slotAreaStart + (i * SLOT_SIZE); - BinaryPrimitives.WriteUInt16LittleEndian(bufferSpan.Slice(slotOffset), page.Slots[i].offset); - BinaryPrimitives.WriteUInt16LittleEndian(bufferSpan.Slice(slotOffset + 2), page.Slots[i].length); + var slot = page.Slots[i]; + + // A slot holds only offset and length (SLOT_SIZE = 4), so a deleted record is stored with length 0 — + // a length no record has (a serialized row is never empty) — and ReadPage reads it back as deleted. + // Up to 2.1.4 the Deleted flag was never written and ReadPage marked every slot Active, so every + // PageBased DELETE was undone by a reopen: the deleted rows came back. A file written before keeps + // its old deleted slots' real lengths and still reads as it always did. + bool deleted = (slot.flags & RecordFlags.Deleted) == RecordFlags.Deleted; + BinaryPrimitives.WriteUInt16LittleEndian(bufferSpan.Slice(slotOffset), slot.offset); + BinaryPrimitives.WriteUInt16LittleEndian(bufferSpan.Slice(slotOffset + 2), deleted ? (ushort)0 : slot.length); } pagesFile.Seek(offset, SeekOrigin.Begin); diff --git a/tests/SharpCoreDB.Tests/BatchUpdateTransactionTests.cs b/tests/SharpCoreDB.Tests/BatchUpdateTransactionTests.cs new file mode 100644 index 00000000..159d048b --- /dev/null +++ b/tests/SharpCoreDB.Tests/BatchUpdateTransactionTests.cs @@ -0,0 +1,153 @@ +// +// Copyright (c) 2026 MPCoreDeveloper. All rights reserved. +// Licensed under the MIT License. See LICENSE file in the project root for full license information. +// + +namespace SharpCoreDB.Tests; + +using System; +using System.Globalization; +using System.IO; +using System.Linq; +using Microsoft.Extensions.DependencyInjection; +using SharpCoreDB.Interfaces; +using Xunit; + +/// +/// The storage transaction behind BeginBatchUpdate() / EndBatchUpdate() — the one an ADO.NET transaction +/// (and so linq2db's and EF Core's) opens. +/// +/// Up to 2.1.4 two defects made the commit itself fail or lose its result. Committing a transaction that had buffered a +/// whole-file write (the database metadata among them) threw "Collection was modified": the flush sent each buffered +/// write back through Storage.WriteBytes, which — the transaction still being open — buffered it again into the +/// list being flushed. And every commit rebuilt each table's primary-key index by searching the index it had just +/// cleared, so after EndBatchUpdate() a columnar table's scan returned no rows until the database was reopened. +/// A PageBased DELETE — in a transaction or not — was undone by every reopen. +/// +/// +public sealed class BatchUpdateTransactionTests : IDisposable +{ + private readonly DatabaseFactory _factory; + private readonly string _root = Path.Combine(Path.GetTempPath(), $"SCDB_BatchTx_{Guid.NewGuid():N}"); + + public BatchUpdateTransactionTests() + { + var services = new ServiceCollection(); + services.AddSharpCoreDB(); + _factory = services.BuildServiceProvider().GetRequiredService(); + Directory.CreateDirectory(_root); + } + + public static TheoryData Modes => new() { "columnar", "page-based" }; + + public void Dispose() + { + try + { + if (Directory.Exists(_root)) + { + Directory.Delete(_root, true); + } + } + catch (IOException) + { + // Best-effort cleanup of a temp directory. + } + } + + private (IDatabase Db, string Path) Open(string mode, string ddl) + { + var path = Path.Combine(_root, $"db_{Guid.NewGuid():N}"); + var db = _factory.Create(path, "pw"); + db.ExecuteSQL(mode == "page-based" ? ddl + " STORAGE = PAGE_BASED" : ddl); + return (db, path); + } + + private static string[] Rows(IDatabase db, string sql) => + [.. db.ExecuteQuery(sql).Select(r => string.Join(",", r.Values.Select(v => v is null or DBNull ? "NULL" : Convert.ToString(v, CultureInfo.InvariantCulture))))]; + + [Theory] + [MemberData(nameof(Modes))] + public void EndBatchUpdate_WithoutChanges_KeepsEveryRowVisible(string mode) + { + var (db, _) = Open(mode, "CREATE TABLE t (id INTEGER PRIMARY KEY, n INTEGER)"); + db.ExecuteSQL("INSERT INTO t (id, n) VALUES (1, 1), (2, 2)"); + + db.BeginBatchUpdate(); + db.EndBatchUpdate(); + + Assert.Equal(["1,1", "2,2"], Rows(db, "SELECT id, n FROM t").Order()); + Assert.Equal(["1"], Rows(db, "SELECT n FROM t WHERE id = 1")); + } + + [Theory] + [MemberData(nameof(Modes))] + public void EndBatchUpdate_AfterUpdateInsertAndDelete_CommitsAndShowsTheResult(string mode) + { + var (db, path) = Open(mode, "CREATE TABLE t (id INTEGER PRIMARY KEY, n INTEGER, name TEXT)"); + db.ExecuteSQL("CREATE INDEX ix_name ON t (name)"); + db.ExecuteSQL("INSERT INTO t (id, n, name) VALUES (1, 1, 'ann'), (2, 2, 'bob')"); + + db.BeginBatchUpdate(); + db.ExecuteSQL("UPDATE t SET n = 50 WHERE id = 1"); + db.ExecuteSQL("UPDATE t SET name = 'zed' WHERE id = 2"); + db.ExecuteSQL("INSERT INTO t (id, n, name) VALUES (3, 3, 'cy')"); + db.ExecuteSQL("DELETE FROM t WHERE id = 1"); + db.ExecuteSQL("UPDATE t SET n = n + 1 WHERE id = 3"); + db.EndBatchUpdate(); + + Assert.Equal(["2,2,zed", "3,4,cy"], Rows(db, "SELECT id, n, name FROM t").Order()); + Assert.Equal(["2"], Rows(db, "SELECT id FROM t WHERE name = 'zed'")); + + (db as IDisposable)?.Dispose(); + var reopened = _factory.Create(path, "pw"); + Assert.Equal(["2,2,zed", "3,4,cy"], Rows(reopened, "SELECT id, n, name FROM t").Order()); + (reopened as IDisposable)?.Dispose(); + } + + [Theory] + [InlineData(false)] + [InlineData(true)] + public void PageBasedDelete_IsKeptAfterReopen_AndLaterInsertsCoexist(bool inTransaction) + { + // PageBased slots stored only offset and length, so the Deleted flag was never written and a reopen read every + // slot as Active: up to 2.1.4 every PageBased DELETE was undone by a reopen. + var (db, path) = Open("page-based", "CREATE TABLE t (id INTEGER PRIMARY KEY, n INTEGER)"); + db.ExecuteSQL("INSERT INTO t (id, n) VALUES (1, 1), (2, 2), (3, 3)"); + if (inTransaction) + { + db.BeginBatchUpdate(); + } + + db.ExecuteSQL("DELETE FROM t WHERE id = 2"); + if (inTransaction) + { + db.EndBatchUpdate(); + } + + (db as IDisposable)?.Dispose(); + + var reopened = _factory.Create(path, "pw"); + Assert.Equal(["1,1", "3,3"], Rows(reopened, "SELECT id, n FROM t").Order()); + reopened.ExecuteSQL("INSERT INTO t (id, n) VALUES (4, 4)"); + (reopened as IDisposable)?.Dispose(); + + var again = _factory.Create(path, "pw"); + Assert.Equal(["1,1", "3,3", "4,4"], Rows(again, "SELECT id, n FROM t").Order()); + Assert.Empty(again.ExecuteQuery("SELECT n FROM t WHERE id = 2")); + (again as IDisposable)?.Dispose(); + } + + [Fact] + public void CancelBatchUpdate_Columnar_DiscardsTheUpdate() + { + var (db, _) = Open("columnar", "CREATE TABLE t (id INTEGER PRIMARY KEY, n INTEGER)"); + db.ExecuteSQL("INSERT INTO t (id, n) VALUES (1, 1), (2, 2)"); + + db.BeginBatchUpdate(); + db.ExecuteSQL("UPDATE t SET n = 99 WHERE id = 1"); + db.CancelBatchUpdate(); + + Assert.Equal(["1,1", "2,2"], Rows(db, "SELECT id, n FROM t").Order()); + } +} diff --git a/tests/SharpCoreDB.Tests/CheckConstraintOperatorTests.cs b/tests/SharpCoreDB.Tests/CheckConstraintOperatorTests.cs new file mode 100644 index 00000000..02e78c29 --- /dev/null +++ b/tests/SharpCoreDB.Tests/CheckConstraintOperatorTests.cs @@ -0,0 +1,140 @@ +// +// Copyright (c) 2026 MPCoreDeveloper. All rights reserved. +// Licensed under the MIT License. See LICENSE file in the project root for full license information. +// + +namespace SharpCoreDB.Tests; + +using System; +using System.Globalization; +using System.IO; +using System.Linq; +using Microsoft.Extensions.DependencyInjection; +using SharpCoreDB.Interfaces; +using Xunit; + +/// +/// CHECK constraints written with IN, NOT IN, BETWEEN, NOT BETWEEN and GLOB, evaluated as SQLite does. +/// +/// Up to 2.1.4 those operators were left to an older evaluator that accepted every row: CHECK (status IN ('a', 'b')) +/// let 'z' in, and CHECK (n BETWEEN 1 AND 10) let 50 in. They are evaluated with SQL's three-valued +/// logic now, so a NULL still passes. The same operators over a computed value in a WHERE clause +/// (length(name) BETWEEN 3 AND 6, upper(name) IN (…)) use the same evaluator. +/// +/// +public sealed class CheckConstraintOperatorTests : IDisposable +{ + private readonly DatabaseFactory _factory; + private readonly string _root = Path.Combine(Path.GetTempPath(), $"SCDB_CheckOps_{Guid.NewGuid():N}"); + + public CheckConstraintOperatorTests() + { + var services = new ServiceCollection(); + services.AddSharpCoreDB(); + _factory = services.BuildServiceProvider().GetRequiredService(); + Directory.CreateDirectory(_root); + } + + public static TheoryData Modes => new() { "columnar", "page-based" }; + + public void Dispose() + { + try + { + if (Directory.Exists(_root)) + { + Directory.Delete(_root, true); + } + } + catch (IOException) + { + // Best-effort cleanup of a temp directory. + } + } + + private (IDatabase Db, string Path) Open(string mode, string ddl) + { + var path = Path.Combine(_root, $"db_{Guid.NewGuid():N}"); + var db = _factory.Create(path, "pw"); + db.ExecuteSQL(mode == "page-based" ? ddl + " STORAGE = PAGE_BASED" : ddl); + return (db, path); + } + + private static string[] Rows(IDatabase db, string sql) => + [.. db.ExecuteQuery(sql).Select(r => string.Join(",", r.Values.Select(v => v is null or DBNull ? "NULL" : Convert.ToString(v, CultureInfo.InvariantCulture))))]; + + [Theory] + [MemberData(nameof(Modes))] + public void Insert_ValueOutsideACheckInList_IsRejected_AndNullPasses(string mode) + { + var (db, _) = Open(mode, "CREATE TABLE t (id INTEGER PRIMARY KEY, status TEXT CHECK (status IN ('a', 'b')))"); + + db.ExecuteSQL("INSERT INTO t (id, status) VALUES (1, 'a')"); + db.ExecuteSQL("INSERT INTO t (id, status) VALUES (2, NULL)"); + Assert.ThrowsAny(() => db.ExecuteSQL("INSERT INTO t (id, status) VALUES (3, 'z')")); + + Assert.Equal(["1,a", "2,NULL"], Rows(db, "SELECT id, status FROM t").Order()); + } + + [Theory] + [MemberData(nameof(Modes))] + public void Insert_ValueOutsideACheckBetween_IsRejected(string mode) + { + var (db, _) = Open(mode, "CREATE TABLE t (id INTEGER PRIMARY KEY, n INTEGER CHECK (n BETWEEN 1 AND 10))"); + + db.ExecuteSQL("INSERT INTO t (id, n) VALUES (1, 5)"); + Assert.ThrowsAny(() => db.ExecuteSQL("INSERT INTO t (id, n) VALUES (2, 50)")); + + Assert.Equal(["1,5"], Rows(db, "SELECT id, n FROM t")); + } + + [Theory] + [MemberData(nameof(Modes))] + public void Insert_ValueInANegatedCheck_IsRejected(string mode) + { + var (db, _) = Open(mode, + "CREATE TABLE t (id INTEGER PRIMARY KEY, s TEXT CHECK (s NOT IN ('x')), n INTEGER CHECK (n NOT BETWEEN 5 AND 6))"); + + db.ExecuteSQL("INSERT INTO t (id, s, n) VALUES (1, 'a', 1)"); + Assert.ThrowsAny(() => db.ExecuteSQL("INSERT INTO t (id, s, n) VALUES (2, 'x', 1)")); + Assert.ThrowsAny(() => db.ExecuteSQL("INSERT INTO t (id, s, n) VALUES (3, 'a', 5)")); + + Assert.Equal(["1,a,1"], Rows(db, "SELECT id, s, n FROM t")); + } + + [Theory] + [MemberData(nameof(Modes))] + public void Insert_ValueNotMatchingACheckGlob_IsRejected_CaseSensitively(string mode) + { + var (db, _) = Open(mode, "CREATE TABLE t (id INTEGER PRIMARY KEY, code TEXT CHECK (code GLOB '[a-z][0-9]*'))"); + + db.ExecuteSQL("INSERT INTO t (id, code) VALUES (1, 'a1')"); + db.ExecuteSQL("INSERT INTO t (id, code) VALUES (2, 'b22x')"); + Assert.ThrowsAny(() => db.ExecuteSQL("INSERT INTO t (id, code) VALUES (3, 'A1')")); + Assert.ThrowsAny(() => db.ExecuteSQL("INSERT INTO t (id, code) VALUES (4, 'ab')")); + + Assert.Equal(["1,a1", "2,b22x"], Rows(db, "SELECT id, code FROM t").Order()); + } + + [Theory] + [MemberData(nameof(Modes))] + public void Update_ToAValueOutsideACheckInList_IsRejectedAndChangesNothing(string mode) + { + var (db, _) = Open(mode, "CREATE TABLE t (id INTEGER PRIMARY KEY, status TEXT CHECK (status IN ('a', 'b')))"); + db.ExecuteSQL("INSERT INTO t (id, status) VALUES (1, 'a')"); + + Assert.ThrowsAny(() => db.ExecuteSQL("UPDATE t SET status = 'z' WHERE id = 1")); + + Assert.Equal(["1,a"], Rows(db, "SELECT id, status FROM t")); + } + + [Fact] + public void Where_BetweenAndInOverAComputedValue_FilterRows() + { + var (db, _) = Open("columnar", "CREATE TABLE t (id INTEGER PRIMARY KEY, name TEXT)"); + db.ExecuteSQL("INSERT INTO t (id, name) VALUES (1, 'ab'), (2, 'abcd'), (3, 'abcdef'), (4, 'abcdefgh')"); + + Assert.Equal(["2", "3"], Rows(db, "SELECT id FROM t WHERE length(name) BETWEEN 3 AND 6 AND id > 0").Order()); + Assert.Equal(["1", "3"], Rows(db, "SELECT id FROM t WHERE upper(name) IN ('AB', 'ABCDEF')").Order()); + } +} diff --git a/tests/SharpCoreDB.Tests/UpdateReturningTests.cs b/tests/SharpCoreDB.Tests/UpdateReturningTests.cs new file mode 100644 index 00000000..59d28cc1 --- /dev/null +++ b/tests/SharpCoreDB.Tests/UpdateReturningTests.cs @@ -0,0 +1,99 @@ +// +// Copyright (c) 2026 MPCoreDeveloper. All rights reserved. +// Licensed under the MIT License. See LICENSE file in the project root for full license information. +// + +namespace SharpCoreDB.Tests; + +using System; +using System.Globalization; +using System.IO; +using System.Linq; +using Microsoft.Extensions.DependencyInjection; +using SharpCoreDB.Interfaces; +using Xunit; + +/// +/// UPDATE … RETURNING reports the rows the UPDATE changed, as the UPDATE left them. +/// +/// Up to 2.1.4 the rows were found by running the WHERE clause again after the update, so a SET that changed a column +/// the WHERE tests returned nothing: UPDATE t SET status = 'done' WHERE status = 'open' RETURNING id and +/// UPDATE t SET n = n + 10 WHERE n < 5 RETURNING n both came back empty. +/// +/// +public sealed class UpdateReturningTests : IDisposable +{ + private readonly DatabaseFactory _factory; + private readonly string _root = Path.Combine(Path.GetTempPath(), $"SCDB_Returning_{Guid.NewGuid():N}"); + + public UpdateReturningTests() + { + var services = new ServiceCollection(); + services.AddSharpCoreDB(); + _factory = services.BuildServiceProvider().GetRequiredService(); + Directory.CreateDirectory(_root); + } + + public static TheoryData Modes => new() { "columnar", "page-based" }; + + public void Dispose() + { + try + { + if (Directory.Exists(_root)) + { + Directory.Delete(_root, true); + } + } + catch (IOException) + { + // Best-effort cleanup of a temp directory. + } + } + + private (IDatabase Db, string Path) Open(string mode, string ddl) + { + var path = Path.Combine(_root, $"db_{Guid.NewGuid():N}"); + var db = _factory.Create(path, "pw"); + db.ExecuteSQL(mode == "page-based" ? ddl + " STORAGE = PAGE_BASED" : ddl); + return (db, path); + } + + private static string[] Rows(IDatabase db, string sql) => + [.. db.ExecuteQuery(sql).Select(r => string.Join(",", r.Values.Select(v => v is null or DBNull ? "NULL" : Convert.ToString(v, CultureInfo.InvariantCulture))))]; + + [Theory] + [MemberData(nameof(Modes))] + public void UpdateReturning_SetChangesTheWhereColumn_ReturnsTheChangedRows(string mode) + { + var (db, _) = Open(mode, "CREATE TABLE t (id INTEGER PRIMARY KEY, status TEXT)"); + db.ExecuteSQL("INSERT INTO t (id, status) VALUES (1, 'open'), (2, 'open'), (3, 'done')"); + + var returned = Rows(db, "UPDATE t SET status = 'done' WHERE status = 'open' RETURNING id, status"); + + Assert.Equal(["1,done", "2,done"], returned.Order()); + } + + [Theory] + [MemberData(nameof(Modes))] + public void UpdateReturning_ComputedSetValue_ReturnsTheNewValues(string mode) + { + var (db, _) = Open(mode, "CREATE TABLE t (id INTEGER PRIMARY KEY, n INTEGER)"); + db.ExecuteSQL("INSERT INTO t (id, n) VALUES (1, 1), (2, 2), (3, 9)"); + + var returned = Rows(db, "UPDATE t SET n = n + 10 WHERE n < 5 RETURNING id, n"); + + Assert.Equal(["1,11", "2,12"], returned.Order()); + Assert.Equal(["1,11", "2,12", "3,9"], Rows(db, "SELECT id, n FROM t").Order()); + } + + [Theory] + [MemberData(nameof(Modes))] + public void UpdateReturning_Star_ReturnsEveryColumnOfTheUpdatedRow(string mode) + { + var (db, _) = Open(mode, "CREATE TABLE t (id INTEGER PRIMARY KEY, n INTEGER, name TEXT)"); + db.ExecuteSQL("INSERT INTO t (id, n, name) VALUES (1, 1, 'a'), (2, 2, 'b')"); + + Assert.Equal(["1,0,a"], Rows(db, "UPDATE t SET n = 0 WHERE id = 1 RETURNING *")); + } +}