Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions docs/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- **Table-level CHECK constraints rejected every row.** A table-level `CHECK (lo <= hi)` was stored with its keyword, failed to evaluate and so failed every INSERT. CHECK constraints are now evaluated with SQLite's semantics: a NULL result passes (`CHECK (price > 0)` accepts a row without a price — it failed before), `<>` is an operator, and functions and expressions work; syntax that evaluator does not cover keeps the previous evaluator. Pinned by `TableCheckConstraintTests`.
- **ORDER BY on JOIN and subquery queries.** The JOIN path read only the first token of the first ORDER BY key — `ORDER BY LENGTH(p.name) DESC` sorted by nothing, `ORDER BY c.score * -1` sorted ascending by score — and ignored OFFSET; LIMIT 0 returned every row. The AST path (derived tables, IN-subqueries) read each key as a bare name, so `ORDER BY UPPER(name)` left "(name)" unparsed and sorted nothing, and its subquery executor read the first key only. Both now read the whole list — every key with its direction, expression keys — the JOIN path through the single-table path's `SqlOrderBy`, and the JOIN path honours `LIMIT n OFFSET m` and `LIMIT m, n`. Pinned by `JoinAndSubqueryOrderByTests`. Suites: core **2449/0 failed (6 skipped)**, VectorSearch 255/0, EF Core 116/0, linq2DB 96/0, regression canaries 51/0; write-path gate: GATE PASSED (worst best-of-reps ratio 1.24× on raw UPDATE, limit 1.50×).

- **`EndBatchUpdate()` — every ADO.NET, linq2db and EF Core transaction commit — failed or lost its result.** Two defects. (1) Committing a transaction that had buffered a whole-file write (the database metadata among them) threw "Collection was modified": the flush sent each buffered write back through `Storage.WriteBytes`, which — the transaction still being open — buffered it again into the list being flushed; the write was lost, and had the loop not thrown, the already-encrypted bytes would have been encrypted a second time. Buffered writes now go straight to the file in their stored form. (2) Every commit rebuilt each table's primary-key index by searching the index it had just cleared, so after `EndBatchUpdate()` a columnar table's scan returned **no rows** until the database was reopened; the live rows' positions are read before the index is cleared now. Pinned by `BatchUpdateTransactionTests`. Two related gaps stay open, with cause and fix outline: a columnar scan inside a transaction does not see the transaction's own UPDATEs and DELETEs (known issue §12), and PageBased storage cannot roll a transaction back (§13).
- **A PageBased DELETE was undone by every reopen.** A page slot stores only offset and length, so the Deleted flag was never written and a reopen read every slot as active: deleted rows came back, in a transaction or not. A deleted slot is now stored with length 0 — a length no record has — and read back as deleted. Files written before read as they always did (their deleted slots kept real lengths, so rows deleted before this version still come back once). Pinned by `BatchUpdateTransactionTests.PageBasedDelete_IsKeptAfterReopen_AndLaterInsertsCoexist`.
- **`UPDATE … RETURNING` returned no rows when the SET changed a column the WHERE tests.** The returned rows were found by running the WHERE clause again after the update, so `SET status = 'done' WHERE status = 'open' RETURNING id` and `SET n = n + 10 WHERE n < 5 RETURNING n` came back empty. The matched rows are read first and the SET list is applied to them as the update applies it. Pinned by `UpdateReturningTests`.
- **CHECK constraints written with IN, NOT IN, BETWEEN, NOT BETWEEN or GLOB accepted every row.** They were left to an older evaluator that passed them: `CHECK (status IN ('a', 'b'))` let `'z'` in. The SQLite-semantics evaluator reads them now (three-valued: a NULL still passes; GLOB case-sensitive with `*`, `?` and character classes), and the same operators over a computed value in a WHERE clause (`length(name) BETWEEN 3 AND 6`, `upper(name) IN (…)`) use it too. Pinned by `CheckConstraintOperatorTests`. Suites: core **2473/0 failed (6 skipped)**, VectorSearch 255/0, EF Core 116/0, linq2DB 96/0, regression canaries 51/0; write-path gate: GATE PASSED (nothing slower than baseline × 1.50; raw UPDATE 1.31×, default READ 1.33×, default DELETE 1.40× in the watch band on an un-quieted box).

## [2.1.4] - 2026-11-13

> The issue #497 follow-up: a quoted identifier in `CREATE TABLE` is catalogued without its quoting, so a schema
Expand Down
45 changes: 44 additions & 1 deletion docs/sharpcoredb-known-issues.md
Original file line number Diff line number Diff line change
Expand Up @@ -526,6 +526,47 @@ on the tuple — for files written before, there is no way to recover the lost k
unenforced), and only then enforce single-column uniqueness on single-file storage. Pinned by the skipped
single-file cases in `PrimaryKeyUniquenessTests`.

## 12. 🟠 OPEN — Inside a transaction, a columnar full scan does not see the transaction's own UPDATEs and DELETEs

> ⚠️ **STILL TRUE (verified 2026-11-17, `fix/returning-check-in-tx-reads`).** Found while investigating
> read-your-own-writes; the two commit defects found with it are fixed (see the CHANGELOG), this is not.

Inside `BeginBatchUpdate()` / `EndBatchUpdate()` — the transaction an ADO.NET `DbTransaction` opens, and so linq2db's
`TransactionAsync` and EF Core's transactions — a columnar table answers reads inconsistently until the commit:

| Read after `UPDATE t SET n = 50 WHERE id = 1` (then `DELETE FROM t WHERE id = 1`) | Columnar | PageBased |
|---|---|---|
| `SELECT n FROM t WHERE id = 1` (primary-key lookup) | 50 (then still 50) | 50 (then none) |
| `SELECT id, n FROM t` (full scan) | 1 — the old value (then still listed) | 50 (then gone) |
| `SELECT id FROM t WHERE n = 50` (scan with a filter) | no row | the row |

**Cause:** the columnar full scan reads the data file through `Storage.ReadBytesWithRecordOffsets`, which returns what
is on disk; the transaction's in-place overwrites and its deletes are held in the storage layer's buffers until the
commit, and only the primary-key path (`engine.Read`) consults them. INSERTs are visible because appends are placed
where the scan finds them. After the commit every read is correct (`BatchUpdateTransactionTests`).

**Fix outline:** overlay the transaction's buffered overwrites and deletes on the scanned records (keyed by physical
record offset, which the scan already carries for its stale-version filter). It is a read-path change on the hottest
scan, so it needs the write-path gate and a read benchmark; not attempted in this change.

## 13. 🔴 OPEN — PageBased storage cannot roll a transaction back

> ⚠️ **STILL TRUE (verified 2026-11-17, `fix/returning-check-in-tx-reads`).**

`CancelBatchUpdate()` (an ADO.NET `DbTransaction.Rollback()`) does not undo a PageBased table's changes: a rolled-back
UPDATE or INSERT is still there — also after a reopen — and a rolled-back DELETE stays deleted. Columnar tables roll
back correctly (`BatchUpdateTransactionTests.CancelBatchUpdate_Columnar_DiscardsTheUpdate`).

**Cause:** the PageBased engine writes into its page cache and page files directly, outside the storage transaction
buffer, and `PageBasedEngine.Rollback()` is a stub ("For simplicity, we don't support rollback in this POC — a
production implementation would use WAL or shadow paging"). `Database.CancelBatchUpdate` rolls back only the storage
buffer and removes rows it recorded as inserted.

**Fix outline:** an undo image per page touched inside the transaction (copy the page on its first write, restore the
copies and drop the cache entries on rollback), with dirty pages withheld from disk until commit. That is a change to
the PageBased engine's durability path and needs the reopen round-trip matrix and the write-path gate; not attempted
in this change.

---

## Priority summary
Expand All @@ -542,4 +583,6 @@ single-file cases in `PrimaryKeyUniquenessTests`.
| 8 | PK-changing UPDATE loses/duplicates a row | 🔴 High | — | **Fixed on directory storage** — every UPDATE path validates the new key first; single-file part tracked in §11 |
| 9 | Column `UNIQUE` not enforced | 🟠 Medium | — | **Fixed on directory storage** — automatic unique index, composite `UNIQUE (a, b)` included; single-file still open (format decision, §10/§11) |
| 10 | Single-file ignores `COLLATE` | 🟡 Low | — | **Open, owner decision** — needs a column-entry format extension (§10) |
| 11 | PK uniqueness / composite keys | 🔴 High | — | **Fixed on directory storage for new tables** (composite key = UNIQUE over its columns); **open, owner decision** for single-file storage, which enforces no PK uniqueness (§11) |
| 11 | PK uniqueness / composite keys | 🔴 High | — | **Fixed on directory storage for new tables** (composite key = UNIQUE over its columns); **open, owner decision** for single-file storage, which enforces no PK uniqueness (§11) |
| 12 | Columnar scan inside a transaction misses its own UPDATE/DELETE | 🟠 Medium | Read by primary key, or commit first | **Open** — buffered overwrites/deletes are not overlaid on the scan (§12) |
| 13 | PageBased transaction rollback | 🔴 High | Columnar storage for tables that need rollback | **Open** — `PageBasedEngine.Rollback()` is a stub (§13) |
14 changes: 13 additions & 1 deletion src/SharpCoreDB/Core/File/TransactionBuffer.cs
Original file line number Diff line number Diff line change
Expand Up @@ -365,9 +365,21 @@ public void Flush()
}
else
{
// The buffered bytes are already in their on-disk form (encrypted when the store is), so they go
// straight to the file. Up to 2.1.4 they went back through storage.WriteBytes, which — the transaction
// still being open — buffered them again into this very list: the commit threw "Collection was
// modified" and every whole-file write of the transaction (the database metadata among them) was lost.
// Had it not thrown, WriteBytes would have encrypted the bytes a second time.
foreach (var write in pendingWrites)
{
storage.WriteBytes(write.FilePath, write.Data);
if (storage is Services.Storage bufferedStorage)
{
bufferedStorage.WriteStoredBytes(write.FilePath, write.Data);
}
else
{
System.IO.File.WriteAllBytes(write.FilePath, write.Data);
}
}

pendingWrites.Clear();
Expand Down
31 changes: 15 additions & 16 deletions src/SharpCoreDB/DataStructures/Table.BatchUpdateMode.cs
Original file line number Diff line number Diff line change
Expand Up @@ -290,31 +290,30 @@ private void RebuildPrimaryKeyIndexInternal()
if (PrimaryKeyIndex < 0)
return; // No primary key

// ✅ CRITICAL: Clear old index (important for correctness!)
// Old index may have stale or outdated references
Index.Clear();

// ✅ Get all current rows
var allRows = this.SelectInternal(null, null, true, false);

// ✅ Re-insert each row's primary key
foreach (var row in allRows)
// The live rows' positions are read from the index BEFORE it is cleared, then re-inserted: the rebuild keeps
// the entry of every row the table still returns and drops any other. Up to 2.1.4 the index was cleared first
// and each position was then searched for in the empty index, so nothing was re-inserted: after every
// EndBatchUpdate() — every ADO.NET, linq2db or EF Core transaction — a columnar table's scan returned no rows
// until the database was reopened.
var pkCol = Columns[PrimaryKeyIndex];
var entries = new List<(string Key, long Position)>();
foreach (var row in this.SelectInternal(null, null, true, false))
{
var pkCol = Columns[PrimaryKeyIndex];
if (row.TryGetValue(pkCol, out var pkValue) && pkValue != null)
{
var pkStr = pkValue.ToString() ?? string.Empty;

// Get storage position from primary key search
// For both columnar and page-based, positions are maintained in the B-tree
var searchResult = Index.Search(pkStr);
if (searchResult.Found)
{
// Position exists from old index, reuse it
// (row was included in SelectInternal which means it's current version)
Index.Insert(pkStr, searchResult.Value);
entries.Add((pkStr, searchResult.Value));
}
}
}

Index.Clear();
foreach (var (key, position) in entries)
{
Index.Insert(key, position);
}
}
}
43 changes: 39 additions & 4 deletions src/SharpCoreDB/Services/SqlParser.DML.cs
Original file line number Diff line number Diff line change
Expand Up @@ -2426,7 +2426,7 @@
/// <summary>
/// Executes UPDATE statement with RETURNING clause and change tracking support.
/// </summary>
private void ExecuteUpdate(string sql, IWAL? wal)

Check failure on line 2429 in src/SharpCoreDB/Services/SqlParser.DML.cs

View check run for this annotation

SonarQubeCloud / SonarCloud Code Analysis

Refactor this method to reduce its Cognitive Complexity from 17 to the 15 allowed.

See more on https://sonarcloud.io/project/issues?id=MPCoreDeveloper_SharpCoreDB&issues=AaHmhEBgRYzdNdYwVlUr&open=AaHmhEBgRYzdNdYwVlUr&pullRequest=519
{
if (isReadOnly)
throw new InvalidOperationException("Cannot update in readonly mode");
Expand Down Expand Up @@ -2495,22 +2495,57 @@
// Issue #8: single-pass — UpdateAffectedCount applies the update AND returns the affected
// count, so change-tracking no longer needs a separate full Select pass (the old code
// materialized every matching row just to count them).
// RETURNING reports the rows as the UPDATE left them. Up to 2.1.4 they were found by running the WHERE clause
// again after the update, so a SET that changes a column the WHERE tests (`SET status = 'done' WHERE status =
// 'open'`, `SET n = n + 10 WHERE n < 5`) returned no rows. The matched rows are read first and the SET list is
// applied to them exactly as the update applies it: constants, and expressions over the row before the statement.
var returningSource = returningColumns is null
? null
: table.Select(whereClause, orderBy: null, asc: true, noEncrypt: false);

var affectedCount = computed is null
? table.UpdateAffectedCount(whereClause, updates)
: table.UpdateAffectedCount(whereClause, updates!, computed);
_lastChanges = affectedCount;
_totalChanges += affectedCount;

// RETURNING: query updated rows after update
if (returningColumns is not null)
if (returningSource is not null)
{
var updatedRows = table.Select(whereClause, orderBy: null, asc: true, noEncrypt: false);
_pendingQueryResults = ProjectReturningRows(updatedRows, returningColumns);
_pendingQueryResults = ProjectReturningRows(ApplySetList(returningSource, updates, computed), returningColumns!);
}

wal?.Log(sqlWithoutReturning);
}

/// <summary>The rows as an UPDATE's SET list leaves them, for RETURNING.</summary>
private static List<Dictionary<string, object>> ApplySetList(
List<Dictionary<string, object>> rows,
Dictionary<string, object?> updates,
List<KeyValuePair<string, Func<Dictionary<string, object>, object?>>>? computed)
{
var result = new List<Dictionary<string, object>>(rows.Count);
foreach (var row in rows)
{
var updated = new Dictionary<string, object>(row, row.Comparer);
foreach (var (column, value) in updates)
{
updated[column] = value!;
}

if (computed is not null)
{
foreach (var (column, compute) in computed)
{
updated[column] = compute(row)!;
}
}

result.Add(updated);
}

return result;
}

/// <summary>
/// Splits the text after <c>SET</c> into the SET list and the WHERE clause (empty when there is none), at the first
/// WHERE keyword outside a quoted literal, a quoted identifier or parentheses.
Expand Down
Loading
Loading