Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions docs/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- **`UPDATE … RETURNING` returned no rows when the SET changed a column the WHERE tests.** The returned rows were found by running the WHERE clause again after the update, so `SET status = 'done' WHERE status = 'open' RETURNING id` and `SET n = n + 10 WHERE n < 5 RETURNING n` came back empty. The matched rows are read first and the SET list is applied to them as the update applies it. Pinned by `UpdateReturningTests`.
- **CHECK constraints written with IN, NOT IN, BETWEEN, NOT BETWEEN or GLOB accepted every row.** They were left to an older evaluator that passed them: `CHECK (status IN ('a', 'b'))` let `'z'` in. The SQLite-semantics evaluator reads them now (three-valued: a NULL still passes; GLOB case-sensitive with `*`, `?` and character classes), and the same operators over a computed value in a WHERE clause (`length(name) BETWEEN 3 AND 6`, `upper(name) IN (…)`) use it too. Pinned by `CheckConstraintOperatorTests`. Suites: core **2473/0 failed (6 skipped)**, VectorSearch 255/0, EF Core 116/0, linq2DB 96/0, regression canaries 51/0; write-path gate: GATE PASSED (nothing slower than baseline × 1.50; raw UPDATE 1.31×, default READ 1.33×, default DELETE 1.40× in the watch band on an un-quieted box).

- **A transaction rollback did not undo PageBased tables (known issue §13).** `CancelBatchUpdate()` — an ADO.NET `DbTransaction.Rollback()`, and so linq2db's and EF Core's — left a PageBased UPDATE or INSERT in place, also after a reopen: the PageBased engine writes into its page cache and page files directly, outside the storage transaction buffer, and its `Rollback()` was a stub. Each page manager now keeps a page's image on its first change inside the transaction and puts the images back on rollback (pages the transaction allocated return to the free list); every table then rebuilds its primary-key index and lets its hash indexes rebuild, which also fixes a columnar rollback that left the hash index holding the rolled-back value. A write outside a transaction pays one null check. Pinned by `TransactionRollbackTests`. `ExecuteBatchSQL`'s own failure rollback does not use the undo scope yet (recorded in §13).
- **Inside a transaction, a columnar full scan did not see the transaction's own UPDATEs and DELETEs (known issue §12).** The scan read the data file as it was on disk, while the transaction's in-place overwrites and its deletes were buffered until commit: it returned an updated row's old value, still listed a deleted row, and `WHERE n = 50` after `SET n = 50` found nothing; a primary-key lookup still returned a deleted row. The buffered overwrites and tombstones are laid over the scanned records now (only while a transaction holds them for that file), and a point read of a deleted record returns nothing. Pinned by `InTransactionReadTests` (encrypted and `NoEncryptMode`). Suites: core **2489/0 failed (6 skipped)**, VectorSearch 255/0, EF Core 116/0, linq2DB 96/0, regression canaries 51/0; write-path gate: GATE PASSED (nothing slower than baseline × 1.50; raw READ 1.30×, default READ 1.27×, default DELETE 1.31× in the watch band, as in the previous run on this box).

## [2.1.4] - 2026-11-13

> The issue #497 follow-up: a quoted identifier in `CREATE TABLE` is catalogued without its quoting, so a schema
Expand Down
27 changes: 20 additions & 7 deletions docs/sharpcoredb-known-issues.md
Original file line number Diff line number Diff line change
Expand Up @@ -526,10 +526,13 @@ on the tuple — for files written before, there is no way to recover the lost k
unenforced), and only then enforce single-column uniqueness on single-file storage. Pinned by the skipped
single-file cases in `PrimaryKeyUniquenessTests`.

## 12. 🟠 OPEN — Inside a transaction, a columnar full scan does not see the transaction's own UPDATEs and DELETEs
## 12. ✅ FIXED — Inside a transaction, a columnar full scan did not see the transaction's own UPDATEs and DELETEs

> ⚠️ **STILL TRUE (verified 2026-11-17, `fix/returning-check-in-tx-reads`).** Found while investigating
> read-your-own-writes; the two commit defects found with it are fixed (see the CHANGELOG), this is not.
> ✅ **FIXED (2026-11-17, `fix/transaction-rollback-and-scan-overlay`).** The scan's data source
> (`Storage.ReadBytesWithRecordOffsets`) lays the transaction's buffered in-place overwrites and commit-time tombstones
> over the records it returns — only while a transaction holds such changes for the file, so other scans are untouched
> — and a point read (`ReadBytesFrom`) of a record the transaction deleted returns nothing. Encrypted and
> `NoEncryptMode` files alike. Pinned by `InTransactionReadTests`. The description below is kept as the record.

Inside `BeginBatchUpdate()` / `EndBatchUpdate()` — the transaction an ADO.NET `DbTransaction` opens, and so linq2db's
`TransactionAsync` and EF Core's transactions — a columnar table answers reads inconsistently until the commit:
Expand All @@ -549,9 +552,19 @@ where the scan finds them. After the commit every read is correct (`BatchUpdateT
record offset, which the scan already carries for its stale-version filter). It is a read-path change on the hottest
scan, so it needs the write-path gate and a read benchmark; not attempted in this change.

## 13. 🔴 OPEN — PageBased storage cannot roll a transaction back
## 13. ✅ FIXED — PageBased storage could not roll a transaction back

> ⚠️ **STILL TRUE (verified 2026-11-17, `fix/returning-check-in-tx-reads`).**
> ✅ **FIXED (2026-11-17, `fix/transaction-rollback-and-scan-overlay`) for `BeginBatchUpdate` / `CancelBatchUpdate` and
> the storage-only transaction the EF Core provider uses** — so ADO.NET, linq2db and EF Core rollbacks. Each
> `PageManager` keeps a page's image on its first change inside the transaction (`BeginUndoScope`); rollback restores
> the images (marked dirty, so a page already written to the file during the transaction is written back) and returns
> pages the transaction allocated to the free list, then every table rebuilds its primary-key index from storage and
> marks its hash indexes stale — which also fixes a columnar rollback leaving the hash index on the rolled-back value.
> Outside a transaction a write pays one null check. Pinned by `TransactionRollbackTests` (including an ADO.NET
> `DbTransaction.Rollback()`). **Still open:** `ExecuteBatchSQL` rolls its own internal transaction back on a failing
> statement without the undo scope, so a PageBased table keeps the statements of a failed batch that ran before the
> failure; hooking it adds a page copy per touched page to the most-measured write path and needs its own gate run.
> And without a page WAL, a crash in the middle of a transaction can still leave pages the cache wrote early.

`CancelBatchUpdate()` (an ADO.NET `DbTransaction.Rollback()`) does not undo a PageBased table's changes: a rolled-back
UPDATE or INSERT is still there — also after a reopen — and a rolled-back DELETE stays deleted. Columnar tables roll
Expand Down Expand Up @@ -584,5 +597,5 @@ in this change.
| 9 | Column `UNIQUE` not enforced | 🟠 Medium | — | **Fixed on directory storage** — automatic unique index, composite `UNIQUE (a, b)` included; single-file still open (format decision, §10/§11) |
| 10 | Single-file ignores `COLLATE` | 🟡 Low | — | **Open, owner decision** — needs a column-entry format extension (§10) |
| 11 | PK uniqueness / composite keys | 🔴 High | — | **Fixed on directory storage for new tables** (composite key = UNIQUE over its columns); **open, owner decision** for single-file storage, which enforces no PK uniqueness (§11) |
| 12 | Columnar scan inside a transaction misses its own UPDATE/DELETE | 🟠 Medium | Read by primary key, or commit first | **Open** — buffered overwrites/deletes are not overlaid on the scan (§12) |
| 13 | PageBased transaction rollback | 🔴 High | Columnar storage for tables that need rollback | **Open** — `PageBasedEngine.Rollback()` is a stub (§13) |
| 12 | Columnar scan inside a transaction misses its own UPDATE/DELETE | 🟠 Medium | — | **Fixed** — buffered overwrites and tombstones are laid over the scan (§12) |
| 13 | PageBased transaction rollback | 🔴 High | — | **Fixed** for `CancelBatchUpdate` (ADO.NET, linq2db, EF Core) via page undo images; `ExecuteBatchSQL`'s internal failure rollback still open (§13) |
74 changes: 74 additions & 0 deletions src/SharpCoreDB/DataStructures/Table.TransactionUndo.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
// <copyright file="Table.TransactionUndo.cs" company="MPCoreDeveloper">
// Copyright (c) 2026 MPCoreDeveloper. All rights reserved.
// Licensed under the MIT License. See LICENSE file in the project root for full license information.
// </copyright>

namespace SharpCoreDB.DataStructures;

using SharpCoreDB.Storage.Engines;
using SharpCoreDB.Storage.Hybrid;

/// <summary>
/// Table partial — rolling a PageBased table back with its database transaction (known issue §13).
/// <para>
/// The PageBased engine writes into its page cache and page files directly, outside the storage transaction buffer that
/// columnar tables write through, and up to 2.1.4 nothing undid those writes: <c>CancelBatchUpdate()</c> — an ADO.NET
/// <c>DbTransaction.Rollback()</c> — left a PageBased UPDATE or INSERT in place, also after a reopen. The engine now keeps
/// each changed page's image for the length of the transaction and puts it back on rollback; the table then rebuilds its
/// primary-key index from the restored pages and lets its hash indexes rebuild on next use.
/// </para>
/// </summary>
public partial class Table
{
/// <summary>Starts the transaction's undo scope (PageBased tables; a no-op for columnar ones).</summary>
internal void BeginTransactionUndo()
{
if (StorageMode == StorageMode.PageBased && GetOrCreateStorageEngine() is PageBasedEngine engine)
{
engine.BeginUndoScope();
}
}

/// <summary>Ends the undo scope keeping every change (the transaction committed).</summary>
internal void CommitTransactionUndo()
{
if (_storageEngine is PageBasedEngine engine)
{
engine.CommitUndoScope();
}
}

/// <summary>
/// After a rollback: puts every page the transaction changed back (PageBased), then rebuilds the primary-key index
/// from storage and lets the hash indexes rebuild on next use. A columnar table's rows are rolled back by the storage
/// transaction buffer, but its in-memory indexes were not: an UPDATE of an indexed column, rolled back, left the hash
/// index holding the rolled-back value, so <c>WHERE name = 'old'</c> found nothing. Both modes rebuild here, as every
/// commit already rebuilds the primary-key index.
/// </summary>
internal void RollbackTransactionUndo()
{
if (_storageEngine is PageBasedEngine engine)
{
engine.RollbackUndoScope();
}
else if (_storageEngine is null)
{
return; // never touched: nothing in memory to rebuild
}

this.rwLock.EnterWriteLock();
try
{
RebuildPrimaryKeyIndexFromDisk();
foreach (var column in this.registeredIndexes.Keys)
{
this.staleIndexes.Add(column);
this._indexReadyCache.TryRemove(column, out _);
}
}
finally
{
this.rwLock.ExitWriteLock();
}
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -96,6 +96,8 @@ public void BeginBatchUpdate()
}
}

BeginTablesUndo();

storage.BeginTransaction();
_batchInsertedRows.Clear();
_batchUpdateActive = true;
Expand All @@ -118,6 +120,8 @@ public void BeginStorageTransactionOnly()
if (isReadOnly)
throw new InvalidOperationException("Cannot begin transaction in readonly mode");

BeginTablesUndo();

storage.BeginTransaction();
_batchInsertedRows.Clear();
_batchUpdateActive = true;
Expand All @@ -136,6 +140,7 @@ public void CommitStorageTransaction()
return;

storage.CommitAsync().GetAwaiter().GetResult();
CommitTablesUndo();
_batchUpdateActive = false;
}
}
Expand Down Expand Up @@ -175,10 +180,47 @@ public void RollbackStorageTransaction()
}

storage.Rollback();
RollbackTablesUndo();
_batchUpdateActive = false;
}
}

/// <summary>Starts every PageBased table's undo scope for the transaction (see <c>Table.TransactionUndo</c>).</summary>
private void BeginTablesUndo()
{
foreach (var table in tables.Values)
{
if (table is Table t)
{
t.BeginTransactionUndo();
}
}
}

/// <summary>Ends every PageBased table's undo scope keeping the changes.</summary>
private void CommitTablesUndo()
{
foreach (var table in tables.Values)
{
if (table is Table t)
{
t.CommitTransactionUndo();
}
}
}

/// <summary>Rolls every PageBased table back to its state at the start of the transaction.</summary>
private void RollbackTablesUndo()
{
foreach (var table in tables.Values)
{
if (table is Table t)
{
t.RollbackTransactionUndo();
}
}
}

/// <summary>
/// Ends the batch UPDATE transaction and commits changes.
/// Performs single WAL flush + bulk index rebuild for optimal performance.
Expand All @@ -194,6 +236,7 @@ public void EndBatchUpdate()
try
{
storage.CommitAsync().GetAwaiter().GetResult();
CommitTablesUndo();

// Ensure data pages are flushed before index rebuild so subsequent
// queries (including those after connection close) see the committed inserts.
Expand Down Expand Up @@ -226,6 +269,7 @@ public void EndBatchUpdate()
catch
{
storage.Rollback();
RollbackTablesUndo();
_batchUpdateActive = false;
throw;
}
Expand Down Expand Up @@ -272,6 +316,7 @@ public void CancelBatchUpdate()

// Now perform the storage-level rollback.
storage.Rollback();
RollbackTablesUndo();

foreach (var table in tables.Values)
{
Expand Down
18 changes: 18 additions & 0 deletions src/SharpCoreDB/Services/Storage.Append.cs
Original file line number Diff line number Diff line change
Expand Up @@ -788,6 +788,17 @@ public bool HasBufferedOverwriteAt(string path, long offset) =>
/// <inheritdoc />
public bool AreRecordsEncrypted(string path) => UseRecordEncryption && FileHasEncryptedHeader(path);

/// <summary>Returns <c>true</c> when the current transaction deleted the record at <paramref name="offset"/>.</summary>
private bool IsBufferedTombstone(string path, long offset)
{
lock (appendLock)
{
return bufferedTombstones.Count > 0 &&
bufferedTombstones.TryGetValue(path, out var tombstones) &&
tombstones.Contains(offset);
}
}

/// <inheritdoc />
public void BufferTombstoneForCommit(string path, long offset)
{
Expand Down Expand Up @@ -1616,6 +1627,13 @@ private bool FlushOverwritePages(string path, long fileLength, int pageBytes, Di
return DecryptBufferedAppendPayload(path, bufferedAppend);
}

// Inside a transaction, a record the transaction deleted is gone, although its tombstone is written only at
// commit. Up to 2.1.4 the read returned it, so a primary-key lookup inside the transaction still found the row.
if (IsInTransaction && bufferedTombstones.Count > 0 && IsBufferedTombstone(path, offset))
{
return null;
}

// B7: inside a transaction, a buffered in-place overwrite takes precedence over the disk
// version (the overwrite is written to disk only at commit). The buffer holds the payload
// only (its length is the record's stored length).
Expand Down
Loading
Loading