Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions docs/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- **A transaction rollback did not undo PageBased tables (known issue §13).** `CancelBatchUpdate()` — an ADO.NET `DbTransaction.Rollback()`, and so linq2db's and EF Core's — left a PageBased UPDATE or INSERT in place, also after a reopen: the PageBased engine writes into its page cache and page files directly, outside the storage transaction buffer, and its `Rollback()` was a stub. Each page manager now keeps a page's image on its first change inside the transaction and puts the images back on rollback (pages the transaction allocated return to the free list); every table then rebuilds its primary-key index and lets its hash indexes rebuild, which also fixes a columnar rollback that left the hash index holding the rolled-back value. A write outside a transaction pays one null check. Pinned by `TransactionRollbackTests`. `ExecuteBatchSQL`'s own failure rollback does not use the undo scope yet (recorded in §13).
- **Inside a transaction, a columnar full scan did not see the transaction's own UPDATEs and DELETEs (known issue §12).** The scan read the data file as it was on disk, while the transaction's in-place overwrites and its deletes were buffered until commit: it returned an updated row's old value, still listed a deleted row, and `WHERE n = 50` after `SET n = 50` found nothing; a primary-key lookup still returned a deleted row. The buffered overwrites and tombstones are laid over the scanned records now (only while a transaction holds them for that file), and a point read of a deleted record returns nothing. Pinned by `InTransactionReadTests` (encrypted and `NoEncryptMode`). Suites: core **2489/0 failed (6 skipped)**, VectorSearch 255/0, EF Core 116/0, linq2DB 96/0, regression canaries 51/0; write-path gate: GATE PASSED (nothing slower than baseline × 1.50; raw READ 1.30×, default READ 1.27×, default DELETE 1.31× in the watch band, as in the previous run on this box).

- **`ExecuteBatchSQL` was not atomic.** A batch with a SELECT, or one mixing statement kinds, ran its statements one at a time without a transaction, so a failing statement kept every statement before it on every storage mode; and the grouped executor's transaction rolled back only the storage buffer, so a PageBased table kept a failed batch's earlier changes. Both paths now run as one transaction with the PageBased undo scope (inside a caller's transaction, that transaction decides). Pinned by `BatchAtomicityTests`.
- **Upsert and INSERT conflict clauses.** `INSERT OR IGNORE|REPLACE|ABORT|FAIL|ROLLBACK` failed with "Unsupported SQL statement", `REPLACE INTO` likewise, and an `ON CONFLICT … DO NOTHING | DO UPDATE SET … [WHERE …]` clause was ignored, so an upsert onto an existing key failed as a duplicate — the conflict handlers existed but nothing called them. They run on the SQL INSERT path now (VALUES and SELECT, single-column and composite keys, inside `ExecuteBatchSQL` too); a DO UPDATE assignment and its WHERE are expressions over the existing row with `excluded.col` the incoming value. The composite conflict lookup compares with each column's declared collation, and its "same row" check with the key's — it was always case-insensitive. Pinned by `UpsertTests`.
- **ORDER BY on derived tables and IN-subqueries ignored COLLATE and NULLS FIRST/LAST, and sorted text case-insensitively.** The AST parser read only ASC/DESC after a key, so `COLLATE NOCASE DESC` lost its direction too; a descending sort moved NULLs to the end whatever the statement said; and text without a COLLATE compared case-insensitively instead of SQLite's BINARY. Pinned by `SubqueryOrderByCollationTests`.
- **Batch UPDATEs with a quoted SET column took the slow path.** `SET "n" = 5` / `SET [n] = 5` (linq2db's and EF Core's form) was looked up with its quotes and left the structured fast path; the name is unquoted first now (no allocation for an unquoted name). Pinned by `BatchUpdateClassificationTests.BatchUpdate_QuotedSetColumn_TakesTheStructuredFastPath`.
- **Composite UNIQUE check without its index.** When the first column of a multi-column constraint has no hash index, the check registers the column's auto index (as reads do) and scans only as a last resort. The constraint's own index cannot be dropped (as in SQLite), so this is hardening. Suites: core **2523/0 failed (6 skipped)**, VectorSearch 255/0, EF Core 116/0, linq2DB 96/0, regression canaries 51/0; write-path gate: GATE PASSED (nothing slower than baseline × 1.50; raw READ 1.39×, default READ 1.42×, default DELETE 1.33× in the watch band on an un-quieted box).

## [2.1.4] - 2026-11-13

> The issue #497 follow-up: a quoted identifier in `CREATE TABLE` is catalogued without its quoting, so a schema
Expand Down
9 changes: 4 additions & 5 deletions docs/sharpcoredb-known-issues.md
Original file line number Diff line number Diff line change
Expand Up @@ -561,10 +561,9 @@ scan, so it needs the write-path gate and a read benchmark; not attempted in thi
> pages the transaction allocated to the free list, then every table rebuilds its primary-key index from storage and
> marks its hash indexes stale — which also fixes a columnar rollback leaving the hash index on the rolled-back value.
> Outside a transaction a write pays one null check. Pinned by `TransactionRollbackTests` (including an ADO.NET
> `DbTransaction.Rollback()`). **Still open:** `ExecuteBatchSQL` rolls its own internal transaction back on a failing
> statement without the undo scope, so a PageBased table keeps the statements of a failed batch that ran before the
> failure; hooking it adds a page copy per touched page to the most-measured write path and needs its own gate run.
> And without a page WAL, a crash in the middle of a transaction can still leave pages the cache wrote early.
> `DbTransaction.Rollback()`). `ExecuteBatchSQL` uses the undo scope too since `fix/batch-rollback-and-sql-gaps`
> (`BatchAtomicityTests`), so a failing batch leaves a PageBased table unchanged. **Still open:** without a page WAL, a
> crash in the middle of a transaction can still leave pages the cache wrote early.

`CancelBatchUpdate()` (an ADO.NET `DbTransaction.Rollback()`) does not undo a PageBased table's changes: a rolled-back
UPDATE or INSERT is still there — also after a reopen — and a rolled-back DELETE stays deleted. Columnar tables roll
Expand Down Expand Up @@ -598,4 +597,4 @@ in this change.
| 10 | Single-file ignores `COLLATE` | 🟡 Low | — | **Open, owner decision** — needs a column-entry format extension (§10) |
| 11 | PK uniqueness / composite keys | 🔴 High | — | **Fixed on directory storage for new tables** (composite key = UNIQUE over its columns); **open, owner decision** for single-file storage, which enforces no PK uniqueness (§11) |
| 12 | Columnar scan inside a transaction misses its own UPDATE/DELETE | 🟠 Medium | — | **Fixed** — buffered overwrites and tombstones are laid over the scan (§12) |
| 13 | PageBased transaction rollback | 🔴 High | — | **Fixed** for `CancelBatchUpdate` (ADO.NET, linq2db, EF Core) via page undo images; `ExecuteBatchSQL`'s internal failure rollback still open (§13) |
| 13 | PageBased transaction rollback | 🔴 High | — | **Fixed** for `CancelBatchUpdate` (ADO.NET, linq2db, EF Core) and `ExecuteBatchSQL` via page undo images; crash atomicity without a page WAL still open (§13) |
13 changes: 8 additions & 5 deletions src/SharpCoreDB/DataStructures/Table.CRUD.cs
Original file line number Diff line number Diff line change
Expand Up @@ -5302,7 +5302,9 @@ private bool TryFindCompositeUniqueConflict(
candidateRow.TryGetValue(constraintColumn, out var existingValue);
row.TryGetValue(constraintColumn, out var incomingValue);

if (!SqlParser.AreValuesEqual(existingValue, incomingValue))
// The column's declared collation decides, as it does for INSERT and UPDATE (Table.UniqueKeys): a
// NOCASE column conflicts on 'ABC' against 'abc', a BINARY one does not.
if (!CompositeComponentEquals(constraintColumn, existingValue, incomingValue))
{
allColumnsMatch = false;
break;
Expand All @@ -5321,7 +5323,7 @@ private bool TryFindCompositeUniqueConflict(
return false;
}

private static bool TryResolveConflictingPrimaryKey(
private bool TryResolveConflictingPrimaryKey(
Dictionary<string, object> existingRow,
string primaryKeyColumn,
object? currentPrimaryKeyValue,
Expand All @@ -5334,10 +5336,11 @@ private static bool TryResolveConflictingPrimaryKey(

if (currentPrimaryKeyValue is not null && currentPrimaryKeyValue is not DBNull)
{
var samePrimaryKey = string.Equals(
// "The same row" is decided with the primary key's collation; up to 2.1.4 it was always case-insensitive,
// so on a BINARY text key a row 'ABC' was taken for the row 'abc' and its UNIQUE conflict was missed.
var samePrimaryKey = UniqueKeyComparer(primaryKeyColumn).Equals(
existingPrimaryKeyValue.ToString(),
currentPrimaryKeyValue.ToString(),
StringComparison.OrdinalIgnoreCase);
currentPrimaryKeyValue.ToString());

if (samePrimaryKey)
return false;
Expand Down
7 changes: 6 additions & 1 deletion src/SharpCoreDB/DataStructures/Table.UniqueKeys.cs
Original file line number Diff line number Diff line change
Expand Up @@ -484,7 +484,12 @@ private List<Dictionary<string, object>> FindCompositeKeyHolders(List<string> co
{
var holders = new List<Dictionary<string, object>>();
var seed = constraint[0];
foreach (var candidate in this.registeredIndexes.ContainsKey(seed)

// The constraint's own index (EnsureUniqueConstraintIndexes) normally exists; if it was dropped, the column's
// auto index is registered again, which ordinary reads do too. Only a column that cannot be indexed is scanned —
// a full scan per checked row, so it stays the last resort. (The table lock allows recursion, so neither path
// can deadlock under the write lock the callers hold.)
foreach (var candidate in EnsureAutoHashIndexRegistered(seed)
? FindByIndex(seed, values[seed])
: SelectInternal(null, orderBy: null, asc: true, noEncrypt: false))
{
Expand Down
Loading
Loading