Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions docs/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- **An UPDATE that moved a PRIMARY KEY onto an existing key lost a row (directory storage).** `UPDATE t SET id = 1 WHERE id = 3` with a row 1 present was accepted: the primary-key index was re-pointed at the moved row and the original row 1 became unreachable. Every UPDATE path — `Table.UpdateAffectedCount` (SQL UPDATE), `UpdateMultipleCore` (`ExecuteBatchSQL`, the structured batch API) and `UpdateByPrimaryKey` — now validates new key values before any row is written, counting what earlier operations of a batch claim and release; a SET of the key matching more than one row is rejected too. The check starts with a test that allocates nothing and returns at once for an UPDATE that does not touch a key. Pinned by `PrimaryKeyUpdateTests` (columnar and page-based, in the session and after a reopen). Known issue §8; its single-file half stays with §11.
- **A UNIQUE constraint was never enforced; a UNIQUE index only partly.** A column declared `UNIQUE` (or `UNIQUE (col)`) accepted duplicates on every storage mode: the INSERT-time check was a stub. A UNIQUE *index* was checked only by a single-row INSERT into a columnar table — not on page-based tables, not by a multi-row INSERT, not by an UPDATE — and rejected a second NULL as a duplicate. A single-column UNIQUE now gets an automatic unique hash index on directory storage, as SQLite's `sqlite_autoindex_…` (registered at CREATE TABLE and on every open, so existing tables are covered; rows stored earlier never block an open). One validator covers INSERT, both batch critical sections and every UPDATE path; NULLs never conflict; and a table with a unique index keeps the batched — atomic — multi-row INSERT, which it used to give up for a per-row loop that left earlier rows behind. Pinned by `UniqueConstraintTests` (columnar, page-based) and `MultiRowInsertBatchingTests.CanUseBatchedInsert_IsTrue_WithUniqueSecondaryIndex_BecauseTheBatchCoreEnforcesIt`, which replaces the test that pinned the old per-row fallback. Known issue §9. Still open: single-file storage, whose persisted `ColumnFlags.Unique` is also written for every inline PRIMARY KEY column, so a reopened file cannot tell a declared UNIQUE from an EF Core composite key's column (the format decision of §10/§11; enforcing it was tried and backed out when `CompleteExampleIntegrationTests.CompositeKey_*` failed); and a composite `UNIQUE (a, b)` on INSERT. Suites: core **2392 / 0 failed (6 skipped: documented single-file cases)**, VectorSearch 255 / 0, EF Core 116 / 0, linq2DB 96 / 0, regression canaries 51 / 0; write-path gate: GATE PASSED on a NOISY box (WSearch and Defender need elevation), every metric ok, worst best-of-reps ratio 1.22x on raw UPDATE against a 1.50x limit while its median ratio was 0.60x (faster).

- **Composite keys and multi-column UNIQUE constraints (directory storage).** A table-level `PRIMARY KEY (a, b)` was read as a column called `PRIMARY` — with `CONSTRAINT pk PRIMARY KEY (a, b)`, a column called `CONSTRAINT` — that became the primary key: always NULL, so the table held one row and every further INSERT was a key violation. `PRIMARY KEY` written on several columns, as EF Core's `EnsureCreated` declares a composite key, made the last column the whole key, so rows sharing it were refused. And a multi-column `UNIQUE (a, b)` was never checked. A composite key is now enforced as SQLite enforces it in a rowid table: a UNIQUE constraint over all its columns (the row id stays the storage key), checked on INSERT, multi-row INSERT and every UPDATE path, with a hash index on the constraint's first column narrowing the check. NULLs never conflict. `PRIMARY KEY (id)` on one column makes that column the key. A directory table created before keeps the key it was created with; re-create it to drop the false violations. Pinned by `CompositeKeyTests`. Known issues §9 and §11 (single-file storage is unchanged: owner decision).
- **`ExecuteBatchSQL` executed some UPDATEs as DELETEs.** When the batch UPDATE fast path declined a statement — a quoted SET column such as `SET "n" = 5`, which linq2db and EF Core write — the DELETE classifier tried next accepted the canonical UPDATE shape and ran it as `DELETE … WHERE <col> = <literal>`: the batch deleted the rows it was asked to update (columnar and page-based, durable after reopen). A statement the scanner reads as an UPDATE is never a DELETE now, and the reverse. Pinned by `BatchUpdateClassificationTests`.
- **An UPDATE's SET value can be an expression.** Every SET value was parsed as a literal of the column's type: `SET n = n + 5` failed with "Invalid value 'n + 5'", and `SET label = name` on a TEXT column stored the four letters `name`. A value that reads a column of the table or calls a function is computed per row from the row as it was before the statement (so `SET a = b, b = a` swaps), on directory and single-file storage and through `ExecuteBatchSQL`; constant arithmetic keeps its literal reading and is evaluated only when that fails (`SET n = 2 * 3`). Pinned by `UpdateSetExpressionTests`.
- **A multi-row UPDATE that failed part-way kept the rows before the failure.** NOT NULL and CHECK were checked row by row as rows were written, so `UPDATE t SET n = 2 WHERE id > 0` failing a CHECK on its second row kept the first row changed. Every matched row is validated before the first is written now (SQLite's statement atomicity); the NOT NULL check reads through the SET list without allocating, and only a table with CHECK constraints builds the updated row. Pinned by `UpdateSetExpressionTests.Update_LiteralFailingAColumnCheckOnALaterRow_LeavesEveryRowUnchanged`.
- **An UPDATE without WHERE threw "Invalid UPDATE syntax"**, and a SET literal containing the word `where` was cut at it. SQLite's `UPDATE t SET n = 0` changes every row; the SET list and WHERE clause are now split at the first WHERE outside quotes and parentheses.
- **Table-level CHECK constraints rejected every row.** A table-level `CHECK (lo <= hi)` was stored with its keyword, failed to evaluate and so failed every INSERT. CHECK constraints are now evaluated with SQLite's semantics: a NULL result passes (`CHECK (price > 0)` accepts a row without a price — it failed before), `<>` is an operator, and functions and expressions work; syntax that evaluator does not cover keeps the previous evaluator. Pinned by `TableCheckConstraintTests`.
- **ORDER BY on JOIN and subquery queries.** The JOIN path read only the first token of the first ORDER BY key — `ORDER BY LENGTH(p.name) DESC` sorted by nothing, `ORDER BY c.score * -1` sorted ascending by score — and ignored OFFSET; LIMIT 0 returned every row. The AST path (derived tables, IN-subqueries) read each key as a bare name, so `ORDER BY UPPER(name)` left "(name)" unparsed and sorted nothing, and its subquery executor read the first key only. Both now read the whole list — every key with its direction, expression keys — the JOIN path through the single-table path's `SqlOrderBy`, and the JOIN path honours `LIMIT n OFFSET m` and `LIMIT m, n`. Pinned by `JoinAndSubqueryOrderByTests`. Suites: core **2449/0 failed (6 skipped)**, VectorSearch 255/0, EF Core 116/0, linq2DB 96/0, regression canaries 51/0; write-path gate: GATE PASSED (worst best-of-reps ratio 1.24× on raw UPDATE, limit 1.50×).

## [2.1.4] - 2026-11-13

> The issue #497 follow-up: a quoted identifier in `CREATE TABLE` is catalogued without its quoting, so a schema
Expand Down
24 changes: 16 additions & 8 deletions docs/sharpcoredb-known-issues.md
Original file line number Diff line number Diff line change
Expand Up @@ -455,10 +455,11 @@ SELECT id FROM t;
> registered at CREATE TABLE and on every open, so existing tables are covered. INSERT, multi-row INSERT (now atomic on
> tables with a unique index) and every UPDATE path are checked; NULLs never conflict. On the way: a UNIQUE *index* was
> enforced only by a single-row INSERT into a columnar table, and a second NULL was rejected as a duplicate. Pinned by
> `UniqueConstraintTests` (columnar and page-based). **Still open:** single-file storage (its persisted
> `ColumnFlags.Unique` is also written for every inline PRIMARY KEY column, so a reopened file cannot tell a declared
> UNIQUE from an EF Core composite key's column — the format decision of §10/§11; tried and backed out when
> `CompleteExampleIntegrationTests.CompositeKey_*` failed) and a composite `UNIQUE (a, b)` on INSERT.
> `UniqueConstraintTests` (columnar and page-based). A multi-column `UNIQUE (a, b)` is enforced too since
> `fix/composite-keys-update-atomicity-orderby` (INSERT, multi-row INSERT and UPDATE; `CompositeKeyTests`). **Still
> open:** single-file storage (its persisted `ColumnFlags.Unique` is also written for every inline PRIMARY KEY column, so
> a reopened file cannot tell a declared UNIQUE from an EF Core composite key's column — the format decision of
> §10/§11; tried and backed out when `CompleteExampleIntegrationTests.CompositeKey_*` failed).

Found 2026-11-17. A column declared `UNIQUE` accepts duplicates on both storage modes:

Expand Down Expand Up @@ -494,8 +495,15 @@ on-disk format extension. A `Locale(…)` collation would still need a name fiel

## 11. 🔴 OPEN (owner decision) — PRIMARY KEY uniqueness and composite keys

> ⚠️ **STILL TRUE (verified 2026-11-17), pending an owner decision** on modelling composite keys. Single-file
> enforcement was implemented and backed out in the same change because of the compatibility hazard below.
> ⚠️ **STILL TRUE for single-file storage (verified 2026-11-17), pending an owner decision** on modelling composite
> keys there. Single-file enforcement was implemented and backed out because of the compatibility hazard below.
> **Directory storage is fixed for tables created from `fix/composite-keys-update-atomicity-orderby` on:** a composite
> key — table-level `PRIMARY KEY (a, b)`, `CONSTRAINT name PRIMARY KEY (a, b)`, or `PRIMARY KEY` written on several
> columns as EF Core's `EnsureCreated` writes it — is enforced as SQLite enforces it in a rowid table: a UNIQUE
> constraint over all its columns, the row id staying the storage key (`CompositeKeyTests`). A directory table created
> earlier keeps the key it was created with (its last key column; the other columns were never recorded) and needs to
> be re-created to drop the false violations. The table-level form was worse than described below: it was read as a
> column called `PRIMARY` (or `CONSTRAINT`) that became the key, always NULL, so such a table held one row.

Found 2026-11-17 while fixing single-file constraint enforcement.

Expand Down Expand Up @@ -532,6 +540,6 @@ single-file cases in `PrimaryKeyUniquenessTests`.
| 6 | INTEGER → Int32 | 🟡 Low | `BIGINT` columns | **Still the default** — opt in with `UseSqliteIntegerAffinity` |
| 7 | Bound text re-parsed as SQL | 🔴 High | Write through `ITable.Insert` instead of SQL text | **Fixed** — one `SqlLiteralScanner` (SQL/SQLite: a doubled quote is one quote, a backslash is data) behind every scanner; `ParseValue` strips one pair and un-collapses doubling; a tuple with too many values is rejected. `ParameterLiteralRoundTripTests` 74/74 |
| 8 | PK-changing UPDATE loses/duplicates a row | 🔴 High | — | **Fixed on directory storage** — every UPDATE path validates the new key first; single-file part tracked in §11 |
| 9 | Column `UNIQUE` not enforced | 🟠 Medium | — | **Fixed on directory storage** — automatic unique index; single-file (format decision, §10/§11) and composite `UNIQUE (a, b)` still open |
| 9 | Column `UNIQUE` not enforced | 🟠 Medium | — | **Fixed on directory storage** — automatic unique index, composite `UNIQUE (a, b)` included; single-file still open (format decision, §10/§11) |
| 10 | Single-file ignores `COLLATE` | 🟡 Low | — | **Open, owner decision** — needs a column-entry format extension (§10) |
| 11 | PK uniqueness / composite keys | 🔴 High | — | **Open, owner decision** — single-file enforces no PK uniqueness; composite keys are modelled as their last column (§11) |
| 11 | PK uniqueness / composite keys | 🔴 High | — | **Fixed on directory storage for new tables** (composite key = UNIQUE over its columns); **open, owner decision** for single-file storage, which enforces no PK uniqueness (§11) |
67 changes: 64 additions & 3 deletions src/SharpCoreDB/DataStructures/Table.CRUD.cs
Original file line number Diff line number Diff line change
Expand Up @@ -1772,6 +1772,15 @@
ValidateKeyChanges([(rows.ConvertAll(r => r.Row), updates)]);
}

// NOT NULL and CHECK are checked for every matched row before the first one is written, so a statement that
// fails on its third row leaves the first two unchanged — SQLite's statement atomicity. Up to 2.1.4 each row
// was validated as it was written and a failure kept the rows before it.
bool hasCheckConstraints = HasCheckConstraints();
foreach (var (_, row) in rows)
{
ValidateRowAfterUpdate(row, updates, hasCheckConstraints);
}

foreach (var (rowPos, row) in rows)
{
affected++;
Expand Down Expand Up @@ -1814,14 +1823,12 @@
.Where(row.ContainsKey)
.ToDictionary(key => key, key => row[key]);

// Apply updates to the row
// Apply updates to the row (validated by the caller, for every matched row, before any was written)
foreach (var update in updates)
{
row[update.Key] = update.Value;
}

ValidateUpdatedRow(row);

if (StorageMode == StorageMode.Columnar)
{
UpdateColumnarRow(row, engine, updates, oldPkValue, oldHashKeys, rowPos, touchesIndex);
Expand Down Expand Up @@ -1876,6 +1883,60 @@
return false;
}

/// <summary>Returns <c>true</c> when the table has a column-level or table-level CHECK constraint. Allocates nothing.</summary>
private bool HasCheckConstraints()
{
if (this.TableCheckConstraints.Count > 0)
{
return true;
}

foreach (var expression in this.ColumnCheckExpressions) // NOSONAR:S3267 - write hot path: LINQ would allocate
{
if (expression is not null)
{
return true;
}
}

return false;
}

/// <summary>
/// Validates <paramref name="row"/> as <paramref name="updates"/> would leave it, without changing it: NOT NULL is
/// read through the SET list and allocates nothing; only a table with CHECK constraints builds the updated row.
/// </summary>
private void ValidateRowAfterUpdate(Dictionary<string, object> row, Dictionary<string, object> updates, bool hasCheckConstraints)
{
for (int i = 0; i < this.Columns.Count && i < this.IsNotNull.Count; i++)
{
if (!this.IsNotNull[i])
{
continue;
}

var column = this.Columns[i];
var value = TryGetUpdateValue(updates, column, out var updated)
? updated
: row.TryGetValue(column, out var current) ? current : null;

Check warning on line 1921 in src/SharpCoreDB/DataStructures/Table.CRUD.cs

View check run for this annotation

SonarQubeCloud / SonarCloud Code Analysis

Extract this nested ternary operation into an independent statement.

See more on https://sonarcloud.io/project/issues?id=MPCoreDeveloper_SharpCoreDB&issues=AaHmcoQCazpyc0UchiLI&open=AaHmcoQCazpyc0UchiLI&pullRequest=518
if (value is null or DBNull)
{
throw new InvalidOperationException($"Column '{column}' cannot be NULL");
}
}

if (hasCheckConstraints)
{
var updatedRow = new Dictionary<string, object>(row, row.Comparer);
foreach (var (column, value) in updates)
{
updatedRow[column] = value;
}

ValidateUpdatedRow(updatedRow);
}
}

private void ValidateUpdatedRow(Dictionary<string, object> row)
{
// ✅ NOT NULL validation for UPDATE
Expand Down
91 changes: 91 additions & 0 deletions src/SharpCoreDB/DataStructures/Table.ComputedUpdate.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
// <copyright file="Table.ComputedUpdate.cs" company="MPCoreDeveloper">
// Copyright (c) 2026 MPCoreDeveloper. All rights reserved.
// Licensed under the MIT License. See LICENSE file in the project root for full license information.
// </copyright>

namespace SharpCoreDB.DataStructures;

using System;
using System.Collections.Generic;
using SharpCoreDB.Storage.Hybrid;

/// <summary>
/// Table partial — an UPDATE whose SET values are computed per row (<c>SET n = n + 1</c>, <c>SET name = upper(name)</c>).
/// <para>
/// Up to 2.1.4 a SET value was always parsed as a literal of the column's type, so <c>SET n = n + 5</c> failed ("Invalid
/// value 'n + 5'") and <c>SET label = name</c> on a TEXT column stored the text <c>name</c>. As in SQLite, every
/// expression reads the row as it was before the statement, and every value is computed and every row validated before
/// the first row is written, so an expression or constraint that fails leaves the table unchanged.
/// </para>
/// </summary>
public partial class Table
{
/// <inheritdoc />
public int UpdateAffectedCount(
string? where,
Dictionary<string, object> updates,
IReadOnlyList<KeyValuePair<string, Func<Dictionary<string, object>, object?>>> computed)
{
ArgumentNullException.ThrowIfNull(updates);
ArgumentNullException.ThrowIfNull(computed);
if (computed.Count == 0)
{
return UpdateAffectedCount(where, updates);
}

if (this.isReadOnly) throw new InvalidOperationException("Cannot update in readonly mode");

this.rwLock.EnterWriteLock();
try
{
var engine = GetOrCreateStorageEngine();
EnsureAllRegisteredIndexesLoaded();

var rows = ResolveUpdateRows(where);
var rowUpdates = new List<Dictionary<string, object>>(rows.Count);
foreach (var (_, row) in rows)
{
var values = new Dictionary<string, object>(updates);
foreach (var (column, compute) in computed)
{
values[column] = compute(row)!;
}

rowUpdates.Add(values);
}

if (rows.Count > 0 && UpdateTouchesKeyColumns(rowUpdates[0]))
{
var changes = new List<(IReadOnlyList<Dictionary<string, object>> Rows, Dictionary<string, object> Updates)>(rows.Count);
for (int i = 0; i < rows.Count; i++)
{
changes.Add(([rows[i].Row], rowUpdates[i]));
}

ValidateKeyChanges(changes);
}

bool hasCheckConstraints = HasCheckConstraints();
for (int i = 0; i < rows.Count; i++)
{
ValidateRowAfterUpdate(rows[i].Row, rowUpdates[i], hasCheckConstraints);
}

for (int i = 0; i < rows.Count; i++)
{
UpdateSingleRow(rows[i].Row, engine, rowUpdates[i], rows[i].Position);
}

if (StorageMode == StorageMode.Columnar)
{
TryAutoCompact();
}

return rows.Count;
}
finally
{
this.rwLock.ExitWriteLock();
}
}
}
Loading
Loading