Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions docs/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- **Batch UPDATEs with a quoted SET column took the slow path.** `SET "n" = 5` / `SET [n] = 5` (linq2db's and EF Core's form) was looked up with its quotes and left the structured fast path; the name is unquoted first now (no allocation for an unquoted name). Pinned by `BatchUpdateClassificationTests.BatchUpdate_QuotedSetColumn_TakesTheStructuredFastPath`.
- **Composite UNIQUE check without its index.** When the first column of a multi-column constraint has no hash index, the check registers the column's auto index (as reads do) and scans only as a last resort. The constraint's own index cannot be dropped (as in SQLite), so this is hardening. Suites: core **2523/0 failed (6 skipped)**, VectorSearch 255/0, EF Core 116/0, linq2DB 96/0, regression canaries 51/0; write-path gate: GATE PASSED (nothing slower than baseline × 1.50; raw READ 1.39×, default READ 1.42×, default DELETE 1.33× in the watch band on an un-quieted box).

- **A crash while the database metadata was being saved could leave the database unopenable (known issue §16).** `meta.dat` was written in place (`File.WriteAllBytes`: truncate, write, no fsync); a half-written or empty file made a database refuse to open with "the master password may be incorrect", although every table file was intact. `meta.dat` and `.salt` are now written atomically (temp file, fsync, rename), a metadata save whose text has not changed is skipped, and a file damaged by an older version is reported as an interrupted save with the remedy. Pinned by `CrashDurabilityTests`.
- **Columnar in-place UPDATEs and DELETE markers were not forced to disk (known issue §17).** Appends were written through, but an UPDATE patched into its record and a DELETE's tombstone marker went through a buffered handle that nothing flushed, so a power loss could lose a committed UPDATE or DELETE. Each file written in place is now fsynced once at the next durability boundary — the transaction commit, the end of an autocommit statement, `Flush()`, or the closing of its handle — under `WalDurabilityMode.FullSync` (the default); `Async` leaves them to the operating system. Cost: the gate's raw UPDATE arm (10,000 in-place updates per commit) measured 1.54× its capability baseline, which was recorded without this fsync; re-recording the baseline is an owner decision.
- **Crash-safety investigation recorded (known issues §14–§20).** Measured with a killed child process and with damaged files: PageBased table data is stored unencrypted (§14, owner decision); a PageBased autocommit statement or `ExecuteBatchSQL` is lost by a crash until something flushes (§15); an uncommitted PageBased transaction can reach the disk through page eviction (§18); a torn page loses rows silently (§19); the directory-mode WAL is never written or replayed (§20, owner decision). Columnar storage kept the committed state in every crash case. `CrashDurabilityTests` pins the cases that hold and skips the PageBased ones with their issue. Suites: core **2536/0 failed (8 skipped: 6 single-file, 2 PageBased crash cases of §15)**, VectorSearch 255/0, EF Core 116/0, linq2DB 96/0, regression canaries 51/0; write-path gate: GATE FAILED on raw UPDATE (1.54× the capability baseline) — the cost of the §17 fsync under the default FullSync; baseline re-record is an owner decision.

## [2.1.4] - 2026-11-13

> The issue #497 follow-up: a quoted identifier in `CREATE TABLE` is catalogued without its quoting, so a schema
Expand Down
123 changes: 121 additions & 2 deletions docs/sharpcoredb-known-issues.md
Original file line number Diff line number Diff line change
Expand Up @@ -563,7 +563,7 @@ scan, so it needs the write-path gate and a read benchmark; not attempted in thi
> Outside a transaction a write pays one null check. Pinned by `TransactionRollbackTests` (including an ADO.NET
> `DbTransaction.Rollback()`). `ExecuteBatchSQL` uses the undo scope too since `fix/batch-rollback-and-sql-gaps`
> (`BatchAtomicityTests`), so a failing batch leaves a PageBased table unchanged. **Still open:** without a page WAL, a
> crash in the middle of a transaction can still leave pages the cache wrote early.
> crash in the middle of a transaction can still leave pages the cache wrote early — known issue §18.

`CancelBatchUpdate()` (an ADO.NET `DbTransaction.Rollback()`) does not undo a PageBased table's changes: a rolled-back
UPDATE or INSERT is still there — also after a reopen — and a rolled-back DELETE stays deleted. Columnar tables roll
Expand All @@ -579,6 +579,118 @@ copies and drop the cache entries on rollback), with dirty pages withheld from d
the PageBased engine's durability path and needs the reopen round-trip matrix and the write-path gate; not attempted
in this change.

## 14. 🔴 OPEN (owner decision) — PageBased table data is stored unencrypted

> ⚠️ **STILL TRUE (verified 2026-11-17, `fix/durability-metadata-and-fsync`).** Found in the crash-safety investigation.

A PageBased table's rows are written to `table_<id>.pages` in plaintext, whatever the master password and the encryption
default: a test value `TOPSECRET-CARD-4111111111111111` inserted into a PageBased table is readable verbatim in the page
file, while every columnar file and `meta.dat` of the same database are encrypted. Known issue §1 ("table data not
encrypted") was fixed for columnar storage only.

**Cause:** `Table.CreatePageBasedEngine` creates the engine with `storage: null` ("PageBased doesn't need IStorage"), and
`PageManager` reads and writes pages with a plain `FileStream`; encryption lives in `IStorage`, which this engine never
uses.

**Decision needed:** encrypting pages (per page, AES-256-GCM, with the nonce and tag in the page — which shrinks the
usable page and changes the page file format) is an on-disk format change; existing page files would stay readable as
plaintext and be rewritten encrypted. Until then, PageBased storage should not hold data that relies on encryption.

## 15. 🔴 OPEN — A PageBased statement that returned is not durable until a flush

> ⚠️ **STILL TRUE (verified 2026-11-17, `fix/durability-metadata-and-fsync`).**

Measured with a child process killed (no Dispose, no flush) right after the statements returned, then reopened:

| Scenario, then crash | Columnar | PageBased |
|---|---|---|
| Autocommit INSERT / UPDATE / DELETE | kept | **lost** |
| `ExecuteBatchSQL` that returned | kept | **lost** |
| Autocommit UPDATE + `Flush()` | kept | kept |
| `EndBatchUpdate()` (committed transaction) | kept | kept |

**Cause:** a PageBased write changes the page in the engine's cache; the page reaches the file only on an explicit
`Flush()`, a transaction commit (`EndBatchUpdate` flushes), a cache eviction, or `Dispose`. Neither an autocommit
statement nor `ExecuteBatchSQL`'s commit flushes the page engine. Pinned by the skipped
`CrashDurabilityTests.Crash_AfterAPageBasedAutocommitUpdate_KeepsTheUpdate` and `…_AfterAPageBasedBatch_KeepsTheBatch`.

**Fix outline:** flush (and fsync) the PageBased engine's dirty pages at the end of every autocommit statement and every
batch — small to implement, but a large PageBased write slowdown that needs measuring — or give the engine a page WAL
(§18), which makes the commit a log append instead of a page write.

## 16. ✅ FIXED — The database metadata was written in place, so a crash could leave the database unopenable

> ✅ **FIXED (2026-11-17, `fix/durability-metadata-and-fsync`).** `meta.dat` and the `.salt` file are written atomically
> (`Storage.WriteFileAtomically`: write `path.tmp`, fsync, rename over the file), so a crash leaves the old or the new
> content, never a mix; a metadata save whose text has not changed is skipped, so an explicit `Flush()` without a schema
> change pays no file replace. A metadata file that is empty or shorter than an encrypted empty document is now reported
> as an interrupted save, not as a wrong password. Pinned by `CrashDurabilityTests.Open_With…Metadata…` and
> `…MetadataSave_LeavesNoTempFile_AndAStaleTempFileDoesNotAffectOpen`.

Before: `Storage.Write` used `File.WriteAllBytes` (truncate, then write, no fsync). Cutting `meta.dat` to half its length,
or to zero bytes, made both a columnar and a PageBased database refuse to open, with "Failed to decrypt database
metadata. The master password may be incorrect or the database file is corrupted." — although every table file was
intact. A file damaged by an older version still cannot be opened (there is no second copy to fall back to); the
message now names the cause and the remedy (restore `meta.dat` from a backup).

## 17. ✅ FIXED — Columnar in-place UPDATEs and DELETE markers were not forced to disk

> ✅ **FIXED (2026-11-17, `fix/durability-metadata-and-fsync`).** Every write through a cached in-place write handle
> (an UPDATE patched into its record, a DELETE's tombstone marker, the commit's batched overwrite pages) marks its file
> unsynced; `IStorage.SyncPendingWrites` forces each such file to disk once — at a transaction commit, at the end of an
> autocommit `ExecuteSQL`, on `Flush()`, and before a cached handle is closed. It follows `WalDurabilityMode`: `FullSync`
> (the default, documented as surviving power failures) syncs, `Async` leaves the bytes to the operating system. Not
> observable by a process-kill test (the operating system keeps the bytes); verified by reading the code.
>
> **Cost, and the owner decision it needs:** the write-path gate's raw UPDATE arm — one `ExecuteBatchSQL` of 10,000
> in-place updates — now pays one fsync of the data file in its commit: best rep 310,529 ops/s against the 477,824
> capability baseline (1.54×, `GATE FAILED`; this session's earlier runs without the fsync: 384–394k). The baseline was
> recorded without power-loss durability for these writes, so it has to be re-recorded (an owner decision) for the gate to
> measure the durable default — or the arm configured `WalDurabilityMode.Async` to keep measuring the non-durable mode.

Before: appends were opened with `FileOptions.WriteThrough`, but the cached in-place write handle used `FileOptions.None`
and nothing called `FlushToDisk`, so a power loss (not a process crash) could lose a committed UPDATE or DELETE while
keeping committed INSERTs.

## 18. 🟠 OPEN — An uncommitted PageBased transaction can reach the disk before a crash

> ⚠️ **STILL TRUE (verified 2026-11-17, `fix/durability-metadata-and-fsync`).**

Measured: a transaction updated row 1 and deleted row 2 of a PageBased table of ~1,400 pages, then read the table twice,
and the process was killed before the commit. After reopening, row 1 had the uncommitted value and row 2 was gone.
Columnar storage kept the committed state.

**Cause:** the page cache evicts the least recently used page when it outgrows its capacity (1,024 pages by default),
and an evicted dirty page is written to the file — committed or not. Rollback without a crash is correct (§13: the undo
images are restored and written back); a crash between the eviction and the commit or rollback is not.

**Fix outline:** a page WAL (redo or undo log with commit records, replayed on open), or withholding dirty pages of an
open transaction from eviction (bounded memory then needs a spill file). Engine work; also closes §15 cheaply.

## 19. 🟠 OPEN — A torn PageBased page loses rows silently

> ⚠️ **STILL TRUE (verified 2026-11-17, `fix/durability-metadata-and-fsync`).**

Measured: zeroing the second half of a page's 8 KB — what a torn write leaves when a crash interrupts it — made 25 of 40
rows disappear without an error, and the table kept accepting writes; cutting the page file mid-page lost that page's
rows the same way. **Cause:** a page has no checksum, and its slot directory sits at the end of the page, so a torn tail
reads as empty slots. **Fix outline:** a checksum per page verified on read (a damaged page then raises an error instead
of reading as empty), plus the page WAL of §18 to repair it. Columnar files are append-only and lose only an incomplete
trailing record (measured: 39 of 40 rows, table writable).

## 20. 🟡 OPEN (owner decision) — The directory-mode write-ahead log is never written or replayed

> ⚠️ **STILL TRUE (verified 2026-11-17, `fix/durability-metadata-and-fsync`).**

`UseGroupCommitWal` (enabled by several `DatabaseConfig` presets, with `WalDurabilityMode = FullSync`) creates a
`GroupCommitWAL`, but `Database.ExecuteSQL` passes `wal: null` to the parser, nothing else writes to it, and nothing calls
its `CrashRecovery()` / `RecoverAll()` on open. Directory-mode durability rests on the data files alone (§15–§19), and
the configuration suggests a guarantee that does not exist. The single-file (`.scdb`) format has its own WAL and
`RecoveryManager` (`CrashRecoveryTests`) and is not affected.

**Decision needed:** wire the WAL in (log each committed change, replay on open — the natural home for §15/§18's fix), or
remove it and the WAL settings from directory mode so the configuration is honest.

---

## Priority summary
Expand All @@ -597,4 +709,11 @@ in this change.
| 10 | Single-file ignores `COLLATE` | 🟡 Low | — | **Open, owner decision** — needs a column-entry format extension (§10) |
| 11 | PK uniqueness / composite keys | 🔴 High | — | **Fixed on directory storage for new tables** (composite key = UNIQUE over its columns); **open, owner decision** for single-file storage, which enforces no PK uniqueness (§11) |
| 12 | Columnar scan inside a transaction misses its own UPDATE/DELETE | 🟠 Medium | — | **Fixed** — buffered overwrites and tombstones are laid over the scan (§12) |
| 13 | PageBased transaction rollback | 🔴 High | — | **Fixed** for `CancelBatchUpdate` (ADO.NET, linq2db, EF Core) and `ExecuteBatchSQL` via page undo images; crash atomicity without a page WAL still open (§13) |
| 13 | PageBased transaction rollback | 🔴 High | — | **Fixed** for `CancelBatchUpdate` (ADO.NET, linq2db, EF Core) and `ExecuteBatchSQL` via page undo images; crash atomicity without a page WAL tracked in §18 |
| 14 | PageBased data stored unencrypted | 🔴 High | Columnar storage for sensitive data | **Open, owner decision** — page format change (§14) |
| 15 | PageBased autocommit / batch not durable until a flush | 🔴 High | `Flush()` after writes, or explicit transactions | **Open** (§15) |
| 16 | Non-atomic metadata write → database unopenable after a crash | 🔴 High | — | **Fixed** — temp file + fsync + rename; clear error for a damaged file (§16) |
| 17 | Columnar in-place UPDATE/DELETE not fsynced | 🟠 Medium | — | **Fixed** — synced at commit, autocommit end and flush (§17) |
| 18 | Uncommitted PageBased pages reach disk via eviction | 🟠 Medium | Keep transactions within the page cache | **Open** — needs a page WAL (§18) |
| 19 | Torn PageBased page loses rows silently | 🟠 Medium | — | **Open** — needs page checksums (§19) |
| 20 | Directory-mode WAL never written or replayed | 🟡 Low | — | **Open, owner decision** (§20) |
50 changes: 46 additions & 4 deletions src/SharpCoreDB/Database/Core/Database.Core.cs
Original file line number Diff line number Diff line change
Expand Up @@ -271,7 +271,7 @@ private void Load()
if (metaExists && metaJson is null) // ✅ C# 15: is null pattern
{
throw new InvalidOperationException(
"Failed to decrypt database metadata. The master password may be incorrect or the database file is corrupted.");
DescribeUnreadableMetadata(Path.Combine(_dbPath, PersistenceConstants.MetaFileName)));
}

// ✅ FIX: Handle empty/null metadata gracefully (valid for new databases)
Expand Down Expand Up @@ -550,6 +550,41 @@ config is { } fixedWidthConfig &&
/// ✅ FIX: Ensures immediate flush for durability
/// ✅ FIX: Adds Brotli compression support (60-80% size reduction)
/// </summary>
/// <summary>
/// The error for a metadata file that exists but cannot be read. A file too short to hold even an encrypted empty
/// document is an interrupted save, not a wrong password, and the message says so — up to 2.1.4 both cases reported
/// "the master password may be incorrect", which sends the user after the wrong cause.
/// </summary>
private static string DescribeUnreadableMetadata(string metaPath)
{
long length = File.Exists(metaPath) ? new FileInfo(metaPath).Length : 0; // NOSONAR:S6549 - metaPath is derived from the user-configured storage root.
const int minimumEncryptedLength = CryptoConstants.GCM_NONCE_SIZE + CryptoConstants.GCM_TAG_SIZE;
if (length < minimumEncryptedLength)
{
return $"The database metadata file '{metaPath}' is {(length == 0 ? "empty" : $"truncated ({length} bytes)")}: " +
"a save of it was interrupted (a crash or power loss while it was being written). The table data files " +
$"are not affected; restore {PersistenceConstants.MetaFileName} from a backup to open the database.";
}

return "Failed to decrypt database metadata. The master password may be incorrect, or the metadata file is " +
"damaged (for example by a crash while it was being written).";
}

/// <summary>The metadata this session last wrote outside a transaction; a save of the same text is skipped.</summary>
private string? _lastSavedMetaJson;

/// <summary>
/// After a statement outside a transaction, forces its in-place writes to disk — the autocommit statement's
/// durability boundary (a transaction's is its commit). Free when the statement wrote nothing in place.
/// </summary>
private void SyncAutocommitWrites()
{
if (storage is { IsInTransaction: false } current)
{
current.SyncPendingWrites();
}
}

private void SaveMetadata()
{
// v2 (AOT readiness): strongly-typed DTO preserves the legacy anonymous-type JSON shape.
Expand Down Expand Up @@ -613,10 +648,13 @@ private void SaveMetadata()
System.Diagnostics.Debug.WriteLine($"[SaveMetadata] Saved and flushed metadata ({metaBytes.Length} bytes)");
#endif
}
else
else if (!string.Equals(metaJson, _lastSavedMetaJson, StringComparison.Ordinal))
{
// ✅ Legacy: Use IStorage (file-based) - no compression
// ✅ Legacy: Use IStorage (file-based) - no compression. Written atomically (Storage.Write); skipped when
// the content has not changed since this session last wrote it, so an explicit Flush() with no schema
// change does not pay a file replace and an fsync.
storage.Write(Path.Combine(_dbPath, PersistenceConstants.MetaFileName), metaJson);
_lastSavedMetaJson = storage.IsInTransaction ? null : metaJson;
}

_metadataDirty = false;
Expand Down Expand Up @@ -722,7 +760,8 @@ private static string GetOrCreateDatabaseSalt(string dbPath)
var newSalt = new byte[CryptoConstants.DATABASE_SALT_SIZE];
RandomNumberGenerator.Fill(newSalt);

File.WriteAllBytes(saltFilePath, newSalt);
// Atomic, like the metadata: a salt half-written by a crash would lock the database for good.
Services.Storage.WriteFileAtomically(saltFilePath, newSalt);
File.SetAttributes(saltFilePath, FileAttributes.Hidden);

return Convert.ToBase64String(newSalt);
Expand Down Expand Up @@ -795,6 +834,9 @@ public void Flush()

SaveMetadata();
_metadataDirty = false;

// An explicit flush is a durability boundary for in-place writes as well.
storage?.SyncPendingWrites();
}
catch (Exception ex)
{
Expand Down
Loading
Loading