Skip to content

fix: PageBased autocommit durability, CASE expressions; READ A/B, coverage, durability design note - #523

Open
CastleSoft wants to merge 2 commits into
fix/durability-metadata-and-fsyncfrom
fix/pagebased-durability-case-and-coverage
Open

CastleSoft wants to merge 2 commits into
fix/durability-metadata-and-fsyncfrom
fix/pagebased-durability-case-and-coverage

Conversation

@CastleSoft

Copy link
Copy Markdown
Collaborator

Stacked on #522. This PR targets fix/durability-metadata-and-fsync, so its diff shows only this change. Merge in order: #518 → #519 → #520 → #521 → #522 → this one, retargeting each to master in turn.

Summary

This PR covers the five items of the "while waiting" list.

1. PageBased autocommit durability (known issue §15) — fixed

A PageBased statement that had returned was lost by a crash: its pages stayed in the cache until an explicit Flush(), a commit, an eviction or Dispose.

What changed:

Cost, measured over 2,000 statements on this machine:

  • An autocommit PageBased INSERT or UPDATE went from about 50 µs to about 750 µs.
  • That is the same order as a columnar autocommit write, which has always been written through to disk.
  • Batches and transactions pay once per commit. The page redo log in the design note (item 5) would turn this into one sequential append.

The three PageBased crash cases in CrashDurabilityTests (autocommit UPDATE, INSERT + DELETE, batch) failed on #522's code and pass now. They were skipped in #522.

2. Gate READ in the watch band — investigated: no read regression

I built master (44195519) in a separate worktree and ran the same read probe against it and this stack, interleaved, three rounds each.

Read shape Result
Point lookup by primary key equal
Point lookup by a non-key column about 25% faster on this stack
Full scan of 20,000 rows within ±3%
PageBased reads equal

So the watch-band READ ratios come from this machine against the recorded baseline, not from code changes.

Found and recorded as §21: a PageBased lookup on a non-key column scans the whole table, about 32 ms for 20,000 rows. Hash indexes exist only for columnar tables. This is the same on master.

3. Coverage

Measure Value
Stack overall (tools/measure-coverage.ps1, CI filter) 48.47% (floor 18%; was 47.34% on #516)
New lines in the stack (since master) 84.9% (1,025 of 1,208; codecov needs 50%)

The two biggest gaps now have tests:

  • SubqueryExecutor's ORDER BY had 0 of 6 lines covered. An IN-subquery whose own ORDER BY + LIMIT decides the result now exercises it.
  • The SET value conversions are now tested into REAL, BIGINT, BOOLEAN and DATETIME, plus integer overflow being rejected.

Writing that test exposed a gap: SET flag = length(label) > 0 failed. A comparison is a value in SQLite, and item 4 now covers it.

4. CASE … WHEN … END, and conditions used as values

Before this PR, the scalar evaluator didn't parse CASE at all:

  • SET grade = CASE WHEN n >= 5 THEN 'high' ELSE 'low' END stored the statement text instead of the value.
  • A CHECK constraint written with CASE rejected every row.

Now the following work, in UPDATE SET, CHECK, ORDER BY and the SELECT list:

  • simple, searched and nested CASE;
  • a missing ELSE gives NULL;
  • a condition used as a value: THEN amount < 0, or SET active = length(name) > 0, evaluates to 1, 0 or NULL as in SQLite.

A condition that starts with a CASE is compiled around it, so the CASE's own = and AND aren't mistaken for the outer condition's. CaseExpressionTests has 6 tests, all failing on #522's code.

5. Design note: docs/storage/PAGEBASED_DURABILITY_DESIGN.md

The note covers known issues §14, §18, §19 and §20 with options, a recommendation and a test plan for each.

Proposals:

  • Page checksums (§19): XxHash64 plus a flag byte, stored in the page header's 23 unused bytes. There is no format change, and pages written before stay readable.
  • Page redo log (§18): one log per table holding page after-images and commit records, replayed on open, with checkpoints. This also makes §15 cheaper.
  • AES-GCM page encryption (§14): needs 28 bytes, more than the free header holds, so it is a format bump behind a flag.
  • The directory-mode WAL (§20): two options, laid out with their trade-offs.

Owner decisions it lists:

  • the page format bump for encryption;
  • what happens to UseGroupCommitWal;
  • whether PageBased should be marked not production-ready until §14 and §18 are done.

Tests

Suite Total Failed Skipped
Core 2546 0 6 (the documented single-file cases)
VectorSearch 255 0 0
EF Core 116 0 0
linq2DB 96 0 0
Regression canaries 51 0 0

SharpCoreDB.CI.slnf builds with 0 errors, and the documentation consistency tests pass.

Performance

Write-path gate: GATE PASSED. Nothing is slower than baseline × 1.50.

Metric Capability ratio Verdict
raw INSERT 1.10× ok
raw READ 1.43× watch
raw UPDATE 1.25× watch
raw DELETE 1.26× watch
default INSERT 1.11× ok
default READ 1.30× watch
default UPDATE 1.20× ok
default DELETE 1.38× watch

Note for #522: this build includes #522's commit fsync, and raw UPDATE measured 1.25×. #522's own run measured 1.54× and failed. The benchmark's own notes say this arm's raw UPDATE runs at one of two speeds, about 215k or 420k ops/s. Both runs are recorded as measured. A gate run on a quieted machine should settle whether #522 needs the baseline decision at all.

… as values; durability design note

- PageBased dirty pages are written at the end of every autocommit statement and at every transaction/batch commit,
  synced under WalDurabilityMode.FullSync (known issue 15); FlushDirtyPages skips the sync when nothing is dirty.
  Measured cost: autocommit PageBased write ~50 -> ~750 us per statement, the order of a columnar autocommit write.
- CASE ... WHEN ... END (simple, searched, nested) in the scalar evaluator: UPDATE SET, CHECK, ORDER BY, SELECT list;
  a condition is a value (1/0/NULL) in CASE branches and SET values.
- docs/storage/PAGEBASED_DURABILITY_DESIGN.md: page checksums in the unused header bytes, a page redo log, AES-GCM
  page encryption and the directory-mode WAL, with the owner decisions each needs.
- READ watch-band investigation: interleaved A/B against master shows no read regression; recorded known issue 21
  (PageBased non-key lookups scan the table). Coverage-gap tests for the subquery ORDER BY path and SET conversions.
@sonarqubecloud

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant