Repository navigation
fix: PageBased autocommit durability, CASE expressions; READ A/B, coverage, durability design note - #523
Open
CastleSoft wants to merge 2 commits into
Conversation
… as values; durability design note - PageBased dirty pages are written at the end of every autocommit statement and at every transaction/batch commit, synced under WalDurabilityMode.FullSync (known issue 15); FlushDirtyPages skips the sync when nothing is dirty. Measured cost: autocommit PageBased write ~50 -> ~750 us per statement, the order of a columnar autocommit write. - CASE ... WHEN ... END (simple, searched, nested) in the scalar evaluator: UPDATE SET, CHECK, ORDER BY, SELECT list; a condition is a value (1/0/NULL) in CASE branches and SET values. - docs/storage/PAGEBASED_DURABILITY_DESIGN.md: page checksums in the unused header bytes, a page redo log, AES-GCM page encryption and the directory-mode WAL, with the owner decisions each needs. - READ watch-band investigation: interleaved A/B against master shows no read regression; recorded known issue 21 (PageBased non-key lookups scan the table). Coverage-gap tests for the subquery ORDER BY path and SET conversions.
…urability-case-and-coverage
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



Summary
This PR covers the five items of the "while waiting" list.
1. PageBased autocommit durability (known issue §15) — fixed
A PageBased statement that had returned was lost by a crash: its pages stayed in the cache until an explicit
Flush(), a commit, an eviction orDispose.What changed:
WalDurabilityMode.FullSync, the default.Asyncwrites the pages without the sync, which protects against a process crash but not a power loss, matching fix: atomic metadata writes and fsync of in-place writes; record crash-safety findings (known issues 14-20) #522.FlushDirtyPagesreturns without syncing when nothing is dirty, so a SELECT pays nothing.Cost, measured over 2,000 statements on this machine:
The three PageBased crash cases in
CrashDurabilityTests(autocommit UPDATE, INSERT + DELETE, batch) failed on #522's code and pass now. They were skipped in #522.2. Gate READ in the watch band — investigated: no read regression
I built
master(44195519) in a separate worktree and ran the same read probe against it and this stack, interleaved, three rounds each.So the watch-band READ ratios come from this machine against the recorded baseline, not from code changes.
Found and recorded as §21: a PageBased lookup on a non-key column scans the whole table, about 32 ms for 20,000 rows. Hash indexes exist only for columnar tables. This is the same on
master.3. Coverage
tools/measure-coverage.ps1, CI filter)master)The two biggest gaps now have tests:
SubqueryExecutor's ORDER BY had 0 of 6 lines covered. An IN-subquery whose own ORDER BY + LIMIT decides the result now exercises it.Writing that test exposed a gap:
SET flag = length(label) > 0failed. A comparison is a value in SQLite, and item 4 now covers it.4.
CASE … WHEN … END, and conditions used as valuesBefore this PR, the scalar evaluator didn't parse CASE at all:
SET grade = CASE WHEN n >= 5 THEN 'high' ELSE 'low' ENDstored the statement text instead of the value.Now the following work, in UPDATE SET, CHECK, ORDER BY and the SELECT list:
THEN amount < 0, orSET active = length(name) > 0, evaluates to 1, 0 or NULL as in SQLite.A condition that starts with a CASE is compiled around it, so the CASE's own
=andANDaren't mistaken for the outer condition's.CaseExpressionTestshas 6 tests, all failing on #522's code.5. Design note:
docs/storage/PAGEBASED_DURABILITY_DESIGN.mdThe note covers known issues §14, §18, §19 and §20 with options, a recommendation and a test plan for each.
Proposals:
Owner decisions it lists:
UseGroupCommitWal;Tests
SharpCoreDB.CI.slnfbuilds with 0 errors, and the documentation consistency tests pass.Performance
Write-path gate: GATE PASSED. Nothing is slower than baseline × 1.50.
Note for #522: this build includes #522's commit fsync, and raw UPDATE measured 1.25×. #522's own run measured 1.54× and failed. The benchmark's own notes say this arm's raw UPDATE runs at one of two speeds, about 215k or 420k ops/s. Both runs are recorded as measured. A gate run on a quieted machine should settle whether #522 needs the baseline decision at all.