Repository navigation
fix: PageBased page checksums and pinning of uncommitted pages; page index lookups by design - #524
Open
CastleSoft wants to merge 1 commit into
Conversation
…ex lookups resolved as by design - Every PageBased page written carries a CRC32C (header bytes 9-12) and a format flag (byte 13) - bytes no version wrote, so older pages read unverified; a damaged page raises InvalidDataException naming it, and the PageBased scan's catch-all no longer hides it (known issue 19, detection). ~2.9 us per page. - Dirty pages of an open transaction are pinned: neither a mid-transaction flush (ExecuteQuery inside BeginBatchUpdate) nor an eviction writes them before the commit (known issue 18, interim). The eviction loop no longer spins when the cache is over capacity with an empty LRU list. - Known issue 21 resolved as by design: PageBased tables do not auto-index; CREATE INDEX serves the lookup (~80 ms -> 0.02-0.1 ms on 20,000 rows) and PageBasedSecondaryIndexTests pins its correctness.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



Summary
This PR covers items 2–4 from the to-do list. Item 1, #521's SonarCloud failure, is fixed on #521's own branch; see the last section.
Page checksums (known issue §19) — detection fixed
Before: a torn page, meaning the tail of an 8 KB page left unwritten by an interrupted write, read as a page with empty slots. Its rows vanished with no error, and the table kept accepting writes. The measured case lost 25 of 40 rows.
InvalidDataExceptionnaming the page and the file.ScanPageBasedTablewrapped the whole scan in a catch-all and returned the rows read so far on any error. That is how the damage stayed silent. It now lets the checksum error through; the per-record catch for a malformed record stays.BitOperations.Crc32C(the CPU's CRC instruction, no new dependency), measured at ~2.9 µs per 8 KB page.PageChecksumTests:The two damage cases failed on #523's code.
Pinning uncommitted pages (known issue §18) — interim fix
A dirty page changed by the still-open transaction is now pinned until the commit:
ExecuteQueryflushes pending writes insideBeginBatchUpdate.The page stays in memory, where the transaction's own reads see it, and the commit writes it. The cache may exceed its capacity until the transaction ends.
Also fixed: the eviction loop spun forever when the cache was over capacity and none of its pages were in the LRU list.
Test: a crash test runs with a 4-page cache. It updates and deletes inside a transaction, reads the whole table twice, then kills the process; the reopened table shows the committed state. It failed on #523's code.
Still open: a crash during the commit's multi-page write can still tear the commit. That needs the page redo log.
PageBased non-key lookups (known issue §21) — resolved as by design
Measured on 20,000 rows: a PageBased lookup on a non-key column takes ~80 ms without an index and 0.02–0.1 ms after
CREATE INDEX.PageBased tables deliberately don't auto-create column indexes, unlike columnar tables, and SQLite doesn't either. Turning that on would reverse a documented choice, so that's an owner decision.
PageBasedSecondaryIndexTestspins that the explicit index returns exactly the right rows through INSERT, UPDATE, DELETE and a reopen.Item 1 — #521's SonarCloud failure (on #521's branch, merged up)
The failing condition was duplicated new code: 3.6% against a 3% limit. Three copies of the same quote- and parenthesis-aware scanner are now one helper,
SqlLiteralScanner.FindTopLevel/IsKeywordAt:ON CONFLICTfinder;The flagged smells are cleared too:
ResolveInsertConflictis now static, without its unused parameter;SonarCloud now passes on #521. That commit was merged into #522's and #523's branches.
Tests
SharpCoreDB.CI.slnfbuilds with 0 errors, and the documentation consistency tests pass.One full core run on #521's branch had a single intermittent failure. Three consecutive reruns, plus the run on the top of the stack, were green, and I couldn't capture the failing test's name.
Performance
The write-path gate FAILED on raw READ (1.58×). The investigation shows machine noise, not this code:
SELECT * FROM docs WHERE name = @namewithNoEncryptMode. I A/B'd that statement againstmasterin an interleaved run, 3 rounds each:masterSo there's no regression on the read path.
A gate run on a quieted machine (that needs an elevated shell) should confirm this before the stack is merged.