Skip to content

fix: PageBased page checksums and pinning of uncommitted pages; page index lookups by design - #524

Open
CastleSoft wants to merge 1 commit into
fix/pagebased-durability-case-and-coveragefrom
fix/pagebased-checksums-pinning-indexes
Open

CastleSoft wants to merge 1 commit into
fix/pagebased-durability-case-and-coveragefrom
fix/pagebased-checksums-pinning-indexes

Conversation

@CastleSoft

Copy link
Copy Markdown
Collaborator

Stacked on #523. This PR targets fix/pagebased-durability-case-and-coverage, so its diff shows only this change. Merge order: #518 → #519 → #520 → #521 → #522 → #523 → this one, retargeting each to master in turn.

Summary

This PR covers items 2–4 from the to-do list. Item 1, #521's SonarCloud failure, is fixed on #521's own branch; see the last section.

Page checksums (known issue §19) — detection fixed

Before: a torn page, meaning the tail of an 8 KB page left unwritten by an interrupted write, read as a page with empty slots. Its rows vanished with no error, and the table kept accepting writes. The measured case lost 25 of 40 rows.

  • Format: every page written now carries a CRC32C in header bytes 9–12 and a format flag in byte 13. No previous version wrote those header bytes, so:
    • the page layout doesn't change;
    • a page written before (flag 0) is read unverified, as it always was.
  • On a mismatch: reading the page raises InvalidDataException naming the page and the file.
  • At open: the scan that rebuilds the page index skips verification, so one damaged page doesn't stop the table from opening.
  • Second defect fixed: ScanPageBasedTable wrapped the whole scan in a catch-all and returned the rows read so far on any error. That is how the damage stayed silent. It now lets the checksum error through; the per-record catch for a malformed record stays.
  • Algorithm: BitOperations.Crc32C (the CPU's CRC instruction, no new dependency), measured at ~2.9 µs per 8 KB page.
  • Still open: repairing a damaged page needs the page redo log from the design note.

PageChecksumTests:

  • torn page → error;
  • one flipped data byte → error;
  • pages without a checksum still read and write.

The two damage cases failed on #523's code.

Pinning uncommitted pages (known issue §18) — interim fix

A dirty page changed by the still-open transaction is now pinned until the commit:

  • an eviction no longer writes it;
  • a flush in the middle of the transaction no longer writes it either. This was the main leak, more than eviction: ExecuteQuery flushes pending writes inside BeginBatchUpdate.

The page stays in memory, where the transaction's own reads see it, and the commit writes it. The cache may exceed its capacity until the transaction ends.

Also fixed: the eviction loop spun forever when the cache was over capacity and none of its pages were in the LRU list.

Test: a crash test runs with a 4-page cache. It updates and deletes inside a transaction, reads the whole table twice, then kills the process; the reopened table shows the committed state. It failed on #523's code.

Still open: a crash during the commit's multi-page write can still tear the commit. That needs the page redo log.

PageBased non-key lookups (known issue §21) — resolved as by design

Measured on 20,000 rows: a PageBased lookup on a non-key column takes ~80 ms without an index and 0.02–0.1 ms after CREATE INDEX.

PageBased tables deliberately don't auto-create column indexes, unlike columnar tables, and SQLite doesn't either. Turning that on would reverse a documented choice, so that's an owner decision. PageBasedSecondaryIndexTests pins that the explicit index returns exactly the right rows through INSERT, UPDATE, DELETE and a reopen.

Item 1 — #521's SonarCloud failure (on #521's branch, merged up)

The failing condition was duplicated new code: 3.6% against a 3% limit. Three copies of the same quote- and parenthesis-aware scanner are now one helper, SqlLiteralScanner.FindTopLevel / IsKeywordAt:

  • the UPDATE SET / WHERE split;
  • the ON CONFLICT finder;
  • the ORDER BY item-end finder.

The flagged smells are cleared too:

  • the conflict-clause reader and resolver are split into smaller methods;
  • ResolveInsertConflict is now static, without its unused parameter;
  • nested ternaries are rewritten.

SonarCloud now passes on #521. That commit was merged into #522's and #523's branches.

Tests

Suite Total Failed Skipped
Core 2551 0 6 (the documented single-file cases)
VectorSearch 255 0 0
EF Core 116 0 0
linq2DB 96 0 0
Regression canaries 51 0 0

SharpCoreDB.CI.slnf builds with 0 errors, and the documentation consistency tests pass.

One full core run on #521's branch had a single intermittent failure. Three consecutive reruns, plus the run on the top of the stack, were green, and I couldn't capture the failing test's name.

Performance

The write-path gate FAILED on raw READ (1.58×). The investigation shows machine noise, not this code:

  • What changed: this PR changes only PageBased code. The gate's arms run on columnar tables.
  • The run itself was very noisy. Spread between its reps: raw READ 1.33×, UPDATE 1.76×, DELETE 1.96×; a quiet machine sits near 1.0. The discarded warm-up reps ranged from 66k to 145k READ ops/s.
  • The exact read statement shows no regression. The gate's READ arm runs a parameterized SELECT * FROM docs WHERE name = @name with NoEncryptMode. I A/B'd that statement against master in an interleaved run, 3 rounds each:
Build µs per read
master 7.45–9.05
This stack 7.98–8.21

So there's no regression on the read path.

Metric Capability ratio Verdict
raw INSERT 1.09× ok
raw READ 1.58× REGRESSED (noise, see above)
raw UPDATE 1.26× watch
raw DELETE 1.15× ok
default INSERT 1.08× ok
default READ 1.32× watch
default UPDATE 1.14× ok
default DELETE 1.26× watch

A gate run on a quieted machine (that needs an elevated shell) should confirm this before the stack is merged.

…ex lookups resolved as by design

- Every PageBased page written carries a CRC32C (header bytes 9-12) and a format flag (byte 13) - bytes no version
  wrote, so older pages read unverified; a damaged page raises InvalidDataException naming it, and the PageBased
  scan's catch-all no longer hides it (known issue 19, detection). ~2.9 us per page.
- Dirty pages of an open transaction are pinned: neither a mid-transaction flush (ExecuteQuery inside
  BeginBatchUpdate) nor an eviction writes them before the commit (known issue 18, interim). The eviction loop no
  longer spins when the cache is over capacity with an empty LRU list.
- Known issue 21 resolved as by design: PageBased tables do not auto-index; CREATE INDEX serves the lookup
  (~80 ms -> 0.02-0.1 ms on 20,000 rows) and PageBasedSecondaryIndexTests pins its correctness.
@sonarqubecloud

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant