Skip to content

fix: PageBased transaction rollback and in-transaction columnar scans (known issues 13 and 12) - #520

Open
CastleSoft wants to merge 1 commit into
fix/returning-check-in-tx-readsfrom
fix/transaction-rollback-and-scan-overlay
Open

CastleSoft wants to merge 1 commit into
fix/returning-check-in-tx-readsfrom
fix/transaction-rollback-and-scan-overlay

Conversation

@CastleSoft

Copy link
Copy Markdown
Collaborator

Stacked on #519. This PR targets fix/returning-check-in-tx-reads, so its diff shows only this change. Merge order: #518, then #519 (retargeted to master), then this PR (retargeted to master).

Summary

Fixes known issues §13 (PageBased tables could not roll back) and §12 (inside a transaction, a columnar full scan missed the transaction's own UPDATEs and DELETEs). Both are about the transaction an ADO.NET DbTransaction opens, and linq2db and EF Core transactions open the same one. Every fix has a regression test that was run red on #519's code and green after.

§13 — PageBased rollback

Before: CancelBatchUpdate() (DbTransaction.Rollback()) left a PageBased UPDATE or INSERT in place, and it was still there after a reopen. The PageBased engine writes into its page cache and page files directly, outside the storage transaction buffer, and PageBasedEngine.Rollback() was a stub: "we don't support rollback in this POC".

Fix: an undo scope on each PageManager.

  • On the first change to a page inside the transaction, the manager keeps a copy of the page as it was. Pages the transaction allocates are recorded too.
  • Rollback puts the copies back and marks them dirty, so a page the cache already wrote to disk mid-transaction is written back. Pages the transaction allocated go back to the free list. Every table then rebuilds its primary-key index from storage and marks its hash indexes stale.
  • Commit drops the copies.
  • The scope opens and closes with BeginBatchUpdate / EndBatchUpdate / CancelBatchUpdate and with the storage-only transaction the EF Core provider uses. A table first touched mid-transaction joins the scope.
  • Outside a transaction, a page write pays one null check.

Also fixed: a columnar rollback restored the rows but not the in-memory hash index. After rolling back an UPDATE of an indexed column, WHERE name = 'old' found nothing.

TransactionRollbackTests cover:

  • Both storage modes.
  • A reopen after rollback.
  • A transaction spanning about 600 inserts across many pages.
  • New writes after a rollback.
  • A real ADO.NET SharpCoreDBConnection.BeginTransaction() / Rollback().

§12 — in-transaction columnar scans

Before: inside a transaction, a columnar full scan read the data file as it was on disk, while the transaction's in-place overwrites and its deletes were buffered until commit. So the scan:

  • returned an updated row's old value;
  • still listed a deleted row;
  • found nothing for WHERE n = 50 after SET n = 50.

A primary-key lookup still returned the deleted row.

Fix:

  • Storage.ReadBytesWithRecordOffsets, the scan's data source, overlays the transaction's buffered changes on the records it returns:
    • an overwritten record gets its new payload, decrypted when the file is encrypted;
    • a deleted record becomes a tombstone and leaves the physical-offset map, so later records keep their mapping.
  • This runs only while a transaction holds such changes for that file, so all other scans are untouched.
  • ReadBytesFrom returns nothing for a record the transaction deleted, checking a lock-free count first.

InTransactionReadTests cover:

  • Columnar tables, both encrypted and NoEncryptMode, plus page-based tables.
  • UPDATE, DELETE, and a mix with INSERTs.
  • Reads by full scan, filtered scan, primary-key lookup and index lookup.

Still open (recorded in §13)

  • ExecuteBatchSQL's own failure rollback doesn't use the undo scope. That rollback is internal and runs when a statement in the batch fails. A PageBased table keeps the batch statements that ran before the failure. Hooking it adds a page copy per touched page to the most-measured write path, so it needs its own gate run.
  • There is no page WAL. A crash mid-transaction can still leave pages the cache wrote to disk early.

Tests

New: TransactionRollbackTests and InTransactionReadTests, 16 cases. 11 fail on #519's code. The 5 that pass there are cases that already worked: columnar rollback, the columnar ADO.NET rollback, and page-based in-transaction reads.

Suite Total Failed Skipped
Core 2489 0 6 (the documented single-file cases)
VectorSearch 255 0 0
EF Core 116 0 0
linq2DB 96 0 0
Regression canaries (ReopenRoundTripMatrixTests, FormatCompatPolicyTests and others) 51 0 0

SharpCoreDB.CI.slnf builds with 0 errors, and the documentation consistency tests pass.

Performance

Write-path gate: GATE PASSED. Nothing is slower than baseline × 1.50.

Metric Ratio Verdict
raw INSERT 1.05× ok
raw READ 1.30× watch
raw UPDATE 1.21× ok
raw DELETE 1.16× ok
default INSERT 1.06× ok
default READ 1.27× watch
default UPDATE 1.20× ok
default DELETE 1.31× watch

The watch-band READ and DELETE ratios match the previous run on this machine (#519: default READ 1.33×, default DELETE 1.40×), which didn't change the read path. Outside a transaction, the new read-path code is one false bool check. The machine wasn't quieted first.

… (known issues 13 and 12)

- PageBased rollback: each PageManager keeps a page's image on its first change inside a transaction
  (BeginUndoScope); CancelBatchUpdate / RollbackStorageTransaction restore the images (marked dirty) and free
  pages the transaction allocated, then every table rebuilds its primary-key index and marks hash indexes stale.
  This also fixes a columnar rollback that left the hash index on the rolled-back value. Outside a transaction a
  write pays one null check. ExecuteBatchSQL's internal failure rollback is not hooked yet (recorded in issue 13).
- In-transaction columnar scans: ReadBytesWithRecordOffsets lays the transaction's buffered in-place overwrites
  and commit-time tombstones over the scanned records, and ReadBytesFrom returns nothing for a record the
  transaction deleted. Applies only while a transaction holds such changes for the file.
@sonarqubecloud

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant