Skip to content

fix: transaction commit failures, PageBased DELETE durability, UPDATE RETURNING and CHECK IN/BETWEEN/GLOB - #519

Open
CastleSoft wants to merge 1 commit into
fix/composite-keys-update-atomicity-orderbyfrom
fix/returning-check-in-tx-reads
Open

CastleSoft wants to merge 1 commit into
fix/composite-keys-update-atomicity-orderbyfrom
fix/returning-check-in-tx-reads

Conversation

@CastleSoft

Copy link
Copy Markdown
Collaborator

Stacked on #518. This PR targets fix/composite-keys-update-atomicity-orderby, so its diff shows only this change. Retarget it to master once #518 is merged.

Summary

Fixes items 1 and 2 from the follow-up list and covers the investigation of item 3. Investigating item 3 turned up three defects that cause failed commits or data loss; all three are fixed here. Two more gaps need design-level work, so they're recorded with cause and a fix outline instead of being rushed. Every fix has a regression test that was run red on the unfixed source and green after.

1. UPDATE … RETURNING returned the wrong rows — fixed

The returned rows were found by running the WHERE clause again after the update. So when the SET changed a column the WHERE tests, nothing came back. Both SET status = 'done' WHERE status = 'open' RETURNING id and SET n = n + 10 WHERE n < 5 RETURNING n returned 0 rows. The matched rows are now read first, and the SET list is applied to them the same way the update applies it. UpdateReturningTests.

2. CHECK constraints using IN / BETWEEN / GLOB — fixed

This was the opposite of the suspicion, and worse: they accepted every row. CHECK (status IN ('a','b')) let 'z' in, and CHECK (n BETWEEN 1 AND 10) let 50 in. The SQLite-semantics evaluator now handles IN, NOT IN, BETWEEN, NOT BETWEEN and GLOB:

  • NULL is treated as unknown, so a NULL still passes a CHECK.
  • GLOB is case-sensitive and supports *, ? and […] character classes.
  • The same operators on computed values in a WHERE clause go through it too, for example length(name) BETWEEN 3 AND 6.

CheckConstraintOperatorTests.

3. Reads inside a transaction — investigated

The transaction under investigation is BeginBatchUpdate() / EndBatchUpdate(), which every ADO.NET DbTransaction opens, so linq2db and EF Core transactions too.

Finding Status
🔴 Commit threw "Collection was modified" whenever the transaction had buffered a whole-file write, such as the database metadata. During the flush, Storage.WriteBytes saw the transaction still open and buffered the write again into the list being looped over. The write was lost; had the loop not thrown, the already-encrypted bytes would have been encrypted a second time. Fixed: buffered bytes go straight to the file in their stored form.
🔴 After every commit, a columnar table read as empty until the database was reopened. The commit's primary-key index rebuild cleared the index, then searched that empty index for each row's position. Fixed: positions are read before the index is cleared.
🔴 A PageBased DELETE was undone by every reopen, inside a transaction or not. A page slot stores only offset and length, so the Deleted flag was never written, and loading marked every slot active. Fixed: a deleted slot is written with length 0, a length no real record has, and read back as deleted. Older files read as they always did. Rows deleted before this version still come back once, because their deletion was never recorded.
🟠 Inside a transaction, a columnar full scan doesn't see the transaction's own UPDATEs and DELETEs. Primary-key lookups do. Open, known issue §12. The scan reads the file on disk and ignores the storage layer's buffered overwrites and deletes. Fixing it means overlaying them on the hottest read path, which needs its own gate and a read benchmark.
🔴 PageBased storage cannot roll back: a rolled-back UPDATE or INSERT is still there after reopen. Open, known issue §13. PageBasedEngine.Rollback() is a stub ("we don't support rollback in this POC"). Fixing it needs per-page undo images or shadow paging, which is a change to the engine's durability path.

BatchUpdateTransactionTests pins the fixed rows.

Tests

The 24 new test cases are in UpdateReturningTests, CheckConstraintOperatorTests and BatchUpdateTransactionTests. Twenty fail on the unfixed source. The four that already pass there are expected:

  • RETURNING * where the SET doesn't touch a WHERE column, which already worked.
  • The empty transaction on page-based storage, since the empty-table bug was columnar-only.
  • The columnar rollback check, which guards behaviour that was already correct.
Suite Total Failed Skipped
Core 2473 0 6 (the documented single-file cases)
VectorSearch 255 0 0
EF Core 116 0 0
linq2DB 96 0 0
Regression canaries, including ReopenRoundTripMatrixTests and FormatCompatPolicyTests 51 0 0

SharpCoreDB.CI.slnf builds with 0 errors. The documentation consistency tests pass.

Performance

Write-path gate: GATE PASSED, with nothing slower than baseline × 1.50. Three metrics landed in the watch band:

Metric Ratio
raw UPDATE 1.31×
default READ 1.33×
default DELETE 1.40×

The run wasn't on a quieted machine: a permission rule blocked the quiet script's temp-directory cleanup. The only hot-path change in this PR is one flag test per slot in PageManager.WritePage. The columnar READ and DELETE paths aren't touched.

… RETURNING and CHECK IN/BETWEEN/GLOB

- EndBatchUpdate (every ADO.NET/linq2db/EF Core commit): a buffered whole-file write was re-buffered during the
  flush, so the commit threw "Collection was modified" and the write was lost; buffered bytes now go straight to
  the file in their stored form. The commit's primary-key index rebuild searched the index it had just cleared,
  leaving columnar tables reading as empty until reopen; positions are captured before clearing.
- PageBased DELETE was undone by every reopen (slot flags were never persisted); a deleted slot is now stored
  with length 0 and read back as deleted. Older files read as before.
- UPDATE ... RETURNING re-ran the WHERE after the update and missed rows whose WHERE column the SET changed;
  the matched rows are read first and the SET list applied to them.
- CHECK with IN / NOT IN / BETWEEN / NOT BETWEEN / GLOB accepted every row; the SQLite-semantics evaluator now
  reads them (also for computed operands in WHERE).
- Known issues 12 (columnar scan inside a transaction misses its own UPDATE/DELETE) and 13 (PageBased rollback
  is a stub) recorded with cause and fix outline.
@sonarqubecloud

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant