Skip to content

fix: atomic ExecuteBatchSQL, upsert / INSERT OR conflict clauses, subquery ORDER BY collation, quoted batch SET columns - #521

Open
CastleSoft wants to merge 2 commits into
fix/transaction-rollback-and-scan-overlayfrom
fix/batch-rollback-and-sql-gaps
Open

CastleSoft wants to merge 2 commits into
fix/transaction-rollback-and-scan-overlayfrom
fix/batch-rollback-and-sql-gaps

Conversation

@CastleSoft

Copy link
Copy Markdown
Collaborator

Stacked on #520. This PR targets fix/transaction-rollback-and-scan-overlay, so its diff shows only this change. Merge in order — #518, #519, #520, then this one — retargeting each to master after the one before it merges.

Summary

This PR covers items 4–9 from the follow-up list. Every fix has a regression test that fails on #520's code and passes after.

4. ExecuteBatchSQL is atomic

A failing statement in a batch kept some or all of the statements before it:

  • SELECT-bearing or mixed batches ran their statements one at a time with no transaction. Every storage mode kept the earlier statements.
  • Grouped batches rolled back only the storage buffer, which PageBased tables don't write through. A PageBased table kept the earlier changes.

Both paths now run as a single transaction with the PageBased undo scope from #520. If a caller already has a transaction open, that transaction decides the outcome. BatchAtomicityTests covers UPDATE, mixed and SELECT-bearing batches, a successful mixed batch, and a batch inside an outer transaction.

5. Upsert and INSERT OR …: they didn't run at all

  • INSERT OR IGNORE, OR REPLACE, OR ABORT, OR FAIL, OR ROLLBACK and REPLACE INTO failed with "Unsupported SQL statement".
  • An ON CONFLICT … DO NOTHING | DO UPDATE SET … [WHERE …] clause was silently ignored, so an upsert onto an existing key failed as a duplicate.
  • The conflict handlers in SqlParser.DML.InsertConflict existed, but nothing called them.

All of these now run on the SQL INSERT path:

  • both VALUES and SELECT inserts;
  • single-column and composite keys;
  • inside ExecuteBatchSQL too.

DO UPDATE assignments and their WHERE are evaluated as expressions over the existing row, with excluded.col holding the incoming value, so n = n + excluded.n works. UpsertTests covers it.

6. Subquery and derived-table ORDER BY

  • COLLATE and NULLS FIRST/LAST were ignored. COLLATE NOCASE DESC lost its DESC as well.
  • A descending sort always put NULLs last, whatever the statement said.
  • Text without a COLLATE was compared case-insensitively, where SQLite's default is BINARY. ORDER BY name over a derived table put 'a' before 'C'.

SubqueryOrderByCollationTests covers these.

7. Upsert lookups ignored collation

The composite conflict lookup now compares each column with its declared collation, as INSERT and UPDATE do: NOCASE conflicts on 'ABC'/'abc', BINARY doesn't. Its "same row" primary-key check was always case-insensitive and now uses the key's own collation.

8. Composite UNIQUE check without its index

There's no deadlock risk: the table lock allows recursion. The full-scan fallback now registers the column's auto index first and scans only as a last resort. This is hardening: the constraint's own index can't be dropped, as in SQLite.

9. Quoted batch SET columns take the fast path again

In ExecuteBatchSQL, SET "n" = 5 and SET [n] = 5 were looked up with their quotes, missed, and fell off the structured fast path. The name is now unquoted first; an unquoted name allocates nothing. BatchUpdateClassificationTests.BatchUpdate_QuotedSetColumn_TakesTheStructuredFastPath covers it.

Tests

The new and extended tests are in BatchAtomicityTests, UpsertTests and SubqueryOrderByCollationTests, plus one new fact in BatchUpdateClassificationTests. Of the 33 cases run red on #520's code, 24 fail. The 9 that pass there are expected:

  • INSERT OR ABORT already threw, though for a different reason.
  • The outer-transaction rollback was already fixed in fix: PageBased transaction rollback and in-transaction columnar scans (known issues 13 and 12) #520.
  • The successful mixed batch and the columnar update batch: columnar rollback already worked.
  • The default DESC NULL placement was already correct.
  • The COLLATE NOCASE derived-table case passed only because the default comparison was case-insensitive. That defect is now pinned by DerivedTable_OrderByTextWithoutCollate_SortsAsBinary.
Suite Total Failed Skipped
Core 2523 0 6 (the documented single-file cases)
VectorSearch 255 0 0
EF Core 116 0 0
linq2DB 96 0 0
Regression canaries 51 0 0

SharpCoreDB.CI.slnf builds with 0 errors, and the documentation consistency tests pass.

Performance

Write-path gate: GATE PASSED. Nothing is slower than baseline × 1.50.

Metric Ratio Verdict
raw INSERT 1.04× ok
raw READ 1.39× watch
raw UPDATE 1.24× ok
raw DELETE 1.16× ok
default INSERT 1.04× ok
default READ 1.42× watch
default UPDATE 1.17× ok
default DELETE 1.33× watch

The READ ratios are higher than the last two runs: 1.30×/1.27× on #520 and 1.33× on #519. This PR doesn't change the SELECT path; it changes the batch dispatchers, the INSERT path and AST ORDER BY. The machine wasn't quieted: a permission rule blocks the quiet script's cleanup. I recommend a gate rerun on a quieted machine before merging the stack, to tell noise from a real READ drift.

…query ORDER BY collation, quoted batch SET columns

- ExecuteBatchSQL: SELECT-bearing and mixed batches ran statement by statement without a transaction; they now run
  as one transaction with the PageBased undo scope, and every grouped batch transaction uses the undo scope too, so
  a failing batch leaves every table unchanged.
- INSERT OR IGNORE|REPLACE|ABORT|FAIL|ROLLBACK, REPLACE INTO and ON CONFLICT ... DO NOTHING | DO UPDATE SET ...
  [WHERE ...] run on the SQL INSERT path (VALUES and SELECT, single and composite keys); DO UPDATE assignments are
  expressions with excluded.col. Composite conflict lookups compare with the column collation.
- AST ORDER BY (derived tables, IN-subqueries) honours COLLATE and NULLS FIRST/LAST and compares text as BINARY by
  default; COLLATE NOCASE DESC no longer loses its direction.
- Batch UPDATEs with a quoted SET column take the structured fast path again.
- Composite UNIQUE check registers the column's auto index before falling back to a scan.
…DER BY splitters (SonarCloud duplication)

SqlLiteralScanner.FindTopLevel / IsKeywordAt replace three copies of the same quote- and parenthesis-aware loop
(SplitUpdateSetAndWhere, the ON CONFLICT finder, FindOrderItemEnd), which SonarCloud's quality gate failed on as
duplicated new code (3.6% > 3%). Also clears the flagged smells: the conflict-clause reader and resolver split into
smaller methods (ReadInsertOrPolicy, ReadDoUpdateAssignments, DeleteConflictingRows), ResolveInsertConflict made
static without its unused table-name parameter, and nested ternaries in SqlOrderBy / ParseOrderItemSuffix rewritten.
No behaviour change.
@sonarqubecloud

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant