Skip to content

Test suite for the packages, and the bugs and performance problems it found - #23

Merged
donwibier merged 32 commits into
masterfrom
fix/test-suite-bugs
Nov 4, 2026
Merged

donwibier merged 32 commits into
masterfrom
fix/test-suite-bugs

Conversation

@donwibier

Copy link
Copy Markdown
Owner

Summary

A new unit test suite for the packages found close to 80 bugs, plus a number of performance problems. This PR adds the tests and fixes them.

Security (DX.Data.Xpo.Identity): updates of existing user tokens were never saved. A redeemed two-factor recovery code stayed valid, and resetting the authenticator key or regenerating recovery codes had no effect. With the AutoMapper package, tokens were not stored at all. Lockouts were not persisted either, so lockout after failed attempts never took effect. Each fix has its own commit.

Tests

  • New projects DX.Utils.Tests, DX.Data.Xpo.Tests and DX.Data.EF.Tests (SQLite in-memory). DX.Data.Xpo.Identity.Tests goes from 17 to 250 tests.
  • All three are added to DXWeb.sln and the Tests workflow.
  • The store and Identity tests run against both the Mapster and the AutoMapper flavour.
  • New SQLite-file tests cover the XPO delete path that uses a database transaction.
Tests Pass Skipped
DX.Utils.Tests 317 1
DX.Data.Xpo.Tests 163 1
DX.Data.EF.Tests 109 0
DX.Data.Xpo.Identity.Tests 250 0
DX.Utils.CronJobs.Tests 77 0

These ran locally on net8.0 and net10.0. This machine has no .NET 9 runtime, so net9.0 runs only in CI.

The two skipped tests pin behaviour left alone on purpose:

  • XpoDatabase with DatabaseAndSchema sets the process-wide XpoDefault.DataLayer.
  • ParseEnum("99") returns an undefined value. Rejecting it would also reject valid [Flags] combinations.

Fixes

  • XPO Identity:
    • Tokens and recovery codes are saved, and lockout is persisted.
    • ConcurrencyStamp is checked on update.
    • AddLoginAsync (threw once any login existed) and GetUsersForClaimAsync (always threw) work.
    • Unknown roles throw in AddToRoleAsync.
    • RoleManager.Roles works; it threw NotImplementedException.
    • Cancellation tokens are honoured and null arguments throw ArgumentNullException.
    • The Mapster stores no longer rewrite TypeAdapterConfig.GlobalSettings.
  • DX.Data.EF:
    • Batches are atomic; any batch of 2+ items used to throw.
    • A failed operation no longer poisons the DbContext.
    • UpdateAsync with an empty key no longer inserts.
    • Results contain every item.
    • Execute(..., transactional: false) keeps the work's saved changes.
  • DX.Data.Xpo:
    • Haversine distances over about 6,670 km are correct.
    • PredicateBuilder works with XPO.
    • UpdateAsync with a default key no longer inserts.
    • Results contain every item.
    • XpoDatabase no longer races on the first use of a name.
    • GetDataLayer(name) and AddXpoUnitOfWork(name) honour the name.
  • DX.Utils:
    • Collection.Clear() and the IList indexer setter no longer overflow the stack.
    • MIME lookup works.
    • Bits.Clear no longer always returns 0.
    • Fixed GetDisplay, PropertyExtensions and StripHTMLTags.
    • GetTypesWith is thread safe and picks up assemblies loaded later.

Performance

  • StripHTMLTags is linear: 100 KB went from 1.2 s to 3 ms.
  • MIME definitions are parsed once: about 400× faster.
  • PropertyExtensions no longer starts a Task.Run per lookup.
  • Batch Update/Store/Delete load all keys in one query instead of one per item, in both the XPO and EF stores.
  • EF: one SaveChanges per batch, updates write only changed columns, and GetByKey reads without tracking.
  • Identity role membership: adding or removing it writes the link row directly instead of loading every member of the role.
  • XPO deletes on a pooled (SQL) database run in a database-level transaction on a connection of their own (ExplicitUnitOfWork), with a new BeforeDeleteAsync hook.
    • The Identity stores use the hook to remove role links first, so deleting a user or role no longer loads every member of each role.
    • Measured with a role of 3,000 users on SQLite: user delete 60 → 8 ms, role delete 566 → 74 ms.

Behaviour changes

These are documented in the root README ("Fix release — behaviour changes"), in each affected package's README and in its PackageReleaseNotes. Versions bumped:

Package New version
DX.Utils 26.1.3.41
DX.Data 26.1.3.37
DX.Data.Xpo 26.1.3.40
DX.Data.Xpo.Mapster 26.1.3.36
DX.Data.Xpo.AutoMapper 26.1.3.38
DX.Data.EF 26.1.3.37
DX.Data.EF.Mapster 26.1.3.35
DX.Data.EF.AutoMapper 26.1.3.35
DX.Data.Xpo.Identity 26.1.3.40
DX.Data.Xpo.Identity.Mapster 26.1.3.40
DX.Data.Xpo.Identity.AutoMapper 26.1.3.39

Known issue, not caused by this PR

The sample app DX.Test.Blazor8 (net9.0) fails to compile its Razor pages in a fresh checkout of master on the 10.0.401 SDK, the same as on this branch. CI on master was green 9 days ago, so a floating package version has probably moved since then. If the Tests job fails on that project, the cause is the same and needs a separate look.

🤖 Generated with Claude Code

donwibier and others added 30 commits November 4, 2026 19:43
…ity stores

New projects DX.Utils.Tests, DX.Data.Xpo.Tests, DX.Data.EF.Tests; Identity tests
extended. Confirmed bugs are pinned as tests skipped with a BUG: reason.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Min(1, x) with an int literal made XPO compare as int, so any central
angle >= 60 degrees (> ~6670 km) evaluated to pi*R and GeoInRangeKM
missed far away points.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…of Expression.Invoke

Expression.Invoke is not translatable by the XPO LINQ provider (and by EF
in several cases). The second lambda's parameter is now replaced with the
first one's by an ExpressionVisitor. Signatures unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…Key returns null for unknown keys

- XPMapsterStore.Query<T>() projected with TypeAdapterConfig.GlobalSettings
  instead of the injected Mapper.Config.
- XPAutoMapperStore.GetByKey returned a blank model for an unknown key,
  now null like the Mapster store.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nges

- EFDataStore.TransactionalExecAsync (EF Core and EF6) committed only when
  ChangeTracker.HasChanges(), so work that called SaveChanges itself was
  silently rolled back. It now saves pending changes and commits whenever the
  work completes and autoCommit is true.
- EFDatabase.Execute/Execute<T> with transactional:false opened a transaction
  and never committed it, rolling back the work's own SaveChanges. It now runs
  the work without a transaction (tx argument is null) and saves pending
  changes when commit is true. The EF6 HasChanges() guard is dropped too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… on failure

StoreAsync (Create/Update/Store):
- The transaction was committed inside the item loop, so any batch of 2+
  items threw on the second commit and left the first rows persisted. All
  items are now validated and added/updated first, then SaveChanges runs once
  and the transaction is committed once; on failure it is rolled back once.
- On failure the entities touched by the batch are detached, so a failed
  insert/update (or rejected validation) no longer poisons later saves on the
  same DbContext and rejected values cannot leak into reads or later saves.
- Failures report the mode of the failing item instead of always Create; for a
  DbUpdateException the offending item is identified from its entries.
- UpdateAsync with an empty key fails instead of inserting a new row.
- An empty batch returns Data = [] instead of [null]; success returns all
  items in order instead of only the last one.

DeleteAsync:
- One SaveChanges and one commit per batch; any exception (e.g. a
  DbUpdateException from an FK constraint) now becomes a failed DataResult
  instead of escaping, with the entity type name (was the literal "TDBModel")
  as property name; deleted-marked entities are detached on failure.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…null for missing keys

- EFMapsterDataStore.Query<T>() used ProjectToType<T>() (global
  TypeAdapterConfig), ignoring the mapping rules of the injected mapper; it now
  uses ProjectToType<T>(Mapper.Config).
- EFAutoMapperDataStore.GetByKey returned an empty new TModel for a missing
  key (Mapper.Map(null, dest) returns dest); it now returns null. The Mapster
  store gets the same explicit null check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- UpdateAsync with a default key no longer inserts; it fails with
  'Unable to locate ...' like an unknown key.
- Successful Update/Store results contain every stored item in input
  order (only created items were returned before).
- Updated items are mapped back from the saved db object too (OnSaving
  changes), not only created ones; the same key twice in one batch works.
- Failures report the mode of the failing item (was always Create).
- DeleteAsync returns a failed result for any exception, not only
  ValidationException, and names the db type instead of 'TDBModel'.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s the db model constructor

Update/Store/Delete called GetObjectByKey per item (N selects). The keys
are now loaded first with one Session.GetObjectsByKey, after which the
per-item XPOGetByKey calls hit the identity map (20 item update: 20 -> 1
select). Skipped when XPOGetByKey is overridden, so overrides keep their
own key semantics; results, order and errors are unchanged.
New db objects are created through a compiled (Session) constructor
delegate instead of Activator.CreateInstance (falls back to Activator).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…untracked GetByKey

- Update/Store/Delete batches load all entities with one parameterized
  key query per 64 keys (new protected virtual EFGetByKeysAsync) instead of a
  Find per item. Keys are padded to a power of two to bound query-cache
  shapes; an OR of equalities is used rather than Contains so no
  OPENJSON/json_each support is needed. Single-item calls still use Find. The
  per-key EFGetByKey path is kept for EF6, composite or differently typed keys,
  an overridden EFGetByKey, and any key the batch query did not match.
- Updates no longer force EntityState.Modified on the tracked entity (which
  rewrote every column); EF's change detection writes only changed columns,
  and an unchanged item issues no UPDATE. Untracked entities from a custom
  EFGetByKey are attached first; with AutoDetectChanges disabled the updated
  entities are detected explicitly.
- GetByKey (Mapster/AutoMapper) uses the new protected virtual
  EFGetByKeyNoTracking: an AsNoTracking key query on EF Core, so reads no
  longer grow the change tracker. Updates still load tracked entities.
- Tests: CommandRecorder interceptor counts SQL commands and SaveChanges
  calls. 10-item update: 20 commands/10 SaveChanges as single calls (a batch
  threw) -> 11 commands/1 SaveChanges; 10-item delete: 20 -> 11 commands.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s and more

- Bits.Clear/BitClear returned 'flag & ~flag' (always 0); Bits.Has(ref, flag, clear)
  ignored 'clear'.
- BaseCollection.OnClear/OnSet re-entered CollectionBase (StackOverflow);
  Collection.Dispose enumerated the list while items removed themselves.
- Conversion.GetConfigOption no longer throws on duplicate/blank keys (last wins).
- DateTimeExtensions.DateMin/DateMax preserve DateTimeKind.
- RegExLibrary.GetRegExGroupValue returns the default for missing/unmatched groups.
- MimeType: '(' is rejected, the primary type is validated by the single-arg ctor,
  and Matches(string) compares extensions case-insensitively.

Unskips the corresponding tests. ParseEnum's undefined-numeric case stays skipped
(marked BY DESIGN?).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…k of the same code

Bug fixes:
- StripHTMLTags: multi-line <head> removed, &nbsp; -> space, &lt;/&gt; decoded only
  after tag stripping (escaped text is kept), unknown-entity catch-all restricted to
  real entity syntax (no longer eats "& Jerry;").
- CopyMaxLength: input of exactly maxLen is returned unchanged.
- InsureHtmlParagraph: no stray </p> or duplicate <p> for well-formed/unclosed input.
- FormatFileSize: exact unit boundaries roll over (1000 -> 1 Kb); long.MinValue ok.
- MakeFileNameSEOSafe: names without extension are cleaned too.
- MimeTypes.GetMimeTypes always returned null (extension lookup never worked).
- PropertyExtensions: null exclusions copy everything, public properties only,
  indexers skipped, read-only destination properties skipped by Assign,
  IsCollectionProperty(string) is false, query values encoded once,
  AppendQueryParameters keeps the existing query.
- Attributes.GetAttributeData honours TAttribute; GetTypesWith is thread safe.
- RequiredIfAttribute: "The X field is required." default message, MemberNames set,
  '_' fallback no longer inverted.
- EnumExtensions.GetDisplay: [Display(Name)] without ResourceType and
  undefined/combined values no longer throw.
- Log: 24h timestamp; trace shows the logged object's type and message.

Performance (output unchanged, guarded by new regression tests):
- StripHTMLTags: precompiled static Regex fields, single-pass known-entity
  replacement, linear replacement of the O(n^2) newline/tab collapsing loop
  (100 KB input: ~1190 ms -> ~3.4 ms; small input: ~86 us -> ~3 us per call).
- MimeTypes: embedded definitions parsed once (shared Lazy<MimeTypes>);
  extension lookup uses ToLowerInvariant.
- PropertyExtensions: per-type name -> PropertyInfo cache, no Task.Run per lookup.
- Attributes.GetTypesWith: materialised results in a ConcurrentDictionary.
- EnumExtensions.GetDisplay: per-value cache of the display lookup (resource values
  are still read per call, so localisation keeps working).
- Log: per-thread buffer capped at 1M chars (oldest half dropped).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…pooling of in-memory stores

- GetDataLayer(string) ignored its argument and always returned the first
  data layer; it now returns the named one (empty name = default layer,
  unknown name throws).
- First use of a name was check-then-act, so concurrent callers could
  create several data layers (separate databases). Creation now goes
  through ConcurrentDictionary<string, Lazy<IDataLayer>>; a failed
  creation is not cached.
- In-memory connection strings were wrapped in a DataStorePool, where
  every extra pooled connection is a new empty database; they are no
  longer pooled.
- AddXpoUnitOfWork(connectionName) now binds the named data layer when it
  is one of the registered XpoDatabase's layers (other names fall back to
  the default layer, as before).
- The XpoDefault.DataLayer replacement in the ctor is left as is; its
  test is re-labelled BY DESIGN?.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ework

The switch exists for Medium Trust, which only applies to .NET Framework.
On .NET (Core) XPO's fast accessors stay on (reported 25-40% faster
object loading).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
AddLoginAsync filtered existing logins with lg.UserId.Equals(user.Id), which
XPO turns into an unknown custom function "Equals": as soon as any login
existed, every further AddLoginAsync threw InvalidOperationException. It now
uses a criteria string (and FindObjectAsync instead of counting a collection).

GetUsersForClaimAsync had a stray ')' in its criteria ("Claims[...])") and
threw a CriteriaParserException on every call.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…surface

XPIdentityMapperProfile mapped IdentityUserToken.UserId onto the get-only
XpoDxUserToken.UserId alias, so User stayed null and validation failed; and
XPUserStore.AddUserTokenAsync ignored the failed result, so
SetAuthenticationTokenAsync reported Success and stored nothing (no
authenticator keys, recovery codes or external tokens with AutoMapper).

The profile now ignores the get-only UserId of the XPO token, login and
claim and sets it through IXPUser*.UserId after mapping. AddUserTokenAsync
throws InvalidOperationException (the store failure as inner exception) when
the token could not be stored, instead of reporting success.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tayed valid)

SECURITY. UserStoreBase.SetTokenAsync changes an existing token only on the
object FindTokenAsync returned and relies on an EF change tracker to save it;
XPUserStore never wrote it back. So any update of an existing token was lost:
- RedeemTwoFactorRecoveryCodeAsync did not remove the redeemed code: a
  recovery code could be used again and again;
- GenerateNewTwoFactorRecoveryCodesAsync kept the old set (old codes kept
  working, the new ones did not);
- ResetAuthenticatorKeyAsync kept the old key, so a leaked authenticator key
  could not be rotated;
- SetAuthenticationTokenAsync on an existing token kept the old value.

XPUserStore now overrides SetTokenAsync and writes the new value to the
database: through the new optional IXPUserTokenValueStore<TKey>
(implemented by XPBaseUserTokenStore.SetTokenValueAsync), or, for a token
store of one's own, by replacing the token.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
IdentityUser.LockoutEnd (DateTimeOffset?) and XpoDxUser.LockoutEndDateUtc
(DateTime?) differ in name, and neither the Mapster config nor the AutoMapper
profile mapped them: a lockout was lost on save, so lockout after
MaxFailedAccessAttempts never took effect (no brute-force protection).
Both mappers now map it (stored as UTC, read back with offset 0).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
XPUserStore.UpdateAsync only set a new ConcurrencyStamp and saved, so a stale
copy silently overwrote a newer one. As the EF Core stores do, the update now
compares the stamp of the model with the stored one inside the update's
transaction and fails with IdentityErrorDescriber.ConcurrencyFailure() on a
mismatch; on success the model and the database get a new stamp. The same
for roles (XPRoleStore.UpdateAsync did not renew the stamp at all).

The base stores implement the new optional IXPConcurrencyStampStore<TKey,
TModel> (UpdateWithConcurrencyCheckAsync); a mismatch is reported as an
XPConcurrencyException (a FluentValidation.ValidationException) in the
IDataResult. Plain IQueryableUserStore/IQueryableRoleStore.UpdateAsync is
unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…l checks, recursion)

- AddToRoleAsync with an unknown role silently did nothing (UserManager
  reported Success); it now throws InvalidOperationException, as the EF Core
  UserStore. An unknown user throws too, instead of a NullReferenceException.
- XPRoleStore.Roles threw NotImplementedException although RoleManager
  reports SupportsQueryableRoles; it now queries the role store.
- XPUserStore.FindUserRoleAsync called itself (StackOverflow); it calls the
  data store, which no longer maps a role onto the user-role link (that threw
  with AutoMapper).
- FindByEmail/Id/Name, GetClaims, GetLogins (and FindRole, FindUserRole,
  RemoveUserToken) pass the CancellationToken on and check it.
- GetClaimsAsync / GetLoginsAsync (user) and GetClaimsAsync (role) throw
  ArgumentNullException for null instead of NullReferenceException.
- AddToRoleAsync and GetRoleClaimsAsync use async XPO calls.
- .NET Framework XPUserStore: SetSecurityStampAsync called itself
  (StackOverflow) and Users threw NotImplementedException.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
AutoMapper projected IdentityUser through its IdentityUser(string userName)
constructor (IdentityRole(string roleName) likewise), and XPO cannot
translate a Where on a member set in a constructor call, so
UserManager.Users.Where(u => u.UserName ...) threw NotSupportedException.
The profile now only uses parameterless constructors, so the projection is a
member initialisation XPO can filter on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…yStamp

Both skipped ConcurrencyStamp (part of IXPUser / IXPRole), so it was lost on
every save and load.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…lobalSettings

AddXpoMapsterIdentityStores called RegisterXPIdentityMapsterConfiguration,
which reset the identity rules in Mapster's process-wide GlobalSettings
(NewConfig) on every registration; when other code was mapping at the same
time, Mapster threw "TypeAdapter.Adapt was already called".

The stores now get a mapper of their own: a clone of the application mapper's
TypeAdapterConfig (GlobalSettings for the default Mapper, so global
customisations still apply) with the identity rules merged in, built once per
registration on first use. Query<T>() projects with that config
(ProjectToType<T>(Mapper.Config)) instead of GlobalSettings.

RegisterXPIdentityMapsterConfiguration still writes GlobalSettings for code
that calls it itself; the new ApplyXPIdentityMapsterConfiguration adds the
rules to a given TypeAdapterConfig. IMapper is still registered as before.

The test assembly no longer needs DisableTestParallelization.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ers in the query

AddToRoleAsync and RemoveFromRoleAsync went through usr.RolesList, and XPO
then loads every user of the role to keep the other side of the many-to-many
association in sync. They now add / delete the row of XPO's intermediate
(link) class directly; the schema is unchanged (falls back to the collections
if no unambiguous intermediate class is found). With 3000 users in a role:
rows read per AddToRole 3002 -> 1, per RemoveFromRole 3003 -> 2.

RemoveClaimsAsync loaded all claims of the user and filtered in C#; it now
puts the (type, value) pairs in the query, as ReplaceClaimAsync does, and no
longer loads the user.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The cache remembers which assemblies each result scanned; an AssemblyLoad
counter makes the next call scan only the new ones.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ser or role does not load every role member

XPDataStore.DeleteAsync takes a connection from the pool for the exclusive use
of an ExplicitUnitOfWork (ThreadSafeDataLayer has no database level
transactions, and sharing the pool would let other requests run inside the
transaction). A new BeforeDeleteAsync hook runs just before each delete; the
identity user and role stores use it to delete and flush the role links, so
XPO's association clean-up finds none. Non-pooled layers (in-memory, caching
node) keep the UnitOfWork path.

Role of 3,000 users on SQLite: user delete 60 -> 8 ms, role delete 566 -> 74 ms.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
donwibier and others added 2 commits November 4, 2026 20:31
…he fix release

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
XPOPrefetchByKeys discarded the objects it loaded. With XPO's default weak
identity map (the process only switches to a strong one once an XpoDatabase is
created) a GC between the prefetch and the per-item lookups dropped them, and
every item was loaded again: the batch was back to one query per item. Seen as
a flaky UpdateBatch_LoadsExistingObjectsInOneRoundTrip in CI (20 selects).

XPOPrefetchByKeys (new in this branch) now returns the loaded objects and the
batch keeps them referenced. New tests force a weak identity map and a GC per
item: 20 selects before, 1 after.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@donwibier
donwibier merged commit 4bf10e4 into master Nov 4, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant