Repository navigation
Test suite for the packages, and the bugs and performance problems it found - #23
Merged
Merged
Conversation
…ity stores New projects DX.Utils.Tests, DX.Data.Xpo.Tests, DX.Data.EF.Tests; Identity tests extended. Confirmed bugs are pinned as tests skipped with a BUG: reason. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Min(1, x) with an int literal made XPO compare as int, so any central angle >= 60 degrees (> ~6670 km) evaluated to pi*R and GeoInRangeKM missed far away points. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…of Expression.Invoke Expression.Invoke is not translatable by the XPO LINQ provider (and by EF in several cases). The second lambda's parameter is now replaced with the first one's by an ExpressionVisitor. Signatures unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…Key returns null for unknown keys - XPMapsterStore.Query<T>() projected with TypeAdapterConfig.GlobalSettings instead of the injected Mapper.Config. - XPAutoMapperStore.GetByKey returned a blank model for an unknown key, now null like the Mapster store. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nges - EFDataStore.TransactionalExecAsync (EF Core and EF6) committed only when ChangeTracker.HasChanges(), so work that called SaveChanges itself was silently rolled back. It now saves pending changes and commits whenever the work completes and autoCommit is true. - EFDatabase.Execute/Execute<T> with transactional:false opened a transaction and never committed it, rolling back the work's own SaveChanges. It now runs the work without a transaction (tx argument is null) and saves pending changes when commit is true. The EF6 HasChanges() guard is dropped too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… on failure StoreAsync (Create/Update/Store): - The transaction was committed inside the item loop, so any batch of 2+ items threw on the second commit and left the first rows persisted. All items are now validated and added/updated first, then SaveChanges runs once and the transaction is committed once; on failure it is rolled back once. - On failure the entities touched by the batch are detached, so a failed insert/update (or rejected validation) no longer poisons later saves on the same DbContext and rejected values cannot leak into reads or later saves. - Failures report the mode of the failing item instead of always Create; for a DbUpdateException the offending item is identified from its entries. - UpdateAsync with an empty key fails instead of inserting a new row. - An empty batch returns Data = [] instead of [null]; success returns all items in order instead of only the last one. DeleteAsync: - One SaveChanges and one commit per batch; any exception (e.g. a DbUpdateException from an FK constraint) now becomes a failed DataResult instead of escaping, with the entity type name (was the literal "TDBModel") as property name; deleted-marked entities are detached on failure. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…null for missing keys - EFMapsterDataStore.Query<T>() used ProjectToType<T>() (global TypeAdapterConfig), ignoring the mapping rules of the injected mapper; it now uses ProjectToType<T>(Mapper.Config). - EFAutoMapperDataStore.GetByKey returned an empty new TModel for a missing key (Mapper.Map(null, dest) returns dest); it now returns null. The Mapster store gets the same explicit null check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- UpdateAsync with a default key no longer inserts; it fails with 'Unable to locate ...' like an unknown key. - Successful Update/Store results contain every stored item in input order (only created items were returned before). - Updated items are mapped back from the saved db object too (OnSaving changes), not only created ones; the same key twice in one batch works. - Failures report the mode of the failing item (was always Create). - DeleteAsync returns a failed result for any exception, not only ValidationException, and names the db type instead of 'TDBModel'. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s the db model constructor Update/Store/Delete called GetObjectByKey per item (N selects). The keys are now loaded first with one Session.GetObjectsByKey, after which the per-item XPOGetByKey calls hit the identity map (20 item update: 20 -> 1 select). Skipped when XPOGetByKey is overridden, so overrides keep their own key semantics; results, order and errors are unchanged. New db objects are created through a compiled (Session) constructor delegate instead of Activator.CreateInstance (falls back to Activator). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…untracked GetByKey - Update/Store/Delete batches load all entities with one parameterized key query per 64 keys (new protected virtual EFGetByKeysAsync) instead of a Find per item. Keys are padded to a power of two to bound query-cache shapes; an OR of equalities is used rather than Contains so no OPENJSON/json_each support is needed. Single-item calls still use Find. The per-key EFGetByKey path is kept for EF6, composite or differently typed keys, an overridden EFGetByKey, and any key the batch query did not match. - Updates no longer force EntityState.Modified on the tracked entity (which rewrote every column); EF's change detection writes only changed columns, and an unchanged item issues no UPDATE. Untracked entities from a custom EFGetByKey are attached first; with AutoDetectChanges disabled the updated entities are detected explicitly. - GetByKey (Mapster/AutoMapper) uses the new protected virtual EFGetByKeyNoTracking: an AsNoTracking key query on EF Core, so reads no longer grow the change tracker. Updates still load tracked entities. - Tests: CommandRecorder interceptor counts SQL commands and SaveChanges calls. 10-item update: 20 commands/10 SaveChanges as single calls (a batch threw) -> 11 commands/1 SaveChanges; 10-item delete: 20 -> 11 commands. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s and more
- Bits.Clear/BitClear returned 'flag & ~flag' (always 0); Bits.Has(ref, flag, clear)
ignored 'clear'.
- BaseCollection.OnClear/OnSet re-entered CollectionBase (StackOverflow);
Collection.Dispose enumerated the list while items removed themselves.
- Conversion.GetConfigOption no longer throws on duplicate/blank keys (last wins).
- DateTimeExtensions.DateMin/DateMax preserve DateTimeKind.
- RegExLibrary.GetRegExGroupValue returns the default for missing/unmatched groups.
- MimeType: '(' is rejected, the primary type is validated by the single-arg ctor,
and Matches(string) compares extensions case-insensitively.
Unskips the corresponding tests. ParseEnum's undefined-numeric case stays skipped
(marked BY DESIGN?).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…k of the same code Bug fixes: - StripHTMLTags: multi-line <head> removed, -> space, </> decoded only after tag stripping (escaped text is kept), unknown-entity catch-all restricted to real entity syntax (no longer eats "& Jerry;"). - CopyMaxLength: input of exactly maxLen is returned unchanged. - InsureHtmlParagraph: no stray </p> or duplicate <p> for well-formed/unclosed input. - FormatFileSize: exact unit boundaries roll over (1000 -> 1 Kb); long.MinValue ok. - MakeFileNameSEOSafe: names without extension are cleaned too. - MimeTypes.GetMimeTypes always returned null (extension lookup never worked). - PropertyExtensions: null exclusions copy everything, public properties only, indexers skipped, read-only destination properties skipped by Assign, IsCollectionProperty(string) is false, query values encoded once, AppendQueryParameters keeps the existing query. - Attributes.GetAttributeData honours TAttribute; GetTypesWith is thread safe. - RequiredIfAttribute: "The X field is required." default message, MemberNames set, '_' fallback no longer inverted. - EnumExtensions.GetDisplay: [Display(Name)] without ResourceType and undefined/combined values no longer throw. - Log: 24h timestamp; trace shows the logged object's type and message. Performance (output unchanged, guarded by new regression tests): - StripHTMLTags: precompiled static Regex fields, single-pass known-entity replacement, linear replacement of the O(n^2) newline/tab collapsing loop (100 KB input: ~1190 ms -> ~3.4 ms; small input: ~86 us -> ~3 us per call). - MimeTypes: embedded definitions parsed once (shared Lazy<MimeTypes>); extension lookup uses ToLowerInvariant. - PropertyExtensions: per-type name -> PropertyInfo cache, no Task.Run per lookup. - Attributes.GetTypesWith: materialised results in a ConcurrentDictionary. - EnumExtensions.GetDisplay: per-value cache of the display lookup (resource values are still read per call, so localisation keeps working). - Log: per-thread buffer capped at 1M chars (oldest half dropped). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…pooling of in-memory stores - GetDataLayer(string) ignored its argument and always returned the first data layer; it now returns the named one (empty name = default layer, unknown name throws). - First use of a name was check-then-act, so concurrent callers could create several data layers (separate databases). Creation now goes through ConcurrentDictionary<string, Lazy<IDataLayer>>; a failed creation is not cached. - In-memory connection strings were wrapped in a DataStorePool, where every extra pooled connection is a new empty database; they are no longer pooled. - AddXpoUnitOfWork(connectionName) now binds the named data layer when it is one of the registered XpoDatabase's layers (other names fall back to the default layer, as before). - The XpoDefault.DataLayer replacement in the ctor is left as is; its test is re-labelled BY DESIGN?. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ework The switch exists for Medium Trust, which only applies to .NET Framework. On .NET (Core) XPO's fast accessors stay on (reported 25-40% faster object loading). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
AddLoginAsync filtered existing logins with lg.UserId.Equals(user.Id), which
XPO turns into an unknown custom function "Equals": as soon as any login
existed, every further AddLoginAsync threw InvalidOperationException. It now
uses a criteria string (and FindObjectAsync instead of counting a collection).
GetUsersForClaimAsync had a stray ')' in its criteria ("Claims[...])") and
threw a CriteriaParserException on every call.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…surface XPIdentityMapperProfile mapped IdentityUserToken.UserId onto the get-only XpoDxUserToken.UserId alias, so User stayed null and validation failed; and XPUserStore.AddUserTokenAsync ignored the failed result, so SetAuthenticationTokenAsync reported Success and stored nothing (no authenticator keys, recovery codes or external tokens with AutoMapper). The profile now ignores the get-only UserId of the XPO token, login and claim and sets it through IXPUser*.UserId after mapping. AddUserTokenAsync throws InvalidOperationException (the store failure as inner exception) when the token could not be stored, instead of reporting success. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tayed valid) SECURITY. UserStoreBase.SetTokenAsync changes an existing token only on the object FindTokenAsync returned and relies on an EF change tracker to save it; XPUserStore never wrote it back. So any update of an existing token was lost: - RedeemTwoFactorRecoveryCodeAsync did not remove the redeemed code: a recovery code could be used again and again; - GenerateNewTwoFactorRecoveryCodesAsync kept the old set (old codes kept working, the new ones did not); - ResetAuthenticatorKeyAsync kept the old key, so a leaked authenticator key could not be rotated; - SetAuthenticationTokenAsync on an existing token kept the old value. XPUserStore now overrides SetTokenAsync and writes the new value to the database: through the new optional IXPUserTokenValueStore<TKey> (implemented by XPBaseUserTokenStore.SetTokenValueAsync), or, for a token store of one's own, by replacing the token. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
IdentityUser.LockoutEnd (DateTimeOffset?) and XpoDxUser.LockoutEndDateUtc (DateTime?) differ in name, and neither the Mapster config nor the AutoMapper profile mapped them: a lockout was lost on save, so lockout after MaxFailedAccessAttempts never took effect (no brute-force protection). Both mappers now map it (stored as UTC, read back with offset 0). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
XPUserStore.UpdateAsync only set a new ConcurrencyStamp and saved, so a stale copy silently overwrote a newer one. As the EF Core stores do, the update now compares the stamp of the model with the stored one inside the update's transaction and fails with IdentityErrorDescriber.ConcurrencyFailure() on a mismatch; on success the model and the database get a new stamp. The same for roles (XPRoleStore.UpdateAsync did not renew the stamp at all). The base stores implement the new optional IXPConcurrencyStampStore<TKey, TModel> (UpdateWithConcurrencyCheckAsync); a mismatch is reported as an XPConcurrencyException (a FluentValidation.ValidationException) in the IDataResult. Plain IQueryableUserStore/IQueryableRoleStore.UpdateAsync is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…l checks, recursion) - AddToRoleAsync with an unknown role silently did nothing (UserManager reported Success); it now throws InvalidOperationException, as the EF Core UserStore. An unknown user throws too, instead of a NullReferenceException. - XPRoleStore.Roles threw NotImplementedException although RoleManager reports SupportsQueryableRoles; it now queries the role store. - XPUserStore.FindUserRoleAsync called itself (StackOverflow); it calls the data store, which no longer maps a role onto the user-role link (that threw with AutoMapper). - FindByEmail/Id/Name, GetClaims, GetLogins (and FindRole, FindUserRole, RemoveUserToken) pass the CancellationToken on and check it. - GetClaimsAsync / GetLoginsAsync (user) and GetClaimsAsync (role) throw ArgumentNullException for null instead of NullReferenceException. - AddToRoleAsync and GetRoleClaimsAsync use async XPO calls. - .NET Framework XPUserStore: SetSecurityStampAsync called itself (StackOverflow) and Users threw NotImplementedException. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
AutoMapper projected IdentityUser through its IdentityUser(string userName) constructor (IdentityRole(string roleName) likewise), and XPO cannot translate a Where on a member set in a constructor call, so UserManager.Users.Where(u => u.UserName ...) threw NotSupportedException. The profile now only uses parameterless constructors, so the projection is a member initialisation XPO can filter on. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…yStamp Both skipped ConcurrencyStamp (part of IXPUser / IXPRole), so it was lost on every save and load. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…lobalSettings AddXpoMapsterIdentityStores called RegisterXPIdentityMapsterConfiguration, which reset the identity rules in Mapster's process-wide GlobalSettings (NewConfig) on every registration; when other code was mapping at the same time, Mapster threw "TypeAdapter.Adapt was already called". The stores now get a mapper of their own: a clone of the application mapper's TypeAdapterConfig (GlobalSettings for the default Mapper, so global customisations still apply) with the identity rules merged in, built once per registration on first use. Query<T>() projects with that config (ProjectToType<T>(Mapper.Config)) instead of GlobalSettings. RegisterXPIdentityMapsterConfiguration still writes GlobalSettings for code that calls it itself; the new ApplyXPIdentityMapsterConfiguration adds the rules to a given TypeAdapterConfig. IMapper is still registered as before. The test assembly no longer needs DisableTestParallelization. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ers in the query AddToRoleAsync and RemoveFromRoleAsync went through usr.RolesList, and XPO then loads every user of the role to keep the other side of the many-to-many association in sync. They now add / delete the row of XPO's intermediate (link) class directly; the schema is unchanged (falls back to the collections if no unambiguous intermediate class is found). With 3000 users in a role: rows read per AddToRole 3002 -> 1, per RemoveFromRole 3003 -> 2. RemoveClaimsAsync loaded all claims of the user and filtered in C#; it now puts the (type, value) pairs in the query, as ReplaceClaimAsync does, and no longer loads the user. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The cache remembers which assemblies each result scanned; an AssemblyLoad counter makes the next call scan only the new ones. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ser or role does not load every role member XPDataStore.DeleteAsync takes a connection from the pool for the exclusive use of an ExplicitUnitOfWork (ThreadSafeDataLayer has no database level transactions, and sharing the pool would let other requests run inside the transaction). A new BeforeDeleteAsync hook runs just before each delete; the identity user and role stores use it to delete and flush the role links, so XPO's association clean-up finds none. Non-pooled layers (in-memory, caching node) keep the UnitOfWork path. Role of 3,000 users on SQLite: user delete 60 -> 8 ms, role delete 566 -> 74 ms. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…he fix release Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
XPOPrefetchByKeys discarded the objects it loaded. With XPO's default weak identity map (the process only switches to a strong one once an XpoDatabase is created) a GC between the prefetch and the per-item lookups dropped them, and every item was loaded again: the batch was back to one query per item. Seen as a flaky UpdateBatch_LoadsExistingObjectsInOneRoundTrip in CI (20 selects). XPOPrefetchByKeys (new in this branch) now returns the loaded objects and the batch keeps them referenced. New tests force a weak identity map and a GC per item: 20 selects before, 1 after. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A new unit test suite for the packages found close to 80 bugs, plus a number of performance problems. This PR adds the tests and fixes them.
Security (DX.Data.Xpo.Identity): updates of existing user tokens were never saved. A redeemed two-factor recovery code stayed valid, and resetting the authenticator key or regenerating recovery codes had no effect. With the AutoMapper package, tokens were not stored at all. Lockouts were not persisted either, so lockout after failed attempts never took effect. Each fix has its own commit.
Tests
DX.Utils.Tests,DX.Data.Xpo.TestsandDX.Data.EF.Tests(SQLite in-memory).DX.Data.Xpo.Identity.Testsgoes from 17 to 250 tests.DXWeb.slnand the Tests workflow.These ran locally on net8.0 and net10.0. This machine has no .NET 9 runtime, so net9.0 runs only in CI.
The two skipped tests pin behaviour left alone on purpose:
XpoDatabasewithDatabaseAndSchemasets the process-wideXpoDefault.DataLayer.ParseEnum("99")returns an undefined value. Rejecting it would also reject valid[Flags]combinations.Fixes
AddLoginAsync(threw once any login existed) andGetUsersForClaimAsync(always threw) work.AddToRoleAsync.RoleManager.Rolesworks; it threw NotImplementedException.ArgumentNullException.TypeAdapterConfig.GlobalSettings.UpdateAsyncwith an empty key no longer inserts.Execute(..., transactional: false)keeps the work's saved changes.PredicateBuilderworks with XPO.UpdateAsyncwith a default key no longer inserts.XpoDatabaseno longer races on the first use of a name.GetDataLayer(name)andAddXpoUnitOfWork(name)honour the name.Collection.Clear()and theIListindexer setter no longer overflow the stack.Bits.Clearno longer always returns 0.GetDisplay,PropertyExtensionsandStripHTMLTags.GetTypesWithis thread safe and picks up assemblies loaded later.Performance
StripHTMLTagsis linear: 100 KB went from 1.2 s to 3 ms.PropertyExtensionsno longer starts aTask.Runper lookup.SaveChangesper batch, updates write only changed columns, andGetByKeyreads without tracking.ExplicitUnitOfWork), with a newBeforeDeleteAsynchook.Behaviour changes
These are documented in the root README ("Fix release — behaviour changes"), in each affected package's README and in its
PackageReleaseNotes. Versions bumped:Known issue, not caused by this PR
The sample app
DX.Test.Blazor8(net9.0) fails to compile its Razor pages in a fresh checkout ofmasteron the 10.0.401 SDK, the same as on this branch. CI on master was green 9 days ago, so a floating package version has probably moved since then. If the Tests job fails on that project, the cause is the same and needs a separate look.🤖 Generated with Claude Code