Repository navigation
Remove HermitCrab's analysis-cascade memo - #522
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #522 +/- ##
==========================================
- Coverage 74.57% 74.57% -0.01%
==========================================
Files 456 453 -3
Lines 38472 38314 -158
Branches 5297 5280 -17
==========================================
- Hits 28692 28573 -119
+ Misses 8617 8576 -41
- Partials 1163 1165 +2 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
4490f59 to
a84dc6f
Compare
a84dc6f to
8313ba0
Compare
This comment has been minimized.
This comment has been minimized.
8313ba0 to
f3258c0
Compare
3a63132 to
df2cf21
Compare
f3258c0 to
e723ade
Compare
e723ade to
1650c10
Compare
This comment has been minimized.
This comment has been minimized.
…state loss edge-cases/final-template-merge-key: a final template in a non-blocking lower stratum, and in the last stratum a non-final template rule and a plain rule that both add "s". "sagdss" has two analyses differing only in FinalTemplateState; a merge key that omits it can keep the NonTemplate one and lose the parse. Passes on #480 (master engine) and on #522 with the fix, fails on #522 without commit 4f63793. Ledgers, manifest and pinned counts updated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1650c10 to
88fc8a1
Compare
The merge key dropped FinalTemplateState, which AnalysisStateKey carried. Two analyses that differ only in it (unapplying a template rule last leaves None; unapplying a non-template rule last leaves NonTemplate) were merged, and the canonical's NonTemplate then blocked a final template in a lower, non-blocking stratum that the merged-away None alternative would have allowed, losing the parse. Add a parse test that fails without the key change, and a key test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The sequential CombinationRuleCascade (maxDegreeOfParallelism 1) is still a distinct code path, and only the compounding grammar compared it with the parallel default. Add comparisons for affix templates, a Linear stratum, and Simultaneous epenthesis (buibui parses to one analysis), without the memo hit-counter assertions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two words with the same multiset of unapplied rules reached in different orders must produce equal keys and equal hashes; that is what the XOR multiset combine exists for. Replacing it with a rolling combine now fails this test, and previously passed every AnalysisMergeKey test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The constructor mutates its argument by freezing SyntacticFeatureStruct, which Word.FreezeImpl leaves unfrozen; a plain constructor does not signal that, so say so at the constructor. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… parses The compounding test compared two empty results for pʰutdidat, and the Linear test compared an empty digusagd; both now assert the words that really parse and say which word does not. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The hermitcrab checklist lists exactly the AnalysisMergeKey fields, MaxDegreeOfParallelism is documented again as a results-neutral performance knob, and the merge-order test comment is stated in terms of AnalysisMergeKey. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…acuously The parallelism-cap test compared capped and unbounded results only, so two empty results for pʰutdat would pass. It now asserts the unbounded parse of pʰutdat is non-empty, matching the compounding parity test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The doc said every value produces equal parse results. With MaxAlternatives set the parallel cascade counts alternatives before removing duplicates, so it can throw where a value of 1 parses. The doc is a rule again, as on master, and names that exception. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…geKey AnalysisStateKeyTests covered both behaviors; the port dropped them, so removing either from AnalysisMergeKey passed every test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On a Linear stratum the di and gu prefixes still both apply, but only in rule order, so gudisagd parses and digusagd does not. The test and the fixture comment said neither nesting parsed; the test now asserts gudisagd is non-empty and compares it across both cascades. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ParallelCombinationRuleCascade and ParallelPermutationRuleCascade added every rule output to the count before removing duplicates, so with MaxAlternatives set they could throw where the sequential cascades, which count a set, do not. Collect output in a set and count only new results. The MaxDegreeOfParallelism doc no longer needs the exception. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#519 landed on master with tests that set Word.AnalysisScope, which this branch removes along with the memo. The transform does not read it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The parallel cascades now build a ConcurrentDictionary from Comparer, whose netstandard2.0 constructor rejects null; before, Distinct(null) fell back to the default. ParallelCombinationRuleCascade also called Comparer.Equals directly, so a null comparer already crashed there. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
c5376ac to
a00aaea
Compare
|
Review: approve
API / framework / parity: no public signature change. Findings: F1 addressed (4f63793); F2 addressed (8987126); F3 withdrawn; F4 addressed (e2579c6); F5 accepted, unverified; F6 addressed (1650c10); F7 addressed (c0fb7d7); F8 addressed (c0fb7d7); F9 addressed (72851b3); F10 addressed (72851b3); F11 addressed (72851b3); F12 addressed (1106397, c5376ac); F13 addressed (d900f0f); F14 addressed (27a22a9); F15 addressed (a00aaea). Reviewed at a00aaea |
…state loss edge-cases/final-template-merge-key: a final template in a non-blocking lower stratum, and in the last stratum a non-final template rule and a plain rule that both add "s". "sagdss" has two analyses differing only in FinalTemplateState; a merge key that omits it can keep the NonTemplate one and lose the parse. Passes on #480 (master engine) and on #522 with the fix, fails on #522 without commit 4f63793. Ledgers, manifest and pinned counts updated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…state loss edge-cases/final-template-merge-key: a final template in a non-blocking lower stratum, and in the last stratum a non-final template rule and a plain rule that both add "s". "sagdss" has two analyses differing only in FinalTemplateState; a merge key that omits it can keep the NonTemplate one and lose the parse. Passes on #480 (master engine) and on #522 with the fix, fails on #522 without commit 4f63793. Ledgers, manifest and pinned counts updated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The analysis memo is going away (#522), so --no-memoization, the useMemoization parameters and their tests go with it, along with the design and plan that introduced them. Self-check now takes both the result set and the trace from a single traced parse instead of an untraced parse plus a second traced one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PR #491 added FinalTemplateState to the key, and #522 then removed AnalysisStateKey with the analysis memo. Name master's AnalysisMergeKey, which carries the listed fields plus FinalTemplateState. Also say that the two engine facts in section 4 concern the removed memo. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#522 removed AnalysisStateKey along with the analysis-cascade memo and kept the same identity as AnalysisStratumRule.AnalysisMergeKey, so the shared-walk equivalence test now compares outputs under that key. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The analysis memo that MemoEntry belonged to was removed in #522 and will not return, so its arrival-count constraint no longer applies. The remaining order-invariance constraint becomes the section's only item. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Section 4 kept two engine facts for any revival of the analysis memo that #522 removed. The memo is not coming back, so both facts now describe code that does not exist. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Quick summary
Removes HermitCrab's opt-in analysis-cascade memo; the sequential rule cascade
and
MergeEquivalentAnalysesstill work exactly as before. Measured as apackage (#490), it cost 28-32% and ~29 GB on Sena for no net speedup on FLEx's
parallel path. Pruning (#474 landed; #491/#494/#519 open) is untouched and is
where the real parse-time wins come from.
Where to look
AnalysisMergeKey(AnalysisStratumRule.csL229-318) still distinguishes non-head count andper-rule unapplication counts -- pinned by
AnalysisMergeKeyTests.csandMergeEquivalentAnalysesTests.cs.FinalTemplateStateis part of that key. A non-blocking stratum inherits it from the stratum above,so merging two analyses that differ only there can drop a parse; pinned by
AffixTemplateTests.FinalTemplateInLowerStratum_ParsesWhenUpperAnalysesDifferOnlyInFinalTemplateState.AnalysisScope.cs,AnalysisStateKey.cs,MemoizedCombinationRuleCascade.csdeleted) --
maxDegreeOfParallelismand the sequential cascade are unchanged public API.docs/review/hermitcrab.mdand others).ParallelCombinationRuleCascadeandParallelPermutationRuleCascade(src/SIL.Machine/Rules) now countdistinct results toward
MaxAlternatives, as the sequential cascades do, so the parallelism setting no longerdecides whether a parse throws
MaxAlternativesExceededException; pinned byRuleCascadeTests.cs.Validation
dotnet test tests/SIL.Machine.Morphology.HermitCrab.Tests-- 108 passed, 0 failed.dotnet test tests/SIL.Machine.Tests-- 889 passed, 3 skipped, 0 failed.dotnet csharpier check .-- 715 files checked, clean.pwsh -NoProfile -File scripts/comment-hygiene.ps1 -BaseRef origin/master-- clean.git diff origin/master..HEAD --stat-- 20 files, +474/-1478. Beyond the memo removal, productioncode adds
FinalTemplateStatetoAnalysisMergeKeyand fixes the parallel cascades'MaxAlternativescount; tests add the lost-parse test, merge-key tests, cascade-count tests and single-threaded-vs-parallel
parity tests.
./local_check.sh --agent-strict: comment hygiene clean; the Release build stops at the native SentencePiecedownload on this machine, so the test commands above were run directly.
Reading this a year from now
The memo only ran when
maxDegreeOfParallelism == 1and tracing was off. FLEx usesthe default parallelism, so production never executed it. #490's ledger measured
the optimization round as a package: no net speedup, and cross-word memo sharing
was 28-32% slower on Sena and cost ~29 GB of memory. The gains keep coming from
search-space pruning that doesn't need a replay cache: #474 (landed), and #491,
#494, and #519 (open at the time of writing). Those shrink the work instead of
caching it, and they apply on the parallel path FLEx actually uses. The memo was
~1,400 lines of subtle code (
ReplayOntografting, arrival-count bookkeeping)whose correctness depended on an unenforced key invariant (#490, "narrowing that
key breaks correctness with no test failure").
What stays, and why
Morpher(..., maxDegreeOfParallelism)parameter and the sequentialcascade. They are API and still useful without the memo.
MergeEquivalentAnalysesfix. It keyed onAnalysisStateKey; that keynow lives as an internal
AnalysisMergeKeyinAnalysisStratumRule, covering thesame dimensions (shape, stratum, syntactic and realizational feature structures,
non-head count, per-rule unapplication counts, and
FinalTemplateState).Prior measurements (not reproduced by this review round)
These are earlier author measurements from the memo-removal work, not commands run
as part of this pass's validation, and no reproducible artifact for them is checked
into the tree:
each, all with at least one parse: analysis multisets (gloss, allomorph identity,
root index, feature structures) were reported identical between master memo-off
and this branch in both default and sequential modes, 25/25 match.
than this branch's sequential mode (e.g. 3,286 ms memo-on vs 3,122 ms without it;
412 vs 350 ms; the rest within noise).
🤖 Generated with Claude Code
This change is