HermitCrab optimization round record: what was tried, and a faster package that is not mergeable

docs/hermitcrab-perf-2026-09-counts.md

@@ -0,0 +1,94 @@
+# HermitCrab optimization round, September 2026: final disposition
+
+This is the final record for the count-driven round that followed
+`hermitcrab-optimization-ledger.md`. Temporary counters were used to choose targets and were then removed;
+the final branch (`perf/hc-optimization-archive`, not master) carries only product changes, focused tests, the
+parity/timing harness, and these records.
+
+## Measurement scope
+
+The exploration used sequential parsing, one warm-up followed by repeated samples, and canonical analysis
+signature sets as the parity gate. Deterministic work counts came from 45 Sena words (five known-heavy probes
+plus the first 40 list entries) and 464 words from 33 conformance fixtures. Timing was never run concurrently.
+The final branch-wide sample compares production commit `d7347226` (harness-only baseline tip `4f677735`)
+with candidate code tip `7370e2dd`.
+
+The probe counters are intentionally absent from the final diff. They put volatile gates in hot code across
+two assemblies and would create permanent maintenance cost for a completed experiment. The reconstruction
+design remains in `hermitcrab-probe-design.md`.
+
+## Retained optimizations
+
+Each retained mechanism has one record with its invariant, implementation, evidence, parity coverage,
+portability, and limits:
+
+| Optimization | Principal evidence | Record |
+| --- | --- | --- |
+| Deferred template clone | 1.054x on the 45-word Sena battery; `WordAnalysisSignature` parity (morph counts only on Sena) | `optimizations/deferred-template-clone.md` |
+| Feature-structure traversal-state reuse | equality/freezing allocation left the hot profile; not isolated | `optimizations/feature-struct-visited-state-reuse.md` |
+| Frozen flat-tree equality | removes visited sets from the proved tree case; not isolated | `optimizations/feature-struct-frozen-tree-equality.md` |
+| Frozen flat-tree clone | removes the copies map from the proved tree case; not isolated | `optimizations/feature-struct-frozen-tree-clone.md` |
+| Single-pass `Shape.CopyTo` mapping | removes a second enumeration/dictionary build; not isolated | `optimizations/shape-copy-single-pass-map.md` |
+| Copy-on-write shape sharing | clone 36% → 9% of profile; peak working set 11 GB → 6 GB | `optimizations/shape-sharing-on-clone.md` |
+| Flat FST register arrays | `CreateInstanceMDArray` 11.5% → 0.2% of profile | `optimizations/flat-register-arrays.md` |
+| Analysis edge-segment prefilter | 87% fewer FST traversals and 54% fewer arc checks on 45 Sena words | `optimizations/edge-segment-prefilter.md` |
+| Copy-on-write syntactic feature structures | removes repeated clone/freeze/hash work measured at ~15% of a heavy Sena profile | `optimizations/syntactic-feature-structure-sharing.md` |
+
+Only the deferred clone was timed alone. Profile movement and deterministic work removal justify the other
+retained mechanisms; the five-language result below measures the branch as a package and must not be divided
+among them.
+
+## Rejected or removed experiments
+
+| Experiment | Deciding evidence | Final state |
+| --- | --- | --- |
+| Two-pass nondeterministic traversal | 2.3x more traversal instances, only 1.6% fewer arc checks, 27% more clones | removed; `rejected-optimizations/two-pass-nondeterministic-traversal.md` |
+| Final-template interleaving prune | zero prunes on Sena, Amharic, and Mbugwe because partial morphemes disable it | removed; `rejected-optimizations/final-template-prune.md` |
+| `ExpandAlternatives` memo/hoist | word clones unchanged at 4,747,013; no retained difference from production ordering | removed; `rejected-optimizations/alternatives-memo-and-hoist.md` |
+| Compounding split prefilter | only ~2% of clones and ~9% of FST traversals were in scope | not built; `rejected-optimizations/compounding-split-prefilter.md` |
+
+All associated production switches, alternate traversal code, and research counters were deleted. The two
+retained internal toggles exist solely for correctness/performance A/B of the edge prefilter and syntactic
+feature-structure sharing.
+
+## Five-language branch-wide sample
+
+Each row used three fixed word-list indices and two reverse-ordered process rounds (baseline/candidate, then
+candidate/baseline). Every process parsed sequentially, discarded one warm-up, and recorded three repetitions,
+for six measured samples per arm. Times are the sum of each word's minimum sample across both rounds. Baseline
+and candidate used identical grammar names and word keys; `scripts/hc-bench-compare.ps1` compared error
+outcomes and rejected missing keys or signature differences. All ten arm comparisons passed the legacy
+`WordAnalysisSignature` check, which on these FLEx-exported grammars compares morph counts only (`Morpheme.Id`
+is empty; see `hermitcrab-perf-2026-09-disposition.md`), and the saved outputs contained zero parse or load
+errors.
+
+| Language | Word-list indices | Baseline ms | Candidate ms | Ratio | Interpretation |
+| --- | --- | ---: | ---: | ---: | --- |
+| Sena | 49, 52, 74 | 1,000.7 | 448.3 | 2.23x | directional: 0/3 probes clear both the 50 ms and spread rules |
+| Amharic | 8, 9, 267 | 6,410.0 | 3,646.3 | **1.76x** | stable: 3/3 probes clear both rules |
+| Indonesian | 0, 9, 44 | 99.0 | 56.0 | 1.77x | exploratory: 0/3 probes clear both rules |
+| Mbugwe | 0, 65, 1272 | 8,979.4 | 4,901.6 | **1.83x** | stable: 3/3 probes clear both rules |
+| Aweti | 2, 3, 27 | 134.2 | 54.4 | 2.47x | exploratory: 0/3 probes clear both rules |
+
+The ratios are deliberately unpooled and are not extrapolated to whole corpora. An initially selected heavy
+Sena trio did not complete the baseline within five minutes, and an Aweti sample containing index 0 crossed
+two minutes; both whole testhosts were terminated and replaced symmetrically before any comparison. This is
+sampling evidence, not a claim about those excluded words.
+
+## Correctness gate
+
+The minimized code tip was merged into the conformance integration branch and run through the complete
+HermitCrab conformance-inclusive project:
+
+- NUnit discovered and ran 604 non-explicit tests: 603 passed, zero failed.
+- One symlink-security test was skipped because the Windows host lacks symlink privilege.
+- Fixture manifests, DTD/schema checks, source/graph audits, 452 fixture rows, and 348 coverage items passed.
+- The ordinary HermitCrab suite also passed with syntactic feature-structure sharing disabled.
+
+## What remains
+
+The edge prefilter leaves roughly 44% of remaining traversals successful, so another edge check has little
+headroom. Compounding construction is too small on the measured heavy word. Alternatives and packed-readout
+schemes remain bounded by the ordered pending trail. The next credible large change is a persistent/shared
+trail representation that can avoid memo-replay and alternative-materialization clones without merging
+linguistically distinct derivations; it is a data-structure redesign, not a safe incremental tweak.

docs/hermitcrab-perf-2026-09-disposition.md

@@ -0,0 +1,143 @@
+# HermitCrab optimization round, September 2026: final disposition of the branch
+
+**Date:** 2026-09-03. **Branch:** `perf/hc-optimization-pr` @ `179708d2`, renamed `perf/hc-optimization-archive`.
+Origin now holds that branch as one squashed commit, `3a0a3316`; that is the authoritative tip. **Decision:** do not
+merge the branch. Isolate one mechanism onto master as its own small PR; archive the rest with its records. Re-focus
+on sillsdev/machine PR #491 and on grammar hygiene, which are where the large returns are.
+
+## Why the package is not merged
+
+- 896 source lines across 28 files; about a third inside `SIL.Machine.FeatureModel.FeatureStruct`, the rest in FST
+  traversal internals, `Word`, `Shape`, and the analysis rules. Not reviewable as one change.
+- Only one mechanism (deferred template clone, 1.054x) was ever timed alone. The package figure (1.76x Amharic,
+  1.83x Mbugwe) cannot be attributed to its parts (`hermitcrab-perf-2026-09-counts.md`).
+- The parity gate that blessed the package compared morph counts, not analyses: `MorpherTests.WordAnalysisSignature`
+  joins `Morpheme.Id`, which comes from `<MorphemeId>` and is empty for every FLEx export used. On Sena, Mbugwe
+  and Amharic the signature therefore degenerates to a morph count per analysis, so every earlier "exact parity"
+  claim on those grammars verified counts, not analyses. Anything re-measured on them needs a
+  gloss + allomorph-index + syntactic-feature-structure signature instead.
+- Meanwhile PR #491's mechanism, measured with the guard lifted (`AlwaysEnforceFinalTemplates = true`), is
+  8–33x on Mbugwe and Sena with identical analysis sets, and 298x on the single heaviest Sena word; Amharic
+  barely moves because its per-word cost is not in the template interleaving. The branch is worth keeping only
+  where #491 cannot fire (Amharic-shaped grammars), and only in its most reviewable piece.
+
+## Disposition per mechanism
+
+| Mechanism | Lines / files | Evidence on the branch | Disposition |
+| --- | --- | --- | --- |
+| Analysis edge-segment prefilter | ~120 in 2 rule files + 9 in `Morpher` + tests | 87% fewer FST traversals on 45 Sena words (deterministic counts); one invariant; internal toggle | **Isolated onto master** as `perf/hc-edge-prefilter` (`be7f241d`), see below — **that standalone branch and its own archive tag were later deleted, but the mechanism is preserved in the whole-round archive branch `perf/hc-optimization-archive`** |
+| Copy-on-write `Shape` sharing | `Word`, `Shape`, every clone-then-mutate site | clone 36% → 9% of profile; peak working set 11 GB → 6 GB | Archived. Aliasing rules are subtle; re-open only if memory becomes the constraint. Commit `dfd1be94`, `e18ca86d` |
+| Copy-on-write syntactic `FeatureStruct` sharing | `Word`, `Morpher`, rules | ~15% of a heavy Sena profile; never isolated | Archived. `4a6a47ed`, `7f9fdef3`, `da35ad4e`, `f08fefa7` |
+| `FeatureStruct` equality/clone/freeze fast paths, visited-set pooling | 312 lines in core FeatureModel | "not isolated" | Archived. `325f8419`, `424a3c23`, `b23a7c99`, `00ece2ba`, `2f4dabe0` |
+| Flat FST register arrays | FST internals | `CreateInstanceMDArray` 11.5% → 0.2% of profile | Archived. `b1692a7a` |
+| Deferred template clone | `AnalysisAffixTemplateRule` | 1.054x alone | Archived. `57703d6b`, `24d75c82` |
+| `ExpandAlternatives` memo/hoist | `Morpher` | no retained difference | Already removed on the branch. `d8886b0c`, `f4195be1` |
+| Two-pass nondeterministic traversal | FST | negative | Already removed. `c3366fcc` |
+| Final-template prune (own version) | analysis rules | zero prunes under the guard | Already removed; superseded by #491 work. `7e6ec59d` |
+| PerfBench / compare script / ParseCounters | tests, scripts | harness | Archived. Its signature is degenerate on FLEx exports; any revival must switch to a gloss/allomorph/feature-structure signature first |
+| Research records (`docs/optimizations/*`, `docs/rejected-optimizations/*`, ledger, counts, probe design, this file, the #491 re-review) | docs | — | Kept on the archive branch as the record of what was tried and why |
+
+## The isolated prefilter
+
+**Status update:** `perf/hc-edge-prefilter` was deleted after this round closed, and its own archive tag was
+later deleted too, but the mechanism is not lost: it remains on the whole-round archive
+branch `perf/hc-optimization-archive`, which still carries
+`Morpher.EdgePrefilterEnabled`, the edge constraints in `AnalysisAffixProcessRule`, and
+`AnalysisEdgePrefilterTests`. The description below is kept as a standalone record of the isolated, reviewable
+form, in case that smaller diff is preferred over pulling the mechanism out of the full archive again.
+
+Branch `perf/hc-edge-prefilter`, worktree `.worktrees/prefilter`, one commit `be7f241d` on `origin/master`
+(`d7347226`). Content: `AnalysisAffixProcessAllomorphRuleSpec` captures the left/right top-level `Constraint`
+feature structures at compile time; `AnalysisAffixProcessRule.Apply` rejects an allomorph before its matcher runs
+when the shape's first/last Segment node provably cannot unify with them, using the matcher's own `IsUnifiable`
+call and `UseDefaults`; optional edge nodes and non-Constraint edges always pass; tracing emits the same
+`MorphologicalRuleNotUnapplied` the matcher would have. `Morpher.EdgePrefilterEnabled` (internal, default true) is
+the same-binary A/B switch. The research counters were not carried over. `AnalysisEdgePrefilterTests` (final branch
+version, no counter dependency) came with it. HermitCrab suite: 104/104.
+
+### Standalone measurement (same binary, toggle off = master behavior, toggle on; sequential, memo on)
+
+Signature: gloss + allomorph index + syntactic feature structure, compared as multisets. Counters: throwaway
+increments at the two nondeterministic `Traverse` entries and at `CheckInputMatch` (not committed).
+
+| Grammar | Words | Wall ms off → on | FST traversals off → on | Arc checks off → on | Parity |
+| --- | ---: | ---: | ---: | ---: | --- |
+| Sena | 42 (49,52,74,0–39; one invalid line skipped) | 71,296 → 69,900 (−2%) | 8,760,382 → 1,270,122 (−85%) | 12,618,566 → 6,004,919 (−52%) | identical 42/42 |
+| Mbugwe | 22 (0,65,1272,1–19) | 110,609 → 96,293 (−13%) | 28,054,872 → 11,419,808 (−59%) | 36,811,510 → 25,960,854 (−29%) | identical 22/22 |
+| Amharic | 10 (8,9,267,5,10,12,15,20,25,30) | 111,617 → 103,403 (−7%) | 145,050 → 129,802 (−10%) | 16,705,704 → 15,777,072 (−6%) | identical 10/10 |
+
+**Reading.** The count reduction reproduces (Sena −85% traversals, matching the branch's −87%), and parity holds
+under a real signature. But the traversals removed are the cheap ones: they would have died at their first arc
+check, so wall time moves only 2–13%. The branch's package figure (1.76x Amharic, 1.83x Mbugwe) therefore came
+mostly from the allocation, clone, and GC work, not from this mechanism. Standalone, the prefilter is correct,
+small, and reviewable, but it is **not a large return**. Recommendation at the time: keep `perf/hc-edge-prefilter`
+ready as an optional small PR (2–13% on Bantu grammars, where it stacks with #491's prune on the surviving
+branches), and do not spend review budget on it ahead of #491. The standalone single-commit branch is deleted
+(see status note above); a PR now starts from the mechanism preserved in the whole-round archive
+(branch `perf/hc-optimization-archive`) rather than resuming
+that branch.
+
+Raw logs: session scratchpad `pf-sena.log`, `pf-mbugwe.log`, `pf-amharic.log`.
+
+## Branch and worktree inventory for the round (2026-08-20 to 2026-09-03)
+
+Every branch below is local unless marked. Worktrees for archived branches were removed on 2026-09-03; the
+branches remain.
+
+| Branch | Tip | What it is | State |
+| --- | --- | --- | --- |
+| `perf/hc-optimization-archive` (was `perf/hc-optimization-pr`) | `3a0a3316` (squashed) | the whole round, code + records | archived, unmerged; worktree `.worktrees/ledger` |
+| `perf/hc-defer-template-clone` | `4a6a47ed` | working branch of the round | fully contained in the archive; worktree removed |
+| `perf/hc-engine-alloc` | `f22f88a8` | allocation/two-pass side branch | fully contained in the archive; worktree removed |
+| `perf/hc-conformance-check` | `0799a977` | archive merged into the conformance integration branch for the 603-test gate | verification only; worktree `.worktrees/hc-conf` can go |
+| `perf/hc-edge-prefilter` | `be7f241d` | the one isolated mechanism on `origin/master` | optional small PR at the time; **branch and its own archive tag have both since been deleted, but the mechanism is preserved on branch `perf/hc-optimization-archive`** |
+| `pr-491` | `96c97f76` | jtmaxwell3's PR head at review time | merged to master as `b240b7b5` on 2026-09-28; worktree `.worktrees/pr491` holds uncommitted throwaway counters + `FinalTemplateUpperBound` harness |
+| `docs/hc-optimization-ledger` (remote, PR #490) | the PR head | the ledger, rows 1-30 | open docs PR |
+| `feature/forest-memo`, `feature/synthesis-fold-probes`, `feature/synthesis-fold-sharing`, `feature/per-node-cost` | — | evidence branches of the ledger round | see `hermitcrab-optimization-ledger.md` |
+| `investigate-memoization-rule-order` | `4f5aeeb4` | despite the name, conformance-suite commits | belongs to the conformance line, not this round |
+
+Records in this directory: `hermitcrab-optimization-ledger.md` (rows 1-30), `hermitcrab-perf-2026-09-counts.md`
+(count round), `hermitcrab-probe-design.md`, `optimizations/*` and `rejected-optimizations/*` (one record per
+mechanism), and this file. The round's execution checklist has been folded into this record; it is not kept as
+a separate plan document.
+
+Four further records of the round were deliberately left out of this set, because they concern another author's
+open PR and a sibling project rather than this round's own mechanisms. They are described here so that nothing
+depends on reading them, and they remain on the `perf/hc-optimization-archive` branch:
+
+- `pr491-final-template-prune-upper-bound.md` — the measurement record behind the 8–33x figures above: PR #491's
+  prune with its partiality guard lifted, per-grammar wall time, affix-rule applications and template entries,
+  plus the analysis-set parity check and the signature caveat quoted earlier. It also proves the guard
+  over-broad, since partial *lexical entries* can never rescue a pruned branch (only partial rules can, and only
+  at or below the template's stratum).
+- `pr491-review-draft.md` — the review comment drafted from that record: the win with the guard lifted, a
+  duplicate-parse bug when enforcement is active (an all-slots-skipped clone differing from its input only by
+  `FinalTemplateState` survives the `ApplyTemplates` equality filter), the rules-only per-stratum guard, and
+  state transitions that run even while the prune is disabled and cost memo hits.
+- `pangloss-final-template-prune-handoff.md` — the same prune handed to PanGloss (the Rust port) for
+  implementation there.
+- `pangloss-hc-rust-speedups-handoff-2026-09-02.md` — speedup leads handed the other way, from PanGloss into
+  this round. Archived; its section 4 is superseded.
+
+Privacy: word forms and glosses from the private grammars appear only as word-list indices and rule ids in these
+records. The raw A/B logs (which contain word forms) live only in the session scratchpad and are not to be
+committed. The main checkout's `.git/info/exclude` now lists the local grammar backups, `Mbugwe.xml`, the Aweti
+word list, and the rulestats HTML report so they cannot be staged by accident.
+
+## What to do next, in order
+
+1. ~~Fix and merge PR #491~~ — merged on 2026-09-28 as `b240b7b5`. On master the duplicate parse is fixed (a
+   final template that would follow a non-template now returns no output instead of marking its clone), and
+   the affix-process rules set `FinalTemplateState` only while the prune is enabled. The guard,
+   `Morpher.IsPartial`, is still grammar-wide and still counts partial lexical entries.
+2. Classify the unclassified affixes in the FLEx projects (Mbugwe 1, Amharic 1, Sena 6); that turns #491's default
+   prune on for Mbugwe and Amharic. Sena stays partial because of its 25 partial lexical entries, so classifying
+   its affixes does not enable the prune there. Whether a narrower guard could safely ignore partial lexical
+   entries is an open question, tracked in sillsdev/machine#551: a partial root makes the word partial, and
+   synthesis then permits a non-partial rule after a final template, so ignoring lexical entries may not be safe.
+   Add the finding to the grammar advisor.
+3. ~~Open the prefilter PR from `perf/hc-edge-prefilter`~~ — that branch and its own archive tag are both
+   deleted; a PR now starts from the mechanism preserved on branch
+   `perf/hc-optimization-archive` rather than resuming the deleted branch.
+4. Port the prune to PanGloss per the handoff, `docs/pangloss-final-template-prune-handoff.md` on branch
+   `perf/hc-optimization-archive`.