Repository navigation
HermitCrab: investigate unstable zero-width morpheme identities across fresh processes #506
Description
Identical inputs have been reported to produce different morpheme identities across fresh C# processes. Until the cause is isolated, those outputs cannot reliably establish parser parity.
Who is affected: HermitCrab consumers of morpheme identities and PanGloss maintainers recording fixtures or comparing against Machine. This is not a demonstrated Rust bug.
Keep this investigation separate from #500's dropped-identity fix. The reported association with tied-node ordering is a hypothesis, not an established root cause; choosing a random seed is not a correctness fix.
Done when: pinned before/after runs reproduce the instability, a small deterministic test isolates the mechanism, and independently justified identity multisets remain correct across every exercised ordering. Add a conformance fixture without freezing an arbitrary incorrect output.
Reproduction and evidence limits
PR #500 uses root sag, A2B (N→V, adds u), B2A (V→N, adds i), and THIRD (N→V, copies input without adding segments).
Expected sagui analyses retain A2B and B2A, with and without THIRD. The reported incorrect signature drops B2A from the THIRD analysis.
The PR reports three outputs over 20 unseeded runs and 12/12 identical correct outputs after fixing the BidirList seed. These observations motivate investigation; this issue's filing did not independently repeat them.
Record exact signatures and revisions, including #500 before/after. Test annotation/tie ordering directly rather than treating repeatability under one seed as proof. PanGloss is affected through its oracle dependency even if it does not use BidirList.
Activity
Fix a dropped morpheme ID when a zero-width rule wraps one ending at …
Fix a dropped morpheme ID when a zero-width rule wraps one ending at …
johnml1135 commentedon Oct 23, 2026
New reproduction on current master (
25992041). It is the same phenomenon, with no compounding involved, and the measurements point atBidirList's random skip-list levels. That contradicts the claim in #500 thatBidirListhas been ruled out.What varies. Grammar
conformance/edge-cases/truncate-morphotactic/grammar.xmlfrom PR #480 (three truncation rules in an unordered stratum, with noMorphemeIds). Parsingaorgaalways returns the same number of analyses (2 and 5) with the same rule-application sequences. What changes is which morphs remain annotated on the output shape. Root32or one of the truncation rules' zero-width morphs is sometimes missing. For example, analysis32 → 3SGB → 3SGAcomes back with morphs3SGA,32,3SGB,3SGA,32or3SGA.What it is not. It is not analysis deduplication, the parallel synthesize
ConcurrentBag, or per-process hash seeds:MaxDegreeOfParallelism = 1at the same rate as with the default (0, unbounded).ParseWordcalls on a singleMorpherinstance in one process.What removes it. Either change below makes every run return the full trails, where each analysis has as many morphs as morphemes applied:
BidirList._rand(src/SIL.Machine/DataStructures/BidirList.cs:15), orBidirList.Find(BidirList.cs:265-269) to return the leftmost equal node instead of the first one an express lane happens to reach.Mechanism, confirmed part.
AnnotationList.Find(offset)searches with a fresh probe annotation (AnnotationList.cs:202,:208). The probe'sListIDis -1, soAnnotation.CompareToreturns 0 against every annotation with the same range (Annotation.cs:167-168). TheListIDtie-break therefore does not apply to these lookups. When zero-width morphs share a node, as truncation morphs do afterSynthesisAffixProcessAllomorphRuleSpec.cs:173and:192attach them toShape.LastandShape.First, which equal nodeFindreturns depends on the skip-list levels drawn atBidirList.cs:58.Mechanism, hypothesis. Which caller turns that into a dropped morph is not traced. The likely path is
GetNodes(AnnotationList.cs:257,:264) during the subsume branch ofAnnotationList.Add(:135), reached fromWord.MarkMorph'sChildren.AddRange(Word.cs:278). Not checked: whether the leftmost-tie change is correct in general, or whether the HermitCrab test suite passes with it.Reproduction and counts
conformance/edge-cases/truncate-morphotactic/grammar.xmlatb49433d7(integrate-conformance-framework, PR Add a conformance suite for HermitCrab, with its adequacy argument #480).origin/masterat25992041.aandga. Neither is a fixture word; the fixture's own words were not checked at scale.src/SIL.Machine.Morphology.HermitCrab. It callsXmlLanguageLoader.Load(path), thennew Morpher(new TraceManager(), lang[, dop]), thenParseWord(word). For each analysis it printsAllomorphsInMorphOrder(Morpheme.Idjoined by+, plus glosses) andMorphemesInApplicationOrder.30 fresh processes per configuration. "Full" means every analysis has a complete morph trail.
aaMaxDegreeOfParallelism = 1gagaMaxDegreeOfParallelism = 1Within one process.
ParseWord("a")calls on oneMorpher(dop=1), in each of three processes: 9, 9 and 9 distinct outputs.XmlLanguageLoader.Load+Morpherconstructions in one process: 9 distinct outputs, in each of two processes.Controls. Three processes, each doing 50 reparses at dop=1 and 20 reloads at default parallelism:
BidirListseeded withnew Random(12345): 1 distinct output, all full.Find, unseeded: 1 distinct output, all full.ListIDbeforebase.AddinAnnotationList.Add: did not help; outputs still varied.Full output for
a, which is stable under both controls:Examples of dropped trails on unpatched master:
The fixture word
gas(++|gas;+|gas) was identical in the handful of runs made; this was not measured at scale.