Repository navigation
HermitCrab conformance: verify blocked_by labels against the trace in the dead-rule gate #552
Description
Summary
The conformance suite's dead-rule gate credits a rule as exercised whenever an expected-failure word names it in blocked_by. It never checks that the rule actually blocked the word. A wrong or stale label therefore counts as coverage.
FailureRuleAttributor.WordLevelFailureRuleIds is documented as upgrading those labels to verified, but it has no callers.
This is a proposal. #480 is adding two weaker steps:
- the gate fails when a label is not a rule id in the fixture's grammar;
- the docs say that
blocked_byis asserted, not verified.
This issue tracks the stronger step: verify each label against the parse trace.
Who is affected: maintainers of the HermitCrab conformance suite (#480) and its PanGloss port.
Done when:
- the gate requires the trace to attribute each word's failure to the rule named in
blocked_by; - the gate keeps an explicit, reviewed exemption list for words whose failure cannot be attributed from the trace (the attributor's doc names
zimed); - a mutated label fails the gate.
Code, on #480 at b49433d
CoverageReport.cs:113-114adds everyword.BlockedByid to the exercised set, with no check.FailureRuleAttributor.cs:66definesWordLevelFailureRuleIds. The only reference outside its own file is a string inOwnedSymbolKeyTests.cs:132.- At that head, all 108
blocked_bylabels name real rule ids in their fixture grammars, so no label is wrong today.
Activity
Check blocked_by labels against grammar rule ids; categorize ledger f…
johnml1135 commentedon Oct 23, 2026
PanGloss has nothing to port this to. I checked sillsdev/PanGloss at 14975991 (v0.10.0):
pg-conformance-fixturesparsesblocked_byfrom words.yaml and validates its shape (unique strings,expect_fail: truerequired, not allowed in variants), but nothing consumes the labels. There is no dead-rule gate, norules.csvordead-rules-allowed.tsv, and no coverage credit forblocked_by. So no PanGloss change, PR or merge was made, and no labels needed exemptions or were found wrong there.Consequence for this issue: the verification gate is machine-only, and PanGloss's shared fixtures carry the same unverified labels without anything reading them. #480 now fails the gate on labels that are not grammar rule ids (49ba481); trace verification (this issue) is not implemented.
Check blocked_by labels against grammar rule ids; categorize ledger f…