Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,8 @@ Three committed contributor docs carry the parser's normative rules and their re

**Guard tests** SHOULD carry a recorded negative control — the answer with the guard off, stored as data (the _EXCLUSION_EFFECT shape; see mechanisms.md's Verification shapes).

**Invariant tests over a generated grid share ONE parsed grid, and a grid pairs a text only with configurations that can read it differently.** Both rules are in `tests/v2/test_properties.py` (`_connective_findings`, `_off_switch_findings`, `_rows`) and both were bought at a price: four tests each walking the same 170,100-row grid, and a grid that was a full cross product of texts × lexicons × policies, together took that module from 11s to 115s and CI's build jobs from ~5 minutes to 17–25 (under coverage every parse costs several times more, and every later PR pays it). A new invariant over an existing grid joins that grid's walk instead of opening its own; a new grid states its measured runtime where it is built. Trim a grid by SHAPE — a variant lexicon reaches only the texts holding the word it adds — never by sampling, and prove the dropped rows were duplicates by signature before dropping them. Mutation counts in the tests' docstrings are counts over the grid, so re-measure them whenever the grid moves -- and so do the DESIGN DOCS that quote them. A recorded control reaches `docs/design/decisions.md` and `docs/design/mechanisms.md` as well as the docstring, nothing checks those copies, and a grid change silently makes them wrong about a number they present as measured: the #397 second review moved four such figures at once (2026-09-20). Move them in the same commit, and where the older number is worth keeping, label it with the population it was measured over rather than deleting it.

## Commands

```bash
Expand Down Expand Up @@ -380,7 +382,7 @@ Add a dedicated `copy.deepcopy()` round-trip test for it too (see `test_regexes_

**`_normalize` must reach a fixed point** — storage and match-time share the one fold, and `Lexicon.__setstate__` re-validates, so a value that changes on re-normalization changes under its owner. `strip().strip(".")` alone is not idempotent (`'. a .'` → `' a '` → `'a'`). The loop is the fix; keep any new stripping inside it. **Anything built on `_normalize` must converge too** — `_fold_words` runs `_normalize` per word and DROPS the words that fold away (`_title_key` is that list space-joined, and `_run_addresses_by_given` reads the list itself, so its last-word arm is the last word of the FOLDED key by construction); keeping the empty slot stored `'lt .'` as `'lt '`, a key match-time can never rebuild (so the entry is silently inert) and `__setstate__` rejects on the next round-trip as "not written by this version".

**Perf regressions are caught by the scaling test, not the absolute-time ones** — `tests/v2/test_benchmark.py::test_parse_cost_grows_no_worse_than_linearly` times a repeated unit at n vs 4n over twelve shapes (one per pipeline inner loop) and bounds the ratio; the `_thousand_names` tests use constant-size, delimiter-free input and are structurally blind to a complexity regression. Two rules when touching it: calibrate `_MAX_RATIO` against the WEAKEST quadratic's signal (a mixed quadratic surfaces far below the textbook 16×, so the operating point `_BASE` matters more than the bound), and confirm a planted regression fails it across REPEATED runs — one failure is a coin-flip on a timing test. The twelve shapes cover different dimensions (segment count only via `commas`, intra-piece accumulation only via `particles`/`conjunctions`, non-ASCII input only via `honorifics` — the other eleven are pure ASCII, so `script_segment` returns at its bail and the CJK stages go unmeasured, and M2's clause view only via `maiden_clause`, whose unit has to END on a class member: `MA nee ` holds the same two words, the peel stops at the trailing marker, and the shape reaches nothing); measure before pruning one. **A shape the CLOCK cannot reach needs a FRAME-count guard instead**, which is the second scaling test in that file (`test_a_trailing_credential_run_does_not_cost_exponentially`, #531): where the defect is an exponential rather than a quadratic, the input length that separates the curves on a timing test does not finish, so the guard counts frames over 8 units against 16 and bounds THAT ratio. One pair does not see every curve, and the fix round for #531 measured why: at 2× the input the per-member LINEAR work swamps a quadratic (2.08× for a genuine one against 1.73× clean), so that pair guards the exponential alone and a second, longer pair — 16 against 64, where the same quadratic reads 7.42× against 3.53× clean — is what can see one. Assert them in that order: an exponential never returns from the longer run, so the cheap pair has to have failed first. Frame counts do not move under load, so this shape needs no repeated-run calibration — but it does need the same reachability assertion `_POLICY_SHAPES` rows carry, since the walk under measurement runs only while every unit still reads as a credential. A stage gated on an opt-in `Policy` field needs a `_POLICY_SHAPES` entry instead, since bare `parse()` never enters it — and that table's rows carry a **reachability probe** run before the measurement, because a precedence change can quietly stop the shape reaching the stage and leave a green test measuring a no-op (`_POLICY_SHAPES` is also asserted non-empty: an empty `parametrize` is a skip, not a failure, so deleting its last row would retire the guard silently).
**Perf regressions are caught by the scaling test, not the absolute-time ones** — `tests/v2/test_benchmark.py::test_parse_cost_grows_no_worse_than_linearly` times a repeated unit at n vs 4n over thirteen shapes (one per pipeline inner loop) and bounds the ratio; the `_thousand_names` tests use constant-size, delimiter-free input and are structurally blind to a complexity regression. Two rules when touching it: calibrate `_MAX_RATIO` against the WEAKEST quadratic's signal (a mixed quadratic surfaces far below the textbook 16×, so the operating point `_BASE` matters more than the bound), and confirm a planted regression fails it across REPEATED runs — one failure is a coin-flip on a timing test. The thirteen shapes cover different dimensions (segment count only via `commas`, intra-piece accumulation only via `particles`/`conjunctions`, non-ASCII input only via `honorifics` — the other twelve are pure ASCII, so `script_segment` returns at its bail and the CJK stages go unmeasured, M2's clause view only via `maiden_clause`, whose unit has to END on a class member: `MA nee ` holds the same two words, the peel stops at the trailing marker, and the shape reaches nothing — and connective RUN LENGTH only via `link_run`, whose unit has to be MIXED CASE and hold a connective of the generational class: `i und ` reads the letter as an initial and reaches nothing where `i Und ` reaches everything); measure before pruning one. **A shape whose input needs a PREFIX cannot be a `_SHAPES` row at all**, since that table repeats a unit and nothing else — a maiden clause needs a name word and a marker before the run it is about, and `"nee i Und " * n` reaches the clause rule not at all, measuring the identical ratio on a broken tree and a fixed one. That one is `test_a_clause_link_run_does_not_cost_quadratically`, which builds its own input and counts FRAMES. **A shape the CLOCK cannot reach needs a FRAME-count guard instead**, which is the second scaling test in that file (`test_a_trailing_credential_run_does_not_cost_exponentially`, #531): where the defect is an exponential rather than a quadratic, the input length that separates the curves on a timing test does not finish, so the guard counts frames over 8 units against 16 and bounds THAT ratio. One pair does not see every curve, and the fix round for #531 measured why: at 2× the input the per-member LINEAR work swamps a quadratic (2.08× for a genuine one against 1.73× clean), so that pair guards the exponential alone and a second, longer pair — 16 against 64, where the same quadratic reads 7.42× against 3.53× clean — is what can see one. Assert them in that order: an exponential never returns from the longer run, so the cheap pair has to have failed first. Frame counts do not move under load, so this shape needs no repeated-run calibration — but it does need the same reachability assertion `_POLICY_SHAPES` rows carry, since the walk under measurement runs only while every unit still reads as a credential. A stage gated on an opt-in `Policy` field needs a `_POLICY_SHAPES` entry instead, since bare `parse()` never enters it — and that table's rows carry a **reachability probe** run before the measurement, because a precedence change can quietly stop the shape reaching the stage and leave a green test measuring a no-op (`_POLICY_SHAPES` is also asserted non-empty: an empty `parametrize` is a skip, not a failure, so deleting its last row would retire the guard silently).

**Expected-failure tests use `@pytest.mark.xfail`** — the conftest parametrized fixture breaks `@unittest.expectedFailure`; always use `@pytest.mark.xfail` instead.

Expand Down
17 changes: 14 additions & 3 deletions docs/customize.rst
Original file line number Diff line number Diff line change
Expand Up @@ -299,9 +299,10 @@ ambiguity is recorded and it becomes part of the surname — under any
A single letter written against the name's own case is an initial and
one written with it is the connective — but a name written wholly in
one case, all upper or all lower, says nothing either way, and this is
the set that decides it there. ``e`` is the one entry shipped: a bare
``E`` initial is common where an ``e`` between two surnames is rare, and
``y`` runs the other way, so ``y`` joins even written as a bare capital.
the set that decides it there. ``e`` and ``i`` are the two entries
shipped: a bare ``E`` or ``I`` initial is common where those letters
between two surnames are rarer, and ``y`` runs the other way, so ``y``
joins even written as a bare capital.

.. doctest::

Expand Down Expand Up @@ -329,6 +330,16 @@ does in Spanish, take it out and the connective reading comes back:
>>> Parser(lexicon=lex).parse("jose e maria santos").given
'jose e maria'

If your data is Catalan or Polish, where ``i`` links two surnames the
way ``y`` does in Spanish, take that one out instead and the link
joins in a one-case name too:

.. doctest::

>>> lex = Lexicon.default().remove(conjunctions_ambiguous={"i"})
>>> Parser(lexicon=lex).parse("josep carod i rovira").family
'carod i rovira'

If your data is Dutch, where a bare single letter is an initial and
never a connective, add the other one instead:

Expand Down
Loading
Loading