From 17e27cf7cf0bfe9a0d10c94fe3a94a267bf37a8d Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Tue, 27 Oct 2026 01:09:41 -0700 Subject: [PATCH 1/6] Move the ambiguity-emitter rules from AGENTS.md to nameparser/_pipeline/AGENTS.md The Ambiguities bullet had grown to 1,488 words in the root file, which every session loads. It is guidance for work inside the pipeline stages, so it moves, unchanged in substance and split under subheadings, into a nested AGENTS.md that loads when a session reads a stage. Root keeps the principle and pointers to it and to mechanisms.md's contract entry; the two test comments quoting "Pin the decision, not the vocabulary" now name the new file. Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 2 +- nameparser/_pipeline/AGENTS.md | 43 ++++++++++++++++++++++++++++++ tests/v2/pipeline/test_assemble.py | 3 ++- tests/v2/pipeline/test_group.py | 4 +-- 4 files changed, 48 insertions(+), 4 deletions(-) create mode 100644 nameparser/_pipeline/AGENTS.md diff --git a/AGENTS.md b/AGENTS.md index 15c476e63..11abc2a0f 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -224,7 +224,7 @@ The 2.0 rewrite lands as underscore-private modules alongside the v1 code. These - **Method organization**, fixed section order in every class: fields + `__post_init__` validation → alternative constructors → dunders (construction/equality → protocol → operators) → properties → public methods by concern (access → editing → comparison → rendering delegates) → private helpers last, except a helper serving exactly one section may sit at that section's head. Sanctioned deviation, facade layer only: `HumanName` and the shim `Constants` organize by v1 concern groups (`# -- render defaults --`, `# -- config / parsing --`, `# -- fields --`, ..., dunders and pickle last) — the classes mirror v1's own surface and die in 3.0; the canonical order still binds every core type. - **Validation is eager and fail-loud**: every `raise` states the offending value, the expected form, and the fix. Exception taxonomy: wrong type — including wrong element type inside a collection, bare `str` where an iterable of strings is expected, or a `Mapping` where a plain iterable is expected — raises `TypeError`; well-typed but unacceptable values raise `ValueError`; failed enum lookups stay `ValueError` for any input (stdlib `EnumType` precedent). **When the message hands the reader code to paste, that code has to survive a type checker, and then RUN** — nameparser ships `py.typed`, and a placeholder can type-check and still raise: the 2.4 release sweep's first fix to the `capitalization_exceptions` hint offered `('key', 'Value')`, which passes mypy and fails #459's mask check, so `test_add_capitalization_exceptions_raises_pointing_at_replace` now executes the offered code. #337's segmenterless warning offered `Policy(segment_scripts=())`, an `arg-type` error, because these fields are annotated with what they STORE rather than everything the constructor accepts. Prefer the `frozenset()` / `()` spellings in messages and docstrings, and pin the offered spelling in a test — the warning tests matched on `ja_segmenter` and never checked the actionable half of the message. **A warning emitted in `Parser.__post_init__` needs `parser_for` to re-emit it from its own frame** (the `catch_warnings(record=True)` block at its return): `__post_init__`'s `stacklevel` is sized for direct `Parser(...)` construction, and through `parser_for`'s extra frame the default one-line rendering attributes the warning to the library's own `return Parser(...)` — the exact call the message tells the user to change becomes invisible. No single stacklevel serves both entry points; a new construction warning gets the re-emission for free, but a new CONSTRUCTION SITE for `Parser` inside this package needs its own re-emission or its callers get library-attributed warnings (#337 review). - **Guard, hint, and emit for the WHOLE family, and parametrize the test over it**: a check added to one member of a set belongs on all of it, and the test must sweep the family, not one example. This session shipped `_reject_str_and_mapping` on `Policy` but not `PolicyPatch`, the bytes decode hint on three of five config entry points, and a regex-sync roster missing four of its copies — each a separate follow-up bug that a `{class} × {field} × {bad-value}` parametrization would have caught and a per-example test hid. When you find you're guarding member N, grep for the other members first. -- **Ambiguities are emitted at the DECISION site**: an `Ambiguity` records a fork the parse had to call, not a token that sits in an ambiguous vocabulary. Emit where the branch is taken — the trailing-suffix peel (read once in `_group` for a main segment since #614, its picks reported by `_assign` once the roles they took are known, except a pick the particle chain takes into the name, which the chain reports and the read then gives up), the delimiter escape's follow-up in `post_rules` — never by scanning for a `vocab:*-ambiguous` tag. The same tagged token is a genuine fork in one position and unremarkable in another (`do` mid-name in "Joao da Silva do Amaral de Souza" chooses nothing). **A branch that runs but changes nothing is not a decision either** -- the prefix chain's `merge_pieces(pieces, ptags, k, j)` executes even when `j == k + 1`, folding a piece into itself, and keying on "the code got here" reported a fork for all ambiguous particles on "Do Van Jr." (`Dr.` when that was written, before #367 made a plain title transparent and put the shape out of the loop's reach entirely), where the particle stayed a lone leading name piece — the GIVEN name under the default order, the family name under `FAMILY_FIRST` — and `_assign` reported the same token again. Check that the branch actually claimed something (`j > k + 1`, or a run of particles group merged into the piece ahead of S2's read, which the chain then claims nothing past, #625) before recording. Structure often settles the question before it arises, which is why `PARTICLE_OR_GIVEN` is not emitted on the `FAMILY_COMMA` path's WHOLLY-FAMILY read -- the comma fixed which piece is the family -- and `SUFFIX_OR_NAME` is not emitted for "Ma, Jack". A tail segment is the same case from the other side: assign reads it as suffixes, a maiden marker there included since #601, and since #603 a title word past the second comma as a title (rules/C2, so `Freiherr` below reads title) (`Jane Doe, PhD, Jr nee van Ma` reads suffix `PhD, Jr nee van Ma`, the marker an ordinary word, rules/M2), so group makes no particle chain there at all (#636; its two emitters had been handed no report list there), after either comma — `John Smith, Jr., Freiherr von Richthofen` leaves `von` unchained and reports only `comma-structure` (rules/C2, 2026-09-28). Read that scope narrowly: the comma settles nothing about a particle trailing the given name, so P6's attachment decides that fork on the same path and reports it (#405; in `post_rules` until #613, in `_assign` since, right after the walk that reads the given part), in the kind naming the reading it OVERRODE, which is the reading assign made and not the word's vocabulary: `SUFFIX_OR_NAME` where assign had read the run as a post-nominal (`vd`, `mc`), else `PARTICLE_OR_GIVEN` where the run holds an ambiguous particle (`van`, and `do`, which is in the suffix vocabulary too but in its AMBIGUOUS half, so no credential reading was overridden), else silence. The decision site also has the token index and the detail text in hand, which the tag scan would have to reconstruct -- all but the FIELD a report names, which a rule in post_rules can still change (H1's move behind a title, P1's family-first fold, P6's attachment, and under opt-in policies the patronymic rotations, O6's two family names and `middle_as_family`'s fold): an emitter naming one passes the rest of its sentence as `field_tail` and assemble words the field from the final role (#626, where `Kim Min Do` under `FAMILY_FIRST` was told 'Do' was a middle name with 'Do' in the family). **If a fork's two branches are taken in DIFFERENT stages, every one of them needs the emitter** -- `PARTICLE_OR_GIVEN` is decided in `_assign` when the ambiguous particle stays a lone leading piece, in `_group` when something shifts it off the name's leading piece and the prefix chain claims it, and wherever P6's attachment takes a trailing particle into the family -- in `_assign` after a comma that names a family (since #613), in `post_rules` at the end of a name read family-first (#467) and after a comma with nothing before it, where H1 and M4 must read the name first -- so every one of them reports; for two years only the first did. What can still do the shifting is narrow, and #367 is why: a plain title no longer can (`Dr. Van Johnson` reads as `Van Johnson` does and reports from `_assign`), so the `_group` emitter needs a word that is BOTH a title and a particle — measured, `TITLES ∩ particles_ambiguous` is `{freiherr, st}` in the default vocabulary (`do` left TITLES in #296's audit; the particle-line block in decisions/vocabulary-exclusions records the before and after), plus any overlap a caller's config creates — standing ahead of the chained particle as the LEADING NAME word. Titles may precede it, so `Dr. St van Johnson` reaches the emitter and `St van Johnson` does too; a given name may not, so `Jan Freiherr von Richthofen` does not reach it while `Freiherr von Richthofen` and `Dr. Freiherr von Richthofen` do. Two shapes that look like they should reach it and do NOT, both measured by stepping `STAGES` and watching where `ambiguities` grows: `Dr. Do van Johnson` and `Do St Johnson` report from `assign`, not `group`, because `do` is no longer a title and so stays the leading name piece assign reports on — a both-vocabulary word CHAINED (`Jan St Johnson`) reports nothing at all. When checking whether that emitter is dead, a both-vocabulary word in the leading name position is the thing to look for, and the answer is that it is not dead. The stage-ownership map in `tests/v2/pipeline/test_state.py` must list `ambiguities` for each such stage, and it passes vacuously until a case row exercises the path, so add the row too. Report BOTH directions of a two-way fork — "John Smith MA" (read as a suffix) and "Jack MA" (read as the family name) are equally guesses. **Every emitter also names the reading it did not take** (`PendingAmbiguity.alternative`, built with `_vocab.suffix_reading`/`name_reading` or `PARTICLE_READING`/`TITLE_READING`, or a `PendingReading` written out where none fits, classify's connective and O5's lone word; 2.7, #752 phase 2m): a non-name reading names its field by what it is, and a name reading's field is the positional read's with every other word kept, or the piece in front where that piece would take the word in (`_assign._name_alt`: a particle chain, a bound given name). A stage that runs before positions exist defers the field (`_state.Deferred`: classify's connective, group's chain), assign names it, and one assign cannot place surfaces as no alternative rather than a guess (a connective letter opening its part, which has no word before it to join). The field is `compare=False`, so a site that REBUILDS an `Ambiguity` (`_render.capitalized` does, mapping its tokens) must pass it on and is tested by comparing the alternatives themselves: equal names say nothing about it, and #902's review found `capitalized()` dropping every one. `test_an_alternative_names_the_field_the_other_reading_gives` (tests/v2/test_cases.py) checks every row's report under all three orders, a name reading by re-parsing with a plain stand-in word, so a new emitter that leaves it unset fails there. The case rows are not enough for a change to how a name reading's field is named: it models the parse the word did not get, so every rule that would have read that parse is a shape to model (a chain, P1's fold, P5's two licences), and two reviews of #752 phase 2m found seven shapes no row held, the second on the first's fixes, by running the same oracle over every one- to three-word run of a 21-word pool (particles, bound given names, numerals, credentials, titles, a marker) in six comma shapes under nine policies; add a found shape to `_REVIEW_SHAPES`. The oracle cannot reach a report whose stand-in is read by the same decision (a credential run's, a one-case connective's, where `Q` is an initial too); those are rows of `_EXPECTED_ALTERNATIVES` beside it, which also pins each reading's kind and source, and a third review of the same PR found a connective inside a joined piece unnamed in a third of one corpus's connective reports for want of such a row. And work a report does must not walk the name: the given-name title test, asked once per report, made three `_SHAPES` rows quadratic (`credential_run` 13.9x for 4x the input). Every kind needs a trigger in `tests/v2/test_contracts.py::_AMBIGUITY_TRIGGERS` (an explicit `None`, strict-xfail, while reserved), and case-table rows pin expected kinds exactly, so a new emitter shows up in both immediately. **Pin the decision, not the vocabulary**: the only titled-particle test used an UNAMBIGUOUS particle, so it walked the right code path and proved nothing about the branch under test -- two criticals passed 1539 tests. A row contrasting the two readings ("John Smith V" against "John Smith B") is what makes an emitter's absence meaningful. +- **Ambiguities are emitted at the DECISION site**: an `Ambiguity` records a fork the parse had to call, not a token that sits in an ambiguous vocabulary, so emit where the branch is taken, in every stage that takes one, and name the reading not taken. The contract is mechanisms.md#AMBIGUITY-AT-THE-DECISION-SITE; the working rules (branches that change nothing, the field a report names, the stages that decide `PARTICLE_OR_GIVEN`, `alternative`, and how to test an emitter) are in `nameparser/_pipeline/AGENTS.md`, which loads when a session reads a pipeline stage. - **A kind is worth adding only if a reader would hesitate too**: the test is not "does the code take a branch" but whether a person reading that input would genuinely be unsure. "Smith, John V" reads as a middle initial to anyone -- the comma settles it -- so reporting it would be noise that teaches callers to ignore the field, which costs more than the missing report. Reachability of the second branch is necessary, not sufficient. Prefer leaving a fork silent and documenting the omission over emitting on input nobody finds ambiguous. - **Parser owns config-dependent conveniences**: `Parser.matches`/`Parser.capitalized`/`Parser.revise` exist because the `ParsedName` equivalents fall back to DEFAULT config for str/omitted arguments (documented loudly in both docstrings). `revise` harvests tokens from a full sub-parse of each replacement value (tags kept minus `FOLDED_TAG`, roles forced, the R1 entry pass `suffix_entries` re-run over the forced state so a suffix value's entries follow its own commas, ambiguities discarded); the merge tail is shared with `replace()` via `ParsedName._with_field_tokens`. `Parser.capitalized` delegates through `name.capitalized(self.lexicon)` specifically so `_parser` never imports `_render` — keep it that way. - **Per-word vocabulary fields warn on multi-word entries** (`_normset`/`_normpairs` via `_warn_dead_entry`, UserWarning, never a raise — see the given_name_titles Gotcha for why raising is wrong). `given_name_titles` is the one multi-word-matched field and is exempt; `_edit` passes `warn=False` (add() warns once via the new instance's `__post_init__`; remove() stores nothing). The default vocabulary and every locale pack must stay warning-free (`test_default_lexicon_builds_warning_free`, `test_pack_vocabulary_entries_are_single_words`). diff --git a/nameparser/_pipeline/AGENTS.md b/nameparser/_pipeline/AGENTS.md new file mode 100644 index 000000000..1b917d684 --- /dev/null +++ b/nameparser/_pipeline/AGENTS.md @@ -0,0 +1,43 @@ +# Working in nameparser/_pipeline/ + +Loaded when a session reads or edits a pipeline stage. The root AGENTS.md carries the 2.0 conventions every stage follows (one stage per module, the `Consumes:`/`Produces:`/`Reads:` header, the frame budget, checkable stage-attribution claims); this file carries what only work inside the stages needs. The contract behind the ambiguity rules below is mechanisms.md#AMBIGUITY-AT-THE-DECISION-SITE, and rules/A1 is its observable side. + +## Ambiguity reports + +### Emit at the decision site + +An `Ambiguity` records a fork the parse had to call, not a token that sits in an ambiguous vocabulary. Emit where the branch is taken — the trailing-suffix peel (read once in `_group` for a main segment since #614, its picks reported by `_assign` once the roles they took are known, except a pick the particle chain takes into the name, which the chain reports and the read then gives up), the delimiter escape's follow-up in `post_rules` — never by scanning for a `vocab:*-ambiguous` tag. The same tagged token is a genuine fork in one position and unremarkable in another (`do` mid-name in "Joao da Silva do Amaral de Souza" chooses nothing). + +**A branch that runs but changes nothing is not a decision either** -- the prefix chain's `merge_pieces(pieces, ptags, k, j)` executes even when `j == k + 1`, folding a piece into itself, and keying on "the code got here" reported a fork for all ambiguous particles on "Do Van Jr." (`Dr.` when that was written, before #367 made a plain title transparent and put the shape out of the loop's reach entirely), where the particle stayed a lone leading name piece — the GIVEN name under the default order, the family name under `FAMILY_FIRST` — and `_assign` reported the same token again. Check that the branch actually claimed something (`j > k + 1`, or a run of particles group merged into the piece ahead of S2's read, which the chain then claims nothing past, #625) before recording. + +### Structure settles some forks before they arise + +That is why `PARTICLE_OR_GIVEN` is not emitted on the `FAMILY_COMMA` path's WHOLLY-FAMILY read -- the comma fixed which piece is the family -- and `SUFFIX_OR_NAME` is not emitted for "Ma, Jack". A tail segment is the same case from the other side: assign reads it as suffixes, a maiden marker there included since #601, and since #603 a title word past the second comma as a title (rules/C2, so `Freiherr` below reads title) (`Jane Doe, PhD, Jr nee van Ma` reads suffix `PhD, Jr nee van Ma`, the marker an ordinary word, rules/M2), so group makes no particle chain there at all (#636; its two emitters had been handed no report list there), after either comma — `John Smith, Jr., Freiherr von Richthofen` leaves `von` unchained and reports only `comma-structure` (rules/C2, 2026-09-28). + +Read that scope narrowly: the comma settles nothing about a particle trailing the given name, so P6's attachment decides that fork on the same path and reports it (#405; in `post_rules` until #613, in `_assign` since, right after the walk that reads the given part), in the kind naming the reading it OVERRODE, which is the reading assign made and not the word's vocabulary: `SUFFIX_OR_NAME` where assign had read the run as a post-nominal (`vd`, `mc`), else `PARTICLE_OR_GIVEN` where the run holds an ambiguous particle (`van`, and `do`, which is in the suffix vocabulary too but in its AMBIGUOUS half, so no credential reading was overridden), else silence. + +### The field a report names is assemble's + +The decision site also has the token index and the detail text in hand, which the tag scan would have to reconstruct -- all but the FIELD a report names, which a rule in post_rules can still change (H1's move behind a title, P1's family-first fold, P6's attachment, and under opt-in policies the patronymic rotations, O6's two family names and `middle_as_family`'s fold): an emitter naming one passes the rest of its sentence as `field_tail` and assemble words the field from the final role (#626, where `Kim Min Do` under `FAMILY_FIRST` was told 'Do' was a middle name with 'Do' in the family). + +### A fork decided in several stages needs an emitter in each + +**If a fork's two branches are taken in DIFFERENT stages, every one of them needs the emitter** -- `PARTICLE_OR_GIVEN` is decided in `_assign` when the ambiguous particle stays a lone leading piece, in `_group` when something shifts it off the name's leading piece and the prefix chain claims it, and wherever P6's attachment takes a trailing particle into the family -- in `_assign` after a comma that names a family (since #613), in `post_rules` at the end of a name read family-first (#467) and after a comma with nothing before it, where H1 and M4 must read the name first -- so every one of them reports; for two years only the first did. + +What can still do the shifting is narrow, and #367 is why: a plain title no longer can (`Dr. Van Johnson` reads as `Van Johnson` does and reports from `_assign`), so the `_group` emitter needs a word that is BOTH a title and a particle — measured, `TITLES ∩ particles_ambiguous` is `{freiherr, st}` in the default vocabulary (`do` left TITLES in #296's audit; the particle-line block in decisions/vocabulary-exclusions records the before and after), plus any overlap a caller's config creates — standing ahead of the chained particle as the LEADING NAME word. Titles may precede it, so `Dr. St van Johnson` reaches the emitter and `St van Johnson` does too; a given name may not, so `Jan Freiherr von Richthofen` does not reach it while `Freiherr von Richthofen` and `Dr. Freiherr von Richthofen` do. Two shapes that look like they should reach it and do NOT, both measured by stepping `STAGES` and watching where `ambiguities` grows: `Dr. Do van Johnson` and `Do St Johnson` report from `assign`, not `group`, because `do` is no longer a title and so stays the leading name piece assign reports on — a both-vocabulary word CHAINED (`Jan St Johnson`) reports nothing at all. When checking whether that emitter is dead, a both-vocabulary word in the leading name position is the thing to look for, and the answer is that it is not dead. + +The stage-ownership map in `tests/v2/pipeline/test_state.py` must list `ambiguities` for each such stage, and it passes vacuously until a case row exercises the path, so add the row too. Report BOTH directions of a two-way fork — "John Smith MA" (read as a suffix) and "Jack MA" (read as the family name) are equally guesses. + +### Name the reading not taken + +**Every emitter also names the reading it did not take** (`PendingAmbiguity.alternative`, built with `_vocab.suffix_reading`/`name_reading` or `PARTICLE_READING`/`TITLE_READING`, or a `PendingReading` written out where none fits, classify's connective and O5's lone word; 2.7, #752 phase 2m): a non-name reading names its field by what it is, and a name reading's field is the positional read's with every other word kept, or the piece in front where that piece would take the word in (`_assign._name_alt`: a particle chain, a bound given name). A stage that runs before positions exist defers the field (`_state.Deferred`: classify's connective, group's chain), assign names it, and one assign cannot place surfaces as no alternative rather than a guess (a connective letter opening its part, which has no word before it to join). + +The field is `compare=False`, so a site that REBUILDS an `Ambiguity` (`_render.capitalized` does, mapping its tokens) must pass it on and is tested by comparing the alternatives themselves: equal names say nothing about it, and #902's review found `capitalized()` dropping every one. + +`test_an_alternative_names_the_field_the_other_reading_gives` (tests/v2/test_cases.py) checks every row's report under all three orders, a name reading by re-parsing with a plain stand-in word, so a new emitter that leaves it unset fails there. The case rows are not enough for a change to how a name reading's field is named: it models the parse the word did not get, so every rule that would have read that parse is a shape to model (a chain, P1's fold, P5's two licences), and two reviews of #752 phase 2m found seven shapes no row held, the second on the first's fixes, by running the same oracle over every one- to three-word run of a 21-word pool (particles, bound given names, numerals, credentials, titles, a marker) in six comma shapes under nine policies; add a found shape to `_REVIEW_SHAPES`. The oracle cannot reach a report whose stand-in is read by the same decision (a credential run's, a one-case connective's, where `Q` is an initial too); those are rows of `_EXPECTED_ALTERNATIVES` beside it, which also pins each reading's kind and source, and a third review of the same PR found a connective inside a joined piece unnamed in a third of one corpus's connective reports for want of such a row. + +And work a report does must not walk the name: the given-name title test, asked once per report, made three `_SHAPES` rows quadratic (`credential_run` 13.9x for 4x the input). + +### Testing an emitter + +Every kind needs a trigger in `tests/v2/test_contracts.py::_AMBIGUITY_TRIGGERS` (an explicit `None`, strict-xfail, while reserved), and case-table rows pin expected kinds exactly, so a new emitter shows up in both immediately. **Pin the decision, not the vocabulary**: the only titled-particle test used an UNAMBIGUOUS particle, so it walked the right code path and proved nothing about the branch under test -- two criticals passed 1539 tests. A row contrasting the two readings ("John Smith V" against "John Smith B") is what makes an emitter's absence meaningful. diff --git a/tests/v2/pipeline/test_assemble.py b/tests/v2/pipeline/test_assemble.py index 873ad2e24..3eedd0f8e 100644 --- a/tests/v2/pipeline/test_assemble.py +++ b/tests/v2/pipeline/test_assemble.py @@ -185,7 +185,8 @@ def test_the_withdrawal_reads_the_role_and_not_the_word() -> None: Nothing shipped builds this state -- a marked letter roled TITLE -- which is exactly why it is built here: an implementation keyed on the suffix role alone would pass every row above and - fail this one (AGENTS.md: "Pin the decision, not the vocabulary"). + fail this one (_pipeline/AGENTS.md: "Pin the decision, not the + vocabulary"). """ import dataclasses from nameparser._pipeline._state import PendingAmbiguity diff --git a/tests/v2/pipeline/test_group.py b/tests/v2/pipeline/test_group.py index a8ff07f91..7f480a23f 100644 --- a/tests/v2/pipeline/test_group.py +++ b/tests/v2/pipeline/test_group.py @@ -1397,8 +1397,8 @@ def test_the_marker_placements_reach_both_answers() -> None: # vocabulary, so each test below builds the overlap it is about. That # is the point of the rule -- it is keyed on the CLASS, never on the # letter -- and a test using the shipped sets would walk the right -# branch while proving nothing about it (AGENTS.md: "Pin the decision, -# not the vocabulary"). +# branch while proving nothing about it (_pipeline/AGENTS.md: "Pin the +# decision, not the vocabulary"). _LINK_LEX = _LEX.add(conjunctions={"i"}, suffix_words={"i"}) #: the same letter as a connective that is NOT generational vocabulary _PLAIN_LEX = _LEX.add(conjunctions={"i"}) From 19e768e67dedead1d6517e0e19d70d3a6fcbd481 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Tue, 27 Oct 2026 01:11:55 -0700 Subject: [PATCH 2/6] Cut AGENTS.md's design-philosophy paragraph to its doctrine and an index of the script rules The paragraph had reached 1,458 words, most of it the glued-honorific peel's comma reach (#312, #319, #613) restated from code. Root keeps the positional, never-detect-language doctrine, the script-itself exception and its three-way test, and lists each behavior it licenses with the rule that states it. The facts no rule or decisions entry held move to decisions/W2 as a dated entry, every parse re-run today: the reach and second-condition examples, where the credential lands by spelling, why the run predicate is not the token predicate pluralized, what lenient_comma_suffixes=False does, the four drifts that made the peel ask _comma.decide, and that neither policy opt-out reaches the peel. The #760 entry, which deferred two of those facts to this paragraph, now says where they went. The opt-out example is spelled segment_scripts=frozenset(), the denylist having refused the () spelling (#334). Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 13 +++++++++++-- docs/design/decisions/W2-glued-honorific-peel.md | 3 ++- 2 files changed, 13 insertions(+), 3 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 11abc2a0f..cfb87da18 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -180,7 +180,16 @@ logging.getLogger('HumanName').setLevel(logging.DEBUG) The library has two layers: data — the per-word TOML in `nameparser/vocab/`, compiled into the `nameparser/config/` modules — and logic, the 2.0 pipeline in the underscore-private `nameparser/_*.py` modules (see 2.0 API modules below). `nameparser/parser.py` and `nameparser/config/__init__.py` are only re-exports of the v1-compat facade (`_facade.py`, `_config_shim.py`); the v1 parser itself was deleted at the M11 swap and is readable as `git show 2d5d8c2:nameparser/parser.py`. -**Design philosophy — positional and language-agnostic.** The parser assigns parts by *position* plus small sets of words that join to neighbors; it never detects language. A name's language can't be reliably inferred from Latin-script transliteration ("Ali" is Arabic or Italian; "Van"/"Della"/"Bin" are first names in some cultures, particles in others), so language-specific rules belong in opt-in `Constants` config, never global defaults. Many "wrong for language X" reports (#133, #150, #130, #85, #103, #146, #83) are irreducible ambiguities — e.g. `de Mesnil` (want last name) vs `Van Johnson` (want first name) are the same `[prefix][word]` shape. Before adding a rule, confirm it doesn't break the opposite case (run the full suite — Portuguese and "Van Johnson" tests are the usual canaries). **The one scoped exception (2.1, #271/#272): script-conditional behavior is permitted exactly where the SCRIPT ITSELF — not statistics about it — determines the convention.** The never-detect-language rule above is about Latin *transliteration*, where the signal genuinely is destroyed; native script is a different question, and it is answered per behavior rather than per script. Five defaults fall out of it, plus a sixth that applies the same not-a-guess standard to specific WORDS rather than to a script (#308's honorific peel, below). Wholly-Han, wholly-Hangul and kana-licensed names read family-first (`Policy.script_orders`) — no language detection needed, because zh and ja both write family-first in native script, so order cannot be misread even though the language is unknowable. Unspaced hangul splits into surname + given name (`config/surnames.py` ships the Korean census list as DEFAULT vocabulary) — nothing but Korean is written in hangul and the surnames are a closed census set, and the vocabulary is self-selecting besides: a hangul entry can only ever match hangul text. Hiragana licenses Japanese (#272) — a name whose characters stay inside Han∪kana while carrying at least one kana cannot be Chinese (the kana rules it out) and is not a transcription (foreign names are transcribed in katakana ALONE, マイケル has no kanji), so 高橋みなみ and 山田 エミ read family-first too; mechanically they resolve to the HIRAGANA entry, the license's carrier key. PURE katakana — full-width or halfwidth, the halfwidth block U+FF65–FF9F being katakana in the script table since #594 — is excluded and keeps the positional default: マイケル・ジャクソン is a transcribed foreign name in its source order, and a Japanese reading written in katakana (legacy halfwidth data spells Japanese names that way too) cannot be told from one, so the script settles nothing and a caller who knows opts in through `script_orders`. And the 间隔号 U+00B7 (#298) is the transcription marker for scripts that HAVE no transcription script: a name it divides (威廉·莎士比亚 — flanked by classified characters on both sides, so Catalan's Gal·la is untouched) keeps the declared order and never segments — the orthography names the convention, with the divider carrying the signal that stands the script's family-first reading down, so the default reads it in its written order and a declared `FAMILY_FIRST` reverses it like any other name; pure katakana reaches the same reading by the opposite route, its script settling nothing so the declared order stands. And a listed CJK honorific glued to the END of a name token is split off it (#308) — 田中さん is 田中 plus さん — on the same orthography-settles-it test, narrowed for the glued position: an entry peels only where it can never end a name, so 씨/님/さん/様/先生 peel while 양/군/氏/博士/殿 stay spaced-only (김지양 and 田中博士 are names, and ~90 Japanese surnames end in 殿) and 君 is in NEITHER set (王君 is a complete Chinese name), though its kana spelling くん peels. Like the nakaguro's tokenize-level separation described next, it is reached by neither policy opt-out — but for its own reason: the vocabulary carries the license itself rather than borrowing the script's, so `segment_scripts` has nothing to say about it. Since #312 it also crosses the 间隔号, which still stops the surname split standing right beside it: it answers where a name DIVIDES into surname and given, and the peel never asks that question. Whether it also crosses the FAMILY comma is tolerated rather than settled: the 2026-09-01 demotion (rules/W3) narrowed that half from contract to best-effort, since no CJK writing system's own convention puts a comma between family and given at all — so `김, 민준씨` reads today exactly as the spaced `김 민준씨` does (family 김, given 민준, suffix 씨) while the split stands down as before, but that reading is watched in the snapshot rather than pinned as contract. Its site is accordingly the name-bearing segment runs — `segments[:2]` under a family comma, and `segments[0]` as before otherwise, the family comma being the one structure that splits the name itself across two runs, with the honorific as often glued to the given side as to the family. That is the whole reach and nothing past it (`김, 민준 지훈씨` peels; `김, 민준, 지훈씨` and `김,, 민준씨` do not, both landing in a third run), and whether `segments[1]` is name text at all is now ASKED rather than inferred from the structure — `segment` does not guarantee it, since a one-word part before the comma reads as FAMILY_COMMA even when the part after it is entirely suffix-shaped, and the peel walking into such a run took `V.` for its site, found no listed tail and abandoned (#319). The question is asked in two parts. First the comma's own decision: `_comma.decide` on a classified copy of the name as written, since #613's /simplify -- a hand copy of it here drifted from decide four times in that PR's review (the ambiguous class's count, the numeral refusal, a suffix before the comma, the split `Ph. D.`), each drift dividing a name decide read as a family comma or leaving whole one it read as a suffix comma, and asking the real decision is what ended it. Where that leaves the family comma, C1's suffix-word predicate, lifted into `_vocab.is_wholly_suffix` so the stages cannot drift: a wholly suffix-shaped second run is declined and the scan stays in `segments[0]`, so `田中さん, V.` and `田中さん, Ph. D.` give さん up as `田中さん, PhD` always did. The test is necessary but NOT sufficient, and the second condition is not decoration: every honorific tail is also a suffix word, so a glued honorific is itself part of what makes its run read as suffix-shaped, and declining a run that holds the ONLY site loses the peel outright. `segments[0]` must therefore offer a peel site of its own before the second run is declined — `이, J.씨` and `선생님, J.씨` pass the suffix test and are scanned anyway, keeping the pre-#319 reading, while `김민준씨, J.씨` has a site on both sides and peels the person's own 씨 rather than the junk one behind the comma. Uniform in the PEEL, that is — where the credential itself lands is `assign`'s question and still differs by spelling (`V.` → `given`, `PhD` and `Ph. D.` → `suffix`). Not `_is_post_nominal` pluralized: the run predicate says yes both to what the token predicate vetoes (`V.`, `V`, `I` — the class the defect was reported as) and to what the token predicate never sees at all, since `period_joined_vocab` and the delimiter routes are the run predicate's alone (`Msc.Ed.` and `J.씨` reach it that way, and `田中さん, Msc.Ed.` moves with the rest). `Policy(lenient_comma_suffixes=False)` (rules/C14) drops this call to the strict token test too — so those three read as name text again and keep the pre-#319 answer, while `Ph. D.` peels under the knob regardless, its merged `phd` passing the strict test. `田中さん, 太郎` is unchanged, and not because of its comma — the honorific there is not at the end of the name, 太郎 is. The nakaguro belongs to the same doctrine but is decided a layer down: U+30FB and its halfwidth twin U+FF65 separate tokens like whitespace, unconditionally and in tokenize, so neither policy opt-out (`script_orders={}`, `segment_scripts=()`) reaches it — the codepoints are CJK-only and appear in no other script's names, which is what licenses a tokenize-level rule where U+00B7 (also the Catalan punt volat, interior to Gal·la) needs the flanked-by-classified-script guard `_tokenize_region` gives it (#298). Han segmentation stays OPT-IN (`locales.ZH` for Chinese, `locales.JA` for Japanese) — a zh surname list corrupts Japanese kanji names, since 高 is a common Chinese surname and 高橋一郎 would split 高+橋一郎 where the correct reading is 高橋+一郎; no surname list divides a kanji name at all, so `locales.JA` activates the stage and a pluggable `Parser(segmenter=...)` does the dividing. Latin-script input is never touched by any of this: "Kim Min-jun" is genuinely order-ambiguous and stays governed by `name_order` and opt-in packs. Before adding a script-conditional rule, work out which of the three it is — certain, certain for this one behavior only, or a statistical guess wearing a script's clothes. +**Design philosophy — positional and language-agnostic.** The parser assigns parts by *position* plus small sets of words that join to neighbors; it never detects language. A name's language can't be reliably inferred from Latin-script transliteration ("Ali" is Arabic or Italian; "Van"/"Della"/"Bin" are first names in some cultures, particles in others), so language-specific rules belong in opt-in `Constants` config, never global defaults. Many "wrong for language X" reports (#133, #150, #130, #85, #103, #146, #83) are irreducible ambiguities — e.g. `de Mesnil` (want last name) vs `Van Johnson` (want first name) are the same `[prefix][word]` shape. Before adding a rule, confirm it doesn't break the opposite case (run the full suite — Portuguese and "Van Johnson" tests are the usual canaries). **The one scoped exception (2.1, #271/#272): script-conditional behavior is permitted exactly where the SCRIPT ITSELF — not statistics about it — determines the convention.** The never-detect-language rule is about Latin *transliteration*, where the signal is destroyed; native script is a different question, answered per behavior rather than per script. What it licenses today, each stated in its rule, with the history in the decisions file of the same ID: + +- Wholly-Han, wholly-Hangul and kana-licensed names read family-first (`Policy.script_orders`): zh and ja both write family-first in native script, so order cannot be misread though the language is unknowable. Hiragana licenses Japanese (#272), while pure katakana, full-width or halfwidth, keeps the positional default, being how foreign names are transcribed (#594) — rules/W4. +- Unspaced hangul splits into surname + given name on the Korean census surnames shipped as default vocabulary: nothing but Korean is written in hangul, and the surnames are a closed set — rules/W1. +- The 间隔号 U+00B7 between classified characters marks a transcription, so the name keeps its declared order and never segments; it divides only when flanked, being also the Catalan punt volat (`Gal·la`) — rules/T3 and W4 (#298). +- The nakaguro (U+30FB and halfwidth U+FF65) separates words unconditionally, in tokenize, so neither policy opt-out (`script_orders={}`, `segment_scripts=frozenset()`) reaches it: its codepoints are CJK-only — rules/T2. +- A listed CJK honorific glued to the end of a name splits off, but only an entry that can never end a name (#308): the same not-a-guess standard applied to WORDS rather than a script, the vocabulary carrying the license itself, so neither opt-out reaches it either. Its reach across a family comma is tolerated rather than contract — rules/W2, rules/W3, decisions/W2. +- Han segmentation stays OPT-IN (`locales.ZH`; `locales.JA` with a `Parser(segmenter=...)`): a Chinese surname list corrupts Japanese kanji names (高 is a common Chinese surname, so 高橋一郎 would split 高+橋一郎), and no surname list divides a kanji name at all — rules/W1. + +Latin-script input is never touched by any of this: "Kim Min-jun" is genuinely order-ambiguous and stays governed by `name_order` and opt-in packs. Before adding a script-conditional rule, work out which of the three it is — certain, certain for this one behavior only, or a statistical guess wearing a script's clothes. **A constant's membership is a question you may reopen.** Proposing that a word be ADDED, REMOVED or MOVED between vocabulary sets is ordinary design work — a shipped entry is not evidence that anyone judged it. `SUFFIX_ACRONYMS` arrived in `af5bdab` as a bulk Wikipedia import never reviewed against surname collisions: 572 of its 577 alphabetic entries leave `family` empty in `"John "` against five ambiguous-gated exceptions (recomputed 2026-09-07; the fifth is `ba`; 568 of 575 against seven, measured 2026-09-25 after #540), and `sa`, `se` and `om` are borne as surnames (measured 2026-08-23; all three since decided, `sa` removed from the vocabulary on 2026-10-25, #857), as were `rai`, `cha`, `ba` and `mc` before `decisions/suffix-acronym-collisions` decided all four — `rai` and `cha` removed, `ba` marked ambiguous, `mc` left alone as no borne name at all. The same entry marked `meng` and `lac` ambiguous on 2026-09-25 (#540). When a fix starts to look like new machinery, check the vocabulary first. Criterion: `decisions/vocabulary-collisions`, with #360's positional qualifier. @@ -222,7 +231,7 @@ The 2.0 rewrite lands as underscore-private modules alongside the v1 code. These - **Layering is enforced by `tests/v2/test_layering.py`** (exact-module matching; `if TYPE_CHECKING:` imports don't count): `_types` imports nothing internal at module level — its rendering delegates import `_render` at call time; `_lexicon` and `_policy` sit above `_types` independently (`_lexicon` may import `nameparser.config.*` DATA modules only — vocabulary is single-sourced from the v1 data modules through 2.x, e.g. `config/maiden_markers.py`); `_locale` sits on `_lexicon`+`_policy` (plus `_types` for the shared pickle mixin); `_render` imports `_types` and `_lexicon` (for `Lexicon.default()` and `_normalize`); `_pipeline/*` imports `_types`+`_lexicon`+`_policy` plus in-package `_pipeline` helpers; `_parser` sits on everything except `_render`; the facade layer (`_facade`, `_config_shim`, `parser`, `config/__init__`, `__main__`) may import anything public plus `_render`; locale pack modules (`locales/*.py`) import `_locale`/`_lexicon`/`_policy`/`_types` only (`_types` joined the list in #272 — `locales/ja.py`'s segmenter factory constructs a `Segmentation`), and the `locales/__init__` additionally lazy-imports its packs (PEP 562). Extend the test's `ALLOWED` table when adding a module. - **Canonical field order** — the seven roles in `Role` enum declaration order, defined once and derived everywhere (properties, `as_dict`, reprs, `comparison_key`). Never restate the order literally. `Role` is a `StrEnum`: members compare as their field-name strings, and `tokens_for()` coerces strings via `_coerce_enum`. - **Method organization**, fixed section order in every class: fields + `__post_init__` validation → alternative constructors → dunders (construction/equality → protocol → operators) → properties → public methods by concern (access → editing → comparison → rendering delegates) → private helpers last, except a helper serving exactly one section may sit at that section's head. Sanctioned deviation, facade layer only: `HumanName` and the shim `Constants` organize by v1 concern groups (`# -- render defaults --`, `# -- config / parsing --`, `# -- fields --`, ..., dunders and pickle last) — the classes mirror v1's own surface and die in 3.0; the canonical order still binds every core type. -- **Validation is eager and fail-loud**: every `raise` states the offending value, the expected form, and the fix. Exception taxonomy: wrong type — including wrong element type inside a collection, bare `str` where an iterable of strings is expected, or a `Mapping` where a plain iterable is expected — raises `TypeError`; well-typed but unacceptable values raise `ValueError`; failed enum lookups stay `ValueError` for any input (stdlib `EnumType` precedent). **When the message hands the reader code to paste, that code has to survive a type checker, and then RUN** — nameparser ships `py.typed`, and a placeholder can type-check and still raise: the 2.4 release sweep's first fix to the `capitalization_exceptions` hint offered `('key', 'Value')`, which passes mypy and fails #459's mask check, so `test_add_capitalization_exceptions_raises_pointing_at_replace` now executes the offered code. #337's segmenterless warning offered `Policy(segment_scripts=())`, an `arg-type` error, because these fields are annotated with what they STORE rather than everything the constructor accepts. Prefer the `frozenset()` / `()` spellings in messages and docstrings, and pin the offered spelling in a test — the warning tests matched on `ja_segmenter` and never checked the actionable half of the message. **A warning emitted in `Parser.__post_init__` needs `parser_for` to re-emit it from its own frame** (the `catch_warnings(record=True)` block at its return): `__post_init__`'s `stacklevel` is sized for direct `Parser(...)` construction, and through `parser_for`'s extra frame the default one-line rendering attributes the warning to the library's own `return Parser(...)` — the exact call the message tells the user to change becomes invisible. No single stacklevel serves both entry points; a new construction warning gets the re-emission for free, but a new CONSTRUCTION SITE for `Parser` inside this package needs its own re-emission or its callers get library-attributed warnings (#337 review). +- **Validation is eager and fail-loud**: every `raise` states the offending value, the expected form, and the fix. Exception taxonomy: wrong type — including wrong element type inside a collection, bare `str` where an iterable of strings is expected, or a `Mapping` where a plain iterable is expected — raises `TypeError`; well-typed but unacceptable values raise `ValueError`; failed enum lookups stay `ValueError` for any input (stdlib `EnumType` precedent). **When the message hands the reader code to paste, that code has to survive a type checker, and then RUN** — nameparser ships `py.typed`, and a placeholder can type-check and still raise: the 2.4 release sweep's first fix to the `capitalization_exceptions` hint offered `('key', 'Value')`, which passes mypy and fails #459's mask check, so `test_add_capitalization_exceptions_raises_pointing_at_replace` now executes the offered code. #337's segmenterless warning offered `Policy(segment_scripts=frozenset())`, an `arg-type` error, because these fields are annotated with what they STORE rather than everything the constructor accepts. Prefer the `frozenset()` / `()` spellings in messages and docstrings, and pin the offered spelling in a test — the warning tests matched on `ja_segmenter` and never checked the actionable half of the message. **A warning emitted in `Parser.__post_init__` needs `parser_for` to re-emit it from its own frame** (the `catch_warnings(record=True)` block at its return): `__post_init__`'s `stacklevel` is sized for direct `Parser(...)` construction, and through `parser_for`'s extra frame the default one-line rendering attributes the warning to the library's own `return Parser(...)` — the exact call the message tells the user to change becomes invisible. No single stacklevel serves both entry points; a new construction warning gets the re-emission for free, but a new CONSTRUCTION SITE for `Parser` inside this package needs its own re-emission or its callers get library-attributed warnings (#337 review). - **Guard, hint, and emit for the WHOLE family, and parametrize the test over it**: a check added to one member of a set belongs on all of it, and the test must sweep the family, not one example. This session shipped `_reject_str_and_mapping` on `Policy` but not `PolicyPatch`, the bytes decode hint on three of five config entry points, and a regex-sync roster missing four of its copies — each a separate follow-up bug that a `{class} × {field} × {bad-value}` parametrization would have caught and a per-example test hid. When you find you're guarding member N, grep for the other members first. - **Ambiguities are emitted at the DECISION site**: an `Ambiguity` records a fork the parse had to call, not a token that sits in an ambiguous vocabulary, so emit where the branch is taken, in every stage that takes one, and name the reading not taken. The contract is mechanisms.md#AMBIGUITY-AT-THE-DECISION-SITE; the working rules (branches that change nothing, the field a report names, the stages that decide `PARTICLE_OR_GIVEN`, `alternative`, and how to test an emitter) are in `nameparser/_pipeline/AGENTS.md`, which loads when a session reads a pipeline stage. - **A kind is worth adding only if a reader would hesitate too**: the test is not "does the code take a branch" but whether a person reading that input would genuinely be unsure. "Smith, John V" reads as a middle initial to anyone -- the comma settles it -- so reporting it would be noise that teaches callers to ignore the field, which costs more than the missing report. Reachability of the second branch is necessary, not sufficient. Prefer leaving a fork silent and documenting the omission over emitting on input nobody finds ambiguous. diff --git a/docs/design/decisions/W2-glued-honorific-peel.md b/docs/design/decisions/W2-glued-honorific-peel.md index e1f483279..237071cfa 100644 --- a/docs/design/decisions/W2-glued-honorific-peel.md +++ b/docs/design/decisions/W2-glued-honorific-peel.md @@ -25,4 +25,5 @@ Open: none in this file. - 2026-10-17 #671 (its no-glued-peel verdict superseded by #707 below for every word but 部長/部长) — SPACED AND AFTER-COMMA ONLY, no glued peel (Derek, 2026-10-17). 議員/议员, 国会議員/國會議員, 太平紳士/太平绅士, 医学博士/醫學博士, 二世 and the job titles 社長/部長/課長 and the simplified 社长/部长/课长 (Derek, 2026-10-17: the characters are shared and the offices are real in both languages; no borne-name data either way, #707) join `SUFFIX_WORDS`; none joins `GLUED_HONORIFICS`. The spaced bar is only that a word standing alone is no name, so they ship; the glued bar ("can it END a name?") needs real-name counts that this change did not get — the Wikidata suffix query timed out and the native-script Wikidata corpus holds none of the words — and 部長 fails it on argument alone (長 is a borne given name, 阿部長). A follow-up can vet 医学博士, 議員 and 太平紳士 one at a time. Consequence measured on `parser_for(locales.ZH)`: after a listed surname each word divides off, `王二世` family 王, suffix 二世 (it read given 二世) and `李部長` family 李, suffix 部長, because the segmenter's division is not a peel and the split-off 二世 is then a whole-token suffix word, the way 王博士 already reads; `陳怡君二世` still reads given 怡君二世 (glued, nothing divides it; superseded for the glued question by the #707 entry below). ジュニア was already shipped by #661, and #679 declined 二世 as CLDR's generation word ("regnal, and glued"); here it ships spaced-only as the Chinese/Japanese numeral, which does not reverse that (corrected 2026-10-21, #762: wrong — #671 put 二世 in `SUFFIX_WORDS` as the generational numeral (a `generation` reading since #737) and #707 let it peel glued, which reverses #679's refusal of it, as decisions/generational-words' 2026-10-21 #762 entry records). Excluded as the issue stated: 小 (a prefix and a common given-name character) and 冯/馮 for "von" (a top-30 surname). - 2026-10-17 #707 — the #671 words meet the glued bar, all but 部長/部长 (Derek, 2026-10-17). 議員/议员, 国会議員/國會議員, 太平紳士/太平绅士, 医学博士/醫學博士, 二世, 社長/課長 and 社长/课长 join `GLUED_HONORIFICS`; 部長/部长 stay spaced-only on the #671 argument, since 長 is a borne given name (Wikidata holds it as a Chinese given name, Q96240389; the Japanese reading rests on the argument) and 阿部長 can be 阿部 + 長 — the measurement below found no such bearer, and a zero that is a lower bound does not refute the argument. The measurement is `tools/name_usage.py`'s ends-with row, new in #707: a local index of every CJK label on eleven name-item classes (family name, given name and their Japanese, Chinese, Korean, male, female, unisex and undifferentiated-kanji subclasses; 215,798 index entries on 128,695 items, 215,696 distinct item-label pairs, fetched 2026-10-17 by the first index, whose character class stopped at the Basic Multilingual Plane; the rebuilt index that takes supplementary-plane Han too (215,820 entries, same day) gave the same answer for every word, 殿 and the boundary pairs below), matched locally, with the matching items' bearers counted live. Two questions per word: does any name item's label END in it, and does any person bear a family name ending in the word's head and a given name equal to its tail (the 阿部 + 長 shape)? Neither found anything for any spelling, with one exception that is a data error: the only item ending in 二世 is Q18339792, the given name Odo, whose Chinese label carries a regnal numeral (厄德二世), and its 109 bearers are Western people named Odo. Of the boundary shapes, only 二+世, 社+長, 社+长, 部+長 and 部+长 had a family name and a given-name item to pair at all, and each counted 0 people. What makes the zeros evidence: the index agreed exactly with a live `STRENDS` query on all 20 class × word cells it was cross-checked on (five subclasses × 殿, 子, 郎, 田, the largest 403 items); 子 found 439 items and 2,436 given-name bearers, 2,081 of them Japanese; and 田 + 太郎 counted 22 people across the boundary. What limits them: Wikidata's coverage, not the index's — 殿 ends 13 name items where a surname dictionary lists about 90, and Japanese family-name items number 4,919. Two shortcuts were measured and rejected before the index: `STRENDS` over every label of a name class timed out at WDQS's 60s with no person join, and WDQS full-text search found 4 family names ending in 殿, matching only where its tokenizer split the character out. Consequence: the snapshot moved four CLDR renderings, each losing its glued word to `suffix` (`王雅婷婷婷二世議員`, both spellings of `…小惠儀太平紳士`, `馮·陳·怡君·達印·二世醫學博士`), and `陳怡君二世`, which the case table and rules.md W2 pin; cldr_agreement records one suffix field newly agreeing (`二世 醫學博士`, where the peel takes 醫學博士 and the 二世 left behind is a whole-token suffix word) and none regressing. The peel still takes one word: `王雅婷婷婷二世議員` peels 議員 and keeps 二世 in the name. Recompute: `uv run python tools/name_usage.py 議員 议员 国会議員 國會議員 太平紳士 太平绅士 医学博士 醫學博士 二世 社長 部長 課長 社长 部长 课长`; the first run builds the index (33 minutes on 2026-10-17, resumable), and `--refresh-index` rebuilds it. ACCEPTED COST, found by the design-docs review and kept (Derek, 2026-10-17): the vetting asked whether a NAME ends in each word and never whether a longer, unlisted OFFICE word does, and common ones do — 衆議院議員 and 参議院議員, 副社長 and 代表取締役社長, 元議員 and 市议员, 副課長, 獣医学博士. The peel matches at the end of the token, spaced or glued, so each splits at the listed tail: `岸田文雄衆議院議員` reads family 岸田文雄衆議院, suffix 議員, and `田中 副社長` given 副, suffix 社長, where before the word stayed whole in the name. Neither reading is right; the compound's head is what the peel leaves behind. No corpus held a compound office word when the review found this, which is why the snapshot was silent; the two W2 examples now put `岸田文雄衆議院議員` and `田中 副社長` in corpus_rules.jsonl. Older entries carry the same cost unrecorded until now (`김민준의학박사` reads given 민준의학, suffix 박사). Listing a compound whole would peel it whole, longest first. - 2026-10-18 (vocabulary phase 1, decisions/vocabulary-model) — RECORDED HERE from the #307/#308 vetting note in nameparser/config/suffixes.py, which the move of the vocabulary to TOML retired; its per-word facts went to each word's TOML entry (`uv run python tools/vocab/query.py --issue 308`). Three class-level points had no other home. The spaced and glued bars are different, and an entry can be safe spaced and unsafe glued, never the other way round, because the spaced position is a token boundary its writer drew. A Han, kana or hangul entry is self-selecting, like the Korean surnames: it can only ever match CJK text. And kana is no license by itself: hiragana spells Japanese GIVEN names (高橋みなみ, pinned in the case table), which is why さん, さま, くん and ちゃん were vetted one at a time as never name-final rather than waved through as kana. Further spaced candidates (여사, 太太) wait on the same case-by-case argument. -- 2026-10-19 #760 — MOVED FROM the peel's code comment in `_script_segment.py` (`_peel_honorific_tail`), the facts the #312/#319 entries above and AGENTS.md's script paragraph did not hold (that paragraph keeps why the run predicate is not the token predicate's plural, and what `lenient_comma_suffixes=False` does here); parses re-run 2026-10-19. WHY ONLY THE FAMILY COMMA CROSSES: a comma is how a writer says which runs are the name, and a family comma is the one structure that splits the name itself across two of them (`김, 민준씨` is runs (0,) and (1,)), with the honorific as often glued to the given side as to the family. Every other structure keeps the whole name in segments[0], so the asymmetry is the point rather than an accident of two cases. "The rest is post-nominals" is true of the SUFFIX comma only: under NO_COMMA segment returns one run and a trailing post-nominal is INSIDE it (`김민준씨 Jr.` is one run of two tokens, family '김', suffix '씨 Jr.'), which is why the scan-back steps over such a token rather than never reaching it. WHY THE RUN PREDICATE IS ASKED: the structure says nothing about whether segments[1] is name text — segment hands every comma form over as FAMILY_COMMA (#613), and a one-word part turned a wholly suffix-shaped remainder into FAMILY_COMMA anyway (`田中さん, V.`). THE SCOPE HOLDS OFF A JUNK TAIL past the second run: `Dr 김민준씨, Jr., 박씨` is a postnominal part behind a whole name (`_postnominal_behind_a_whole_name`), so the scan stays in segments[0] and 박씨 in a third run is never reached (family '김', suffix '씨, Jr., 박씨'). AN EMPTY SECOND RUN stays in scope and contributes nothing: `is_wholly_suffix` is False on it by its own contract (v1 read `Doe,, Jr.` as a family comma), which is the reading the peel wants, flattening an empty run adding no site. FLATTENING SEGMENTS, NOT `state.tokens`, is load-bearing: extracted nickname and maiden content is in tokens but in no segment, and scanning tokens would put the site on a nickname (`김민준씨 (Jimmy)` would peel nothing); the case row `ko_honorific_glued_given_nickname` pins it (family '김', given '민준', suffix '씨', nickname 'Jimmy'). WHY THE PREDICATE IS CIRCULAR HERE ALONE, and so needs the second condition: not because `_comma.decide` asks it too (at group's head, after this stage), but because decide SPENDS the answer differently — it reads the run's shape and stops, the answer being the structure, while the peel reads the same shape and then decides whether to strip the very honorific that produced it. Declining a run that holds the only site does not fall back to another site; it loses the peel outright, and with it the given name, which lands in suffix (`이, J.씨` would read suffix 'J.씨'). DROPPED AS STALE, not moved: the comment said the declined credential lands "`PhD` a title"; it is a suffix (`田中さん, PhD` → suffix 'さん, PhD', rules/W3's example line), as AGENTS.md's script paragraph already says. +- 2026-10-19 #760 (the two facts it left to AGENTS.md's script paragraph moved into this file on 2026-10-27, below) — MOVED FROM the peel's code comment in `_script_segment.py` (`_peel_honorific_tail`), the facts the #312/#319 entries above and AGENTS.md's script paragraph did not hold (that paragraph keeps why the run predicate is not the token predicate's plural, and what `lenient_comma_suffixes=False` does here); parses re-run 2026-10-19. WHY ONLY THE FAMILY COMMA CROSSES: a comma is how a writer says which runs are the name, and a family comma is the one structure that splits the name itself across two of them (`김, 민준씨` is runs (0,) and (1,)), with the honorific as often glued to the given side as to the family. Every other structure keeps the whole name in segments[0], so the asymmetry is the point rather than an accident of two cases. "The rest is post-nominals" is true of the SUFFIX comma only: under NO_COMMA segment returns one run and a trailing post-nominal is INSIDE it (`김민준씨 Jr.` is one run of two tokens, family '김', suffix '씨 Jr.'), which is why the scan-back steps over such a token rather than never reaching it. WHY THE RUN PREDICATE IS ASKED: the structure says nothing about whether segments[1] is name text — segment hands every comma form over as FAMILY_COMMA (#613), and a one-word part turned a wholly suffix-shaped remainder into FAMILY_COMMA anyway (`田中さん, V.`). THE SCOPE HOLDS OFF A JUNK TAIL past the second run: `Dr 김민준씨, Jr., 박씨` is a postnominal part behind a whole name (`_postnominal_behind_a_whole_name`), so the scan stays in segments[0] and 박씨 in a third run is never reached (family '김', suffix '씨, Jr., 박씨'). AN EMPTY SECOND RUN stays in scope and contributes nothing: `is_wholly_suffix` is False on it by its own contract (v1 read `Doe,, Jr.` as a family comma), which is the reading the peel wants, flattening an empty run adding no site. FLATTENING SEGMENTS, NOT `state.tokens`, is load-bearing: extracted nickname and maiden content is in tokens but in no segment, and scanning tokens would put the site on a nickname (`김민준씨 (Jimmy)` would peel nothing); the case row `ko_honorific_glued_given_nickname` pins it (family '김', given '민준', suffix '씨', nickname 'Jimmy'). WHY THE PREDICATE IS CIRCULAR HERE ALONE, and so needs the second condition: not because `_comma.decide` asks it too (at group's head, after this stage), but because decide SPENDS the answer differently — it reads the run's shape and stops, the answer being the structure, while the peel reads the same shape and then decides whether to strip the very honorific that produced it. Declining a run that holds the only site does not fall back to another site; it loses the peel outright, and with it the given name, which lands in suffix (`이, J.씨` would read suffix 'J.씨'). DROPPED AS STALE, not moved: the comment said the declined credential lands "`PhD` a title"; it is a suffix (`田中さん, PhD` → suffix 'さん, PhD', rules/W3's example line), as AGENTS.md's script paragraph already says. +- 2026-10-27 — MOVED FROM AGENTS.md's script paragraph, which now keeps the doctrine and points at the rules; every parse below re-run 2026-10-27. THE REACH, by example: under a family comma the scan reads the first two runs and nothing past them, so `김, 민준 지훈씨` peels (family '김', given '민준', middle '지훈', suffix '씨') while `김, 민준, 지훈씨` (suffix '지훈씨') and `김,, 민준씨` (suffix '민준씨') do not, both landing the honorific in a third run. THE SECOND CONDITION, by example: the part before the comma must offer a site of its own before a suffix-shaped second run is declined, so `선생님, J.씨` is scanned anyway like `이, J.씨` (given 'J.', suffix '씨'), and `김민준씨, J.씨`, with a site on both sides, peels the person's own 씨 rather than the one behind the comma (family '김민준', suffix '씨, J.씨'). UNIFORM IN THE PEEL, not in where the credential lands, which is assign's question and differs by spelling: `田中さん, V.` reads given 'V.', while `田中さん, PhD` and `田中さん, Ph. D.` read suffix 'さん, PhD' and 'さん, Ph. D.'. THE RUN PREDICATE IS NOT THE TOKEN PREDICATE (`_is_post_nominal`) PLURALIZED: it says yes both to what the token predicate vetoes (`V.`, `V`, `I`, the class the defect was reported as) and to what the token predicate never sees, since `period_joined_vocab` and the delimiter routes are the run predicate's alone (`Msc.Ed.` and `J.씨` reach it that way, and `田中さん, Msc.Ed.` reads suffix 'さん, Msc.Ed.' with the rest). UNDER `Policy(lenient_comma_suffixes=False)` (rules/C14) the call drops to the strict token test, so `田中さん, V.`, `田中さん, V` and `田中さん, I` read their credential as name text and keep the pre-#319 answer (family '田中さん'), while `田中さん, Ph. D.` peels regardless, its merged `phd` passing the strict test. `田中さん, 太郎` is untouched (family '田中さん', given '太郎'), and not because of its comma: the honorific is not at the end of the name, 太郎 is. WHY THE PEEL ASKS `_comma.decide` (on a classified copy of the name as written, since #613's /simplify) RATHER THAN A HAND COPY: the copy drifted from decide four times in that PR's review (the ambiguous class's count, the numeral refusal, a suffix before the comma, the split `Ph. D.`), each drift dividing a name decide read as a family comma or leaving whole one it read as a suffix comma; decisions/C1's 2026-10-06 (Derek), #613 entry. Neither policy opt-out reaches the peel (`Policy(script_orders={}, segment_scripts=frozenset())` still reads `田中さん` suffix 'さん' and `김민준씨` suffix '씨'): the vocabulary carries the license, so the script switches have nothing to say about it. From 9f96dfa6487efb69874b49e86f03897ce13e6ea6 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Tue, 27 Oct 2026 01:13:00 -0700 Subject: [PATCH 3/6] Move the spelling sweep from AGENTS.md to a nameparser/AGENTS.md The spelling sweep (1,326 words) is needed by any change to a word class, vocabulary-only changes included, so it moves to a nested AGENTS.md at nameparser/ rather than under _pipeline/: reading a vocab TOML loads it too. The text moves unchanged in substance, split under a subheading per class (numerals, credentials and titles, separators, full stops inside a word, capitals, adjacent pairs). Root keeps the lead and a pointer; the Workflow paragraph, which says where a missed shape is recorded, and the two cases.py notes that cite the sweep now name the new file. Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 4 ++-- nameparser/AGENTS.md | 43 +++++++++++++++++++++++++++++++++++++++++++ tests/v2/cases.py | 6 +++--- 3 files changed, 48 insertions(+), 5 deletions(-) create mode 100644 nameparser/AGENTS.md diff --git a/AGENTS.md b/AGENTS.md index cfb87da18..676752232 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -12,7 +12,7 @@ After a review round, **review the fix commit too** — second-round passes on # **Check whether the PR description still needs updating before merging.** A review that moves behavior or retracts a measurement leaves it stale, and a stale body reads as authoritative. -**When a defect is found that an earlier check would have caught, write the check down where the next person will meet it, in the same PR.** A review finding, a failing sweep or a regression that arrived because nobody thought to test a shape is a lesson about the repository, not only about the change, and it is lost when the PR merges unless it lands somewhere a later session reads before making the same mistake. Ask, before merging: what should I have checked, and where would I have looked? Then put it there — a missed input shape in the sweep paragraphs under Architecture (comma shapes, then `name_order`, then spellings, each earned this way); a trap in a module's behavior under Gotchas; a design-review miss in docs/design/AGENTS.md's axes; a measurement or test-design trap in mechanisms.md's Verification shapes or Field notes; a reusable design move in mechanisms.md proper. Name the PR or issue that earned it and the input that showed it, so the rule can be checked against its case. #610 is the shape: its sweep test found that the maiden take missed a shape-only numeral, and the spelling sweep below records the class (`VI` beside `V` and `III`). This is how the file accumulates expertise rather than history — prefer amending an existing paragraph to adding a new one, and delete a lesson the code has since made impossible. +**When a defect is found that an earlier check would have caught, write the check down where the next person will meet it, in the same PR.** A review finding, a failing sweep or a regression that arrived because nobody thought to test a shape is a lesson about the repository, not only about the change, and it is lost when the PR merges unless it lands somewhere a later session reads before making the same mistake. Ask, before merging: what should I have checked, and where would I have looked? Then put it there — a missed input shape in the sweeps (comma shapes, then `name_order`, under Architecture; then spellings, in `nameparser/AGENTS.md`; each earned this way); a trap in a module's behavior under Gotchas; a design-review miss in docs/design/AGENTS.md's axes; a measurement or test-design trap in mechanisms.md's Verification shapes or Field notes; a reusable design move in mechanisms.md proper. Name the PR or issue that earned it and the input that showed it, so the rule can be checked against its case. #610 is the shape: its sweep test found that the maiden take missed a shape-only numeral, and the spelling sweep in `nameparser/AGENTS.md` records the class (`VI` beside `V` and `III`). This is how the file accumulates expertise rather than history — prefer amending an existing paragraph to adding a new one, and delete a lesson the code has since made impossible. **A scripted multi-edit that asserts as it goes discards everything when a late pattern misses.** Assert every pattern before applying any, then verify the edit landed — three times in one session a docs batch failed its last assertion, wrote nothing, and reported success, invisible because prose edits fail no test. @@ -195,7 +195,7 @@ Latin-script input is never touched by any of this: "Kim Min-jun" is genuinely o **Sweep the forms a change can reach: comma shapes, then `name_order`.** No comma, a FULL name before the comma, a ONE-WORD name before the comma and NOTHING before the comma are four paths, and the last two are the misses — #429, #430 and #432 are all the one-word path disagreeing with the full-name path on inputs the full-name path reads correctly, and #613's PR review found `, PhD` and `, Jr.` (an empty surname field, as CSV data writes it) read as given and title by a decision that returned early on an empty head, after two review rounds whose every grid had a name before the comma. Put an empty head and a nickname-only head (`(Bob), PhD`) in every comma grid. Count the one-word path the way C3 counts, which is neither tokens nor pieces: a particle run and the one word it attaches to are one word (#575), a connective join and a bound given-name pair count as their words (#613, so `Ortega y Gasset, Dr.` is a whole name), and a suffix beside the surname counts as no name word at all. #603's review found the suffix half (`Vega y Lopez Jr., PhD Jones` lost its family where `Vega y Lopez, PhD Jones` kept it); #613 then reversed the connective half, and both now read given 'Vega', middle 'y', family 'Lopez' — so test a particle surname, a connective surname and a surname with a suffix, each before a comma. For orders the sweep already exists (`tests/v2/test_cases.py` runs every row under all three) but carries one assertion, R2's family partition, so it checks nothing a new change moves; coverage there has been vacuous before (PR #394's review found the suite passed with `name_order` discarded from grouping). Parse your change's names in each comma shape and each order, and read the ones you did not predict. And read a generated grid's moves by SHAPE, not by corpus membership: #614's first design passed the suite, the five gates and a fingerprint of every corpus name, because its grid moves were sorted into 'corpus name moved' and 'garbage', and the review found the moves that mattered among the 'garbage' -- a bare salutation (`Mr. and Mrs.` lost its connective to the family name) and a name with one trailing word (`Freiherr von Berg X.Y.Z.`). Pull out the head-plus-one-word slice of the grid and a salutation with nothing after it, and read every move there. Two more were the second review's (#614): no grid put TWO words that are both a title and a particle at the head (`Freiherr St van Berg MA`, where an index into the pieces was applied to a merged view of them -- `TITLES ∩ particles_ambiguous` is `{freiherr, st}`, so put both in every head list, each in its bare and its period-marked spelling -- #637's grid had `St` and `Freiherr` alone and missed that `St.` and `Freiherr.` are declined by the bound join and then attached by P6), and the grids compared ambiguity KINDS, so a report telling the caller the wrong field (`Freiherr von Berg Ma`, 'read as a given name' of a word in the family) passed every pin. #701's review added one: put a title/suffix dual (`Ms`, `Sr.`, `Cpl`) SECOND in a title run, behind an ordinary title. A stop that asks only "is this word suffix vocabulary" passes every row that opens with the dual and fails `Rev. Sr. Smith, Mary`; it was found by comparing `T Smith, John` against `T John Smith` over every title word, and a comma-versus-comma-less comparison is the cheap check for any rule that moves a word between the two slots. Compare each report's `detail` against the field its tokens land in -- since #626 `test_a_report_names_the_field_its_word_lands_in` (tests/v2/test_cases.py) does it over every non-locale case row under all three orders (a row carrying its own policy only as declared), for the phrasings its `_FIELD_CLAIMS` lists -- so a row with the shape is the test only where the report's wording is listed: a new emitter's wording goes in that list, and segment's report on a part past the second comma names no field (#629). #626's own first grid then missed an emitter for want of one comma shape (`, Ma Dr.`: an empty head, an ambiguous word, a trailing title, which H1 moves to the family): a trailing title belongs in every empty-head comma grid too. And #620's: an exclusion list in code being replaced is a set of rules, not noise -- #614's merged copy excluded every word the trailing read weighs from its units, the first #620 prototype dropped the list as a symptom of the copy, and its review found the list had been keeping positional counting's rule that a member in front counts as a word (`Freiherr von Berg MA X.Y.Z.` lost its suffix). Before deleting one, mutate each entry out and read what moves. The converse holds for replacing a hand copy WITH the shared predicate: it is a behavior change wherever the two disagreed, and #735's /simplify shipped one as "no reading moves" because the suite and the snapshot were its only check (`St Sir de Mesnil` family-first, a token test replaced by P1's piece test); diff the two predicates' answers over a grid before calling the swap a refactor. -**Then sweep the spellings: a class reached by vocabulary is also reached by SHAPE.** A test that uses only the listed spelling of a word class walks only the vocabulary path, and the shape path is a separate branch that can be missing while every test passes. Roman numerals are the sharpest case: `ii`, `iii` and `iv` are suffix vocabulary; `i` and `v` are vocabulary too but initial-shaped, so `is_suffix_piece` vetoes them and only the numeral fork reads them; and `vi` through `x` are in no list at all, read by the fork's shape test alone (checked 2026-10-04 against `Lexicon.default()`) -- which stops at `x`: `_vocab._ROMAN` matches I to X, so `John Smith XI` reads `XI` as a name word (re-checked 2026-10-07, #614's /simplify, which found a docstring claiming "any length"). And the fork counts a ONE-letter numeral besides, since #718 (rules/S10): it is the suffix only where the name has a word to spare, so `John V` and `John Smith V` take different branches of one fork, a one-case name being counted like capitals whatever the letter -- since #721 a one-case one-letter numeral carries the `initial` tag its capital does, and since #723 every one-case bare letter does (rules/T4), and a bare letter in front of the numeral is initial-shaped as its capital is, so `john v` reads as `JOHN V` does over `test_a_one_case_name_reads_as_its_capitals`'s grid (comma shapes, orders, a maiden clause, a particle chain, a bare letter in front, a trailing title, a non-ASCII letter in front: 315 of its then 1,872 pairs disagreed at 2bfa9533, before #721, and none after). Its first version claimed "every shape" with no bare letter in front and no trailing title in the grid, and #721's reviews found both broken; a grid's claim is the grid -- put a head-plus-numeral row beside every words-to-spare row. So a change that handles `V` and `III` can still miss `VI` — #610 found exactly that, the maiden take's candidate loop admitting a shape-only numeral in one position where the peel reads it in two (`Jane Doe nee Smith VI Prof.` kept maiden `Smith VI` while `V` and `III` worked). #656's sweep found the same class from the other side: C12 takes the word ending the name before a comma as EVIDENCE, and its first draft asked the suffix-word tag, so `John Smith VI, AD` missed the rule `III` reached, and then counted the whole part, so `Smith VI, AD` fired with `VI` as its own name word and `Smith V, AD` with a `V` that S10 reads as the family. A rule that weighs a numeral as a generation must ask the predicate that READS it as one (the trailing fork, `_vocab.reads_as_generation`, and its words-to-spare count), not the vocabulary that lists some of its spellings. The same split runs through the credentials (`MA` listed, `X.Y.Z.` by the dotted shape, `XYZ` by the caps shape, `MD/PhD` by its listed chunks and `PU-PH` by the caps shape's joined spelling, #752, each with its joiner glued, spaced and glued to one side, which are three token streams -- and a new chunk reading inherits the old one's exceptions, which #752's first draft did not (`Jack I-V` lost its family to the listed numerals, and the fix's length test counted a chunk's period, so `Jack I.-V.` did too, found by the second review), and `Ph. D.` merged by group into one flagged piece the peel's walk never holds -- #603 found #602's run-start predicate refusing it too, so `Smith, John Ph. D. Jones` kept middle 'Jones' where `Smith, John PhD Jones` read suffix 'PhD Jones') and the titles (`Prof.` period-marked and read at the end of a name, `Prof` bare and a name word there, and `Mr. and Mrs.` a run P3 joins, carrying a piece tag a one-word vocabulary test never sees -- PR #735's review found C12's hand copy of C13's title test (C8's until its 2026-10-19 split) reading `John Smith Jr., Mr. and Mrs. AD` as all suffix; ask C13's predicates in C13's order, `not is_suffix_piece` then `is_title_piece`, not the token tags, since `is_title_piece` alone reads the duals `Ms` and `Sr.` as titles). When a change touches a word class, test one spelling from each path. That holds for a word a rule only asks ABOUT, too: #810's first draft asked whether any word after a bracketed credential lacked the suffix TAG, and kept `John Smith (MA) VI`, `(MA) Ph. D.` and `(MA) Dr.` nicknames while `(MA) III` and `(MA) PhD` escaped; asking the role the parse read for the following word answers for every spelling. The same holds for a separator's SPACING: a comma (or any punctuation) can stand alone, glue to the word before it or glue to the word after, and each is a different token stream. #653's first fix swept `(née, Jones)` and missed `(née , Jones)`, where the kept comma became a word after the marker and M3 read the clause as maiden ', Jones'; its review found it. Put all three spacings in any grid around a separator. And a separator glued to a word hides the word from every vocabulary test, not only the one being changed: the semicolon delimiter's first draft read a glued word's stem in C1's run test alone and put 365 of ClinicalTrials.gov's reversed names right, missing the ones C13, C4 and C6 read off a word's TAGS (`MBBS;`, `MA;`); tagging the stem put 412 right, and reading it again in the comma decision's shape tests, which read a word's text rather than its tags (`MBBS;`), 431 (decisions/C10, 2026-10-24). A rule over a glued spelling reads the stem everywhere the word is read, its tags and every test of its text, and is swept against the space twin -- with a declared word delimiter too, which a substring test finds inside other words ('Stand' under ' and ', the review of the same change). A full stop INSIDE a word has four spellings of its own: initials run together (`J.R.R.`), joined words each closed by a stop (`Lt.Gov.`, `Msc.Ed.`), a joined word or name closed by none (`St.John`, `J.Smith`), and an abbreviation's own ending, closed or not (Italian `Sig.ra`, Latvian `Dr.philol.`). Shape tells the last from none of the others, so a rule over the shape decides which it costs; #838 found the ending only through the snapshot's CLDR row `Dut.ra Prof.ra`, and its first narrowing traded `St.John` for `Sig.ra` while the snapshot listed the `Dr.philol.` move among moves read as intended, until the design-docs review named it. Initials run together are also how some SURNAMES are written (Nepali `B.K.`, `K.C.`), and a caseless script writes its abbreviations, degrees included, as dotted letters (Thai `พ.บ.`): #864 found S3's position reading taking the first from 2.4.0 on, visible only once #866's Nepal candidates gave the corpora a population that writes them, and its first fix moved CLDR's Thai degrees into the name. A rule over dotted letters reads both before it ships. Capitals between two name words have the same second reading: some writers put their own initials there (`Maria Ana MA Silva`), so a rule that reads a word's capitals as a credential reads the initials too; #785 met it only in ctgov's field moves, no grid having held the shape, and Derek chose the initials (rules/S8: a name word behind capitals keeps them a name word where no family comma stands). A predicate over ADJACENT PAIRS of words meets two shapes no grid of one word holds: an overlapping pair and a joining word beside an initial. #752's split-credential test (phase 2e) read `Dr. Ph.` in `Jane Doe Dr. Ph. D.` as the listed `drph`, leaving `D.` a word of its own (found by the maiden take's candidate grid), and a particle beside a surname written short as the listed `ded` (`DE D JUNIOR`, found only in tse-donors' readings), so put a title, a particle and a connective beside each pair spelling, and a pair overlapping another, in the grid. +**Then sweep the spellings: a class reached by vocabulary is also reached by SHAPE.** A test that uses only the listed spelling of a word class walks only the vocabulary path, and the shape path is a separate branch that can be missing while every test passes, so when a change touches a word class, test one spelling from each path. The classes where a path was missed (roman numerals, credentials and titles, a separator's spacing, a full stop inside a word, capitals between name words, adjacent pairs), each with the PR that found it, are in `nameparser/AGENTS.md`, which loads when a session reads anything under `nameparser/`, the vocabulary TOML included. **And the encoding: a letter class written `[^\W\d_]` or tested with `isalpha()` is blind to a combining mark.** Python's `\w` and `isalpha()` do not match Mn, so a letter written as base plus accent (NFD) ends a word early; the vocabulary path is immune because `_normalize` NFC-folds, the shape path is not. #652: `Dr. inž. Janez Novak` read `inž.` as the given name when the ž was decomposed, and `Janez Novak, ŽŠČ` reversed, in two families that share the blind spot — the letter-shape regexes (`period_abbreviation`, `_DOTTED`, `_CHUNKED`, `_PAIRED_INITIALS`), now spelled with `_vocab._LETTER` (a letter plus the general combining blocks; `test_regex_sync` compares the two `_MARKS` strings and pins `period_abbreviation` against config's copy), and the caps predicates, which NFC-compose a non-ASCII word before `isalpha()` (`is_caps_word`, behind C-level `isalpha()`/`is_normalized` tests, so already-composed text pays no frame — measured 2026-10-12, `Smith, José` was 193 frames before and after; the first version called it for every non-ASCII word and cost one frame each, and no guard saw it because the reference name is ASCII). A width test must count letters, not code points (`composed_len`): admitting a decomposed word to the caps set put `John Smith, MÁ` on the pair exception's wrong side, and the review's grid found it where the rows (all three letters long) did not — put a two-letter word in every caps grid. The initial shape (`_INITIAL`, `\w\.`, v1-verbatim and public) is not widened but asked of the COMPOSED word, in `is_initial_shaped` and in `_render`'s two copies; and the single-letter connective tests count composed letters, since a decomposed `й` is two code points (#665, where `É. y É. Smith` and `John й Smith` read differently decomposed). Compose the WHOLE predicate, not one half: `is_initial` is a shape ANDed with the #320 script veto, and composing the shape alone made a decomposed `씨.` an initial, since decomposed hangul is conjoining jamo outside every script range (#665's first version; `in_initialless_script` composes too now). Not covered, deliberately: the marks only some scripts write (Indic vowel signs, Arabic and Hebrew points), declined in decisions/nfd-shapes because widening the class would move readings nobody had measured; rules/H2's Accepted clause records the one consequence it names, that a period-marked abbreviation in Bengali or Devanagari never reads as a title by shape. The guard is `test_a_decomposed_name_renders_as_its_composed_twin` over the CLDR corpus plus `test_a_decomposed_abbreviation_reads_as_its_composed_twin` (rows × three orders × EVERYWHERE); the corpus guard reaches a shape only if some corpus name does, which is how the caps shape was missed until review, so when adding a shape rule add a row there. Four of #652's rows and two of #665's pass with every fix reverted (measured) and are parity pins, and the test compares ambiguity kinds and `initials()` too, since #665's `John й Smith` differed in its report; `_DOTTED` has no observable input and is pinned by a direct call. diff --git a/nameparser/AGENTS.md b/nameparser/AGENTS.md new file mode 100644 index 000000000..488a885fd --- /dev/null +++ b/nameparser/AGENTS.md @@ -0,0 +1,43 @@ +# Working in nameparser/ + +Loaded when a session reads or edits anything under `nameparser/`: the pipeline, the vocabulary TOML, the config shim. The root AGENTS.md carries the sweeps every behavior change runs (comma shapes, then `name_order`, under Architecture) and the 2.0 conventions; `nameparser/_pipeline/AGENTS.md` carries the ambiguity-report rules. This file holds the third sweep, which a vocabulary-only change needs as much as a code change does. + +## Sweep the spellings: a class reached by vocabulary is also reached by SHAPE + +A test that uses only the listed spelling of a word class walks only the vocabulary path, and the shape path is a separate branch that can be missing while every test passes. When a change touches a word class, test one spelling from each path. Each subsection below is a class where a path was missed, with the PR that found it; add a new class here when a review finds one (root AGENTS.md's Workflow section says how). + +### Roman numerals + +Roman numerals are the sharpest case: `ii`, `iii` and `iv` are suffix vocabulary; `i` and `v` are vocabulary too but initial-shaped, so `is_suffix_piece` vetoes them and only the numeral fork reads them; and `vi` through `x` are in no list at all, read by the fork's shape test alone (checked 2026-10-04 against `Lexicon.default()`) -- which stops at `x`: `_vocab._ROMAN` matches I to X, so `John Smith XI` reads `XI` as a name word (re-checked 2026-10-07, #614's /simplify, which found a docstring claiming "any length"). + +And the fork counts a ONE-letter numeral besides, since #718 (rules/S10): it is the suffix only where the name has a word to spare, so `John V` and `John Smith V` take different branches of one fork, a one-case name being counted like capitals whatever the letter -- since #721 a one-case one-letter numeral carries the `initial` tag its capital does, and since #723 every one-case bare letter does (rules/T4), and a bare letter in front of the numeral is initial-shaped as its capital is, so `john v` reads as `JOHN V` does over `test_a_one_case_name_reads_as_its_capitals`'s grid (comma shapes, orders, a maiden clause, a particle chain, a bare letter in front, a trailing title, a non-ASCII letter in front: 315 of its then 1,872 pairs disagreed at 2bfa9533, before #721, and none after). Its first version claimed "every shape" with no bare letter in front and no trailing title in the grid, and #721's reviews found both broken; a grid's claim is the grid -- put a head-plus-numeral row beside every words-to-spare row. + +So a change that handles `V` and `III` can still miss `VI` — #610 found exactly that, the maiden take's candidate loop admitting a shape-only numeral in one position where the peel reads it in two (`Jane Doe nee Smith VI Prof.` kept maiden `Smith VI` while `V` and `III` worked). #656's sweep found the same class from the other side: C12 takes the word ending the name before a comma as EVIDENCE, and its first draft asked the suffix-word tag, so `John Smith VI, AD` missed the rule `III` reached, and then counted the whole part, so `Smith VI, AD` fired with `VI` as its own name word and `Smith V, AD` with a `V` that S10 reads as the family. A rule that weighs a numeral as a generation must ask the predicate that READS it as one (the trailing fork, `_vocab.reads_as_generation`, and its words-to-spare count), not the vocabulary that lists some of its spellings. + +### Credentials and titles + +The same split runs through the credentials (`MA` listed, `X.Y.Z.` by the dotted shape, `XYZ` by the caps shape, `MD/PhD` by its listed chunks and `PU-PH` by the caps shape's joined spelling, #752, each with its joiner glued, spaced and glued to one side, which are three token streams -- and a new chunk reading inherits the old one's exceptions, which #752's first draft did not (`Jack I-V` lost its family to the listed numerals, and the fix's length test counted a chunk's period, so `Jack I.-V.` did too, found by the second review), and `Ph. D.` merged by group into one flagged piece the peel's walk never holds -- #603 found #602's run-start predicate refusing it too, so `Smith, John Ph. D. Jones` kept middle 'Jones' where `Smith, John PhD Jones` read suffix 'PhD Jones') and the titles (`Prof.` period-marked and read at the end of a name, `Prof` bare and a name word there, and `Mr. and Mrs.` a run P3 joins, carrying a piece tag a one-word vocabulary test never sees -- PR #735's review found C12's hand copy of C13's title test (C8's until its 2026-10-19 split) reading `John Smith Jr., Mr. and Mrs. AD` as all suffix; ask C13's predicates in C13's order, `not is_suffix_piece` then `is_title_piece`, not the token tags, since `is_title_piece` alone reads the duals `Ms` and `Sr.` as titles). + +### A word a rule only asks ABOUT + +That holds for a word a rule only asks ABOUT, too: #810's first draft asked whether any word after a bracketed credential lacked the suffix TAG, and kept `John Smith (MA) VI`, `(MA) Ph. D.` and `(MA) Dr.` nicknames while `(MA) III` and `(MA) PhD` escaped; asking the role the parse read for the following word answers for every spelling. + +### A separator's spacing + +The same holds for a separator's SPACING: a comma (or any punctuation) can stand alone, glue to the word before it or glue to the word after, and each is a different token stream. #653's first fix swept `(née, Jones)` and missed `(née , Jones)`, where the kept comma became a word after the marker and M3 read the clause as maiden ', Jones'; its review found it. Put all three spacings in any grid around a separator. + +And a separator glued to a word hides the word from every vocabulary test, not only the one being changed: the semicolon delimiter's first draft read a glued word's stem in C1's run test alone and put 365 of ClinicalTrials.gov's reversed names right, missing the ones C13, C4 and C6 read off a word's TAGS (`MBBS;`, `MA;`); tagging the stem put 412 right, and reading it again in the comma decision's shape tests, which read a word's text rather than its tags (`MBBS;`), 431 (decisions/C10, 2026-10-24). A rule over a glued spelling reads the stem everywhere the word is read, its tags and every test of its text, and is swept against the space twin -- with a declared word delimiter too, which a substring test finds inside other words ('Stand' under ' and ', the review of the same change). + +### A full stop inside a word + +A full stop INSIDE a word has four spellings of its own: initials run together (`J.R.R.`), joined words each closed by a stop (`Lt.Gov.`, `Msc.Ed.`), a joined word or name closed by none (`St.John`, `J.Smith`), and an abbreviation's own ending, closed or not (Italian `Sig.ra`, Latvian `Dr.philol.`). Shape tells the last from none of the others, so a rule over the shape decides which it costs; #838 found the ending only through the snapshot's CLDR row `Dut.ra Prof.ra`, and its first narrowing traded `St.John` for `Sig.ra` while the snapshot listed the `Dr.philol.` move among moves read as intended, until the design-docs review named it. + +Initials run together are also how some SURNAMES are written (Nepali `B.K.`, `K.C.`), and a caseless script writes its abbreviations, degrees included, as dotted letters (Thai `พ.บ.`): #864 found S3's position reading taking the first from 2.4.0 on, visible only once #866's Nepal candidates gave the corpora a population that writes them, and its first fix moved CLDR's Thai degrees into the name. A rule over dotted letters reads both before it ships. + +### Capitals between two name words + +Capitals between two name words have the same second reading: some writers put their own initials there (`Maria Ana MA Silva`), so a rule that reads a word's capitals as a credential reads the initials too; #785 met it only in ctgov's field moves, no grid having held the shape, and Derek chose the initials (rules/S8: a name word behind capitals keeps them a name word where no family comma stands). + +### Adjacent pairs of words + +A predicate over ADJACENT PAIRS of words meets two shapes no grid of one word holds: an overlapping pair and a joining word beside an initial. #752's split-credential test (phase 2e) read `Dr. Ph.` in `Jane Doe Dr. Ph. D.` as the listed `drph`, leaving `D.` a word of its own (found by the maiden take's candidate grid), and a particle beside a surname written short as the listed `ded` (`DE D JUNIOR`, found only in tse-donors' readings), so put a title, a particle and a connective beside each pair spelling, and a pair overlapping another, in the grid. diff --git a/tests/v2/cases.py b/tests/v2/cases.py index 7a0fc7a71..3867762be 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -7869,7 +7869,7 @@ def _check_cjk_shape_purity(self) -> None: "one token, so 'Ph. D.' started no run where 'PhD' did, " "and this read middle 'Jones', suffix 'Ph. D.' (2.3.0 " "too). Found by #603, whose opened part asks the same " - "predicate (AGENTS.md's spelling sweep)"), + "predicate (nameparser/AGENTS.md's spelling sweep)"), Case("a_split_credential_opens_the_part_as_the_joined_one_does", "John Smith, Ph. D. and Mary Jones", {"given": "John", "family": "Smith", @@ -12995,8 +12995,8 @@ def _check_cjk_shape_purity(self) -> None: "shape, so S2's count keeps it out of the particle " "chain's unit; folded in with the plain words it was " "family 'de la Vega VI' (#614's review, the #610 class " - "AGENTS.md's spelling sweep records) (1.4.0 family 'de la Vega" - " VI', bisected to the #424 chain stop)"), + "nameparser/AGENTS.md's spelling sweep records) (1.4.0 family " + "'de la Vega VI', bisected to the #424 chain stop)"), Case("a_particle_takes_the_dual_behind_it_at_the_head", "Freiherr von vd", {"title": "Freiherr", "family": "von vd"}, From ed51e5306aeccdd10525e5dbd37cccd956de323c Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Tue, 27 Oct 2026 01:19:48 -0700 Subject: [PATCH 4/6] Fix what the design-docs review found in the relocation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Restore #337's `Policy(segment_scripts=())` in the validation bullet: the frozenset() replacement meant for the new opt-out example also hit this history line, which records () as the spelling that shipped wrong. - Retarget the stale pointers: decisions/C14 (honorific-peel paragraph), _pieces.py and cases.py:2859 (the spelling sweep's #603 example and separator spacing). - Root index: restore the katakana reason (its script settles nothing, not "foreign"), cite W1's interpunct Except for "never segments", name decisions/T3 as the nakaguro's history, keep 高橋+一郎. - decisions/W2: the #760 entry's last pointer, and the facts the cut left only in usage.rst or nowhere (#319's failure mechanism, the pre-#319 readings, why 博士/殿/君 are spaced-only), re-run 2026-10-27. - nameparser/AGENTS.md: give the moved "That holds" sentence an antecedent. Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 12 ++++++------ ...trict-mode-a-suffix-word-written-as-an-initial.md | 2 +- docs/design/decisions/W2-glued-honorific-peel.md | 4 ++-- nameparser/AGENTS.md | 2 +- nameparser/_pipeline/_pieces.py | 4 ++-- tests/v2/cases.py | 3 ++- 6 files changed, 14 insertions(+), 13 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 676752232..0d517e86a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -180,14 +180,14 @@ logging.getLogger('HumanName').setLevel(logging.DEBUG) The library has two layers: data — the per-word TOML in `nameparser/vocab/`, compiled into the `nameparser/config/` modules — and logic, the 2.0 pipeline in the underscore-private `nameparser/_*.py` modules (see 2.0 API modules below). `nameparser/parser.py` and `nameparser/config/__init__.py` are only re-exports of the v1-compat facade (`_facade.py`, `_config_shim.py`); the v1 parser itself was deleted at the M11 swap and is readable as `git show 2d5d8c2:nameparser/parser.py`. -**Design philosophy — positional and language-agnostic.** The parser assigns parts by *position* plus small sets of words that join to neighbors; it never detects language. A name's language can't be reliably inferred from Latin-script transliteration ("Ali" is Arabic or Italian; "Van"/"Della"/"Bin" are first names in some cultures, particles in others), so language-specific rules belong in opt-in `Constants` config, never global defaults. Many "wrong for language X" reports (#133, #150, #130, #85, #103, #146, #83) are irreducible ambiguities — e.g. `de Mesnil` (want last name) vs `Van Johnson` (want first name) are the same `[prefix][word]` shape. Before adding a rule, confirm it doesn't break the opposite case (run the full suite — Portuguese and "Van Johnson" tests are the usual canaries). **The one scoped exception (2.1, #271/#272): script-conditional behavior is permitted exactly where the SCRIPT ITSELF — not statistics about it — determines the convention.** The never-detect-language rule is about Latin *transliteration*, where the signal is destroyed; native script is a different question, answered per behavior rather than per script. What it licenses today, each stated in its rule, with the history in the decisions file of the same ID: +**Design philosophy — positional and language-agnostic.** The parser assigns parts by *position* plus small sets of words that join to neighbors; it never detects language. A name's language can't be reliably inferred from Latin-script transliteration ("Ali" is Arabic or Italian; "Van"/"Della"/"Bin" are first names in some cultures, particles in others), so language-specific rules belong in opt-in `Constants` config, never global defaults. Many "wrong for language X" reports (#133, #150, #130, #85, #103, #146, #83) are irreducible ambiguities — e.g. `de Mesnil` (want last name) vs `Van Johnson` (want first name) are the same `[prefix][word]` shape. Before adding a rule, confirm it doesn't break the opposite case (run the full suite — Portuguese and "Van Johnson" tests are the usual canaries). **The one scoped exception (2.1, #271/#272): script-conditional behavior is permitted exactly where the SCRIPT ITSELF — not statistics about it — determines the convention.** The never-detect-language rule is about Latin *transliteration*, where the signal is destroyed; native script is a different question, answered per behavior rather than per script. What it licenses today, each stated in its rule, with the history in the decisions file named beside it or, where none is, the one of the same ID: -- Wholly-Han, wholly-Hangul and kana-licensed names read family-first (`Policy.script_orders`): zh and ja both write family-first in native script, so order cannot be misread though the language is unknowable. Hiragana licenses Japanese (#272), while pure katakana, full-width or halfwidth, keeps the positional default, being how foreign names are transcribed (#594) — rules/W4. +- Wholly-Han, wholly-Hangul and kana-licensed names read family-first (`Policy.script_orders`): zh and ja both write family-first in native script, so order cannot be misread though the language is unknowable. Hiragana licenses Japanese (#272), while pure katakana, full-width or halfwidth, keeps the positional default: it may be a transcribed foreign name already in its source order or a Japanese reading written family-first, so its script settles nothing and a caller who knows opts in through `script_orders` (#594) — rules/W4. - Unspaced hangul splits into surname + given name on the Korean census surnames shipped as default vocabulary: nothing but Korean is written in hangul, and the surnames are a closed set — rules/W1. -- The 间隔号 U+00B7 between classified characters marks a transcription, so the name keeps its declared order and never segments; it divides only when flanked, being also the Catalan punt volat (`Gal·la`) — rules/T3 and W4 (#298). -- The nakaguro (U+30FB and halfwidth U+FF65) separates words unconditionally, in tokenize, so neither policy opt-out (`script_orders={}`, `segment_scripts=frozenset()`) reaches it: its codepoints are CJK-only — rules/T2. +- The 间隔号 U+00B7 between classified characters marks a transcription, so the name keeps its declared order (rules/W4) and never segments (rules/W1's interpunct Except); it divides only when flanked, being also the Catalan punt volat (`Gal·la`) — rules/T3 (#298). +- The nakaguro (U+30FB and halfwidth U+FF65) separates words unconditionally, in tokenize, so neither policy opt-out (`script_orders={}`, `segment_scripts=frozenset()`) reaches it: its codepoints are CJK-only — rules/T2, history in decisions/T3. - A listed CJK honorific glued to the end of a name splits off, but only an entry that can never end a name (#308): the same not-a-guess standard applied to WORDS rather than a script, the vocabulary carrying the license itself, so neither opt-out reaches it either. Its reach across a family comma is tolerated rather than contract — rules/W2, rules/W3, decisions/W2. -- Han segmentation stays OPT-IN (`locales.ZH`; `locales.JA` with a `Parser(segmenter=...)`): a Chinese surname list corrupts Japanese kanji names (高 is a common Chinese surname, so 高橋一郎 would split 高+橋一郎), and no surname list divides a kanji name at all — rules/W1. +- Han segmentation stays OPT-IN (`locales.ZH`; `locales.JA` with a `Parser(segmenter=...)`): a Chinese surname list corrupts Japanese kanji names (高 is a common Chinese surname, so 高橋一郎 would split 高+橋一郎 where the correct reading is 高橋+一郎), and no surname list divides a kanji name at all — rules/W1. Latin-script input is never touched by any of this: "Kim Min-jun" is genuinely order-ambiguous and stays governed by `name_order` and opt-in packs. Before adding a script-conditional rule, work out which of the three it is — certain, certain for this one behavior only, or a statistical guess wearing a script's clothes. @@ -231,7 +231,7 @@ The 2.0 rewrite lands as underscore-private modules alongside the v1 code. These - **Layering is enforced by `tests/v2/test_layering.py`** (exact-module matching; `if TYPE_CHECKING:` imports don't count): `_types` imports nothing internal at module level — its rendering delegates import `_render` at call time; `_lexicon` and `_policy` sit above `_types` independently (`_lexicon` may import `nameparser.config.*` DATA modules only — vocabulary is single-sourced from the v1 data modules through 2.x, e.g. `config/maiden_markers.py`); `_locale` sits on `_lexicon`+`_policy` (plus `_types` for the shared pickle mixin); `_render` imports `_types` and `_lexicon` (for `Lexicon.default()` and `_normalize`); `_pipeline/*` imports `_types`+`_lexicon`+`_policy` plus in-package `_pipeline` helpers; `_parser` sits on everything except `_render`; the facade layer (`_facade`, `_config_shim`, `parser`, `config/__init__`, `__main__`) may import anything public plus `_render`; locale pack modules (`locales/*.py`) import `_locale`/`_lexicon`/`_policy`/`_types` only (`_types` joined the list in #272 — `locales/ja.py`'s segmenter factory constructs a `Segmentation`), and the `locales/__init__` additionally lazy-imports its packs (PEP 562). Extend the test's `ALLOWED` table when adding a module. - **Canonical field order** — the seven roles in `Role` enum declaration order, defined once and derived everywhere (properties, `as_dict`, reprs, `comparison_key`). Never restate the order literally. `Role` is a `StrEnum`: members compare as their field-name strings, and `tokens_for()` coerces strings via `_coerce_enum`. - **Method organization**, fixed section order in every class: fields + `__post_init__` validation → alternative constructors → dunders (construction/equality → protocol → operators) → properties → public methods by concern (access → editing → comparison → rendering delegates) → private helpers last, except a helper serving exactly one section may sit at that section's head. Sanctioned deviation, facade layer only: `HumanName` and the shim `Constants` organize by v1 concern groups (`# -- render defaults --`, `# -- config / parsing --`, `# -- fields --`, ..., dunders and pickle last) — the classes mirror v1's own surface and die in 3.0; the canonical order still binds every core type. -- **Validation is eager and fail-loud**: every `raise` states the offending value, the expected form, and the fix. Exception taxonomy: wrong type — including wrong element type inside a collection, bare `str` where an iterable of strings is expected, or a `Mapping` where a plain iterable is expected — raises `TypeError`; well-typed but unacceptable values raise `ValueError`; failed enum lookups stay `ValueError` for any input (stdlib `EnumType` precedent). **When the message hands the reader code to paste, that code has to survive a type checker, and then RUN** — nameparser ships `py.typed`, and a placeholder can type-check and still raise: the 2.4 release sweep's first fix to the `capitalization_exceptions` hint offered `('key', 'Value')`, which passes mypy and fails #459's mask check, so `test_add_capitalization_exceptions_raises_pointing_at_replace` now executes the offered code. #337's segmenterless warning offered `Policy(segment_scripts=frozenset())`, an `arg-type` error, because these fields are annotated with what they STORE rather than everything the constructor accepts. Prefer the `frozenset()` / `()` spellings in messages and docstrings, and pin the offered spelling in a test — the warning tests matched on `ja_segmenter` and never checked the actionable half of the message. **A warning emitted in `Parser.__post_init__` needs `parser_for` to re-emit it from its own frame** (the `catch_warnings(record=True)` block at its return): `__post_init__`'s `stacklevel` is sized for direct `Parser(...)` construction, and through `parser_for`'s extra frame the default one-line rendering attributes the warning to the library's own `return Parser(...)` — the exact call the message tells the user to change becomes invisible. No single stacklevel serves both entry points; a new construction warning gets the re-emission for free, but a new CONSTRUCTION SITE for `Parser` inside this package needs its own re-emission or its callers get library-attributed warnings (#337 review). +- **Validation is eager and fail-loud**: every `raise` states the offending value, the expected form, and the fix. Exception taxonomy: wrong type — including wrong element type inside a collection, bare `str` where an iterable of strings is expected, or a `Mapping` where a plain iterable is expected — raises `TypeError`; well-typed but unacceptable values raise `ValueError`; failed enum lookups stay `ValueError` for any input (stdlib `EnumType` precedent). **When the message hands the reader code to paste, that code has to survive a type checker, and then RUN** — nameparser ships `py.typed`, and a placeholder can type-check and still raise: the 2.4 release sweep's first fix to the `capitalization_exceptions` hint offered `('key', 'Value')`, which passes mypy and fails #459's mask check, so `test_add_capitalization_exceptions_raises_pointing_at_replace` now executes the offered code. #337's segmenterless warning offered `Policy(segment_scripts=())`, an `arg-type` error, because these fields are annotated with what they STORE rather than everything the constructor accepts. Prefer the `frozenset()` / `()` spellings in messages and docstrings, and pin the offered spelling in a test — the warning tests matched on `ja_segmenter` and never checked the actionable half of the message. **A warning emitted in `Parser.__post_init__` needs `parser_for` to re-emit it from its own frame** (the `catch_warnings(record=True)` block at its return): `__post_init__`'s `stacklevel` is sized for direct `Parser(...)` construction, and through `parser_for`'s extra frame the default one-line rendering attributes the warning to the library's own `return Parser(...)` — the exact call the message tells the user to change becomes invisible. No single stacklevel serves both entry points; a new construction warning gets the re-emission for free, but a new CONSTRUCTION SITE for `Parser` inside this package needs its own re-emission or its callers get library-attributed warnings (#337 review). - **Guard, hint, and emit for the WHOLE family, and parametrize the test over it**: a check added to one member of a set belongs on all of it, and the test must sweep the family, not one example. This session shipped `_reject_str_and_mapping` on `Policy` but not `PolicyPatch`, the bytes decode hint on three of five config entry points, and a regex-sync roster missing four of its copies — each a separate follow-up bug that a `{class} × {field} × {bad-value}` parametrization would have caught and a per-example test hid. When you find you're guarding member N, grep for the other members first. - **Ambiguities are emitted at the DECISION site**: an `Ambiguity` records a fork the parse had to call, not a token that sits in an ambiguous vocabulary, so emit where the branch is taken, in every stage that takes one, and name the reading not taken. The contract is mechanisms.md#AMBIGUITY-AT-THE-DECISION-SITE; the working rules (branches that change nothing, the field a report names, the stages that decide `PARTICLE_OR_GIVEN`, `alternative`, and how to test an emitter) are in `nameparser/_pipeline/AGENTS.md`, which loads when a session reads a pipeline stage. - **A kind is worth adding only if a reader would hesitate too**: the test is not "does the code take a branch" but whether a person reading that input would genuinely be unsure. "Smith, John V" reads as a middle initial to anyone -- the comma settles it -- so reporting it would be noise that teaches callers to ignore the field, which costs more than the missing report. Reachability of the second branch is necessary, not sufficient. Prefer leaving a fork silent and documenting the omission over emitting on input nobody finds ambiguous. diff --git a/docs/design/decisions/C14-strict-mode-a-suffix-word-written-as-an-initial.md b/docs/design/decisions/C14-strict-mode-a-suffix-word-written-as-an-initial.md index 63b9173d8..181464c86 100644 --- a/docs/design/decisions/C14-strict-mode-a-suffix-word-written-as-an-initial.md +++ b/docs/design/decisions/C14-strict-mode-a-suffix-word-written-as-an-initial.md @@ -8,4 +8,4 @@ Open: none in this file; the switch's reach into the count before the comma (`Sm ## Log -- 2026-10-19 (Derek), #761 — STRICT MODE IS A RULE OF ITS OWN. The strict comma-suffix switch (`Policy(lenient_comma_suffixes=False)`) was a sentence of rules/C1 and a clause of rules/C13, and rules/C9 and rules/T4 restated it by pointing at C1. It reads nothing in the written name, only the caller's configuration, so under docs/design/AGENTS.md's one rule per piece of evidence it is a rule rather than a clause of the hub; split out while the C section moved to the skeleton format. Moved here unchanged in meaning: C1's sentence and its two strict examples (`John Smith, V.`, `john smith, v`), C13's `Smith, PSM I.` exception, which becomes this rule's boundary, and C9's clause that the run ends at an initial-shaped word. C1, C9, C13 and T4 now cite this rule, and so does W3 for the honorific split-off, whose comma-side question the switch also reaches (`田中さん, V.` keeps さん glued under it; root AGENTS.md's honorific-peel paragraph and the case row `ja_honorific_glued_family_comma_strict_knob`). The switch's own history stays where it was decided: decisions/C1's 2026-07 and 2026-07-12/13 entries (its blast radius, the one-letter roman numerals, and its date) and decisions/C13's #603 STRICT MODE paragraph (the opened run takes the vetoed word). The given part's walk after a family comma is strict-blind, measured on d7e1d47a (`Smith, John V`, `Smith, John PhD I` and `Smith, John Jr. V.` read the same under the switch as without it; C14's `Smith, John V` boundary pins the first), so this rule scopes the veto to the comma's decision and to a part after the comma that is nothing but suffix words. decisions/C9's #430/#432 sentence that "the strict knob is consulted" and "the run ends at the numeral" means that credential-run continuation, not the given part's walk. Two drafts of this rule overstated the reach (a run "inside a comma part", then "in the part after the comma", which in the listing form is the given part too); the PR reviews found both. One reach of the switch no rule states, the count before the comma reading the vetoed word as a name word (`Smith V., Ma` → family 'V.'), stays recorded as decisions/S4's 2026-09-18 rough edge. +- 2026-10-19 (Derek), #761 — STRICT MODE IS A RULE OF ITS OWN. The strict comma-suffix switch (`Policy(lenient_comma_suffixes=False)`) was a sentence of rules/C1 and a clause of rules/C13, and rules/C9 and rules/T4 restated it by pointing at C1. It reads nothing in the written name, only the caller's configuration, so under docs/design/AGENTS.md's one rule per piece of evidence it is a rule rather than a clause of the hub; split out while the C section moved to the skeleton format. Moved here unchanged in meaning: C1's sentence and its two strict examples (`John Smith, V.`, `john smith, v`), C13's `Smith, PSM I.` exception, which becomes this rule's boundary, and C9's clause that the run ends at an initial-shaped word. C1, C9, C13 and T4 now cite this rule, and so does W3 for the honorific split-off, whose comma-side question the switch also reaches (`田中さん, V.` keeps さん glued under it; decisions/W2's 2026-10-27 entry, which holds what was then root AGENTS.md's honorific-peel paragraph, and the case row `ja_honorific_glued_family_comma_strict_knob`). The switch's own history stays where it was decided: decisions/C1's 2026-07 and 2026-07-12/13 entries (its blast radius, the one-letter roman numerals, and its date) and decisions/C13's #603 STRICT MODE paragraph (the opened run takes the vetoed word). The given part's walk after a family comma is strict-blind, measured on d7e1d47a (`Smith, John V`, `Smith, John PhD I` and `Smith, John Jr. V.` read the same under the switch as without it; C14's `Smith, John V` boundary pins the first), so this rule scopes the veto to the comma's decision and to a part after the comma that is nothing but suffix words. decisions/C9's #430/#432 sentence that "the strict knob is consulted" and "the run ends at the numeral" means that credential-run continuation, not the given part's walk. Two drafts of this rule overstated the reach (a run "inside a comma part", then "in the part after the comma", which in the listing form is the given part too); the PR reviews found both. One reach of the switch no rule states, the count before the comma reading the vetoed word as a name word (`Smith V., Ma` → family 'V.'), stays recorded as decisions/S4's 2026-09-18 rough edge. diff --git a/docs/design/decisions/W2-glued-honorific-peel.md b/docs/design/decisions/W2-glued-honorific-peel.md index 237071cfa..55822a262 100644 --- a/docs/design/decisions/W2-glued-honorific-peel.md +++ b/docs/design/decisions/W2-glued-honorific-peel.md @@ -25,5 +25,5 @@ Open: none in this file. - 2026-10-17 #671 (its no-glued-peel verdict superseded by #707 below for every word but 部長/部长) — SPACED AND AFTER-COMMA ONLY, no glued peel (Derek, 2026-10-17). 議員/议员, 国会議員/國會議員, 太平紳士/太平绅士, 医学博士/醫學博士, 二世 and the job titles 社長/部長/課長 and the simplified 社长/部长/课长 (Derek, 2026-10-17: the characters are shared and the offices are real in both languages; no borne-name data either way, #707) join `SUFFIX_WORDS`; none joins `GLUED_HONORIFICS`. The spaced bar is only that a word standing alone is no name, so they ship; the glued bar ("can it END a name?") needs real-name counts that this change did not get — the Wikidata suffix query timed out and the native-script Wikidata corpus holds none of the words — and 部長 fails it on argument alone (長 is a borne given name, 阿部長). A follow-up can vet 医学博士, 議員 and 太平紳士 one at a time. Consequence measured on `parser_for(locales.ZH)`: after a listed surname each word divides off, `王二世` family 王, suffix 二世 (it read given 二世) and `李部長` family 李, suffix 部長, because the segmenter's division is not a peel and the split-off 二世 is then a whole-token suffix word, the way 王博士 already reads; `陳怡君二世` still reads given 怡君二世 (glued, nothing divides it; superseded for the glued question by the #707 entry below). ジュニア was already shipped by #661, and #679 declined 二世 as CLDR's generation word ("regnal, and glued"); here it ships spaced-only as the Chinese/Japanese numeral, which does not reverse that (corrected 2026-10-21, #762: wrong — #671 put 二世 in `SUFFIX_WORDS` as the generational numeral (a `generation` reading since #737) and #707 let it peel glued, which reverses #679's refusal of it, as decisions/generational-words' 2026-10-21 #762 entry records). Excluded as the issue stated: 小 (a prefix and a common given-name character) and 冯/馮 for "von" (a top-30 surname). - 2026-10-17 #707 — the #671 words meet the glued bar, all but 部長/部长 (Derek, 2026-10-17). 議員/议员, 国会議員/國會議員, 太平紳士/太平绅士, 医学博士/醫學博士, 二世, 社長/課長 and 社长/课长 join `GLUED_HONORIFICS`; 部長/部长 stay spaced-only on the #671 argument, since 長 is a borne given name (Wikidata holds it as a Chinese given name, Q96240389; the Japanese reading rests on the argument) and 阿部長 can be 阿部 + 長 — the measurement below found no such bearer, and a zero that is a lower bound does not refute the argument. The measurement is `tools/name_usage.py`'s ends-with row, new in #707: a local index of every CJK label on eleven name-item classes (family name, given name and their Japanese, Chinese, Korean, male, female, unisex and undifferentiated-kanji subclasses; 215,798 index entries on 128,695 items, 215,696 distinct item-label pairs, fetched 2026-10-17 by the first index, whose character class stopped at the Basic Multilingual Plane; the rebuilt index that takes supplementary-plane Han too (215,820 entries, same day) gave the same answer for every word, 殿 and the boundary pairs below), matched locally, with the matching items' bearers counted live. Two questions per word: does any name item's label END in it, and does any person bear a family name ending in the word's head and a given name equal to its tail (the 阿部 + 長 shape)? Neither found anything for any spelling, with one exception that is a data error: the only item ending in 二世 is Q18339792, the given name Odo, whose Chinese label carries a regnal numeral (厄德二世), and its 109 bearers are Western people named Odo. Of the boundary shapes, only 二+世, 社+長, 社+长, 部+長 and 部+长 had a family name and a given-name item to pair at all, and each counted 0 people. What makes the zeros evidence: the index agreed exactly with a live `STRENDS` query on all 20 class × word cells it was cross-checked on (five subclasses × 殿, 子, 郎, 田, the largest 403 items); 子 found 439 items and 2,436 given-name bearers, 2,081 of them Japanese; and 田 + 太郎 counted 22 people across the boundary. What limits them: Wikidata's coverage, not the index's — 殿 ends 13 name items where a surname dictionary lists about 90, and Japanese family-name items number 4,919. Two shortcuts were measured and rejected before the index: `STRENDS` over every label of a name class timed out at WDQS's 60s with no person join, and WDQS full-text search found 4 family names ending in 殿, matching only where its tokenizer split the character out. Consequence: the snapshot moved four CLDR renderings, each losing its glued word to `suffix` (`王雅婷婷婷二世議員`, both spellings of `…小惠儀太平紳士`, `馮·陳·怡君·達印·二世醫學博士`), and `陳怡君二世`, which the case table and rules.md W2 pin; cldr_agreement records one suffix field newly agreeing (`二世 醫學博士`, where the peel takes 醫學博士 and the 二世 left behind is a whole-token suffix word) and none regressing. The peel still takes one word: `王雅婷婷婷二世議員` peels 議員 and keeps 二世 in the name. Recompute: `uv run python tools/name_usage.py 議員 议员 国会議員 國會議員 太平紳士 太平绅士 医学博士 醫學博士 二世 社長 部長 課長 社长 部长 课长`; the first run builds the index (33 minutes on 2026-10-17, resumable), and `--refresh-index` rebuilds it. ACCEPTED COST, found by the design-docs review and kept (Derek, 2026-10-17): the vetting asked whether a NAME ends in each word and never whether a longer, unlisted OFFICE word does, and common ones do — 衆議院議員 and 参議院議員, 副社長 and 代表取締役社長, 元議員 and 市议员, 副課長, 獣医学博士. The peel matches at the end of the token, spaced or glued, so each splits at the listed tail: `岸田文雄衆議院議員` reads family 岸田文雄衆議院, suffix 議員, and `田中 副社長` given 副, suffix 社長, where before the word stayed whole in the name. Neither reading is right; the compound's head is what the peel leaves behind. No corpus held a compound office word when the review found this, which is why the snapshot was silent; the two W2 examples now put `岸田文雄衆議院議員` and `田中 副社長` in corpus_rules.jsonl. Older entries carry the same cost unrecorded until now (`김민준의학박사` reads given 민준의학, suffix 박사). Listing a compound whole would peel it whole, longest first. - 2026-10-18 (vocabulary phase 1, decisions/vocabulary-model) — RECORDED HERE from the #307/#308 vetting note in nameparser/config/suffixes.py, which the move of the vocabulary to TOML retired; its per-word facts went to each word's TOML entry (`uv run python tools/vocab/query.py --issue 308`). Three class-level points had no other home. The spaced and glued bars are different, and an entry can be safe spaced and unsafe glued, never the other way round, because the spaced position is a token boundary its writer drew. A Han, kana or hangul entry is self-selecting, like the Korean surnames: it can only ever match CJK text. And kana is no license by itself: hiragana spells Japanese GIVEN names (高橋みなみ, pinned in the case table), which is why さん, さま, くん and ちゃん were vetted one at a time as never name-final rather than waved through as kana. Further spaced candidates (여사, 太太) wait on the same case-by-case argument. -- 2026-10-19 #760 (the two facts it left to AGENTS.md's script paragraph moved into this file on 2026-10-27, below) — MOVED FROM the peel's code comment in `_script_segment.py` (`_peel_honorific_tail`), the facts the #312/#319 entries above and AGENTS.md's script paragraph did not hold (that paragraph keeps why the run predicate is not the token predicate's plural, and what `lenient_comma_suffixes=False` does here); parses re-run 2026-10-19. WHY ONLY THE FAMILY COMMA CROSSES: a comma is how a writer says which runs are the name, and a family comma is the one structure that splits the name itself across two of them (`김, 민준씨` is runs (0,) and (1,)), with the honorific as often glued to the given side as to the family. Every other structure keeps the whole name in segments[0], so the asymmetry is the point rather than an accident of two cases. "The rest is post-nominals" is true of the SUFFIX comma only: under NO_COMMA segment returns one run and a trailing post-nominal is INSIDE it (`김민준씨 Jr.` is one run of two tokens, family '김', suffix '씨 Jr.'), which is why the scan-back steps over such a token rather than never reaching it. WHY THE RUN PREDICATE IS ASKED: the structure says nothing about whether segments[1] is name text — segment hands every comma form over as FAMILY_COMMA (#613), and a one-word part turned a wholly suffix-shaped remainder into FAMILY_COMMA anyway (`田中さん, V.`). THE SCOPE HOLDS OFF A JUNK TAIL past the second run: `Dr 김민준씨, Jr., 박씨` is a postnominal part behind a whole name (`_postnominal_behind_a_whole_name`), so the scan stays in segments[0] and 박씨 in a third run is never reached (family '김', suffix '씨, Jr., 박씨'). AN EMPTY SECOND RUN stays in scope and contributes nothing: `is_wholly_suffix` is False on it by its own contract (v1 read `Doe,, Jr.` as a family comma), which is the reading the peel wants, flattening an empty run adding no site. FLATTENING SEGMENTS, NOT `state.tokens`, is load-bearing: extracted nickname and maiden content is in tokens but in no segment, and scanning tokens would put the site on a nickname (`김민준씨 (Jimmy)` would peel nothing); the case row `ko_honorific_glued_given_nickname` pins it (family '김', given '민준', suffix '씨', nickname 'Jimmy'). WHY THE PREDICATE IS CIRCULAR HERE ALONE, and so needs the second condition: not because `_comma.decide` asks it too (at group's head, after this stage), but because decide SPENDS the answer differently — it reads the run's shape and stops, the answer being the structure, while the peel reads the same shape and then decides whether to strip the very honorific that produced it. Declining a run that holds the only site does not fall back to another site; it loses the peel outright, and with it the given name, which lands in suffix (`이, J.씨` would read suffix 'J.씨'). DROPPED AS STALE, not moved: the comment said the declined credential lands "`PhD` a title"; it is a suffix (`田中さん, PhD` → suffix 'さん, PhD', rules/W3's example line), as AGENTS.md's script paragraph already says. -- 2026-10-27 — MOVED FROM AGENTS.md's script paragraph, which now keeps the doctrine and points at the rules; every parse below re-run 2026-10-27. THE REACH, by example: under a family comma the scan reads the first two runs and nothing past them, so `김, 민준 지훈씨` peels (family '김', given '민준', middle '지훈', suffix '씨') while `김, 민준, 지훈씨` (suffix '지훈씨') and `김,, 민준씨` (suffix '민준씨') do not, both landing the honorific in a third run. THE SECOND CONDITION, by example: the part before the comma must offer a site of its own before a suffix-shaped second run is declined, so `선생님, J.씨` is scanned anyway like `이, J.씨` (given 'J.', suffix '씨'), and `김민준씨, J.씨`, with a site on both sides, peels the person's own 씨 rather than the one behind the comma (family '김민준', suffix '씨, J.씨'). UNIFORM IN THE PEEL, not in where the credential lands, which is assign's question and differs by spelling: `田中さん, V.` reads given 'V.', while `田中さん, PhD` and `田中さん, Ph. D.` read suffix 'さん, PhD' and 'さん, Ph. D.'. THE RUN PREDICATE IS NOT THE TOKEN PREDICATE (`_is_post_nominal`) PLURALIZED: it says yes both to what the token predicate vetoes (`V.`, `V`, `I`, the class the defect was reported as) and to what the token predicate never sees, since `period_joined_vocab` and the delimiter routes are the run predicate's alone (`Msc.Ed.` and `J.씨` reach it that way, and `田中さん, Msc.Ed.` reads suffix 'さん, Msc.Ed.' with the rest). UNDER `Policy(lenient_comma_suffixes=False)` (rules/C14) the call drops to the strict token test, so `田中さん, V.`, `田中さん, V` and `田中さん, I` read their credential as name text and keep the pre-#319 answer (family '田中さん'), while `田中さん, Ph. D.` peels regardless, its merged `phd` passing the strict test. `田中さん, 太郎` is untouched (family '田中さん', given '太郎'), and not because of its comma: the honorific is not at the end of the name, 太郎 is. WHY THE PEEL ASKS `_comma.decide` (on a classified copy of the name as written, since #613's /simplify) RATHER THAN A HAND COPY: the copy drifted from decide four times in that PR's review (the ambiguous class's count, the numeral refusal, a suffix before the comma, the split `Ph. D.`), each drift dividing a name decide read as a family comma or leaving whole one it read as a suffix comma; decisions/C1's 2026-10-06 (Derek), #613 entry. Neither policy opt-out reaches the peel (`Policy(script_orders={}, segment_scripts=frozenset())` still reads `田中さん` suffix 'さん' and `김민준씨` suffix '씨'): the vocabulary carries the license, so the script switches have nothing to say about it. +- 2026-10-19 #760 (the two facts it left to AGENTS.md's script paragraph moved into this file on 2026-10-27, below) — MOVED FROM the peel's code comment in `_script_segment.py` (`_peel_honorific_tail`), the facts the #312/#319 entries above and AGENTS.md's script paragraph did not hold (that paragraph keeps why the run predicate is not the token predicate's plural, and what `lenient_comma_suffixes=False` does here); parses re-run 2026-10-19. WHY ONLY THE FAMILY COMMA CROSSES: a comma is how a writer says which runs are the name, and a family comma is the one structure that splits the name itself across two of them (`김, 민준씨` is runs (0,) and (1,)), with the honorific as often glued to the given side as to the family. Every other structure keeps the whole name in segments[0], so the asymmetry is the point rather than an accident of two cases. "The rest is post-nominals" is true of the SUFFIX comma only: under NO_COMMA segment returns one run and a trailing post-nominal is INSIDE it (`김민준씨 Jr.` is one run of two tokens, family '김', suffix '씨 Jr.'), which is why the scan-back steps over such a token rather than never reaching it. WHY THE RUN PREDICATE IS ASKED: the structure says nothing about whether segments[1] is name text — segment hands every comma form over as FAMILY_COMMA (#613), and a one-word part turned a wholly suffix-shaped remainder into FAMILY_COMMA anyway (`田中さん, V.`). THE SCOPE HOLDS OFF A JUNK TAIL past the second run: `Dr 김민준씨, Jr., 박씨` is a postnominal part behind a whole name (`_postnominal_behind_a_whole_name`), so the scan stays in segments[0] and 박씨 in a third run is never reached (family '김', suffix '씨, Jr., 박씨'). AN EMPTY SECOND RUN stays in scope and contributes nothing: `is_wholly_suffix` is False on it by its own contract (v1 read `Doe,, Jr.` as a family comma), which is the reading the peel wants, flattening an empty run adding no site. FLATTENING SEGMENTS, NOT `state.tokens`, is load-bearing: extracted nickname and maiden content is in tokens but in no segment, and scanning tokens would put the site on a nickname (`김민준씨 (Jimmy)` would peel nothing); the case row `ko_honorific_glued_given_nickname` pins it (family '김', given '민준', suffix '씨', nickname 'Jimmy'). WHY THE PREDICATE IS CIRCULAR HERE ALONE, and so needs the second condition: not because `_comma.decide` asks it too (at group's head, after this stage), but because decide SPENDS the answer differently — it reads the run's shape and stops, the answer being the structure, while the peel reads the same shape and then decides whether to strip the very honorific that produced it. Declining a run that holds the only site does not fall back to another site; it loses the peel outright, and with it the given name, which lands in suffix (`이, J.씨` would read suffix 'J.씨'). DROPPED AS STALE, not moved: the comment said the declined credential lands "`PhD` a title"; it is a suffix (`田中さん, PhD` → suffix 'さん, PhD', rules/W3's example line), as AGENTS.md's script paragraph then said (moved into this file on 2026-10-27). +- 2026-10-27 (#907's follow-up, the AGENTS.md relocation) — MOVED FROM AGENTS.md's script paragraph, which now keeps the doctrine and points at the rules; every parse below re-run 2026-10-27. THE REACH, by example: under a family comma the scan reads the first two runs and nothing past them, so `김, 민준 지훈씨` peels (family '김', given '민준', middle '지훈', suffix '씨') while `김, 민준, 지훈씨` (suffix '지훈씨') and `김,, 민준씨` (suffix '민준씨') do not, both landing the honorific in a third run. WHAT #319 FIXED: before it, the peel walked into such a second run, took `V.` for its site, found no listed tail and abandoned. THE SECOND CONDITION, by example: the part before the comma must offer a site of its own before a suffix-shaped second run is declined, so `선생님, J.씨` is scanned anyway like `이, J.씨` (given 'J.', suffix '씨'), both keeping the pre-#319 reading, and `김민준씨, J.씨`, with a site on both sides, peels the person's own 씨 rather than the one behind the comma (family '김민준', suffix '씨, J.씨'). UNIFORM IN THE PEEL, not in where the credential lands, which is assign's question and differs by spelling: `田中さん, V.` reads given 'V.', while `田中さん, PhD` and `田中さん, Ph. D.` read suffix 'さん, PhD' and 'さん, Ph. D.'. THE RUN PREDICATE IS NOT THE TOKEN PREDICATE (`_is_post_nominal`) PLURALIZED: it says yes both to what the token predicate vetoes (`V.`, `V`, `I`, the class the defect was reported as) and to what the token predicate never sees, since `period_joined_vocab` and the delimiter routes are the run predicate's alone (`Msc.Ed.` and `J.씨` reach it that way, and `田中さん, Msc.Ed.` reads suffix 'さん, Msc.Ed.' with the rest). UNDER `Policy(lenient_comma_suffixes=False)` (rules/C14) the call drops to the strict token test, so `田中さん, V.`, `田中さん, V` and `田中さん, I` read their credential as name text and keep the pre-#319 answer (family '田中さん'), while `田中さん, Ph. D.` peels regardless, its merged `phd` passing the strict test. `田中さん, 太郎` is untouched (family '田中さん', given '太郎'), and not because of its comma: the honorific is not at the end of the name, 太郎 is. WHY THE PEEL ASKS `_comma.decide` (on a classified copy of the name as written, since #613's /simplify) RATHER THAN A HAND COPY: the copy drifted from decide four times in that PR's review (the ambiguous class's count, the numeral refusal, a suffix before the comma, the split `Ph. D.`), each drift dividing a name decide read as a family comma or leaving whole one it read as a suffix comma; decisions/C1's 2026-10-06 (Derek), #613 entry. WHY THE SPACED-ONLY ENTRIES ARE: 김지양 and 田中博士 are names, and about 90 Japanese surnames end in 殿; 君 is in neither set because 王君 is a complete Chinese name. Neither policy opt-out reaches the peel (`Policy(script_orders={}, segment_scripts=frozenset())` still reads `田中さん` suffix 'さん' and `김민준씨` suffix '씨'): the vocabulary carries the license, so the script switches have nothing to say about it. diff --git a/nameparser/AGENTS.md b/nameparser/AGENTS.md index 488a885fd..b56209e2d 100644 --- a/nameparser/AGENTS.md +++ b/nameparser/AGENTS.md @@ -20,7 +20,7 @@ The same split runs through the credentials (`MA` listed, `X.Y.Z.` by the dotted ### A word a rule only asks ABOUT -That holds for a word a rule only asks ABOUT, too: #810's first draft asked whether any word after a bracketed credential lacked the suffix TAG, and kept `John Smith (MA) VI`, `(MA) Ph. D.` and `(MA) Dr.` nicknames while `(MA) III` and `(MA) PhD` escaped; asking the role the parse read for the following word answers for every spelling. +Testing one spelling from each path holds for a word a rule only asks ABOUT, too: #810's first draft asked whether any word after a bracketed credential lacked the suffix TAG, and kept `John Smith (MA) VI`, `(MA) Ph. D.` and `(MA) Dr.` nicknames while `(MA) III` and `(MA) PhD` escaped; asking the role the parse read for the following word answers for every spelling. ### A separator's spacing diff --git a/nameparser/_pipeline/_pieces.py b/nameparser/_pipeline/_pieces.py index ca5aeada0..b9e7630b5 100644 --- a/nameparser/_pipeline/_pieces.py +++ b/nameparser/_pipeline/_pieces.py @@ -493,8 +493,8 @@ def starts_a_credential_run(piece: Sequence[int], ptags: Set[str], The split credential group merges ('Ph. D.', the one piece carrying the "suffix" piece tag) starts it as its one-token spelling 'PhD' does: a run that started for one spelling of a word and not the - other is the vocabulary-against-shape split AGENTS.md's spelling - sweep names (#603 found it, 'Smith, John Ph. D. Jones' keeping + other is the vocabulary-against-shape split nameparser/AGENTS.md's + spelling sweep names (#603 found it, 'Smith, John Ph. D. Jones' keeping middle 'Jones' where 'Smith, John PhD Jones' reads suffix 'PhD Jones'). The no-comma spelling is not reached: `run_start` asks only of the pieces `peel_walk` keeps, and the walk drops the diff --git a/tests/v2/cases.py b/tests/v2/cases.py index 3867762be..f466ffad2 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -2856,7 +2856,8 @@ def _check_cjk_shape_purity(self) -> None: "Ivoi, Paul d',Jr.", {"given": "Paul", "family": "d' Ivoi", "suffix": "Jr."}, classification="fix(#700)", - notes="AGENTS.md's separator spacing: the comma glued to the " + notes="nameparser/AGENTS.md's separator spacing: the comma " + "glued to the " "apostrophe still ends the particle's token. 1.4.0 " "through 2.4.2 middle \"d'\""), Case("an_elided_particle_after_a_particle_surname", From a9f116fdd582344922bc7f183a78ff0eb19b6660 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Tue, 27 Oct 2026 01:20:03 -0700 Subject: [PATCH 5/6] Cite #909 in decisions/W2's 2026-10-27 entry Co-Authored-By: Claude Opus 5.5 --- docs/design/decisions/W2-glued-honorific-peel.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/design/decisions/W2-glued-honorific-peel.md b/docs/design/decisions/W2-glued-honorific-peel.md index 55822a262..20fed03da 100644 --- a/docs/design/decisions/W2-glued-honorific-peel.md +++ b/docs/design/decisions/W2-glued-honorific-peel.md @@ -26,4 +26,4 @@ Open: none in this file. - 2026-10-17 #707 — the #671 words meet the glued bar, all but 部長/部长 (Derek, 2026-10-17). 議員/议员, 国会議員/國會議員, 太平紳士/太平绅士, 医学博士/醫學博士, 二世, 社長/課長 and 社长/课长 join `GLUED_HONORIFICS`; 部長/部长 stay spaced-only on the #671 argument, since 長 is a borne given name (Wikidata holds it as a Chinese given name, Q96240389; the Japanese reading rests on the argument) and 阿部長 can be 阿部 + 長 — the measurement below found no such bearer, and a zero that is a lower bound does not refute the argument. The measurement is `tools/name_usage.py`'s ends-with row, new in #707: a local index of every CJK label on eleven name-item classes (family name, given name and their Japanese, Chinese, Korean, male, female, unisex and undifferentiated-kanji subclasses; 215,798 index entries on 128,695 items, 215,696 distinct item-label pairs, fetched 2026-10-17 by the first index, whose character class stopped at the Basic Multilingual Plane; the rebuilt index that takes supplementary-plane Han too (215,820 entries, same day) gave the same answer for every word, 殿 and the boundary pairs below), matched locally, with the matching items' bearers counted live. Two questions per word: does any name item's label END in it, and does any person bear a family name ending in the word's head and a given name equal to its tail (the 阿部 + 長 shape)? Neither found anything for any spelling, with one exception that is a data error: the only item ending in 二世 is Q18339792, the given name Odo, whose Chinese label carries a regnal numeral (厄德二世), and its 109 bearers are Western people named Odo. Of the boundary shapes, only 二+世, 社+長, 社+长, 部+長 and 部+长 had a family name and a given-name item to pair at all, and each counted 0 people. What makes the zeros evidence: the index agreed exactly with a live `STRENDS` query on all 20 class × word cells it was cross-checked on (five subclasses × 殿, 子, 郎, 田, the largest 403 items); 子 found 439 items and 2,436 given-name bearers, 2,081 of them Japanese; and 田 + 太郎 counted 22 people across the boundary. What limits them: Wikidata's coverage, not the index's — 殿 ends 13 name items where a surname dictionary lists about 90, and Japanese family-name items number 4,919. Two shortcuts were measured and rejected before the index: `STRENDS` over every label of a name class timed out at WDQS's 60s with no person join, and WDQS full-text search found 4 family names ending in 殿, matching only where its tokenizer split the character out. Consequence: the snapshot moved four CLDR renderings, each losing its glued word to `suffix` (`王雅婷婷婷二世議員`, both spellings of `…小惠儀太平紳士`, `馮·陳·怡君·達印·二世醫學博士`), and `陳怡君二世`, which the case table and rules.md W2 pin; cldr_agreement records one suffix field newly agreeing (`二世 醫學博士`, where the peel takes 醫學博士 and the 二世 left behind is a whole-token suffix word) and none regressing. The peel still takes one word: `王雅婷婷婷二世議員` peels 議員 and keeps 二世 in the name. Recompute: `uv run python tools/name_usage.py 議員 议员 国会議員 國會議員 太平紳士 太平绅士 医学博士 醫學博士 二世 社長 部長 課長 社长 部长 课长`; the first run builds the index (33 minutes on 2026-10-17, resumable), and `--refresh-index` rebuilds it. ACCEPTED COST, found by the design-docs review and kept (Derek, 2026-10-17): the vetting asked whether a NAME ends in each word and never whether a longer, unlisted OFFICE word does, and common ones do — 衆議院議員 and 参議院議員, 副社長 and 代表取締役社長, 元議員 and 市议员, 副課長, 獣医学博士. The peel matches at the end of the token, spaced or glued, so each splits at the listed tail: `岸田文雄衆議院議員` reads family 岸田文雄衆議院, suffix 議員, and `田中 副社長` given 副, suffix 社長, where before the word stayed whole in the name. Neither reading is right; the compound's head is what the peel leaves behind. No corpus held a compound office word when the review found this, which is why the snapshot was silent; the two W2 examples now put `岸田文雄衆議院議員` and `田中 副社長` in corpus_rules.jsonl. Older entries carry the same cost unrecorded until now (`김민준의학박사` reads given 민준의학, suffix 박사). Listing a compound whole would peel it whole, longest first. - 2026-10-18 (vocabulary phase 1, decisions/vocabulary-model) — RECORDED HERE from the #307/#308 vetting note in nameparser/config/suffixes.py, which the move of the vocabulary to TOML retired; its per-word facts went to each word's TOML entry (`uv run python tools/vocab/query.py --issue 308`). Three class-level points had no other home. The spaced and glued bars are different, and an entry can be safe spaced and unsafe glued, never the other way round, because the spaced position is a token boundary its writer drew. A Han, kana or hangul entry is self-selecting, like the Korean surnames: it can only ever match CJK text. And kana is no license by itself: hiragana spells Japanese GIVEN names (高橋みなみ, pinned in the case table), which is why さん, さま, くん and ちゃん were vetted one at a time as never name-final rather than waved through as kana. Further spaced candidates (여사, 太太) wait on the same case-by-case argument. - 2026-10-19 #760 (the two facts it left to AGENTS.md's script paragraph moved into this file on 2026-10-27, below) — MOVED FROM the peel's code comment in `_script_segment.py` (`_peel_honorific_tail`), the facts the #312/#319 entries above and AGENTS.md's script paragraph did not hold (that paragraph keeps why the run predicate is not the token predicate's plural, and what `lenient_comma_suffixes=False` does here); parses re-run 2026-10-19. WHY ONLY THE FAMILY COMMA CROSSES: a comma is how a writer says which runs are the name, and a family comma is the one structure that splits the name itself across two of them (`김, 민준씨` is runs (0,) and (1,)), with the honorific as often glued to the given side as to the family. Every other structure keeps the whole name in segments[0], so the asymmetry is the point rather than an accident of two cases. "The rest is post-nominals" is true of the SUFFIX comma only: under NO_COMMA segment returns one run and a trailing post-nominal is INSIDE it (`김민준씨 Jr.` is one run of two tokens, family '김', suffix '씨 Jr.'), which is why the scan-back steps over such a token rather than never reaching it. WHY THE RUN PREDICATE IS ASKED: the structure says nothing about whether segments[1] is name text — segment hands every comma form over as FAMILY_COMMA (#613), and a one-word part turned a wholly suffix-shaped remainder into FAMILY_COMMA anyway (`田中さん, V.`). THE SCOPE HOLDS OFF A JUNK TAIL past the second run: `Dr 김민준씨, Jr., 박씨` is a postnominal part behind a whole name (`_postnominal_behind_a_whole_name`), so the scan stays in segments[0] and 박씨 in a third run is never reached (family '김', suffix '씨, Jr., 박씨'). AN EMPTY SECOND RUN stays in scope and contributes nothing: `is_wholly_suffix` is False on it by its own contract (v1 read `Doe,, Jr.` as a family comma), which is the reading the peel wants, flattening an empty run adding no site. FLATTENING SEGMENTS, NOT `state.tokens`, is load-bearing: extracted nickname and maiden content is in tokens but in no segment, and scanning tokens would put the site on a nickname (`김민준씨 (Jimmy)` would peel nothing); the case row `ko_honorific_glued_given_nickname` pins it (family '김', given '민준', suffix '씨', nickname 'Jimmy'). WHY THE PREDICATE IS CIRCULAR HERE ALONE, and so needs the second condition: not because `_comma.decide` asks it too (at group's head, after this stage), but because decide SPENDS the answer differently — it reads the run's shape and stops, the answer being the structure, while the peel reads the same shape and then decides whether to strip the very honorific that produced it. Declining a run that holds the only site does not fall back to another site; it loses the peel outright, and with it the given name, which lands in suffix (`이, J.씨` would read suffix 'J.씨'). DROPPED AS STALE, not moved: the comment said the declined credential lands "`PhD` a title"; it is a suffix (`田中さん, PhD` → suffix 'さん, PhD', rules/W3's example line), as AGENTS.md's script paragraph then said (moved into this file on 2026-10-27). -- 2026-10-27 (#907's follow-up, the AGENTS.md relocation) — MOVED FROM AGENTS.md's script paragraph, which now keeps the doctrine and points at the rules; every parse below re-run 2026-10-27. THE REACH, by example: under a family comma the scan reads the first two runs and nothing past them, so `김, 민준 지훈씨` peels (family '김', given '민준', middle '지훈', suffix '씨') while `김, 민준, 지훈씨` (suffix '지훈씨') and `김,, 민준씨` (suffix '민준씨') do not, both landing the honorific in a third run. WHAT #319 FIXED: before it, the peel walked into such a second run, took `V.` for its site, found no listed tail and abandoned. THE SECOND CONDITION, by example: the part before the comma must offer a site of its own before a suffix-shaped second run is declined, so `선생님, J.씨` is scanned anyway like `이, J.씨` (given 'J.', suffix '씨'), both keeping the pre-#319 reading, and `김민준씨, J.씨`, with a site on both sides, peels the person's own 씨 rather than the one behind the comma (family '김민준', suffix '씨, J.씨'). UNIFORM IN THE PEEL, not in where the credential lands, which is assign's question and differs by spelling: `田中さん, V.` reads given 'V.', while `田中さん, PhD` and `田中さん, Ph. D.` read suffix 'さん, PhD' and 'さん, Ph. D.'. THE RUN PREDICATE IS NOT THE TOKEN PREDICATE (`_is_post_nominal`) PLURALIZED: it says yes both to what the token predicate vetoes (`V.`, `V`, `I`, the class the defect was reported as) and to what the token predicate never sees, since `period_joined_vocab` and the delimiter routes are the run predicate's alone (`Msc.Ed.` and `J.씨` reach it that way, and `田中さん, Msc.Ed.` reads suffix 'さん, Msc.Ed.' with the rest). UNDER `Policy(lenient_comma_suffixes=False)` (rules/C14) the call drops to the strict token test, so `田中さん, V.`, `田中さん, V` and `田中さん, I` read their credential as name text and keep the pre-#319 answer (family '田中さん'), while `田中さん, Ph. D.` peels regardless, its merged `phd` passing the strict test. `田中さん, 太郎` is untouched (family '田中さん', given '太郎'), and not because of its comma: the honorific is not at the end of the name, 太郎 is. WHY THE PEEL ASKS `_comma.decide` (on a classified copy of the name as written, since #613's /simplify) RATHER THAN A HAND COPY: the copy drifted from decide four times in that PR's review (the ambiguous class's count, the numeral refusal, a suffix before the comma, the split `Ph. D.`), each drift dividing a name decide read as a family comma or leaving whole one it read as a suffix comma; decisions/C1's 2026-10-06 (Derek), #613 entry. WHY THE SPACED-ONLY ENTRIES ARE: 김지양 and 田中博士 are names, and about 90 Japanese surnames end in 殿; 君 is in neither set because 王君 is a complete Chinese name. Neither policy opt-out reaches the peel (`Policy(script_orders={}, segment_scripts=frozenset())` still reads `田中さん` suffix 'さん' and `김민준씨` suffix '씨'): the vocabulary carries the license, so the script switches have nothing to say about it. +- 2026-10-27 #909 — MOVED FROM AGENTS.md's script paragraph, which now keeps the doctrine and points at the rules; every parse below re-run 2026-10-27. THE REACH, by example: under a family comma the scan reads the first two runs and nothing past them, so `김, 민준 지훈씨` peels (family '김', given '민준', middle '지훈', suffix '씨') while `김, 민준, 지훈씨` (suffix '지훈씨') and `김,, 민준씨` (suffix '민준씨') do not, both landing the honorific in a third run. WHAT #319 FIXED: before it, the peel walked into such a second run, took `V.` for its site, found no listed tail and abandoned. THE SECOND CONDITION, by example: the part before the comma must offer a site of its own before a suffix-shaped second run is declined, so `선생님, J.씨` is scanned anyway like `이, J.씨` (given 'J.', suffix '씨'), both keeping the pre-#319 reading, and `김민준씨, J.씨`, with a site on both sides, peels the person's own 씨 rather than the one behind the comma (family '김민준', suffix '씨, J.씨'). UNIFORM IN THE PEEL, not in where the credential lands, which is assign's question and differs by spelling: `田中さん, V.` reads given 'V.', while `田中さん, PhD` and `田中さん, Ph. D.` read suffix 'さん, PhD' and 'さん, Ph. D.'. THE RUN PREDICATE IS NOT THE TOKEN PREDICATE (`_is_post_nominal`) PLURALIZED: it says yes both to what the token predicate vetoes (`V.`, `V`, `I`, the class the defect was reported as) and to what the token predicate never sees, since `period_joined_vocab` and the delimiter routes are the run predicate's alone (`Msc.Ed.` and `J.씨` reach it that way, and `田中さん, Msc.Ed.` reads suffix 'さん, Msc.Ed.' with the rest). UNDER `Policy(lenient_comma_suffixes=False)` (rules/C14) the call drops to the strict token test, so `田中さん, V.`, `田中さん, V` and `田中さん, I` read their credential as name text and keep the pre-#319 answer (family '田中さん'), while `田中さん, Ph. D.` peels regardless, its merged `phd` passing the strict test. `田中さん, 太郎` is untouched (family '田中さん', given '太郎'), and not because of its comma: the honorific is not at the end of the name, 太郎 is. WHY THE PEEL ASKS `_comma.decide` (on a classified copy of the name as written, since #613's /simplify) RATHER THAN A HAND COPY: the copy drifted from decide four times in that PR's review (the ambiguous class's count, the numeral refusal, a suffix before the comma, the split `Ph. D.`), each drift dividing a name decide read as a family comma or leaving whole one it read as a suffix comma; decisions/C1's 2026-10-06 (Derek), #613 entry. WHY THE SPACED-ONLY ENTRIES ARE: 김지양 and 田中博士 are names, and about 90 Japanese surnames end in 殿; 君 is in neither set because 王君 is a complete Chinese name. Neither policy opt-out reaches the peel (`Policy(script_orders={}, segment_scripts=frozenset())` still reads `田中さん` suffix 'さん' and `김민준씨` suffix '씨'): the vocabulary carries the license, so the script switches have nothing to say about it. From 2d40acbbcee7bd20513dfbeb47b0187042b06e03 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Tue, 27 Oct 2026 01:20:27 -0700 Subject: [PATCH 6/6] Record in AGENTS.md that a whole-file replace rewrites history lines (#909) Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/AGENTS.md b/AGENTS.md index 0d517e86a..c6a1c3131 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -14,7 +14,7 @@ After a review round, **review the fix commit too** — second-round passes on # **When a defect is found that an earlier check would have caught, write the check down where the next person will meet it, in the same PR.** A review finding, a failing sweep or a regression that arrived because nobody thought to test a shape is a lesson about the repository, not only about the change, and it is lost when the PR merges unless it lands somewhere a later session reads before making the same mistake. Ask, before merging: what should I have checked, and where would I have looked? Then put it there — a missed input shape in the sweeps (comma shapes, then `name_order`, under Architecture; then spellings, in `nameparser/AGENTS.md`; each earned this way); a trap in a module's behavior under Gotchas; a design-review miss in docs/design/AGENTS.md's axes; a measurement or test-design trap in mechanisms.md's Verification shapes or Field notes; a reusable design move in mechanisms.md proper. Name the PR or issue that earned it and the input that showed it, so the rule can be checked against its case. #610 is the shape: its sweep test found that the maiden take missed a shape-only numeral, and the spelling sweep in `nameparser/AGENTS.md` records the class (`VI` beside `V` and `III`). This is how the file accumulates expertise rather than history — prefer amending an existing paragraph to adding a new one, and delete a lesson the code has since made impossible. -**A scripted multi-edit that asserts as it goes discards everything when a late pattern misses.** Assert every pattern before applying any, then verify the edit landed — three times in one session a docs batch failed its last assertion, wrote nothing, and reported success, invisible because prose edits fail no test. +**A scripted multi-edit that asserts as it goes discards everything when a late pattern misses.** Assert every pattern before applying any, then verify the edit landed — three times in one session a docs batch failed its last assertion, wrote nothing, and reported success, invisible because prose edits fail no test. And replace by an anchored match, never across a whole file: a history line quotes the wrong spelling on purpose, and #909's `sed` turning `segment_scripts=()` into `frozenset()` also rewrote #337's record of what shipped wrong, until the design-docs review found it. ## Rules documentation (docs/design/)