diff --git a/AGENTS.md b/AGENTS.md index 973953116..15c476e63 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -178,7 +178,7 @@ logging.getLogger('HumanName').setLevel(logging.DEBUG) ## Architecture -The library has two layers: `nameparser/config/` (data) and `nameparser/parser.py` (logic). +The library has two layers: data — the per-word TOML in `nameparser/vocab/`, compiled into the `nameparser/config/` modules — and logic, the 2.0 pipeline in the underscore-private `nameparser/_*.py` modules (see 2.0 API modules below). `nameparser/parser.py` and `nameparser/config/__init__.py` are only re-exports of the v1-compat facade (`_facade.py`, `_config_shim.py`); the v1 parser itself was deleted at the M11 swap and is readable as `git show 2d5d8c2:nameparser/parser.py`. **Design philosophy — positional and language-agnostic.** The parser assigns parts by *position* plus small sets of words that join to neighbors; it never detects language. A name's language can't be reliably inferred from Latin-script transliteration ("Ali" is Arabic or Italian; "Van"/"Della"/"Bin" are first names in some cultures, particles in others), so language-specific rules belong in opt-in `Constants` config, never global defaults. Many "wrong for language X" reports (#133, #150, #130, #85, #103, #146, #83) are irreducible ambiguities — e.g. `de Mesnil` (want last name) vs `Van Johnson` (want first name) are the same `[prefix][word]` shape. Before adding a rule, confirm it doesn't break the opposite case (run the full suite — Portuguese and "Van Johnson" tests are the usual canaries). **The one scoped exception (2.1, #271/#272): script-conditional behavior is permitted exactly where the SCRIPT ITSELF — not statistics about it — determines the convention.** The never-detect-language rule above is about Latin *transliteration*, where the signal genuinely is destroyed; native script is a different question, and it is answered per behavior rather than per script. Five defaults fall out of it, plus a sixth that applies the same not-a-guess standard to specific WORDS rather than to a script (#308's honorific peel, below). Wholly-Han, wholly-Hangul and kana-licensed names read family-first (`Policy.script_orders`) — no language detection needed, because zh and ja both write family-first in native script, so order cannot be misread even though the language is unknowable. Unspaced hangul splits into surname + given name (`config/surnames.py` ships the Korean census list as DEFAULT vocabulary) — nothing but Korean is written in hangul and the surnames are a closed census set, and the vocabulary is self-selecting besides: a hangul entry can only ever match hangul text. Hiragana licenses Japanese (#272) — a name whose characters stay inside Han∪kana while carrying at least one kana cannot be Chinese (the kana rules it out) and is not a transcription (foreign names are transcribed in katakana ALONE, マイケル has no kanji), so 高橋みなみ and 山田 エミ read family-first too; mechanically they resolve to the HIRAGANA entry, the license's carrier key. PURE katakana — full-width or halfwidth, the halfwidth block U+FF65–FF9F being katakana in the script table since #594 — is excluded and keeps the positional default: マイケル・ジャクソン is a transcribed foreign name in its source order, and a Japanese reading written in katakana (legacy halfwidth data spells Japanese names that way too) cannot be told from one, so the script settles nothing and a caller who knows opts in through `script_orders`. And the 间隔号 U+00B7 (#298) is the transcription marker for scripts that HAVE no transcription script: a name it divides (威廉·莎士比亚 — flanked by classified characters on both sides, so Catalan's Gal·la is untouched) keeps the declared order and never segments — the orthography names the convention, with the divider carrying the signal that stands the script's family-first reading down, so the default reads it in its written order and a declared `FAMILY_FIRST` reverses it like any other name; pure katakana reaches the same reading by the opposite route, its script settling nothing so the declared order stands. And a listed CJK honorific glued to the END of a name token is split off it (#308) — 田中さん is 田中 plus さん — on the same orthography-settles-it test, narrowed for the glued position: an entry peels only where it can never end a name, so 씨/님/さん/様/先生 peel while 양/군/氏/博士/殿 stay spaced-only (김지양 and 田中博士 are names, and ~90 Japanese surnames end in 殿) and 君 is in NEITHER set (王君 is a complete Chinese name), though its kana spelling くん peels. Like the nakaguro's tokenize-level separation described next, it is reached by neither policy opt-out — but for its own reason: the vocabulary carries the license itself rather than borrowing the script's, so `segment_scripts` has nothing to say about it. Since #312 it also crosses the 间隔号, which still stops the surname split standing right beside it: it answers where a name DIVIDES into surname and given, and the peel never asks that question. Whether it also crosses the FAMILY comma is tolerated rather than settled: the 2026-09-01 demotion (rules/W3) narrowed that half from contract to best-effort, since no CJK writing system's own convention puts a comma between family and given at all — so `김, 민준씨` reads today exactly as the spaced `김 민준씨` does (family 김, given 민준, suffix 씨) while the split stands down as before, but that reading is watched in the snapshot rather than pinned as contract. Its site is accordingly the name-bearing segment runs — `segments[:2]` under a family comma, and `segments[0]` as before otherwise, the family comma being the one structure that splits the name itself across two runs, with the honorific as often glued to the given side as to the family. That is the whole reach and nothing past it (`김, 민준 지훈씨` peels; `김, 민준, 지훈씨` and `김,, 민준씨` do not, both landing in a third run), and whether `segments[1]` is name text at all is now ASKED rather than inferred from the structure — `segment` does not guarantee it, since a one-word part before the comma reads as FAMILY_COMMA even when the part after it is entirely suffix-shaped, and the peel walking into such a run took `V.` for its site, found no listed tail and abandoned (#319). The question is asked in two parts. First the comma's own decision: `_comma.decide` on a classified copy of the name as written, since #613's /simplify -- a hand copy of it here drifted from decide four times in that PR's review (the ambiguous class's count, the numeral refusal, a suffix before the comma, the split `Ph. D.`), each drift dividing a name decide read as a family comma or leaving whole one it read as a suffix comma, and asking the real decision is what ended it. Where that leaves the family comma, C1's suffix-word predicate, lifted into `_vocab.is_wholly_suffix` so the stages cannot drift: a wholly suffix-shaped second run is declined and the scan stays in `segments[0]`, so `田中さん, V.` and `田中さん, Ph. D.` give さん up as `田中さん, PhD` always did. The test is necessary but NOT sufficient, and the second condition is not decoration: every honorific tail is also a suffix word, so a glued honorific is itself part of what makes its run read as suffix-shaped, and declining a run that holds the ONLY site loses the peel outright. `segments[0]` must therefore offer a peel site of its own before the second run is declined — `이, J.씨` and `선생님, J.씨` pass the suffix test and are scanned anyway, keeping the pre-#319 reading, while `김민준씨, J.씨` has a site on both sides and peels the person's own 씨 rather than the junk one behind the comma. Uniform in the PEEL, that is — where the credential itself lands is `assign`'s question and still differs by spelling (`V.` → `given`, `PhD` and `Ph. D.` → `suffix`). Not `_is_post_nominal` pluralized: the run predicate says yes both to what the token predicate vetoes (`V.`, `V`, `I` — the class the defect was reported as) and to what the token predicate never sees at all, since `period_joined_vocab` and the delimiter routes are the run predicate's alone (`Msc.Ed.` and `J.씨` reach it that way, and `田中さん, Msc.Ed.` moves with the rest). `Policy(lenient_comma_suffixes=False)` (rules/C14) drops this call to the strict token test too — so those three read as name text again and keep the pre-#319 answer, while `Ph. D.` peels under the knob regardless, its merged `phd` passing the strict test. `田中さん, 太郎` is unchanged, and not because of its comma — the honorific there is not at the end of the name, 太郎 is. The nakaguro belongs to the same doctrine but is decided a layer down: U+30FB and its halfwidth twin U+FF65 separate tokens like whitespace, unconditionally and in tokenize, so neither policy opt-out (`script_orders={}`, `segment_scripts=()`) reaches it — the codepoints are CJK-only and appear in no other script's names, which is what licenses a tokenize-level rule where U+00B7 (also the Catalan punt volat, interior to Gal·la) needs the flanked-by-classified-script guard `_tokenize_region` gives it (#298). Han segmentation stays OPT-IN (`locales.ZH` for Chinese, `locales.JA` for Japanese) — a zh surname list corrupts Japanese kanji names, since 高 is a common Chinese surname and 高橋一郎 would split 高+橋一郎 where the correct reading is 高橋+一郎; no surname list divides a kanji name at all, so `locales.JA` activates the stage and a pluggable `Parser(segmenter=...)` does the dividing. Latin-script input is never touched by any of this: "Kim Min-jun" is genuinely order-ambiguous and stays governed by `name_order` and opt-in packs. Before adding a script-conditional rule, work out which of the three it is — certain, certain for this one behavior only, or a statistical guess wearing a script's clothes. @@ -199,7 +199,7 @@ The `config/` SETS are frozen since 2.2 (#293): there is no `.add()`/`.remove()` - `titles.py` — `TITLES` (every title reading; prenominals), `GIVEN_NAME_TITLES` (the readings with `addresses = "given"`, e.g. "Sir", which treat the following name as given, not family — a subset by construction) and `AFTER_GIVEN_TITLES` (the salutation readings with `after_given = true`, read as the title where one ends the given part after a family comma, rules/H8, #835 — a subset by construction, with no v1 `Constants` attribute); `TITLES_AMBIGUOUS` (a title reading beside a `given` one, derived, compiled INSTEAD of into `TITLES`: a title only inside a title phrase, rules/H9, #752 phase 2i), `TITLE_MODIFIERS` and `TITLE_STYLES` (by kind) and `TITLE_PHRASES` (the two-word phrases `attaches` makes), none with a v1 attribute - `suffixes.py` — `SUFFIX_ACRONYMS` (suffix readings with `acronym = true`, matched with every period removed, e.g. "M.D.") and `SUFFIX_WORDS` (the rest, e.g. "Jr."), disjoint because a word has one suffix kind; `SUFFIX_ACRONYMS_AMBIGUOUS` (an acronym reading on a word with a `given` or `family` reading, derived since 2026-10-25: the build refuses the phase 1 flag `may_be_name`); `SUFFIX_WORDS_AMBIGUOUS` (its word twin, a word reading on a word with a name reading, `fils` and the agnomens, gated by the count, rules/S11, since 2026-10-25) with `AGNOMENS` inside it (the `agnomen` kind, whose writing never leans); and `GLUED_HONORIFICS` (#308, `attached`), the subset of `SUFFIX_WORDS` the peel may split off the END of a name token — a separate, harsher bar, since the glued position has no writer-drawn boundary to lean on, and each word refused it carries `excluded = { attached = … }` - `particles.py` — `PARTICLES` (family-name particles, e.g. "de", "van") and `NON_GIVEN_NAME_PARTICLES`, the curated subset that is *never* a standalone given name (the particles whose word has no `given` reading, each stating `may_be_given = false`; a `given` reading on the word is what makes a particle possibly given, the phase 1 flag `may_be_given = true` refused since 2026-10-25). What the subset licenses is rules/P1's fold in `post_rules` (the "rule 1b" of older issue comments — decisions/legacy-rule-numbers maps the old numbers), and the fold has ONE site: the piece that OPENS the name. A lone never-given particle there, with at least one more name word behind it, starts the family. Titles do not move the opening position (rules/P4), so `Sir de Mesnil` reads family `de Mesnil` under every `name_order`, as `de Mesnil` does. How far the fold reaches depends on the order the name was read under: by default it takes the rest of the name (`de Mesnil Jean` → family `de Mesnil Jean`); under either family-first order it takes the particle run plus ONE name unit and lays the rest out by the order (`de Mesnil Jean` → family `de Mesnil`, given `Jean`); after a family comma it takes the rest of that part whatever the order (`Smith, de Mesnil Jean` → family `Smith de Mesnil Jean`). The leading particle stays a lone piece for the fold to find — `group` does not chain it forward, and the fold walks the particles after it itself — so `de la Vega` reads family `de la Vega` under all three orders. The GIVEN position is not a site. It was a second one until #467, which removed it because under a family-first order that slot holds what the caller DECLARED to be the given name: `Mesnil de` under `FAMILY_FIRST` or `FAMILY_FIRST_GIVEN_LAST` reads given `de`, family `Mesnil`. Nor is a particle inside the name: `Juan de la Vega` under a family-first order reads family `Juan`, given `de la Vega`, with no ambiguity recorded — what the vocabulary forbids is the bare particle as a given name, not a name part that begins with one. A bare `de` has nothing to fold into and is placed by position like any one-word name: given `de` by default, family `de` under both family-first orders. A leading particle OUTSIDE the set is left to position — `van Gogh` is given `van`, family `Gogh` by default and family `van`, given `Gogh` under both family-first orders — and records a `PARTICLE_OR_GIVEN` ambiguity under all three; a never-given particle opening the name records none (`de Mesnil`), and neither does an ambiguous one inside it (`Vincent van Gogh`). `Lexicon.particles_ambiguous` is its complement within `PARTICLES`, so the two mark OPPOSITE sets — see the flip warning in `docs/migrate.rst` before translating either -- `bound_given_names.py` — `BOUND_GIVEN_NAMES` (`bound_given` readings, e.g. "abdul", "abu"; one that is also a particle needs a `given` reading, a build check); a group-stage rule joins the first non-title piece to its following piece before roles are assigned, reserving a family word unless a family comma has already fixed it or a given-name title stands ahead (#369, rules/P5) (v1's `_join_bound_first_name`, ported into `_pipeline/_group.py` and gone from the tree — the v1 descriptions further down are history, not current code) +- `bound_given_names.py` — `BOUND_GIVEN_NAMES` (`bound_given` readings, e.g. "abdul", "abu"; one that is also a particle needs a `given` reading, a build check); a group-stage rule joins the first non-title piece to its following piece before roles are assigned, reserving a family word unless a family comma has already fixed it or a given-name title stands ahead (#369, rules/P5) (ported from v1's `_join_bound_first_name`) - `given_names.py` — `GIVEN_NAME_ABBREVIATIONS` (#789, a `given` reading carrying `same_as`, the full name: "chas", "wm", Filipino "ma"), the given names written short that rules/H2's shape yields to; no v1 `Constants` attribute, and nothing else reads it - `conjunctions.py` — `CONJUNCTIONS` (e.g. "and", "of") used to chain multi-word titles and to join name parts (rules/P3), plus `CONJUNCTIONS_AMBIGUOUS` (#383/#479, rules/P8, `may_be_initial`, refused by the build on anything but a cased single letter), the subset of single cased letters that read as an INITIAL in a name written wholly in one case — a marker set with no v1 `Constants` attribute of its own, so the only v1 knob that reaches it is deleting the conjunction - `maiden_markers.py` — `MAIDEN_MARKERS` (`birth_marker` readings, e.g. "née", "geb.") routing the following name to `maiden`; with an `addresses = "given"` title, the one kind a phrase may be (`z domu`) @@ -207,27 +207,11 @@ The `config/` SETS are frozen since 2.2 (#293): there is no `.add()`/`.remove()` - `capitalization.py` — `CAPITALIZATION_EXCEPTIONS` mapping of case MASKS (e.g. `{'phd': 'PhD'}`), compiled from the `cased` property, at most one per word: each value is its key's own letters and digits recased, laid over the word as written, so `ph.d.` repairs to `Ph.D.` and repair never writes the value's own punctuation into the word (#459); `Lexicon` raises `ValueError` on a value that spells anything else - `regexes.py` — dict of compiled regular expressions (wrapped in `RegexTupleManager` by `Constants`) -`config/__init__.py` wraps everything into `SetManager` and `TupleManager` instances inside a `Constants` class. A module-level singleton `CONSTANTS` is shared across all `HumanName` instances by default. +`config/__init__.py` re-exports the v1-compat `Constants`, `CONSTANTS` and managers from `_config_shim.py`. `Constants` wraps the constants in `SetManager` and `TupleManager` instances, and `Constants._snapshot()` translates them into the `Lexicon` and `Policy` the core parses with. A module-level singleton `CONSTANTS` is shared across all `HumanName` instances by default. -**Two-tier config pattern**: `CONSTANTS` is global; passing `None` as the second arg to `HumanName` creates a fresh per-instance `Constants()`. After modifying per-instance config you must call `hn.parse_full_name()` again. `SetManager` normalizes elements (lowercase, leading/trailing periods stripped) at every entry point — constructor, `add()`/`remove()`, set operators, and `in` membership checks (`__contains__`, #244) — and raises `TypeError` for bare strings/bytes and non-str elements, so callers don't need to worry about case. `TupleManager`/`RegexTupleManager` apply the equivalent bare-string/bytes guard to their own constructor, plus a check that no element of an iterable-of-pairs argument is itself a `str`/`bytes` — a 2-character string is a valid dict-update-sequence element, so without the check it silently shreds into a key/value pair (#242). +**Two-tier config pattern**: `CONSTANTS` is global; pass `constants=Constants()` for per-instance config (`None` raises `TypeError` since 2.0, #261). After modifying per-instance config you must call `hn.parse_full_name()` again. `SetManager` normalizes elements (lowercase, leading/trailing periods stripped) at every entry point — constructor, `add()`/`remove()`, set operators, and `in` membership checks (`__contains__`, #244) — and raises `TypeError` for bare strings/bytes and non-str elements, so callers don't need to worry about case. `TupleManager`/`RegexTupleManager` apply the equivalent bare-string/bytes guard to their own constructor, plus a check that no element of an iterable-of-pairs argument is itself a `str`/`bytes` — a 2-character string is a valid dict-update-sequence element, so without the check it silently shreds into a key/value pair (#242). -**`_SetManagerAttribute`/`_CachedUnionMember` descriptors**: all nine `SetManager`-backed `Constants` attributes (`prefixes`, `suffix_acronyms`, `suffix_not_acronyms`, `titles`, `first_name_titles`, `conjunctions`, `bound_first_names`, `non_first_name_prefixes`, `suffix_acronyms_ambiguous`) are managed by a descriptor so a non-`SetManager` assignment (e.g. `c.conjunctions = 'and'`) raises `TypeError` immediately instead of silently degrading later `in` checks into substring tests (#241). The four PST-contributing attrs (`prefixes`, `suffix_acronyms`, `suffix_not_acronyms`, `titles`) additionally use `_CachedUnionMember`, a subclass that also wires the `_pst` cache-invalidation callback on assignment; the other five use the plain `_SetManagerAttribute` base with no cache involved. Both store their values under the *private* name (`_prefixes`, `_conjunctions`, etc.) in the instance `__dict__` so the descriptor's `__set__` owns every assignment. Any code that inspects `__dict__` directly (e.g. `__getstate__`) must map `_xxx` → `xxx` for descriptor-managed attrs rather than filtering on `not k.startswith('_')`. - -**`_EmptyAttributeDefaultAttribute` descriptor** backs `empty_attribute_default` (1.4, #255): unlike `_SetManagerAttribute`, `__get__(obj=None, ...)` returns `''`, not `self` — deliberate, so `Constants.__repr__`'s `getattr(type(self), name)` default comparison still works and the pre-existing `str`-only mypy inference on the attribute (never widened to `str | None`, matching the descriptor-free days) is preserved. Because of that, `__getstate__`/`__setstate__` use `inspect.getattr_static(type(self), name)`, not plain `getattr`, to detect this descriptor (or the `property` used by the legacy-pickle migration shim below) without triggering `__get__` — a plain `getattr` there would silently miss both. `__setstate__` also bypasses the descriptor's own `__set__` (writes `_empty_attribute_default` directly) so restoring pickled/copied state doesn't itself emit the deprecation warning meant for user assignment. - -### Parser (`nameparser/parser.py`) - -`HumanName` is the single public class. Assigning to `full_name` (or instantiating with a string) triggers `parse_full_name()`. - -Parse flow: -1. `pre_process()` — strips nicknames/maiden names (parenthesis/quotes, routed to `nickname_list`/`maiden_list` per `Constants.nickname_delimiters`/`maiden_delimiters`) and emoji, fixes "Ph.D." variant spellings -2. Split on commas → 1 part (no comma), 2 parts (suffix-comma or lastname-comma), 3+ parts -3. `parse_pieces()` — splits on spaces, detects dotted abbreviations like "Lt.Gov." and registers them as per-parse derived titles/suffixes (the `_derived_*` sets — never the config; see Gotchas) -4. `join_on_conjunctions()` — merges pieces adjacent to conjunctions into single tokens (e.g. `['Secretary', 'of', 'State']` → `['Secretary of State']`); also joins prefix particles to the following lastname token — a piece merged with a title *or prefix* neighbor is re-registered into the per-parse derived title/prefix set so later steps still recognize it, e.g. `von`+`und`+`zu` → registered as a derived prefix so the whole phrase joins to the last name (German "von und zu"; PR #191) 4a. `_join_bound_first_name()` — called immediately after step 4 in all three paths that build a first-name-bearing token sequence: no-comma, lastname-comma (post-comma pieces), and suffix-comma (`parts[0]`); merges bound given-name prefixes (e.g. "abdul") with the next piece before the assignment loop runs; suffixes are still in `pieces` at this point, so the `reserve_last` guard must count non-suffix pieces only. Not called for the lastname portion itself (`lastname_pieces` in the lastname-comma path) — that token sequence is a surname, not first-name text, so the join must not apply there. The join lives at each of these three call sites individually rather than inside `parse_pieces()` itself. That's because `parse_pieces()` is also called for the lastname portion with the same `additional_parts_count` value used for the (joinable) post-comma given-names portion, so `additional_parts_count` alone can't disambiguate which callers want the join. -5. Iterates pieces, assigning to `title_list`, `first_list`, `middle_list`, `last_list`, `suffix_list` -6. `post_process()` — `handle_firstnames()` swaps first/last when only a title + one name; then, gated on `patronymic_name_order`, `handle_east_slavic_patronymic_name_order()` and `handle_turkic_patronymic_name_order()` reorder Russian-formal-order and reversed Turkic patronymics; then, gated on `middle_name_as_last`, `handle_middle_name_as_last()` folds `middle_list` into `last_list`; finally `handle_capitalization()` applies optional auto-cap. Any new `self._attr` used by `post_process()` helpers must be initialized in `__init__` (with its default value) — the direct-kwargs path bypasses `parse_full_name()`, so the attribute won't exist otherwise. - -Each named attribute (`title`, `first`, etc.) is a `@property` that joins its corresponding `_list`. Setters call `_set_list()` which runs the value through `parse_pieces()`, so assigning `hn.last = "de la Vega"` correctly re-parses prefix tokens. +**Assigning a non-manager to a `Constants` set field raises `TypeError`** (`c.conjunctions = 'and'`, #241), in `Constants.__setattr__`: a bare string stored there would turn later `in` checks into substring tests. ## 2.0 API modules (`nameparser/_*.py`, `tests/v2/`) @@ -258,26 +242,11 @@ The 2.0 rewrite lands as underscore-private modules alongside the v1 code. These ## Extension Patterns -**Adding a scalar `Constants` attribute + `HumanName` kwarg** (e.g. `initials_separator`, `suffix_delimiter`): -1. Add class attr to `Constants` in `config/__init__.py` with docstring -2. Add `x: str | None = None` to `HumanName.__init__` signature after related kwargs -3. Add `self.x = x if x is not None else self.C.x` in body — use `is not None`, not `or`, to allow falsy values like `""` -4. conftest auto-restores scalar CONSTANTS between tests, but tests that *set* CONSTANTS mid-run still need their own try/finally -5. Update `docs/customize.rst`'s constants list (and `docs/usage.rst` if it affects a documented example) — don't wait for the release checklist, these `.rst` docs aren't covered by CI so a stale one can ship silently. Check `AGENTS.md` itself too (Extension Patterns, Gotchas, the Architecture section) for now-stale attribute/test names or descriptions — it has the same blind spot as the `.rst` docs and the release checklist only catches it at release time. - -**Adding a new mutable/collection `Constants` attribute** (a `SetManager`/`TupleManager`-backed group, e.g. `nickname_delimiters`/`maiden_delimiters`): add it to `_COLLECTION_CONFIG_ATTRS` in `tests/conftest.py`, or tests that mutate the global `CONSTANTS` copy will leak state into later tests. Contents must be deep-copyable (the snapshot uses `copy.deepcopy`) — already true for the existing manager types. (The parse-no-mutation tests in `tests/test_constants.py` need no such registration — they discover collections structurally via `Constants.__getstate__()`; conftest's list is the only manual one.) - -Add a dedicated `copy.deepcopy()` round-trip test for it too (see `test_regexes_deepcopy_roundtrip`/`test_nickname_delimiters_deepcopy_roundtrip` in `tests/test_constants.py`), not just reliance on conftest's autouse snapshot/restore exercising it incidentally. `TupleManager`/`RegexTupleManager.__getattr__` answer *any* unknown attribute lookup — dunder probes like `__deepcopy__` raise `AttributeError` (never answered as config keys), everything else still returns `self.get(attr)`/`EMPTY_REGEX` — so a new manager subtype or a `__getattr__` tweak can silently break `copy.deepcopy` (this bit `RegexTupleManager` before the dunder-lookup guard was added). A direct test on the new attribute's own manager instance catches that where the conftest fixture, which never asserts on the copy, would not. As with the scalar-attribute pattern above, also check `AGENTS.md` itself for now-stale references when you touch this. - -**Unknown-key attribute access on `TupleManager`/`RegexTupleManager` warns (1.4, #256) — 2.0 note: the warning became a hard `AttributeError` naming the known keys (shim `TupleManager`); the paragraph below describes the deleted v1 machinery and applies only when reading v1 history.** — a key not currently in the dict emits `DeprecationWarning` naming the miss and the known keys (`_warn_unknown_key`), before falling back to the same `None`/`EMPTY_REGEX` default as before; `.get()` stays silent. Dunder probes (`__deepcopy__`) still raise `AttributeError` outright, and single-underscore probes (`_repr_html_`, IPython's `_ipython_canary_method_should_not_exist_`, etc.) are excluded from the warning too — no real config key starts with `_`, so both guards just keep protocol/introspection probes from misfiring as "typo" warnings. This means internal parser code that reads `self.C.regexes.` unconditionally (e.g. `squash_bidi`'s `bidi`) now warns if a caller's custom `regexes` dict omits that key — a previously-silent partial-override pattern is on the same deprecation path as an actual typo. - **Adding a word to the vocabulary** — the steps are in `nameparser/vocab/README.md`; this paragraph is the traps they have run into. First check the word's OTHER readings: `uv run python tools/vocab/query.py --word ` lists every reading and exclusion of it in every file, which is where the overlaps between categories now show. Real overlaps exist: `st`/`freiherr` ∈ `PARTICLES` ∩ `TITLES`, and `do`/`mc`/`vd` ∈ `PARTICLES` ∩ `SUFFIX_ACRONYMS`; `abd` = "ABD" ∈ `SUFFIX_ACRONYMS`; `abu` ∈ `PARTICLES` ∩ `BOUND_GIVEN_NAMES` (position-dependent: leading token → given-name join, mid-name → family-name join). Usually position-dependent and harmless, but can force a guard or an exclusion (the `last_base` all-particles guard; `abd`, kept out of `BOUND_GIVEN_NAMES` until decisions/P5 found the credential collision cost nothing at either end). Then check the word against real names, not memory: `uv run python tools/name_usage.py ` counts its bearers as a given name and as a family name (US Census surnames, SSA given names, Wikidata with countries of citizenship), and gives its given-name frequency band per country from Jörg Michael's `nam_dict.txt` (54 countries and an "other countries" column, GFDL, downloaded into the cache and never committed) — the one source that sees a given name outside the US and Wikidata's lower bound, such as Korean `In` (band A in Korea) or Vietnamese `Bay` (6 in Vietnam). A band is a frequency class, not a count, and the dictionary is a selection, so a miss there (`Shehu`) proves nothing; and because a truncated source reads as a run of false zeros, a pinned download is checked against its sha256 on every load from the cache, never against a floor on what it holds — #775's first fix refused fewer than 40,000 names and passed a `nam_dict.txt` missing only its last line (`Zyta`). Pass a name as it is written (`McDonald`): Wikidata labels match exactly, and no rule rebuilds an internal capital from `mcdonald`. A word in a non-Latin script is counted by its romanization too, since the Census, SSA and nam_dict hold only Latin names, and that count weighs only where its bearers write the script: #835 vetted its salutations by their own spellings and its review found `बाबू` flagged while `babu` is a US surname (1,496) and an Indian given-name element (Ram Babu), where the romanization of `মি`, `Mi`, counts Korean and Chinese bearers who never write Bengali. It reports and does not decide — read the counts against the position the set's claim acts on. For a suffix it also has the other side: `--fetch-nppes` counts the US healthcare providers who write the word as a credential (NPPES, the 4.8 million credentialed providers in the pinned August 2026 file, so blind to law, accounting and honours), and `--rank N --unlisted` lists the commonest credentials the vocabulary misses, with where each lands in `John Smith, X` -- the list #797 came from. Where a local corpus is fetched (`tools/differential/local_corpora.py`, mechanisms.md#NAME-DATA-JOBS), a row says how many of its records (for ClinicalTrials.gov, officials) write the word, and how many after a comma: ClinicalTrials.gov's study officials show where a credential actually stands in a name as typed. A word that would peel GLUED (`GLUED_HONORIFICS`) asks a different question, whether any name ENDS in it, and for a CJK word the tool answers that too: an ends-with row from a cached index of Wikidata's CJK name labels (built on first use; 33 minutes on 2026-10-17), plus the names the word ends across the family/given join (阿部 + 長 for 部長). Read the tool's hits by item before counting them, because labels include transcriptions and mislabels: #707's one hit for 二世 was 厄德二世, the given name Odo with a regnal numeral in its Chinese label, borne by 109 Western people named Odo. Then ask a third question the tool cannot: does a common longer word end in it? The peel matches the end of a token, spaced or glued, so a listed tail splits every unlisted compound that ends in it — #707 vetted 議員, 社長 and 医学博士 against names and its review found 衆議院議員, 副社長 and 獣医学博士 cut at the tail (`田中 副社長` → given 副); no corpus held a compound office word until #707's rules examples added two, so the snapshot was silent about the class. List the common compounds whole or record the split as a cost. And check the predicates that test a member's SPELLING rather than its membership: they encode what the set's members happened to look like when they were written. `_pieces.starts_a_credential_run` implemented the credential run's "Latin" (rules/S8 now, part of S2 until #682 split it) as `isascii()`, exact until #661 added `mlađi`, which would have read `mlađi Jones` differently from `stariji Jones`; the design-docs review caught it, after a first draft had credited the test to the wrong clause of S2 and proposed deleting it on a mutation that moved nothing because nothing pinned it. (#679 then dropped the script test on purpose, after measuring what it protected.) Before adding the first member of a new script, or the first accented one, grep the set's consumers for `isascii`, `isalpha` and script tests, and read the rule each one implements. The same goes for the first member holding PUNCTUATION: #700 added `d'` and `az-`, the parse read them right and every snapshot move was an intended one, and `capitalized()` still wrote `D' Ivoi`, because case repair splits a token at its apostrophe or hyphen and asks the vocabulary of the stem. The snapshot records both case views since #826, but only for the names the corpora hold, so parse such a member's names through `capitalized()` and `capitalized(force=True)` in each case before calling it done. The user docs use real words as examples of an UNLISTED one, and listing the word breaks the example: #797 listed `dphil`, and docs/customize.rst's capitalization example (`parse("jane smith dphil").capitalized()` giving `Dphil`) failed in all five CI build jobs while `uv run pytest` passed, because `sphinx-build -b doctest docs` is a separate step. Run it after adding a word, and grep `docs/*.rst` for the word; the example now uses `dclinpsy`, a credential left unlisted on purpose, so list it only after moving the example. Record what the vetting found ON the word, not in a comment or a decisions sentence: `note`, `issue`, a `given`/`family` reading with dated counts, and for a word you decided to keep out an `excluded` entry with its reason, so the next contributor who adds it meets the reason as a build error (decisions/vocabulary-model). A name reading is not only provenance on a word that has a credential, particle or title reading: it IS the clash, and derives the marker (an ambiguous acronym, a particle that may be given, since 2026-10-25, a generation or agnomen that may be a name, rules/S11, since phase 2h the same day, so given-name counts recorded on `junior` would gate it, and a title that may be a given name, rules/H9, since phase 2i), so counts kept as evidence AGAINST marking such a word go in the clashing reading's `note` instead, as `ld`'s do. A word whose reading moves may be the worked example of the class it is leaving: `king` was the rules' and tests' title-that-is-also-a-surname across 35 files when #901 gave it a given reading, and five tests went vacuous before the review found them. Grep rules, decisions Standing lines, the RFCs, AGENTS.md, docs/*.rst, code comments and tests for the word before measuring, and replace it with a stand-in checked to read exactly as the word read on master in every example (`pope` there). A new derivation reads readings other work recorded for another purpose, so list what it would mark before shipping it: phase 2i's first build found thirteen salutations whose given readings #835 had kept as H8's evidence (`mr` among them, so `Mr Smith` would have read given 'Mr'), and moved their counts into notes. **Before adding a short/common word to `PARTICLES` globally**, test it mid-string against realistic 3-token names, not just check for English-word collisions: Korean/Vietnamese given names put a short syllable in the middle slot (`Park In Hwan`, `Nguyen To Nga`), and Western names put a bare initial there (`John V. Smith`). A word that looks safe ("nobody is named 'to'") can still swallow a real middle name/initial into the last name once it's a global prefix — confirmed regressions for `to`/`in`/`an`/`ten`/`then` and bare `v` this way (PR #191). -**Adding a curated sub-set of an existing vocabulary set** (a marker over a base set, as `GIVEN_NAME_TITLES` is over `TITLES`) — it is a behavioral PROPERTY of a reading, not a second word list: a row in `tools/vocab/build.py`'s `BEHAVIORAL` table, a branch in `compile_pool` adding the words whose reading carries it, a constant in its `CONSTANTS` roster and a key of the same name in its `Compiled` TypedDict (an import-time assert holds the two to one list), and the `NAME = _compiled.NAME` assignment in the config module. The subset relation then holds by construction, since the property cannot exist without its reading, so it needs no check. A marker that records a clash with a NAME is the other shape: derived from the word's name reading beside the clashing one (`SUFFIX_ACRONYMS_AMBIGUOUS`, `SUFFIX_WORDS_AMBIGUOUS`, and the complement `NON_GIVEN_NAME_PARTICLES` works from), not written as a property, so it too holds by construction (decisions/vocabulary-model, 2026-10-25). A relation construction does not give (`∩ == ∅` with a set the sub-set cannot logically overlap; a sub-set member that is also a title is silently inert, title handling consuming it first) goes in `check_pool`, with a mutation in `tests/v2/test_vocab.py`'s table. Do **not** forbid `titles ∩ particles` — that overlap is intentional (`freiherr`, `st`). On the v1 side the sub-set is a `_SetManagerAttribute` on `Constants` where v1 had one (like `first_name_titles`; a sub-set new in 2.x, such as `honorific_tails`, `conjunctions_ambiguous`, `after_given_titles` or `suffix_words_ambiguous`, gets none and rides in `_snapshot()` as an intersection with its base, the shim roster under the 2.0 conventions listing each — the v1 ATTRIBUTE names did not move in #293, only the module constants did), **not** a `_CachedUnionMember` — only `prefixes`/`suffix_acronyms`/`suffix_not_acronyms`/`titles` feed the `_pst` hot-path cache, so a sub-set costs nothing at runtime and stays out of `is_rootname`. A runtime `.add()` to the sub-set's `SetManager` does **not** propagate to the parent's (same as `first_name_titles`→`titles`), so a caller adding a brand-new word adds it to both. Two traps a gating set met (#752 phase 2i). Where a caller listing a word in the BASE set has always meant "read it as this everywhere" (the 1.4 recipe `c.titles.add("dean")`), the gated words compile OUT of the base rather than into both, or the gate overrides the caller's listing: `TITLES_AMBIGUOUS` is not a subset of `TITLES` for that reason, and a word in both reads as a title. And `_VOCAB_FIELDS` is read whole by two rosters in `_pipeline/_vocab.py`, the capitals shape's "unlisted" (`in_any_wordlist`) and the comma contrast's "claimed as non-name" (`_NON_NAME_FIELDS`), so a new field joins both by default; a field whose words may be names leaves both, which the grid found as `John Smith, DEAN` reversed where C6 had read a credential; #789's `given_name_abbreviations` met it again (`John Smith, JAS` and `Jas SMITH, XYZ` reversed), found by review because its grid held only the dotted spellings the set was for -- a new field's grid holds the spellings it must NOT move too. - -**Adding a flag-gated post-parse transform** (reorder/adjust) — add a `Constants` boolean (default `False`), implement a `handle_*()` method, and call it in `post_process()` after `handle_firstnames()` and before `handle_capitalization()`, gated on the flag. Default-off keeps existing parses byte-for-byte unchanged. Two shipped examples: `patronymic_name_order` gates both `handle_east_slavic_patronymic_name_order()` (#85) and `handle_turkic_patronymic_name_order()` (#185) — one flag driving two independent handlers, added in the same `post_process()` slot; `middle_name_as_last` gates `handle_middle_name_as_last()` (#133), which folds `middle_list` into `last_list`. +**Adding a curated sub-set of an existing vocabulary set** (a marker over a base set, as `GIVEN_NAME_TITLES` is over `TITLES`) — it is a behavioral PROPERTY of a reading, not a second word list: a row in `tools/vocab/build.py`'s `BEHAVIORAL` table, a branch in `compile_pool` adding the words whose reading carries it, a constant in its `CONSTANTS` roster and a key of the same name in its `Compiled` TypedDict (an import-time assert holds the two to one list), and the `NAME = _compiled.NAME` assignment in the config module. The subset relation then holds by construction, since the property cannot exist without its reading, so it needs no check. A marker that records a clash with a NAME is the other shape: derived from the word's name reading beside the clashing one (`SUFFIX_ACRONYMS_AMBIGUOUS`, `SUFFIX_WORDS_AMBIGUOUS`, and the complement `NON_GIVEN_NAME_PARTICLES` works from), not written as a property, so it too holds by construction (decisions/vocabulary-model, 2026-10-25). A relation construction does not give (`∩ == ∅` with a set the sub-set cannot logically overlap; a sub-set member that is also a title is silently inert, title handling consuming it first) goes in `check_pool`, with a mutation in `tests/v2/test_vocab.py`'s table. Do **not** forbid `titles ∩ particles` — that overlap is intentional (`freiherr`, `st`). On the v1 side a sub-set new in 2.x, such as `honorific_tails`, `conjunctions_ambiguous`, `after_given_titles` or `suffix_words_ambiguous`, gets no `Constants` attribute and rides in `_snapshot()` as an intersection with its base, the shim roster under the 2.0 conventions listing each (the v1 ATTRIBUTE names did not move in #293, only the module constants did). A runtime `.add()` to the sub-set's `SetManager` does **not** propagate to the parent's (same as `first_name_titles`→`titles`), so a caller adding a brand-new word adds it to both. Two traps a gating set met (#752 phase 2i). Where a caller listing a word in the BASE set has always meant "read it as this everywhere" (the 1.4 recipe `c.titles.add("dean")`), the gated words compile OUT of the base rather than into both, or the gate overrides the caller's listing: `TITLES_AMBIGUOUS` is not a subset of `TITLES` for that reason, and a word in both reads as a title. And `_VOCAB_FIELDS` is read whole by two rosters in `_pipeline/_vocab.py`, the capitals shape's "unlisted" (`in_any_wordlist`) and the comma contrast's "claimed as non-name" (`_NON_NAME_FIELDS`), so a new field joins both by default; a field whose words may be names leaves both, which the grid found as `John Smith, DEAN` reversed where C6 had read a credential; #789's `given_name_abbreviations` met it again (`John Smith, JAS` and `Jas SMITH, XYZ` reversed), found by review because its grid held only the dotted spellings the set was for -- a new field's grid holds the spellings it must NOT move too. **Validating a new parsing rule** — before implementing, simulate it in a throwaway script against `TEST_NAMES` (plus a few target-language examples) to catch regressions/false-positives early. E.g. this surfaced the `last_base` `do`/`st`/`mc` empties and the patronymic `"David Michael Abramovich"` false-reorder. After the change, `tests/v2/test_snapshot.py` lists every corpus name it moved, grouped by changed field -- read them, and re-record with `uv run python tools/differential/snapshot.py --record` once every move is intended, so the PR diff of `snapshot.jsonl` shows the reviewer the same list. `uv run python tools/differential/cldr_agreement.py` reports how the tree's agreement with CLDR's stated name fields moved; `tests/v2/test_cldr_agreement.py` fails on any move, and `tools/differential/cldr_agreement.json` doubles as a measured list of names the parser and CLDR read differently. The same command and test score the GND corpus's real names (`gnd_fields.jsonl`, record `gnd_agreement.json`) by the same rules, and `--record` rewrites every record, printing per source what it accepts. A locale-pack change reads the second view the same command prints, per pack (`cldr_agreement_locales.json`, #672, and `wikidata_native_agreement_locales.json` for real Han and kana names); its `JA` rows run only with the `ja` extra installed. A change to Han, Hangul or kana order or to Korean segmentation reads `wikidata_native_agreement.json` too: the only real names in those scripts the corpora hold. It holds no honorific, so a change to the honorific peel moves nothing there. A change to comma or credential handling also reads the local corpus of hand-typed names (`snapshot.py --local ctgov-officials --base `, which reads the base from a baseline kept for its package's files and never the shared snapshot beside the corpus (a base read once from the same fetch is copied rather than parsed again; `--sample N` reads a fixed sample while iterating; the shared snapshot beside the corpus is one file per source for every worktree, and #752's phase 2f found ctgov reporting a sibling branch's 1,334 joined-credential readings as its own moves against it); tools/differential/README.md's "Local corpora"): the one population of names nobody chose for a test or a catalog, so it shows how often each shape is typed, and the one no CI run sees. A change to title handling reads `ukec-individuals` the same way: UK donors' names in spoken order and mixed case, in 8,219 of whose 15,214 `parse()` read a title on 2026-10-22. A change to particles or generational words reads `snapshot.py --local tse-donors` the same way, Brazilian names in spoken order dense with both. A change to initials, to a title word outside English, or to how a script's combining marks are read reads `snapshot.py --local nepal-candidates` the same way: Nepal's local election candidates, each a romanized name (surnames written as initials, `B.K.`, `K.C.`; honorific titles such as `Shekh` in front, and `Md.`, which its period makes a title by shape) beside the same name in Devanagari, written with vowel signs and the virama, combining marks the letter-shape tests do not count as letters (decisions/nfd-shapes), and with the joiners ZWJ and ZWNJ. It is the only large population of South Asian names the corpora hold, and its corpus reads both scripts. A change to non-Latin scripts or to Arabic-script names reads `snapshot.py --local jrc-names` the same way, and `local_corpora.py groups jrc-names`: the spellings of one person across scripts and transliterations, harvested from news text, with 17,074 Arabic-script spellings where the committed corpora hold 315 distinct names with an Arabic letter, 96 of them CLDR's fictional renderings (2026-10-24), and under a licence that keeps even a reading of one of its names in the cache. A change to which field a word lands in also reads `local_agreement.py psc-individuals --base ` the same way: real names with the four boxes their filers typed them into (UK companies' people with significant control), scored field by field and by nationality. It is the only real fielded population outside the committed fields files, and no CI run sees it either. @@ -289,11 +258,11 @@ Add a dedicated `copy.deepcopy()` round-trip test for it too (see `test_regexes_ **A local run on 3.11 cannot see a deprecation newer interpreters raise** — `.python-version` pins 3.11, pytest's `filterwarnings = ["error"]` turns every warning into a failure, and CI runs 3.11 through 3.15, so a stdlib call deprecated in a later release passes here and fails the newer jobs. #795 passed `maxsplit` to `re.split` positionally (deprecated in 3.13) and failed four jobs while 3.11 and 3.12 passed. Before pushing new code, run it on the newest interpreter CI uses too: `uv run --python 3.15 pytest -q` (uv fetches it; the next plain `uv run` rebuilds the 3.11 venv). Pass `maxsplit`, `count` and `flags` to `re.split`/`re.sub` by keyword. -**Parsing must never write into `Constants`** — parse-time derived recognition (dotted abbreviations like "Lt.Gov.", conjunction-joined titles/prefixes like "Mr. and Mrs." / "von und zu") lives in per-instance `HumanName._derived_titles` / `_derived_suffixes` / `_derived_conjunctions` / `_derived_prefixes` sets, consulted by `is_title`/`is_suffix`/`is_conjunction`/`is_prefix`/`is_rootname`, reset at the start of `parse_full_name()`, and backfilled in `__setstate__` for pre-existing pickles. Never `self.C..add()` during parsing — `self.C` is usually the shared `CONSTANTS` singleton, so a write makes parse results order-dependent and thread-unsafe (this was a real bug through 1.2.x). `ParsingDoesNotMutateConfigTests` (`tests/test_constants.py`) enforces the invariant by snapshotting the whole config around a parse; it discovers collections structurally via `Constants.__getstate__()`, so new `Constants` collections are watched automatically — nothing to register. If a new derived category is ever needed: add a `_derived_*` set in `__init__`, reset it in `parse_full_name()`, backfill it in `__setstate__`, consult it in the matching `is_*` predicate (store `lc()`-normalized values, mirroring `SetManager`), and add a leak test with a name that triggers it. +**Parsing must never write into `Constants`** — `self.C` is usually the shared `CONSTANTS` singleton, so a write makes parse results order-dependent and thread-unsafe (a real bug through 1.2.x, when v1 registered dotted abbreviations like `Lt.Gov.` into the config). `ParsingDoesNotMutateConfigTests` (`tests/test_constants.py`) snapshots the whole config around a parse, discovering collections structurally via `Constants.__getstate__()`, so a new collection is watched with nothing to register. **The `nameparser/config` vocabulary constants are frozen, and the 1.x names for them are a bridge for CALLERS only** (2.2, #293) — `TITLES.add("dean")` raises `AttributeError` at the line that writes it, so a runtime addition goes on a config OBJECT instead: `c = Constants(); c.titles.add("dean"); HumanName(name, constants=c)` for the v1 API, `Parser(lexicon=Lexicon.default().add(titles={"dean"}))` for the 2.0 one. Both are warning-free; mutating the shared `CONSTANTS` still works but warns. What the freeze buys is measured in `docs/migrate.rst`: `Lexicon.default()` is `functools.cache`d and reads the constants once, a v1 `Constants` copies them at every construction, and the shared `CONSTANTS` is a copy taken at import — so pre-freeze, an edit after the first parse reached only a freshly built `Constants`, an edit before any parse reached `parse()` and a fresh `Constants` but still not the shared singleton, and which of those happened depended on nothing the reader could see. Separately, every retired 1.x name (`PREFIXES`, `NON_FIRST_NAME_PREFIXES`, `BOUND_FIRST_NAMES`, `FIRST_NAME_TITLES`, `SUFFIX_NOT_ACRONYMS`) still resolves for a caller, with a `DeprecationWarning` naming its new path — but nothing inside `nameparser/` may spell one, in code OR in a comment: `tests/v2/test_config_aliases.py::test_no_internal_code_reads_a_retired_vocabulary_name` scans every `.py` in the package and fails on a hit outside the alias table that owns it. What an internal read costs is attribution: the warning names the file that performed the read, so the caller is told about a deprecation inside `nameparser/` they cannot act on, and a run under `-W error::DeprecationWarning` dies in library code. It does *not* cost the caller their own warning, and the reason is the warnings module rather than anything this bridge does: `__warningregistry__` lives in the READING module's globals, so repeats are suppressed per LOCATION, and a library read at `internal_reader.py:1` plus a caller read at `real_caller.py:1` produce two warnings — the same two that a plain `warnings.warn` from two files produces, verified against that control. What the bridge contributes is the *absence* of a write-back: `config/_deprecated.py` keeps serving retired names through `__getattr__` for the life of the process instead of caching the resolved value into module globals, which would silence every reader after the first — and the first is whoever imported earliest, routinely a dependency whose author is not the person who has to edit anything. Don't "optimize" that lookup. That scan does not reach `docs/` or this file, which is why the rename needed a prose sweep of its own. -**`HumanName.C` is a property backed by `_C`, but pickles under the public key `'C'`** — `__init__`/direct assignment route through the `C` setter, which calls the shared `_validate_constants` staticmethod (also used by `__init__`) so an invalid value raises `TypeError` immediately instead of surfacing later as an unrelated `AttributeError` deep in parsing (#239). `__getstate__`/`__setstate__` deliberately translate `self._C` ↔ a `'C'` key in the pickled dict (with the usual `CONSTANTS`-singleton-becomes-`None` sentinel) rather than pickling `_C` directly, so the on-disk pickle format hasn't changed across this fix — don't "simplify" that translation away or old pickles/tests that hand-build a state dict with a `'C'` key will break. +**`HumanName.C` is a property backed by `_C`, but pickles under the public key `'C'`** — `__init__` and direct assignment both validate, so an invalid value (or `None`, removed in 2.0, #261) raises `TypeError` immediately instead of surfacing later deep in parsing (#239). `__getstate__`/`__setstate__` deliberately translate `self._C` ↔ a `'C'` key in the pickled dict (with the `CONSTANTS`-singleton-becomes-`None` sentinel) rather than pickling `_C` directly, so the on-disk pickle format is v1's — don't "simplify" that translation away or 1.4 pickles and tests that hand-build a state dict with a `'C'` key will break. **Titles permanently shadow first names — be conservative** — any word in `TITLES` is always consumed as a title and can never be parsed as a first name. `"Dean"` is the canonical example: it's a common academic title *and* a common given name, so it is not in `TITLES` but in `TITLES_AMBIGUOUS`, a title only inside a title phrase (`Dean of Students`, `Associate Dean`; rules/H9, #752 phase 2i), and users who need it everywhere add it via opt-in `Constants`. A title that is also a given name now has that machinery: a `given` reading on the word moves it there, decided word by word in measured batches (decisions/vocabulary-model's 2026-10-24 order). A rule that reads a title by its neighbours in the title run meets more phrase shapes than the qualifier-then-head it was designed on: the object of a connective (`President of the United States`, `the Queen's Bench`, whose `united`, `states` and `bench` are modifiers with no head after them), a head before `of` (`Member of Parliament`), and a qualifier written after its head (`Chargé d'Affaires`, `Princess Royal`). Phase 2i's first build, "a modifier needs a head title after it", broke `President of the United States`, `Chargé d'Affaires` and the committed `Queen's Bench` row, found by the case table and the committed corpora, and then read a modifier ENDING the name out of H5's trailing run (`Jane Smith Vice.`), found by the design-docs review; put each shape, and the trailing position, in the grid of any such rule. Before adding a word to `TITLES`, ask: "Could this plausibly be someone's given name in any culture?" If yes, don't add it globally; it belongs in caller-supplied `Constants` instead. This same caution applies to international honorifics — `Prince`, `Sheikh`, `Frau` are all first names in some contexts. It also applies to any prefix sub-set gated on "never a first name": obscure-looking foreign particles are surprisingly often real given names — `Von` (Von Miller), `Vander` (Brazilian, also the Arcane character). When unsure, exclude — a missing member just means that name isn't auto-handled, whereas a wrong member misparses a real person. The decided exception is a word borne as a name only RARELY in the leading position — mostly a surname written last (`graf`, `greve`, `knight`), a few with small given-name or surname-first counts (`herra`, `hrabia`): the title claim acts only in front, so it ships, and a name that begins with it (`Greve Anna`, `Herra Wijaya` → title) is the accepted cost. That is a judgment by incidence, made per word, with the counts taken recorded on the word's entry in `nameparser/vocab/` (herra's, hrabia's; graf, which predates #606, and greve and knight, which #606 proposed, carry none) and the class in `decisions/salutation-titles` (rules/H1's Accepted clause states the cost), not a rule a sweep can apply: a word that commonly leads a real name (Vietnamese `Ông`, the given names `thiru`, `puan`, `marshal`) is still out. `tools/name_usage.py` produces the counts this judgment starts from (role counts, plus Wikidata citizenship to tell where a family name is written first); its Wikidata column is a lower bound (Shehu Shagari has no given-name statement), so a small count there proves little and a zero proves nothing, and its nam_dict row gives a band per country rather than a count, with a missing name proving nothing either. @@ -301,7 +270,7 @@ Add a dedicated `copy.deepcopy()` round-trip test for it too (see `test_regexes_ **Cyrillic suffix regexes need `re.I` even when the pattern is suffix-only** — a Latin title-cased word (`Ivanovich`) keeps its suffix lowercase, so `re.I` seemed skippable; but an irregular Cyrillic suffix can be nearly the whole word (`ильич`), so title-casing capitalizes into the suffix itself (`Ильич`). `east_slavic_patronymic_cyrillic` shipped without `re.I` on the Latin reasoning and silently failed on capitalized irregular forms — don't assume Latin's title-case safety transfers to Cyrillic. (#185) -**`suffix_not_acronyms` vs `is_an_initial` tension** — single-letter roman numeral suffixes (`i`, `v`) are in `suffix_not_acronyms` but also match the `is_an_initial` regex (single uppercase letter), so `is_suffix()` rejects them. The lenient test lives in `is_suffix_lenient()`, which accepts `suffix_not_acronyms` members unconditionally and is only safe in unambiguous positions: (1) suffix-comma detection uses it via `are_suffixes_after_comma()`; (2) lastname-comma post-comma parsing uses it inline, only when `nxt is None` and `len(parts)==2` (no `parts[2]` suffix segment). See issues #136, #144. **The tension is Latin-only since #320** — the veto is now scoped to scripts that HAVE initials, and v1's single `is_an_initial` is now two predicates, of which the tension is about `_vocab.is_initial` (the one `is_suffix()`/`is_suffix_strict` vetoes with; `is_initial_shaped` is the bare v1 shape test and vetoes nothing) — `_vocab.is_initial` ANDs that shape test with "not in `_policy._NO_INITIALS`", that constant listing the four scripts which do NOT (Han, Hangul, Hiragana, Katakana) — so the seven single-character CJK members of `suffix_not_acronyms` (씨, 様, 氏, 군, 양, 님, 殿) no longer collide with it when written with a period: `"김민준 씨."` gives suffix `씨.` where the veto had made it the family name, and `"김민준, 씨."` where it had made it the given name. Latin is untouched — `V.`/`I.` still lose to the veto and still need `is_suffix_lenient()`. +**Suffix words vs the initial veto** — single-letter roman numeral suffixes (`i`, `v`) are suffix words but also initial-shaped, so `_vocab.is_suffix_strict` (v1's `is_suffix`) vetoes them through `_vocab.is_initial`. `_vocab.is_suffix_lenient` accepts suffix words without the veto and is only safe in unambiguous positions, after a comma (#136, #144). **The veto is Latin-only since #320**: `is_initial` ANDs the bare shape test (`is_initial_shaped`) with "not in `_policy._NO_INITIALS`", the four scripts that have no initials (Han, Hangul, Hiragana, Katakana), so the single-character CJK suffix words (씨, 様, 氏, 군, 양, 님, 殿) written with a period read as suffixes: `"김민준 씨."` gives suffix `씨.`. `V.`/`I.` still lose to the veto and still need the lenient test. **Comparing against v1 needs `PYTHONSAFEPATH=1` AND a directory outside the worktree** — `uv run --isolated --no-project --with 'nameparser==1.4.0'` still puts the checkout's `nameparser/` ahead of the pinned wheel on `sys.path`, so the "v1" side silently imports the branch and every comparison reports parity. Run it as `cd && PYTHONSAFEPATH=1 uv run --isolated --no-project --with 'nameparser==1.4.0' python -c "..."`. This produces false confidence rather than an error, so it invalidates results without ever looking wrong. **A `tests/v2/cases.py` row's `parity` means 1.4.0, and nothing checks it** -- the table runs every row against the tree, so a row labelled `parity` that only matches master passes for good. #614 added eight such rows, compared against master; #620 found them by running the wheel this way and relabelled each with the change its reading came from, found by `git bisect run` in a scratch worktree (`git worktree add --detach`, never this one). Run the wheel before writing `parity`. @@ -321,18 +290,14 @@ Add a dedicated `copy.deepcopy()` round-trip test for it too (see `test_regexes_ **A row count does not prove a CSV whole: Python's `csv` returns a body cut inside a row as a short last row, without raising** -- even inside an open quote (`'a,b\r\n"x,y'` gives `[['a', 'b'], ['x,y']]`). So a fetch that checks its row count against the source's stated total, and tolerates a few malformed rows, passes a cut body one name short. #842's review found it in `ukec-individuals` (two reviewers independently); the fix requires the body to end with a line break and its last row not to be short (a cut only shortens a row; a long last row is left to the malformed share), with a test for each arm, since a quoted field holding a line break lets a cut end at one. A new CSV source (a `tools/differential/local_.py` module) takes the same check (`local_ukec._ukec_rows`'s last-row test), and its skip shares are tested at their shipped values, not only patched open for a small fixture. -**Expected-failure tests use `@pytest.mark.xfail`** — the conftest parametrized fixture breaks `@unittest.expectedFailure`; always use `@pytest.mark.xfail` instead. +**Expected-failure tests use `@pytest.mark.xfail`** — the test classes are plain pytest classes, not `unittest.TestCase`, so `@unittest.expectedFailure` is not honored. **`lc()` strips leading and trailing periods** — `'M.D.'` → `'m.d'`, not `'md'` (interior periods are preserved). Shipped keys in `capitalization_exceptions` are dot-free, so lookups must also try `.replace('.', '')`; whatever key matches, its value is a case mask applied to the word as written, never substituted for it (#459). -**`is_suffix()`'s period-stripping is asymmetric** — it does `lc(piece).replace('.', '')` (strips *all* periods) before checking `suffix_acronyms`, but only `lc(piece)` (leading/trailing only) before checking `suffix_not_acronyms`. Code that reimplements this check instead of calling `is_suffix()` must mirror both branches or it will misclassify acronym suffixes with internal-only periods (e.g. `"M.D"` with no trailing dot) — bit `parse_nicknames()`'s `handle_match()` in PR #189. - **`nickname_delimiters`/`maiden_delimiters` built-ins are string sentinels, not compiled patterns** — each default key (`quoted_word`, `double_quotes`, `parenthesis`, and the rest) stores its own *name* as a plain `str`, and `Constants._snapshot()` maps that key to the `(open, close)` pair the 2.0 `Policy` carries (`_SENTINEL_PAIRS` in `_config_shim.py`). Routing a built-in between buckets is still a `pop()` + assign (`maiden_delimiters['parenthesis'] = nickname_delimiters.pop('parenthesis')`) to carry the sentinel over — measured, that flips `"Jane (Jones) Smith"` from `nickname='Jones'` to `maiden='Jones'`. **2.0 note:** the two things this bullet used to justify the sentinel with are both gone, and only the v1 history above still holds. `CONSTANTS.regexes.parenthesis = ...` now raises `TypeError` (2.0 configures parsing through named `Policy` flags), so the sentinel no longer buys post-construction tracking of a `regexes` override; and a caller's *custom* key is refused by `_DelimiterManager` — its message points at `Policy(nickname_delimiters=...)` — rather than accepted as a compiled pattern. No `isinstance(raw_pattern, re.Pattern)` branch survives, and `HumanName.parse_nicknames` is gone as a method — but do NOT sweep the name itself as dead: it is still live in `_facade.py`'s `_V1_HOOKS`, where a v1 subclass that overrides it earns the once-per-subclass hook warning (#280). Don't count the default keys in prose either: this bullet said "three" until #273 added eight typographic pairs in one commit — five Western (`smart_double_quotes`, `low_high_quotes`, `right_double_quotes`, `guillemets`, `reversed_guillemets`) and three CJK — taking it to eleven, and #597 added a twelfth, `halfwidth_corner_brackets`, on 2026-10-03. (#22) **`TupleManager.__setattr__`/`__delattr__` guard dunder names too, not just `__getattr__`** — constructing a subscripted generic, e.g. `TupleManager[re.Pattern[str] | str]({...})` (needed so mypy sees the right value type instead of inferring one from the dict literal), makes `typing`'s `GenericAlias.__call__` set `__orig_class__` on the new instance right after `__init__` returns. Before this guard existed, `__setattr__` was a bare `dict.__setitem__` alias, so that assignment silently inserted a bogus `'__orig_class__'` entry into the dict itself, corrupting `.values()`/iteration for *every* `TupleManager`/`RegexTupleManager` instance, not just the one being constructed — this bit `nickname_delimiters`'s construction (#22) before the guard was added. Same fix shape as the `__getattr__` dunder guard above: fall back to `object.__setattr__`/`object.__delattr__` for dunder names, dict-backed storage for everything else. -**`_join_bound_first_name` guard must exclude trailing suffixes** — suffix tokens are still in `pieces` when the helper runs (suffix detection happens in the assignment loop, later). The `reserve_last` guard must count `if not self.is_suffix(p)` to avoid treating a trailing suffix like "Jr." as a last-name slot; otherwise `"abdul salam jr"` → `last='jr'`. - **Doctests** — docstring examples in `nameparser/*.py` run under `uv run pytest` (`--doctest-modules`; `testpaths` is `tests` + `nameparser` only). The `.rst` doctests in `docs/` (`usage.rst`, `customize.rst`) and `README.rst` are also run in CI: the `python-package.yml` workflow runs `sphinx-build -b doctest docs dist/doctest` (covers every `.rst` under `docs/`) and `python -m doctest README.rst` as separate steps after the existing `-b html` build. Both should pass with zero failures — expect a clean run, not baked-in noise. **An example needing an optional extra cannot be a doctest.** `namedivider-python` is not in the `dev` group, so it reaches only the `ja-extra` job; a `.. doctest::` requiring a segmenter passes locally, where it is incidentally installed, and fails `sphinx-build -b doctest` in all five build jobs. Use `.. code-block:: python` for those. That is why the Japanese examples in `usage.rst` are code-blocks while the Korean and Chinese honorific ones are live doctests — the latter go through VOCABULARY, which ships. Same rule for `README.rst`. Every `.. doctest::` block in one `.rst` file shares a single namespace — the per-document default group — so **both** local variables and global state carry forward, in document order. Two consequences, and the first is the one that bites. A variable bound in one block clobbers that name in every later block, so inserting a section that binds something common breaks examples further down the file that were passing and that you never touched: writing the #271 usage.rst section with `name = parse("김민준")` in it made `name.initials()` six blocks later return `'민. 김.'` instead of `'J. Q. X. V.'`. Pick a distinctive variable name (`minjun`), or reuse whatever the surrounding blocks already bind. Second, examples that mutate the shared `CONSTANTS` singleton (e.g. `capitalize_name`, `titles.clear()`) must reset it at the end of the same block, because nothing else will — an unreset mutation leaks into every later example in the same build (this caused several real failures before it was fixed). If an example isn't actually demonstrating shared-singleton behavior, prefer a local `Constants()` instance instead of `CONSTANTS` — no reset needed, and no ordering dependency on other examples. `SetManager.__repr__` sorts its elements for this reason too: plain `set()` iteration order depends on per-process string hash randomization, so an unsorted repr would make ellipsis-style doc examples (`SetManager({'10th', ..., 'zoologist'})`) unreproducible across runs. @@ -341,14 +306,12 @@ Don't use the bare `python3 -m doctest .rst` CLI (no `optionflags`) to che **A `Constants` attribute docstring is not a place for a doctest, and nothing runs one there (2.0 note; the rest of this paragraph is v1 history).** v1's `Constants` documented attributes like `patronymic_name_order` with a bare string literal after the assignment, Sphinx's attribute-docstring convention. That string never became a real `__doc__`, so `--doctest-modules` (which walks `__doc__` attributes) never saw a `.. doctest::` inside it, and a stale example once slipped through CI (`middle_name_as_last`, #133); `tests/test_config_attribute_docstrings.py` (#195) recovered those literals with `ast` and ran their examples explicitly. Both halves are gone: since the M11 swap the real `Constants` lives in `nameparser/_config_shim.py` with bare type annotations (`patronymic_name_order: bool`) and no attribute docstrings, and the test was deleted in M12 (`a689e1d8`) because its AST walk found no class in the re-export shim and nothing to run in the real one. If an attribute docstring with an example ever returns, remember that `--doctest-modules` will not see it: put the example in a unit test instead, per the lean-docs rule in the 2.0 conventions above. -**`uv run mypy` covers `tests/` too** (`pyproject.toml`'s `[tool.mypy] packages = ["nameparser", "tests"]`, PR #250) — a test that deliberately passes an off-contract value to assert a `TypeError`/`ValueError`, or exercises a documented falsy-disables-the-feature toggle (e.g. `regexes.emoji = False`), needs a `# type: ignore[code]` comment on that line rather than a signature change. Before reaching for an ignore, check whether the "error" is really a source annotation that hasn't caught up with already-supported runtime behavior (e.g. `is_prefix`/`is_conjunction`/`is_suffix` accepting `list[str]`, `add_with_encoding` accepting `bytes`) — widen the annotation instead in that case. +**`uv run mypy` covers `tests/` too** (`pyproject.toml`'s `[tool.mypy] packages = ["nameparser", "tests"]`, PR #250) — a test that deliberately passes an off-contract value to assert a `TypeError`/`ValueError`, or exercises a documented falsy-disables-the-feature toggle (e.g. `regexes.emoji = False`), needs a `# type: ignore[code]` comment on that line rather than a signature change. Before reaching for an ignore, check whether the "error" is really a source annotation that hasn't caught up with already-supported runtime behavior — widen the annotation instead in that case. **`initials_separator` is intra-group only** — it controls the joiner between consecutive initials *within* a name group (e.g. two middle names in `middle_list`). Spaces *between* groups come from `initials_format`. To fully concatenate initials you need both `initials_separator=""` and `initials_format="{first}{middle}{last}"`. **`pr/NNN` local branches** track upstream PRs — don't commit to them by accident. Check `git branch --show-current` before starting work. -**Prefix-join uses value-based `list.index()`** in `join_on_conjunctions` — fragile when a token value repeats (e.g. a trailing title that's also a suffix acronym, or two `van`s); constrain such lookups to start at `i + 1`. See #100. - **Title vs suffix is positional for BARE words, and the leading period-abbreviation rule overrides even that** — a word matching `TITLES` at the front of a name becomes `title`; the same word matching `SUFFIX_ACRONYMS`/`SUFFIX_WORDS` at the end becomes `suffix` (never both, regardless of the word's real-world meaning). The `TITLES`/suffix overlap was audited in #296 (2026-08-23): the pure postnominals (`jr`, `junior`, `phd`, `do`, `se`) left `TITLES`, the v1-residue `dr`/`sra` left the suffix sets, and the ten words still in both (`ms`, `sr`, `lt`, `ra`, `vc`, and the ranks `cpl`, `cpo`, `cpt`, `csm`, `sgm`; `md` and `sa` were duals too until #853 took `md`'s title reading out, a leading `Md` being the given name Muhammad, and #857 took `sa` out of the vocabulary) are deliberate duals that position decides. External test sources (old issue gists, etc.) sometimes assert `suffix` for a leading professional abbreviation like `RA`/`PD`/`Dipl.-Ing.` — that's the source data being wrong, not a parser bug. Verify position before "fixing" it. Two qualifications the older "purely positional" wording papered over, the first measured 2026-08-01 and the second answered 2026-09-08 (#316): a PERIOD-marked leading word is claimed by the shape rule before any vocabulary is read (`"Esq. Smith"` → `title`, though `esq` is suffix-only), save a listed title and, since #789, a given name written short (`"Chas. Smith"` → `given`), and a PERIOD-marked trailing word is claimed by VOCABULARY — first the suffix sets, which the peel reads before anything else (`"John Smith Esq."` → `suffix`), then the titles (`"John Smith Prof."` → `title='Prof.'`, `rules/H5`) — while an unlisted abbreviation there stays a name part (`"John Smith Xyz."` → `family='Xyz.'`) unless it is written as two or more period-separated chunks, which since 2.4 IS a trailing shape rule (`Policy.unlisted_dotted_suffixes`, default on: `"John Smith X.Y.Z."` → `suffix='X.Y.Z.'`, read by the same words-to-spare count a bare ambiguous acronym takes, `rules/S3`), except paired initials in a script with capitals, which stay a name word (`"Ram Bahadur B.K."` → `family='B.K.'`, #864). The SINGLE trailing period is still not one, deliberately — it is the abbreviation shape any word can wear. The bare word is where "positional" still holds whole -- outside a part past the second comma, which holds no name word, so C2 reads a bare title word there as a title since #603 (`Eric H. Holder, Jr., Attorney General`), and outside a listed salutation ending the given part after a family comma, which rules/H8 reads as the title since #835 (`Doe, Jane Mrs`), the comma having marked the surname's place -- and the reason it must: `TITLES` holds words that are in no suffix set — 787 of them on 2026-10-11 (746 before #606's salutations), measured with `L = Parser().lexicon; len(L.titles - L.suffix_acronyms - L.suffix_words)`, a figure that grows with the vocabulary and never shrinks the argument — and many are ordinary surnames (`pope`, `bishop`, `prince`, `judge`, `sheriff`, `baron`, `master`, ...), so a vocabulary-first rule over BARE trailing words would read `"Mary Jane Pope"` as `title='Pope'`, `family='Jane'`. The period is what separates the safe case from that one, and it separates it by being a WRITING convention rather than by making the collision go away: `"Mary Jane Pope"` keeps `family='Pope'` while `"Mary Jane Pope."` reads `title='Pope.'`, `family='Jane'`, a cost accepted under the input-is-a-name premise rather than one the rule prevents (the trailing-position bullet in decisions/vocabulary-exclusions, measured 2026-09-09). Since #316 that sentence describes the shipped rule rather than an aspiration. ### Tests (`tests/`) @@ -361,7 +324,7 @@ A parse-behavior test can still be a disguised constant-content test if it only **Deleting a test module: sort its tests by the code they exercise, not by their names.** A module named for a retired tool also holds tests of code that survives it. #664 deleted `test_ledger_guards.py` and rewrote `test_differential.py`, triaging both by test name, and review found five tests guarding live code in the discard pile: four of `build_issues_corpus.py`'s harvester, and the check that `compare.py` reports the entries it skips. For each test, grep its module-level handles (`load_tool("...")`, imports) and ask whether that code is also being deleted. -**When adding a new aggregate property** (like `given_names` or `surnames`), always include a test for the empty path — a name that produces no value for that property — so the `or self.C.empty_attribute_default` guard is covered. The conftest dual-fixture then automatically exercises both `""` and `None` variants. Example: `HumanName("Williams")` for a given-names property (last-name-only string has no first or middle). +**When adding a new aggregate property** (like `given_names` or `surnames`), always include a test for the empty path — a name that produces no value for that property, which must render `''`. Example: `HumanName("Williams")` for a given-names property (last-name-only string has no first or middle). **`python_classes = ["*Tests", "*TestCase"]`** in `pyproject.toml` — suffix style (`FooTests`), NOT prefix (`TestFoo`); wrong style silently skips discovery.