diff --git a/AGENTS.md b/AGENTS.md index 53fd7724..3bee60b3 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -196,7 +196,7 @@ The library has two layers: `nameparser/config/` (data) and `nameparser/parser.p The `config/` SETS are frozen since 2.2 (#293): there is no `.add()`/`.remove()` on any of them, so a default word list is changed at runtime by configuring an object — a private `Constants` for `HumanName`, a `Lexicon` for the 2.0 API — never by editing the constant, and the shipped default is changed by editing the TOML and rebuilding. `CONSTANTS`/`Constants` still hand out mutable `SetManager`s; the freeze is on the module set constants they copy from. **Neither dict was frozen, and `CAPITALIZATION_EXCEPTIONS` is not covered by anything else either.** `REGEXES` is a compiled-pattern table rather than vocabulary and was never in #293's scope; `CAPITALIZATION_EXCEPTIONS` is vocabulary-shaped and is a decided, in-scope exemption. So the split-default hazard the freeze closes is still live for it, measured on 2.2 and re-measured 2026-09-23 with the key below once `'PhD'` became the shipped value: `CAPITALIZATION_EXCEPTIONS['dphil'] = 'DPhil'` reaches a freshly built `Constants` and neither the cached `Lexicon.default()` nor the shared `CONSTANTS`. Same advice — configure the object (`constants.capitalization_exceptions[...]`, or `dataclasses.replace(lexicon, capitalization_exceptions=...)`). `tests/v2/test_contracts.py::test_every_vocabulary_constant_is_frozen` names both dicts as explicit exemptions rather than letting its `isinstance` filter drop them. -- `titles.py` — `TITLES` (every title reading; prenominals) and `GIVEN_NAME_TITLES` (the readings with `addresses = "given"`, e.g. "Sir", which treat the following name as given, not family — a subset by construction) +- `titles.py` — `TITLES` (every title reading; prenominals), `GIVEN_NAME_TITLES` (the readings with `addresses = "given"`, e.g. "Sir", which treat the following name as given, not family — a subset by construction) and `AFTER_GIVEN_TITLES` (the salutation readings with `after_given = true`, read as the title where one ends the given part after a family comma, rules/H8, #835 — a subset by construction, with no v1 `Constants` attribute) - `suffixes.py` — `SUFFIX_ACRONYMS` (suffix readings with `acronym = true`, matched with every period removed, e.g. "M.D.") and `SUFFIX_WORDS` (the rest, e.g. "Jr."), disjoint because a word has one suffix kind; `SUFFIX_ACRONYMS_AMBIGUOUS` (`may_be_name`); and `GLUED_HONORIFICS` (#308, `attached`), the subset of `SUFFIX_WORDS` the peel may split off the END of a name token — a separate, harsher bar, since the glued position has no writer-drawn boundary to lean on, and each word refused it carries `excluded = { attached = … }` - `particles.py` — `PARTICLES` (family-name particles, e.g. "de", "van") and `NON_GIVEN_NAME_PARTICLES`, the curated subset that is *never* a standalone given name (the particle readings with `may_be_given = false`). What the subset licenses is rules/P1's fold in `post_rules` (the "rule 1b" of older issue comments — decisions/legacy-rule-numbers maps the old numbers), and the fold has ONE site: the piece that OPENS the name. A lone never-given particle there, with at least one more name word behind it, starts the family. Titles do not move the opening position (rules/P4), so `Sir de Mesnil` reads family `de Mesnil` under every `name_order`, as `de Mesnil` does. How far the fold reaches depends on the order the name was read under: by default it takes the rest of the name (`de Mesnil Jean` → family `de Mesnil Jean`); under either family-first order it takes the particle run plus ONE name unit and lays the rest out by the order (`de Mesnil Jean` → family `de Mesnil`, given `Jean`); after a family comma it takes the rest of that part whatever the order (`Smith, de Mesnil Jean` → family `Smith de Mesnil Jean`). The leading particle stays a lone piece for the fold to find — `group` does not chain it forward, and the fold walks the particles after it itself — so `de la Vega` reads family `de la Vega` under all three orders. The GIVEN position is not a site. It was a second one until #467, which removed it because under a family-first order that slot holds what the caller DECLARED to be the given name: `Mesnil de` under `FAMILY_FIRST` or `FAMILY_FIRST_GIVEN_LAST` reads given `de`, family `Mesnil`. Nor is a particle inside the name: `Juan de la Vega` under a family-first order reads family `Juan`, given `de la Vega`, with no ambiguity recorded — what the vocabulary forbids is the bare particle as a given name, not a name part that begins with one. A bare `de` has nothing to fold into and is placed by position like any one-word name: given `de` by default, family `de` under both family-first orders. A leading particle OUTSIDE the set is left to position — `van Gogh` is given `van`, family `Gogh` by default and family `van`, given `Gogh` under both family-first orders — and records a `PARTICLE_OR_GIVEN` ambiguity under all three; a never-given particle opening the name records none (`de Mesnil`), and neither does an ambiguous one inside it (`Vincent van Gogh`). `Lexicon.particles_ambiguous` is its complement within `PARTICLES`, so the two mark OPPOSITE sets — see the flip warning in `docs/migrate.rst` before translating either - `bound_given_names.py` — `BOUND_GIVEN_NAMES` (`bound_given` readings, e.g. "abdul", "abu"; one that is also a particle needs `may_be_given = true`, a build check); a group-stage rule joins the first non-title piece to its following piece before roles are assigned, reserving a family word unless a family comma has already fixed it or a given-name title stands ahead (#369, rules/P5) (v1's `_join_bound_first_name`, ported into `_pipeline/_group.py` and gone from the tree — the v1 descriptions further down are history, not current code) @@ -244,7 +244,7 @@ The 2.0 rewrite lands as underscore-private modules alongside the v1 code. These - **Parser owns config-dependent conveniences**: `Parser.matches`/`Parser.capitalized`/`Parser.revise` exist because the `ParsedName` equivalents fall back to DEFAULT config for str/omitted arguments (documented loudly in both docstrings). `revise` harvests tokens from a full sub-parse of each replacement value (tags kept minus `FOLDED_TAG`, roles forced, the R1 entry pass `suffix_entries` re-run over the forced state so a suffix value's entries follow its own commas, ambiguities discarded); the merge tail is shared with `replace()` via `ParsedName._with_field_tokens`. `Parser.capitalized` delegates through `name.capitalized(self.lexicon)` specifically so `_parser` never imports `_render` — keep it that way. - **Per-word vocabulary fields warn on multi-word entries** (`_normset`/`_normpairs` via `_warn_dead_entry`, UserWarning, never a raise — see the given_name_titles Gotcha for why raising is wrong). `given_name_titles` is the one multi-word-matched field and is exempt; `_edit` passes `warn=False` (add() warns once via the new instance's `__post_init__`; remove() stores nothing). The default vocabulary and every locale pack must stay warning-free (`test_default_lexicon_builds_warning_free`, `test_pack_vocabulary_entries_are_single_words`). - **Invariants guard harm, not no-ops**: add a constructor check when violating it produces a *wrong parse*, not when it produces *nothing*. A false positive costs a working configuration; a true positive on an inert condition costs the user nothing, so that trade is never worth taking. `suffix_acronyms_ambiguous ∩ suffix_words` is guarded because the overlap loses a family name; `given_name_titles` is not, because an unreachable entry is simply never consulted (see Gotchas). Before adding one, construct the config it forbids and check what actually breaks. **Between raise and silence sits the construction-time `UserWarning`**, for a gap that is statically decidable, harmless to SOME deliberate caller, and indistinguishable-from-working for everyone else: the segmenterless activation (#337 — `parser_for(locales.JA)` without a segmenter behaved exactly like a working parser minus the feature) warns rather than raises because the inert JA registration is itself a pinned property, and a warning is filterable by the caller who wants exactly that. The message must carry every applicable remedy and no inapplicable one (the `ja_segmenter` hint fires only when a Japanese script is among the dead ones). Test fuzzers that legitimately construct such configs suppress the warning by MESSAGE, never by category — a blanket `UserWarning` ignore would mask the next construction diagnostic (`_quiet_parser` in `tests/v2/test_properties.py` is the pattern). -- **The shim TRANSLATES; it never raises on a config v1 accepted (one decided exception, #459, at the end of this bullet), and never silently changes the parse**: `Constants._snapshot()` is a translation boundary between v1's model and v2's invariants, and every transformation there carries its v1-reachability argument in a comment. These exist today — `first_name_titles` re-folded per word (v1 joins-then-`lc`, v2 normalizes-then-joins), `suffix_acronyms_ambiguous ∩ acronyms` (a provable no-op), `suffix_words − ambiguous` (v1 already accepts the word via the acronym branch, so the addition is inert there), `particles_ambiguous ∪ (bound ∩ particles)` (a pinned deviation, `test_bound_never_given_prefix_deviates_on_two_pieces`), `honorific_tails = GLUED_HONORIFICS ∩ suffix_words` (#308 behavior with no v1 manager of its own, so the one v1 knob that reaches it is deleting the suffix word — which turns the peel off, `test_snapshot_removing_a_honorific_word_turns_the_peel_off`), `conjunctions_ambiguous = CONJUNCTIONS_AMBIGUOUS ∩ conjunctions` (#383/#479 behavior with no v1 manager of its own either, so the one v1 knob that reaches it is deleting the conjunction — which turns the marking off, `test_snapshot_removing_a_conjunction_turns_the_marker_off`), and `maiden_delimiters − nickname_delimiters` on the POLICY half of the same method (v1 precedence: a pair in both v1 buckets parses as a nickname, while `Policy` resolves the overlap the other way, so the subtraction is what keeps the facade at v1 behavior, `test_snapshot_overlap_keeps_v1_nickname_precedence`), and `_v1_matchable`, which runs ahead of every other one: an entry v1 could never match — empty after `lc()`, or holding edge whitespace, a whitespace run or any whitespace character other than a single space, since no v1 piece does — is dropped from all nine set fields and from the `capitalization_exceptions` keys and named in one `UserWarning` whose remedy runs (#541; translated, it started matching once `Lexicon` stripped it or tripped a `Lexicon` invariant the set algebra could not see, `test_an_entry_v1_could_not_match_is_dropped_with_one_warning`), except the two whitespace typos in 1.4.0's own shipped `TITLES`, which a restored 1.4 pickle carries and are dropped without the warning. The same filter drops an entry `Lexicon` folds to empty — since 2.3.0's `FULL_STOPS` fold, a lone CJK full stop, which 1.4.0 through 2.2.0 could match on a token made of that stop but which since 2.3.0 can be neither held nor matched, so translating it raised wherever `Lexicon` received it (#582, the one departure from 1.4.0 this filter makes) — and the two cross-set computations `Lexicon` re-checks after its own fold (`suffix_words − ambiguous` and `bound ∩ particles`) compare `_normalize`d spellings, while every other field passes raw as before (an edge full stop, `'ma。'` beside the ambiguous `ma`, had raised `Lexicon`'s gate-bypass check since 2.3.0; folding EVERY field instead was measured to move readings on configs that never raised, `test_an_edge_full_stop_entry_outside_the_two_checks_reads_as_before`). Note that the `maiden_delimiters − nickname_delimiters` one is on the `Policy`, not the `Lexicon` — the roster is per-`_snapshot()`, not per-vocabulary-field, so a sweep that only reads the `Lexicon(...)` call misses it. When a v1 config cannot satisfy a v2 invariant, work out what v1 actually *does* with it — usually nothing — and reproduce that; weakening the invariant or letting the raise through are both wrong. **Test the case the translation decides**, not one where both branches agree: a test using an input v1 parses identically with and without the config pins nothing. A `capitalization_exceptions` value that does not spell its key's letters and digits raises `ValueError` at snapshot rather than being translated (#459), because v1 substituted such a value for the word and 2.4 repair only recases, so there is no v1 behavior to reproduce; pinned by `test_a_mismatched_exception_value_raises_at_the_first_parse`. +- **The shim TRANSLATES; it never raises on a config v1 accepted (one decided exception, #459, at the end of this bullet), and never silently changes the parse**: `Constants._snapshot()` is a translation boundary between v1's model and v2's invariants, and every transformation there carries its v1-reachability argument in a comment. These exist today — `first_name_titles` re-folded per word (v1 joins-then-`lc`, v2 normalizes-then-joins), `suffix_acronyms_ambiguous ∩ acronyms` (a provable no-op), `suffix_words − ambiguous` (v1 already accepts the word via the acronym branch, so the addition is inert there), `particles_ambiguous ∪ (bound ∩ particles)` (a pinned deviation, `test_bound_never_given_prefix_deviates_on_two_pieces`), `honorific_tails = GLUED_HONORIFICS ∩ suffix_words` (#308 behavior with no v1 manager of its own, so the one v1 knob that reaches it is deleting the suffix word — which turns the peel off, `test_snapshot_removing_a_honorific_word_turns_the_peel_off`), `conjunctions_ambiguous = CONJUNCTIONS_AMBIGUOUS ∩ conjunctions` (#383/#479 behavior with no v1 manager of its own either, so the one v1 knob that reaches it is deleting the conjunction — which turns the marking off, `test_snapshot_removing_a_conjunction_turns_the_marker_off`), `after_given_titles = AFTER_GIVEN_TITLES ∩ titles` (#835, likewise: deleting the title turns H8 off for it, `test_snapshot_removing_a_title_turns_its_after_given_reading_off`), and `maiden_delimiters − nickname_delimiters` on the POLICY half of the same method (v1 precedence: a pair in both v1 buckets parses as a nickname, while `Policy` resolves the overlap the other way, so the subtraction is what keeps the facade at v1 behavior, `test_snapshot_overlap_keeps_v1_nickname_precedence`), and `_v1_matchable`, which runs ahead of every other one: an entry v1 could never match — empty after `lc()`, or holding edge whitespace, a whitespace run or any whitespace character other than a single space, since no v1 piece does — is dropped from all nine set fields and from the `capitalization_exceptions` keys and named in one `UserWarning` whose remedy runs (#541; translated, it started matching once `Lexicon` stripped it or tripped a `Lexicon` invariant the set algebra could not see, `test_an_entry_v1_could_not_match_is_dropped_with_one_warning`), except the two whitespace typos in 1.4.0's own shipped `TITLES`, which a restored 1.4 pickle carries and are dropped without the warning. The same filter drops an entry `Lexicon` folds to empty — since 2.3.0's `FULL_STOPS` fold, a lone CJK full stop, which 1.4.0 through 2.2.0 could match on a token made of that stop but which since 2.3.0 can be neither held nor matched, so translating it raised wherever `Lexicon` received it (#582, the one departure from 1.4.0 this filter makes) — and the two cross-set computations `Lexicon` re-checks after its own fold (`suffix_words − ambiguous` and `bound ∩ particles`) compare `_normalize`d spellings, while every other field passes raw as before (an edge full stop, `'ma。'` beside the ambiguous `ma`, had raised `Lexicon`'s gate-bypass check since 2.3.0; folding EVERY field instead was measured to move readings on configs that never raised, `test_an_edge_full_stop_entry_outside_the_two_checks_reads_as_before`). Note that the `maiden_delimiters − nickname_delimiters` one is on the `Policy`, not the `Lexicon` — the roster is per-`_snapshot()`, not per-vocabulary-field, so a sweep that only reads the `Lexicon(...)` call misses it. When a v1 config cannot satisfy a v2 invariant, work out what v1 actually *does* with it — usually nothing — and reproduce that; weakening the invariant or letting the raise through are both wrong. **Test the case the translation decides**, not one where both branches agree: a test using an input v1 parses identically with and without the config pins nothing. A `capitalization_exceptions` value that does not spell its key's letters and digits raises `ValueError` at snapshot rather than being translated (#459), because v1 substituted such a value for the word and 2.4 repair only recases, so there is no v1 behavior to reproduce; pinned by `test_a_mismatched_exception_value_raises_at_the_first_parse`. - **Reprs are bounded**: render which fields deviate from a named baseline and by how much, never contents (`Lexicon(default + titles: +2)`). `PolicyPatch`'s repr shows only set (non-UNSET) fields; `_order_repr` must never raise even on an unvalidated patch's garbage `name_order` (PolicyPatch defers validation to apply time); the sweep test in `tests/v2/test_reprs.py` pins that no config repr leaks the UNSET sentinel. - **Every pipeline stage is one module holding one public function of the same name** (`_tokenize.py`/`tokenize`), with the exceptions named where they stand — `_group.py`'s `marker_run_length`, a shared predicate (mechanisms.md#ONE-PREDICATE-PER-QUESTION), and since #511 `_post_rules.py`'s `suffix_entries`, the R1 entry pass as a function, because `Parser.revise` runs that one pass over a forced-role sub-parse and a stage's tail cannot be called on its own — **and its module docstring declares the contract in three labelled lines**: `Consumes:` what it takes from `ParseState`, `Produces:` what it hands back, `Reads:` which `Policy`/`Lexicon` fields it consults. `Reads:` is the load-bearing one — it makes "which stage do I touch for this feature?" a grep rather than a read-through. The authority for the stage set is `_pipeline/__init__.py`'s `STAGES` (eight, and not public API) with the field-ownership map in `ParseState`'s docstring, pinned by `tests/v2/pipeline/test_state.py`; NOT the rules, which are implementation-free by their own preamble and whose `implemented:` names modules honoring a rule rather than stages. A `_pipeline/` module that is not a stage says so in its first line instead (`_assemble.py`: "Not a stage: …", omitting `Reads:` because it consults neither), so absence of the three lines is a claim about the module rather than an oversight. Provenance: §5 of the 2026-07-11 conventions spec, recorded here 2026-08-16. - **Frames are a budget for hot and scaling paths; everywhere else, prefer the shared predicate.** The parser's cost is counted in Python frames (`sys.setprofile` "call" events — deterministic and load-immune, which is why #475 chose them over wall-clock timing), but one frame is small: measured 2026-09-28 on py3.11, a trivial call costs about 13 ns against 40–80 µs for a whole parse, so one frame per parse is roughly 0.03% of it. What the budget exists to catch is how a cost SCALES and how it ACCUMULATES, so weigh an extra frame by four questions, in order. (1) Does it multiply with the input — per token, per member, per pair? Then it matters, and a hand-inlined fast path can earn its place (the inline tag tests ahead of `_pieces.starts_a_credential_run` in its loops, each a necessary half of the predicate so nothing can drift; `_vocab.run_word_fold` was the measured example until #613 deleted the run test it served). Folding a copy into a shared helper is where such a pre-check gets lost: #617 replaced `_comma._title_chains`, whose loop tested the conjunction tag inline, with a call into group's loop, which asks `is_conj_piece` per piece, and every family-comma parse paid two frames per word of the part (`Smith, John` +2) while the reference row fell -- review caught it; the fix gates the call on the tag walk the caller already makes, and `test_a_comma_part_with_no_free_connective_skips_the_join` holds it. Count a comma name, not only the reference row, when a fold touches the comma path. (2) How many names pay it — every parse, where plain names like `Smith, John` must keep their cost, or only a rare shape? (3) Does it move an enforced guard — the ±2% reference band in `test_benchmark.py` (`_CALL_BASELINE`, decisions/parse-cost) or a scaling ratio? (4) What does the inline alternative cost to maintain? A hand copy of a predicate that can drift is the defect class mechanisms.md#ONE-PREDICATE-PER-QUESTION names, and it compounds; a NARROWER question stated exactly is not a copy and may stay inline on that ground alone. A constant frame on a rare shape that no guard sees loses to readability: #544's cleanup declined `is_title_piece` for one frame on comma credential runs (24 ns a parse, no guard moved), a stricter call than this rule asks for; by this rule the inline test is still right, but because it asks the narrower question (a lone word in both vocabularies), not because of the frame. The converse holds too: the frame counter cannot see C-level work (`x in some_list`, slices, string concatenation), so "no new frames" is not "no new cost" — #553's quadratic was a list membership test inside a loop, invisible to every frame guard. @@ -270,11 +270,11 @@ Add a dedicated `copy.deepcopy()` round-trip test for it too (see `test_regexes_ **Unknown-key attribute access on `TupleManager`/`RegexTupleManager` warns (1.4, #256) — 2.0 note: the warning became a hard `AttributeError` naming the known keys (shim `TupleManager`); the paragraph below describes the deleted v1 machinery and applies only when reading v1 history.** — a key not currently in the dict emits `DeprecationWarning` naming the miss and the known keys (`_warn_unknown_key`), before falling back to the same `None`/`EMPTY_REGEX` default as before; `.get()` stays silent. Dunder probes (`__deepcopy__`) still raise `AttributeError` outright, and single-underscore probes (`_repr_html_`, IPython's `_ipython_canary_method_should_not_exist_`, etc.) are excluded from the warning too — no real config key starts with `_`, so both guards just keep protocol/introspection probes from misfiring as "typo" warnings. This means internal parser code that reads `self.C.regexes.` unconditionally (e.g. `squash_bidi`'s `bidi`) now warns if a caller's custom `regexes` dict omits that key — a previously-silent partial-override pattern is on the same deprecation path as an actual typo. -**Adding a word to the vocabulary** — the steps are in `nameparser/vocab/README.md`; this paragraph is the traps they have run into. First check the word's OTHER readings: `uv run python tools/vocab/query.py --word ` lists every reading and exclusion of it in every file, which is where the overlaps between categories now show. Real overlaps exist: `st`/`freiherr` ∈ `PARTICLES` ∩ `TITLES`, and `do`/`mc`/`vd` ∈ `PARTICLES` ∩ `SUFFIX_ACRONYMS`; `abd` = "ABD" ∈ `SUFFIX_ACRONYMS`; `abu` ∈ `PARTICLES` ∩ `BOUND_GIVEN_NAMES` (position-dependent: leading token → given-name join, mid-name → family-name join). Usually position-dependent and harmless, but can force a guard or an exclusion (the `last_base` all-particles guard; `abd`, kept out of `BOUND_GIVEN_NAMES` until decisions/P5 found the credential collision cost nothing at either end). Then check the word against real names, not memory: `uv run python tools/name_usage.py ` counts its bearers as a given name and as a family name (US Census surnames, SSA given names, Wikidata with countries of citizenship), and gives its given-name frequency band per country from Jörg Michael's `nam_dict.txt` (54 countries and an "other countries" column, GFDL, downloaded into the cache and never committed) — the one source that sees a given name outside the US and Wikidata's lower bound, such as Korean `In` (band A in Korea) or Vietnamese `Bay` (6 in Vietnam). A band is a frequency class, not a count, and the dictionary is a selection, so a miss there (`Shehu`) proves nothing; and because a truncated source reads as a run of false zeros, a pinned download is checked against its sha256 on every load from the cache, never against a floor on what it holds — #775's first fix refused fewer than 40,000 names and passed a `nam_dict.txt` missing only its last line (`Zyta`). Pass a name as it is written (`McDonald`): Wikidata labels match exactly, and no rule rebuilds an internal capital from `mcdonald`. It reports and does not decide — read the counts against the position the set's claim acts on. For a suffix it also has the other side: `--fetch-nppes` counts the US healthcare providers who write the word as a credential (NPPES, the 4.8 million credentialed providers in the pinned August 2026 file, so blind to law, accounting and honours), and `--rank N --unlisted` lists the commonest credentials the vocabulary misses, with where each lands in `John Smith, X` -- the list #797 came from. Where a local corpus is fetched (`tools/differential/local_corpora.py`, mechanisms.md#NAME-DATA-JOBS), a row says how many of its records (for ClinicalTrials.gov, officials) write the word, and how many after a comma: ClinicalTrials.gov's study officials show where a credential actually stands in a name as typed. A word that would peel GLUED (`GLUED_HONORIFICS`) asks a different question, whether any name ENDS in it, and for a CJK word the tool answers that too: an ends-with row from a cached index of Wikidata's CJK name labels (built on first use; 33 minutes on 2026-10-17), plus the names the word ends across the family/given join (阿部 + 長 for 部長). Read the tool's hits by item before counting them, because labels include transcriptions and mislabels: #707's one hit for 二世 was 厄德二世, the given name Odo with a regnal numeral in its Chinese label, borne by 109 Western people named Odo. Then ask a third question the tool cannot: does a common longer word end in it? The peel matches the end of a token, spaced or glued, so a listed tail splits every unlisted compound that ends in it — #707 vetted 議員, 社長 and 医学博士 against names and its review found 衆議院議員, 副社長 and 獣医学博士 cut at the tail (`田中 副社長` → given 副); no corpus held a compound office word until #707's rules examples added two, so the snapshot was silent about the class. List the common compounds whole or record the split as a cost. And check the predicates that test a member's SPELLING rather than its membership: they encode what the set's members happened to look like when they were written. `_pieces.starts_a_credential_run` implemented the credential run's "Latin" (rules/S8 now, part of S2 until #682 split it) as `isascii()`, exact until #661 added `mlađi`, which would have read `mlađi Jones` differently from `stariji Jones`; the design-docs review caught it, after a first draft had credited the test to the wrong clause of S2 and proposed deleting it on a mutation that moved nothing because nothing pinned it. (#679 then dropped the script test on purpose, after measuring what it protected.) Before adding the first member of a new script, or the first accented one, grep the set's consumers for `isascii`, `isalpha` and script tests, and read the rule each one implements. The same goes for the first member holding PUNCTUATION: #700 added `d'` and `az-`, the parse read them right and every snapshot move was an intended one, and `capitalized()` still wrote `D' Ivoi`, because case repair splits a token at its apostrophe or hyphen and asks the vocabulary of the stem. The snapshot records both case views since #826, but only for the names the corpora hold, so parse such a member's names through `capitalized()` and `capitalized(force=True)` in each case before calling it done. The user docs use real words as examples of an UNLISTED one, and listing the word breaks the example: #797 listed `dphil`, and docs/customize.rst's capitalization example (`parse("jane smith dphil").capitalized()` giving `Dphil`) failed in all five CI build jobs while `uv run pytest` passed, because `sphinx-build -b doctest docs` is a separate step. Run it after adding a word, and grep `docs/*.rst` for the word; the example now uses `dclinpsy`, a credential left unlisted on purpose, so list it only after moving the example. Record what the vetting found ON the word, not in a comment or a decisions sentence: `note`, `issue`, a `given`/`family` reading with dated counts, and for a word you decided to keep out an `excluded` entry with its reason, so the next contributor who adds it meets the reason as a build error (decisions/vocabulary-model). +**Adding a word to the vocabulary** — the steps are in `nameparser/vocab/README.md`; this paragraph is the traps they have run into. First check the word's OTHER readings: `uv run python tools/vocab/query.py --word ` lists every reading and exclusion of it in every file, which is where the overlaps between categories now show. Real overlaps exist: `st`/`freiherr` ∈ `PARTICLES` ∩ `TITLES`, and `do`/`mc`/`vd` ∈ `PARTICLES` ∩ `SUFFIX_ACRONYMS`; `abd` = "ABD" ∈ `SUFFIX_ACRONYMS`; `abu` ∈ `PARTICLES` ∩ `BOUND_GIVEN_NAMES` (position-dependent: leading token → given-name join, mid-name → family-name join). Usually position-dependent and harmless, but can force a guard or an exclusion (the `last_base` all-particles guard; `abd`, kept out of `BOUND_GIVEN_NAMES` until decisions/P5 found the credential collision cost nothing at either end). Then check the word against real names, not memory: `uv run python tools/name_usage.py ` counts its bearers as a given name and as a family name (US Census surnames, SSA given names, Wikidata with countries of citizenship), and gives its given-name frequency band per country from Jörg Michael's `nam_dict.txt` (54 countries and an "other countries" column, GFDL, downloaded into the cache and never committed) — the one source that sees a given name outside the US and Wikidata's lower bound, such as Korean `In` (band A in Korea) or Vietnamese `Bay` (6 in Vietnam). A band is a frequency class, not a count, and the dictionary is a selection, so a miss there (`Shehu`) proves nothing; and because a truncated source reads as a run of false zeros, a pinned download is checked against its sha256 on every load from the cache, never against a floor on what it holds — #775's first fix refused fewer than 40,000 names and passed a `nam_dict.txt` missing only its last line (`Zyta`). Pass a name as it is written (`McDonald`): Wikidata labels match exactly, and no rule rebuilds an internal capital from `mcdonald`. A word in a non-Latin script is counted by its romanization too, since the Census, SSA and nam_dict hold only Latin names, and that count weighs only where its bearers write the script: #835 vetted its salutations by their own spellings and its review found `बाबू` flagged while `babu` is a US surname (1,496) and an Indian given-name element (Ram Babu), where the romanization of `মি`, `Mi`, counts Korean and Chinese bearers who never write Bengali. It reports and does not decide — read the counts against the position the set's claim acts on. For a suffix it also has the other side: `--fetch-nppes` counts the US healthcare providers who write the word as a credential (NPPES, the 4.8 million credentialed providers in the pinned August 2026 file, so blind to law, accounting and honours), and `--rank N --unlisted` lists the commonest credentials the vocabulary misses, with where each lands in `John Smith, X` -- the list #797 came from. Where a local corpus is fetched (`tools/differential/local_corpora.py`, mechanisms.md#NAME-DATA-JOBS), a row says how many of its records (for ClinicalTrials.gov, officials) write the word, and how many after a comma: ClinicalTrials.gov's study officials show where a credential actually stands in a name as typed. A word that would peel GLUED (`GLUED_HONORIFICS`) asks a different question, whether any name ENDS in it, and for a CJK word the tool answers that too: an ends-with row from a cached index of Wikidata's CJK name labels (built on first use; 33 minutes on 2026-10-17), plus the names the word ends across the family/given join (阿部 + 長 for 部長). Read the tool's hits by item before counting them, because labels include transcriptions and mislabels: #707's one hit for 二世 was 厄德二世, the given name Odo with a regnal numeral in its Chinese label, borne by 109 Western people named Odo. Then ask a third question the tool cannot: does a common longer word end in it? The peel matches the end of a token, spaced or glued, so a listed tail splits every unlisted compound that ends in it — #707 vetted 議員, 社長 and 医学博士 against names and its review found 衆議院議員, 副社長 and 獣医学博士 cut at the tail (`田中 副社長` → given 副); no corpus held a compound office word until #707's rules examples added two, so the snapshot was silent about the class. List the common compounds whole or record the split as a cost. And check the predicates that test a member's SPELLING rather than its membership: they encode what the set's members happened to look like when they were written. `_pieces.starts_a_credential_run` implemented the credential run's "Latin" (rules/S8 now, part of S2 until #682 split it) as `isascii()`, exact until #661 added `mlađi`, which would have read `mlađi Jones` differently from `stariji Jones`; the design-docs review caught it, after a first draft had credited the test to the wrong clause of S2 and proposed deleting it on a mutation that moved nothing because nothing pinned it. (#679 then dropped the script test on purpose, after measuring what it protected.) Before adding the first member of a new script, or the first accented one, grep the set's consumers for `isascii`, `isalpha` and script tests, and read the rule each one implements. The same goes for the first member holding PUNCTUATION: #700 added `d'` and `az-`, the parse read them right and every snapshot move was an intended one, and `capitalized()` still wrote `D' Ivoi`, because case repair splits a token at its apostrophe or hyphen and asks the vocabulary of the stem. The snapshot records both case views since #826, but only for the names the corpora hold, so parse such a member's names through `capitalized()` and `capitalized(force=True)` in each case before calling it done. The user docs use real words as examples of an UNLISTED one, and listing the word breaks the example: #797 listed `dphil`, and docs/customize.rst's capitalization example (`parse("jane smith dphil").capitalized()` giving `Dphil`) failed in all five CI build jobs while `uv run pytest` passed, because `sphinx-build -b doctest docs` is a separate step. Run it after adding a word, and grep `docs/*.rst` for the word; the example now uses `dclinpsy`, a credential left unlisted on purpose, so list it only after moving the example. Record what the vetting found ON the word, not in a comment or a decisions sentence: `note`, `issue`, a `given`/`family` reading with dated counts, and for a word you decided to keep out an `excluded` entry with its reason, so the next contributor who adds it meets the reason as a build error (decisions/vocabulary-model). **Before adding a short/common word to `PARTICLES` globally**, test it mid-string against realistic 3-token names, not just check for English-word collisions: Korean/Vietnamese given names put a short syllable in the middle slot (`Park In Hwan`, `Nguyen To Nga`), and Western names put a bare initial there (`John V. Smith`). A word that looks safe ("nobody is named 'to'") can still swallow a real middle name/initial into the last name once it's a global prefix — confirmed regressions for `to`/`in`/`an`/`ten`/`then` and bare `v` this way (PR #191). -**Adding a curated sub-set of an existing vocabulary set** (a marker over a base set, as `GIVEN_NAME_TITLES` is over `TITLES`) — it is a behavioral PROPERTY of a reading, not a second word list: a row in `tools/vocab/build.py`'s `BEHAVIORAL` table, a branch in `compile_pool` adding the words whose reading carries it, a constant in its `CONSTANTS` roster and a key of the same name in its `Compiled` TypedDict (an import-time assert holds the two to one list), and the `NAME = _compiled.NAME` assignment in the config module. The subset relation then holds by construction, since the property cannot exist without its reading, so it needs no check. A relation construction does not give (`∩ == ∅` with a set the sub-set cannot logically overlap; a sub-set member that is also a title is silently inert, title handling consuming it first) goes in `check_pool`, with a mutation in `tests/v2/test_vocab.py`'s table. Do **not** forbid `titles ∩ particles` — that overlap is intentional (`freiherr`, `st`). On the v1 side the sub-set is a `_SetManagerAttribute` on `Constants` (like `first_name_titles` — the v1 ATTRIBUTE names did not move in #293, only the module constants did), **not** a `_CachedUnionMember` — only `prefixes`/`suffix_acronyms`/`suffix_not_acronyms`/`titles` feed the `_pst` hot-path cache, so a sub-set costs nothing at runtime and stays out of `is_rootname`. A runtime `.add()` to the sub-set's `SetManager` does **not** propagate to the parent's (same as `first_name_titles`→`titles`), so a caller adding a brand-new word adds it to both. +**Adding a curated sub-set of an existing vocabulary set** (a marker over a base set, as `GIVEN_NAME_TITLES` is over `TITLES`) — it is a behavioral PROPERTY of a reading, not a second word list: a row in `tools/vocab/build.py`'s `BEHAVIORAL` table, a branch in `compile_pool` adding the words whose reading carries it, a constant in its `CONSTANTS` roster and a key of the same name in its `Compiled` TypedDict (an import-time assert holds the two to one list), and the `NAME = _compiled.NAME` assignment in the config module. The subset relation then holds by construction, since the property cannot exist without its reading, so it needs no check. A relation construction does not give (`∩ == ∅` with a set the sub-set cannot logically overlap; a sub-set member that is also a title is silently inert, title handling consuming it first) goes in `check_pool`, with a mutation in `tests/v2/test_vocab.py`'s table. Do **not** forbid `titles ∩ particles` — that overlap is intentional (`freiherr`, `st`). On the v1 side the sub-set is a `_SetManagerAttribute` on `Constants` where v1 had one (like `first_name_titles`; a sub-set new in 2.x, such as `honorific_tails`, `conjunctions_ambiguous` or `after_given_titles`, gets none and rides in `_snapshot()` as an intersection with its base, the shim roster under the 2.0 conventions listing each — the v1 ATTRIBUTE names did not move in #293, only the module constants did), **not** a `_CachedUnionMember` — only `prefixes`/`suffix_acronyms`/`suffix_not_acronyms`/`titles` feed the `_pst` hot-path cache, so a sub-set costs nothing at runtime and stays out of `is_rootname`. A runtime `.add()` to the sub-set's `SetManager` does **not** propagate to the parent's (same as `first_name_titles`→`titles`), so a caller adding a brand-new word adds it to both. **Adding a flag-gated post-parse transform** (reorder/adjust) — add a `Constants` boolean (default `False`), implement a `handle_*()` method, and call it in `post_process()` after `handle_firstnames()` and before `handle_capitalization()`, gated on the flag. Default-off keeps existing parses byte-for-byte unchanged. Two shipped examples: `patronymic_name_order` gates both `handle_east_slavic_patronymic_name_order()` (#85) and `handle_turkic_patronymic_name_order()` (#185) — one flag driving two independent handlers, added in the same `post_process()` slot; `middle_name_as_last` gates `handle_middle_name_as_last()` (#133), which folds `middle_list` into `last_list`. @@ -296,7 +296,7 @@ Add a dedicated `copy.deepcopy()` round-trip test for it too (see `test_regexes_ **Titles permanently shadow first names — be conservative** — any word in `TITLES` is always consumed as a title and can never be parsed as a first name. `"Dean"` is the canonical example: it's a common academic title *and* a common given name, so it is intentionally absent from the default titles (see `docs/customize.rst` — users who need it add it via opt-in `Constants`). Before adding a word to `TITLES`, ask: "Could this plausibly be someone's given name in any culture?" If yes, don't add it globally; it belongs in caller-supplied `Constants` instead. This same caution applies to international honorifics — `Prince`, `Sheikh`, `Frau` are all first names in some contexts. It also applies to any prefix sub-set gated on "never a first name": obscure-looking foreign particles are surprisingly often real given names — `Von` (Von Miller), `Vander` (Brazilian, also the Arcane character). When unsure, exclude — a missing member just means that name isn't auto-handled, whereas a wrong member misparses a real person. The decided exception is a word borne as a name only RARELY in the leading position — mostly a surname written last (`graf`, `greve`, `knight`), a few with small given-name or surname-first counts (`herra`, `hrabia`): the title claim acts only in front, so it ships, and a name that begins with it (`Greve Anna`, `Herra Wijaya` → title) is the accepted cost. That is a judgment by incidence, made per word, with the counts taken recorded on the word's entry in `nameparser/vocab/` (herra's, hrabia's; graf, which predates #606, and greve and knight, which #606 proposed, carry none) and the class in `decisions/salutation-titles` (rules/H1's Accepted clause states the cost), not a rule a sweep can apply: a word that commonly leads a real name (Vietnamese `Ông`, the given names `thiru`, `puan`, `marshal`) is still out. `tools/name_usage.py` produces the counts this judgment starts from (role counts, plus Wikidata citizenship to tell where a family name is written first); its Wikidata column is a lower bound (Shehu Shagari has no given-name statement), so a small count there proves little and a zero proves nothing, and its nam_dict row gives a band per country rather than a count, with a missing name proving nothing either. -**The period-abbreviation title inference runs at the head of the GIVEN-NAME part, not the head of the name** — an unrecognized multi-letter word ending in a single trailing period (`_pieces._PERIOD_ABBREV`, a hand copy of the `period_abbreviation` regex, `{2,}` letters — it was assign's until #424 and group's until #439) is treated as a title in the leading title run, e.g. `"Insp. Jane Morse"` → `title='Insp.'`. "Leading" is per SEGMENT: `_peel_leading_titles` is called for NO_COMMA segment 0, SUFFIX_COMMA segment 0, and FAMILY_COMMA **segment 1**, so `"Morse, Det. Insp. Jane"` → `title='Det. Insp.'` and a lone `"Smith, Xyz."` → `title='Xyz.'` — long-standing, verified against 1.4.0, and the mechanism behind #296 (`"Smith, Jr."` read title in 1.4.0; the comma's credential read takes it as suffix `Jr.` now). The docs said "leading word" until 2026-08-01 and were wrong for every comma path. FAMILY_COMMA **segment 0** peels too since #701, but by VOCABULARY only (rules/H6, in assign's family-comma branch rather than `_peel_leading_titles`): `Dr. Smith, John` → `title='Dr.'`, while `Xyz. Smith, John` keeps family `Xyz. Smith`, and a title-particle (`St`, `Freiherr`) stops the run, as does a title with nothing but post-nominals behind it (`Dr. Jr., John` keeps family `Dr.`); a title/suffix dual with a name word behind it still peels (`Rev. Sr. Smith, Mary`). It does not mutate `C.titles`, so the periodless form (`"Insp"`) is unaffected elsewhere. The `{2,}` length requirement — not a separate initials check — is what excludes single-letter initials like `"J."`; the same word after the given name is left as a middle name. **The inference OUTRANKS vocabulary where it runs**: `"Esq. Smith"` → `title='Esq.'` even though `esq` is suffix-only vocabulary, because the shape rule fires before anything consults the suffix sets. **The INFERENCE still runs in one direction only, and that is what the two slots share and where they part** (rewritten 2026-09-08, #316): a period-marked word is claimed by SHAPE at the front and by VOCABULARY at the back, so an unlisted abbreviation opening a name is a title while an unlisted abbreviation ending one is a NAME word (`"John Smith Xyz."` → `family='Xyz.'`). A trailing period-marked word the vocabulary knows as a title now IS one (`"John Smith Prof."` → `title='Prof.'`, `rules/H5`) — the comma path always read it that way and the two agree now — and a trailing period-marked word the UNAMBIGUOUS SUFFIX vocabulary knows is still a suffix, the suffix run being peeled first (`"John Smith Esq."` → `suffix='Esq.'`). A BARE trailing title word stays a name word (`"John Smith Sir"`, `"Mary Jane King"`), which is the doctrine line below. Meanwhile `period_joined_vocab` resolves INTERIOR-period tokens (`Lt.Gov.`, `Msc.Ed.`) to title-or-suffix by vocabulary, and `_extract._suffix_shaped` treats any period-final delimited content as not-a-nickname. Four sites, and the trailing one now resolves by two vocabularies in order rather than one; unifying the rest is open design work, not settled. (#109; see `docs/usage.rst` "Titles you didn't configure") +**The period-abbreviation title inference runs at the head of the GIVEN-NAME part, not the head of the name** — an unrecognized multi-letter word ending in a single trailing period (`_pieces._PERIOD_ABBREV`, a hand copy of the `period_abbreviation` regex, `{2,}` letters — it was assign's until #424 and group's until #439) is treated as a title in the leading title run, e.g. `"Insp. Jane Morse"` → `title='Insp.'`. "Leading" is per SEGMENT: `_peel_leading_titles` is called for NO_COMMA segment 0, SUFFIX_COMMA segment 0, and FAMILY_COMMA **segment 1**, so `"Morse, Det. Insp. Jane"` → `title='Det. Insp.'` and a lone `"Smith, Xyz."` → `title='Xyz.'` — long-standing, verified against 1.4.0, and the mechanism behind #296 (`"Smith, Jr."` read title in 1.4.0; the comma's credential read takes it as suffix `Jr.` now). The docs said "leading word" until 2026-08-01 and were wrong for every comma path. FAMILY_COMMA **segment 0** peels too since #701, but by VOCABULARY only (rules/H6, in assign's family-comma branch rather than `_peel_leading_titles`): `Dr. Smith, John` → `title='Dr.'`, while `Xyz. Smith, John` keeps family `Xyz. Smith`, and a title-particle (`St`, `Freiherr`) stops the run, as does a title with nothing but post-nominals behind it (`Dr. Jr., John` keeps family `Dr.`); a title/suffix dual with a name word behind it still peels (`Rev. Sr. Smith, Mary`). It does not mutate `C.titles`, so the periodless form (`"Insp"`) is unaffected elsewhere. The `{2,}` length requirement — not a separate initials check — is what excludes single-letter initials like `"J."`; the same word after the given name is left as a middle name. **The inference OUTRANKS vocabulary where it runs**: `"Esq. Smith"` → `title='Esq.'` even though `esq` is suffix-only vocabulary, because the shape rule fires before anything consults the suffix sets. **The INFERENCE still runs in one direction only, and that is what the two slots share and where they part** (rewritten 2026-09-08, #316): a period-marked word is claimed by SHAPE at the front and by VOCABULARY at the back, so an unlisted abbreviation opening a name is a title while an unlisted abbreviation ending one is a NAME word (`"John Smith Xyz."` → `family='Xyz.'`). A trailing period-marked word the vocabulary knows as a title now IS one (`"John Smith Prof."` → `title='Prof.'`, `rules/H5`) — the comma path read a lone `Smith, Prof.` that way already, and the given part's `Smith, John Prof.` since 2.3.0 (1.4.0 through 2.2.0 read `Doe, Jane Mrs.` middle 'Mrs.'), so the two agree now — and a trailing period-marked word the UNAMBIGUOUS SUFFIX vocabulary knows is still a suffix, the suffix run being peeled first (`"John Smith Esq."` → `suffix='Esq.'`). A BARE trailing title word stays a name word (`"John Smith Sir"`, `"Mary Jane King"`), which is the doctrine line below, save a listed salutation ending the given part after a family comma (rules/H8, `"Doe, Jane Mrs"` → `title='Mrs'`). Meanwhile `period_joined_vocab` resolves INTERIOR-period tokens (`Lt.Gov.`, `Msc.Ed.`) to title-or-suffix by vocabulary, and `_extract._suffix_shaped` treats any period-final delimited content as not-a-nickname. Four sites, and the trailing one now resolves by two vocabularies in order rather than one; unifying the rest is open design work, not settled. (#109; see `docs/usage.rst` "Titles you didn't configure") **Cyrillic suffix regexes need `re.I` even when the pattern is suffix-only** — a Latin title-cased word (`Ivanovich`) keeps its suffix lowercase, so `re.I` seemed skippable; but an irregular Cyrillic suffix can be nearly the whole word (`ильич`), so title-casing capitalizes into the suffix itself (`Ильич`). `east_slavic_patronymic_cyrillic` shipped without `re.I` on the Latin reasoning and silently failed on capitalized irregular forms — don't assume Latin's title-case safety transfers to Cyrillic. (#185) @@ -304,7 +304,7 @@ Add a dedicated `copy.deepcopy()` round-trip test for it too (see `test_regexes_ **Comparing against v1 needs `PYTHONSAFEPATH=1` AND a directory outside the worktree** — `uv run --isolated --no-project --with 'nameparser==1.4.0'` still puts the checkout's `nameparser/` ahead of the pinned wheel on `sys.path`, so the "v1" side silently imports the branch and every comparison reports parity. Run it as `cd && PYTHONSAFEPATH=1 uv run --isolated --no-project --with 'nameparser==1.4.0' python -c "..."`. This produces false confidence rather than an error, so it invalidates results without ever looking wrong. **A `tests/v2/cases.py` row's `parity` means 1.4.0, and nothing checks it** -- the table runs every row against the tree, so a row labelled `parity` that only matches master passes for good. #614 added eight such rows, compared against master; #620 found them by running the wheel this way and relabelled each with the change its reading came from, found by `git bisect run` in a scratch worktree (`git worktree add --detach`, never this one). Run the wheel before writing `parity`. -**A group-side reading of a family comma's given part that asks `tail_reading` disagrees with the given part's own walk, and the disagreement is accepted.** `tail_reading` reads the part as the comma-less name reads it; assign's walk over the given part after a family comma reads by its own predicates -- a shape-only numeral VI to X stays a name word there (`Smith, John van der Berg VI` reads middle 'van der Berg VI'), V before a third comma part is read strictly (#144), and P6 attaches a period-marked title-particle (`St.`) the comma-less reading takes as a title. So a bound or a decline group computes with `tail_reading` acts where the walk then reads no title: #623's chain bound (`Smith, John Mc Prof. VI`) and #637's join decline (`Smith, abdul Prof. VI` → middle 'Prof. VI', `Smith, abdul St.` → family 'St. Smith'). Both were recorded rather than narrowed (Derek: garbage input; decisions/H5 and #P5, rules/P5's Accepted clause). Before writing another group-side predictor of the given part, sweep those three shapes and decide, rather than rediscovering the gap in review. And a change to the trailing peel ITSELF reaches the given part through those predictors without touching them: #718's words-to-spare count for a one-letter numeral, asked through `trailing_run_start`, made the chain take the V in `Smith, Freiherr von Berg V` (given 'von Berg V', unreported) while assign's walk, which never counts there (S4: "read without it"), read it as the suffix -- found by review on a grid of family-comma and empty-head shapes behind a title-particle; the predictor now passes `count_numeral=False`. When a change moves what the peel takes, put a family comma with a title-particle head in its grid. +**A group-side reading of a family comma's given part that asks `tail_reading` disagrees with the given part's own walk, and the disagreement is accepted.** `tail_reading` reads the part as the comma-less name reads it, plus the bare salutations rules/H8 adds after a family comma, which a predictor of this part asks for with `salutations=True` (#835); assign's walk over the given part after a family comma reads by its own predicates -- a shape-only numeral VI to X stays a name word there (`Smith, John van der Berg VI` reads middle 'van der Berg VI'), V before a third comma part is read strictly (#144), and P6 attaches a period-marked title-particle (`St.`) the comma-less reading takes as a title. So a bound or a decline group computes with `tail_reading` acts where the walk then reads no title: #623's chain bound (`Smith, John Mc Prof. VI`) and #637's join decline (`Smith, abdul Prof. VI` → middle 'Prof. VI', `Smith, abdul St.` → family 'St. Smith'). Both were recorded rather than narrowed (Derek: garbage input; decisions/H5 and #P5, rules/P5's Accepted clause). Before writing another group-side predictor of the given part, sweep those three shapes and decide, rather than rediscovering the gap in review. And a change to the trailing peel ITSELF reaches the given part through those predictors without touching them: #718's words-to-spare count for a one-letter numeral, asked through `trailing_run_start`, made the chain take the V in `Smith, Freiherr von Berg V` (given 'von Berg V', unreported) while assign's walk, which never counts there (S4: "read without it"), read it as the suffix -- found by review on a grid of family-comma and empty-head shapes behind a title-particle; the predictor now passes `count_numeral=False`. When a change moves what the peel takes, put a family comma with a title-particle head in its grid. **A comparison that runs both sides in one tree reports 0 differences, which is what success looks like.** `tools/differential/compare.py` has been hardened against this — it generates its baseline worker into a temp dir, strips `PYTHONPATH` from the child, and aborts unless both the version AND the resolved path check out on each side; its README's three invocation traps are the analysis behind that design, worth reading before changing the harness. **The exposure is ad-hoc comparisons you write yourself**, where the same collapse has a cause the harness cannot disarm for you: **the shell's working directory persists between tool calls**, so a two-tree comparison written as two `cd`s silently runs both halves in whichever tree it landed in. Pin absolute paths, and assert `nameparser.__file__` on both sides the way `compare.py` does. The general rule outlives any particular trap: before believing a null result, prove the harness can report a difference — a clean run and a broken harness are the same output. The proof has to come from the input class the null result is about: #683 moved `tools/name_usage.py` to a POST and checked it with `shehu`, which is ASCII and so cannot show that the server decodes a Cyrillic body. A misdecoded query still runs and matches nothing, so the Cyrillic zeros it was meant to vouch for proved nothing until `Иван` (17,119 given) went through the same path. The converse trap is a mutation that moves nothing: it shows the code is unexercised by your grids, not that it is dead. #638 was filed calling group's chain re-ask dead on a grid with no title-particle head before a particle run and a dotted acronym; the next review found 400 moves (`Smith, Freiherr van Berg X.Y.Z.`) and the issue closed the same day. Before calling code dead, look for the input that reaches it. A CACHE of readings is the same trap one step removed: a wrong key copies one tree's readings in as another's, and the check then reports nothing. #859 first keyed `snapshot.py --local`'s kept baselines by `git rev-parse HEAD:nameparser`, which answers for the enclosing checkout, so a gitignored copy inside one (`build/lib`) took the checkout's key; two reviewers found it. Key a cached reading by what the import reads -- the files on disk, hashed again after the read -- never by what version control says should be there. @@ -348,7 +348,7 @@ Don't use the bare `python3 -m doctest .rst` CLI (no `optionflags`) to che **Prefix-join uses value-based `list.index()`** in `join_on_conjunctions` — fragile when a token value repeats (e.g. a trailing title that's also a suffix acronym, or two `van`s); constrain such lookups to start at `i + 1`. See #100. -**Title vs suffix is positional for BARE words, and the leading period-abbreviation rule overrides even that** — a word matching `TITLES` at the front of a name becomes `title`; the same word matching `SUFFIX_ACRONYMS`/`SUFFIX_WORDS` at the end becomes `suffix` (never both, regardless of the word's real-world meaning). The `TITLES`/suffix overlap was audited in #296 (2026-08-23): the pure postnominals (`jr`, `junior`, `phd`, `do`, `se`) left `TITLES`, the v1-residue `dr`/`sra` left the suffix sets, and the eleven words still in both (`ms`, `sa`, `sr`, `lt`, `ra`, `vc`, and the ranks `cpl`, `cpo`, `cpt`, `csm`, `sgm`; `md` was a twelfth until #853 took its title reading out, a leading `Md` being the given name Muhammad) are deliberate duals that position decides. External test sources (old issue gists, etc.) sometimes assert `suffix` for a leading professional abbreviation like `RA`/`PD`/`Dipl.-Ing.` — that's the source data being wrong, not a parser bug. Verify position before "fixing" it. Two qualifications the older "purely positional" wording papered over, the first measured 2026-08-01 and the second answered 2026-09-08 (#316): a PERIOD-marked leading word is claimed by the shape rule before any vocabulary is read (`"Esq. Smith"` → `title`, though `esq` is suffix-only), and a PERIOD-marked trailing word is claimed by VOCABULARY — first the suffix sets, which the peel reads before anything else (`"John Smith Esq."` → `suffix`), then the titles (`"John Smith Prof."` → `title='Prof.'`, `rules/H5`) — while an unlisted abbreviation there stays a name part (`"John Smith Xyz."` → `family='Xyz.'`) unless it is written as two or more period-separated chunks, which since 2.4 IS a trailing shape rule (`Policy.unlisted_dotted_suffixes`, default on: `"John Smith X.Y.Z."` → `suffix='X.Y.Z.'`, read by the same words-to-spare count a bare ambiguous acronym takes, `rules/S3`), except paired initials in a script with capitals, which stay a name word (`"Ram Bahadur B.K."` → `family='B.K.'`, #864). The SINGLE trailing period is still not one, deliberately — it is the abbreviation shape any word can wear. The bare word is where "positional" still holds whole -- outside a part past the second comma, which holds no name word, so C2 reads a bare title word there as a title since #603 (`Eric H. Holder, Jr., Attorney General`) -- and the reason it must: `TITLES` holds words that are in no suffix set — 787 of them on 2026-10-11 (746 before #606's salutations), measured with `L = Parser().lexicon; len(L.titles - L.suffix_acronyms - L.suffix_words)`, a figure that grows with the vocabulary and never shrinks the argument — and many are ordinary surnames (`king`, `bishop`, `prince`, `pope`, `judge`, `sheriff`, `baron`, `master`, ...), so a vocabulary-first rule over BARE trailing words would read `"Mary Jane King"` as `title='King'`, `family='Jane'`. The period is what separates the safe case from that one, and it separates it by being a WRITING convention rather than by making the collision go away: `"Mary Jane King"` keeps `family='King'` while `"Mary Jane King."` reads `title='King.'`, `family='Jane'`, a cost accepted under the input-is-a-name premise rather than one the rule prevents (the trailing-position bullet in decisions/vocabulary-exclusions, measured 2026-09-09). Since #316 that sentence describes the shipped rule rather than an aspiration. +**Title vs suffix is positional for BARE words, and the leading period-abbreviation rule overrides even that** — a word matching `TITLES` at the front of a name becomes `title`; the same word matching `SUFFIX_ACRONYMS`/`SUFFIX_WORDS` at the end becomes `suffix` (never both, regardless of the word's real-world meaning). The `TITLES`/suffix overlap was audited in #296 (2026-08-23): the pure postnominals (`jr`, `junior`, `phd`, `do`, `se`) left `TITLES`, the v1-residue `dr`/`sra` left the suffix sets, and the eleven words still in both (`ms`, `sa`, `sr`, `lt`, `ra`, `vc`, and the ranks `cpl`, `cpo`, `cpt`, `csm`, `sgm`; `md` was a twelfth until #853 took its title reading out, a leading `Md` being the given name Muhammad) are deliberate duals that position decides. External test sources (old issue gists, etc.) sometimes assert `suffix` for a leading professional abbreviation like `RA`/`PD`/`Dipl.-Ing.` — that's the source data being wrong, not a parser bug. Verify position before "fixing" it. Two qualifications the older "purely positional" wording papered over, the first measured 2026-08-01 and the second answered 2026-09-08 (#316): a PERIOD-marked leading word is claimed by the shape rule before any vocabulary is read (`"Esq. Smith"` → `title`, though `esq` is suffix-only), and a PERIOD-marked trailing word is claimed by VOCABULARY — first the suffix sets, which the peel reads before anything else (`"John Smith Esq."` → `suffix`), then the titles (`"John Smith Prof."` → `title='Prof.'`, `rules/H5`) — while an unlisted abbreviation there stays a name part (`"John Smith Xyz."` → `family='Xyz.'`) unless it is written as two or more period-separated chunks, which since 2.4 IS a trailing shape rule (`Policy.unlisted_dotted_suffixes`, default on: `"John Smith X.Y.Z."` → `suffix='X.Y.Z.'`, read by the same words-to-spare count a bare ambiguous acronym takes, `rules/S3`), except paired initials in a script with capitals, which stay a name word (`"Ram Bahadur B.K."` → `family='B.K.'`, #864). The SINGLE trailing period is still not one, deliberately — it is the abbreviation shape any word can wear. The bare word is where "positional" still holds whole -- outside a part past the second comma, which holds no name word, so C2 reads a bare title word there as a title since #603 (`Eric H. Holder, Jr., Attorney General`), and outside a listed salutation ending the given part after a family comma, which rules/H8 reads as the title since #835 (`Doe, Jane Mrs`), the comma having marked the surname's place -- and the reason it must: `TITLES` holds words that are in no suffix set — 787 of them on 2026-10-11 (746 before #606's salutations), measured with `L = Parser().lexicon; len(L.titles - L.suffix_acronyms - L.suffix_words)`, a figure that grows with the vocabulary and never shrinks the argument — and many are ordinary surnames (`king`, `bishop`, `prince`, `pope`, `judge`, `sheriff`, `baron`, `master`, ...), so a vocabulary-first rule over BARE trailing words would read `"Mary Jane King"` as `title='King'`, `family='Jane'`. The period is what separates the safe case from that one, and it separates it by being a WRITING convention rather than by making the collision go away: `"Mary Jane King"` keeps `family='King'` while `"Mary Jane King."` reads `title='King.'`, `family='Jane'`, a cost accepted under the input-is-a-name premise rather than one the rule prevents (the trailing-position bullet in decisions/vocabulary-exclusions, measured 2026-09-09). Since #316 that sentence describes the shipped rule rather than an aspiration. ### Tests (`tests/`) diff --git a/docs/customize.rst b/docs/customize.rst index 4f2e3af8..99f977f3 100644 --- a/docs/customize.rst +++ b/docs/customize.rst @@ -102,8 +102,8 @@ Removing works the same way, and drops the word from recognition: >>> Parser(lexicon=lean).parse("Professor Robert Johns").title '' -A few fields mark a subset of another — ``given_name_titles`` over -``titles``, ``particles_ambiguous`` over ``particles``, +A few fields mark a subset of another — ``given_name_titles`` and +``after_given_titles`` over ``titles``, ``particles_ambiguous`` over ``particles``, ``suffix_acronyms_ambiguous`` over ``suffix_acronyms``, ``conjunctions_ambiguous`` over ``conjunctions``, and ``honorific_tails`` over ``suffix_words``. Entries belong in the base @@ -111,13 +111,13 @@ field too, so add to both and remove from the marker first. Three of them enforce that — ``particles_ambiguous``, ``suffix_acronyms_ambiguous`` and ``honorific_tails`` raise ``ValueError`` naming the orphans, because an orphan in each of those does real harm rather than nothing. The -other two are deliberately unchecked because an orphan there is inert: +other three are deliberately unchecked because an orphan there is inert: ``given_name_titles`` matches a title run as one space-joined string, or by that run's last word, so a legitimate entry like ``"sir and dame"`` -is no single word in ``titles``; and a ``conjunctions_ambiguous`` entry -is only ever read for a word that is a conjunction, so -``remove(conjunctions={"e"})`` simply works and the stale marker entry -is never consulted. +is no single word in ``titles``; and a ``conjunctions_ambiguous`` or +``after_given_titles`` entry is only ever read for a word that is a +conjunction or a title, so ``remove(conjunctions={"e"})`` simply works +and the stale marker entry is never consulted. .. _title-detection-off: diff --git a/docs/design/decisions/H5-trailing-title-run.md b/docs/design/decisions/H5-trailing-title-run.md index 8e73c513..c0cfada3 100644 --- a/docs/design/decisions/H5-trailing-title-run.md +++ b/docs/design/decisions/H5-trailing-title-run.md @@ -3,7 +3,7 @@ ## Standing - After the trailing suffix run, successive period-marked words the title vocabulary lists chain into the title from the end, leaving one name word standing, and join the title field in input order: `John Smith Prof.` → title 'Prof.', family 'Smith'; `John Smith Prof. Dr.` → title 'Prof. Dr.'; `Dr. John Smith Prof.` → title 'Dr. Prof.' — 2026-09-08 #316. -- The back is claimed by vocabulary and the front by shape, so an unlisted abbreviation (`John Smith Xyz.` → family 'Xyz.') and a bare title word (`John Smith Sir`, `Mary Jane King`) are name words there, the walk asking `is_title_piece` and never `is_leading_title` — 2026-09-08 #316. +- The back is claimed by vocabulary and the front by shape, so an unlisted abbreviation (`John Smith Xyz.` → family 'Xyz.') and a bare title word (`John Smith Sir`, `Mary Jane King`) are name words there, the walk asking `is_title_piece` and never `is_leading_title` — 2026-09-08 #316. A listed salutation ending the given part after a family comma is the exception, read as the title bare (decisions/H8). - No fork is reported; where the doubt is real, the word left standing carries it (H4) — 2026-09-08 #316. - The given part after a family comma gets the walk: `Smith, John Prof.` → title 'Prof.', given 'John' — 2026-09-08 #316. - The trailing peel and the title chain are one reading run to a fixed point (`_pieces.tail_reading`), so a trailing title is transparent to the suffix reading (`John Smith Jr. Prof.` → suffix 'Jr.'; `abdul rahman MA Prof.` reads as `abdul rahman MA` does, plus the title), and the bound join takes no piece the chain takes (`Sir abdul Prof.` → given 'abdul') — 2026-09-09 (the /simplify round on the bundle's PR). @@ -18,7 +18,7 @@ Open: none in this file. ## Log -- 2026-09-08 (the #316/#489 bundle; closes #316(a)) (A1's scope superseded by review round 2 and its single re-peel by the /simplify round, both below; A1's `John Prof. MA` family 'MA' by #289, decisions/S6, `John MA` reading suffix 'MA'; A2's `Smith, John Prof. MA` middle by #531, decisions/S5; the detail mismatch by #626, decisions/A1; Q3's unconditional H2 in part by #761, decisions/H2, C13 winning after a family comma) — a trailing run of period-marked title words is a title. `John Smith Prof.` read family `Prof.` and lost the surname while `Smith, Prof.` read title `Prof.`; the two comma paths disagreed and #316 is that question. They agree now. +- 2026-09-08 (the #316/#489 bundle; closes #316(a)) (A1's scope superseded by review round 2 and its single re-peel by the /simplify round, both below; A1's `John Prof. MA` family 'MA' by #289, decisions/S6, `John MA` reading suffix 'MA'; A2's `Smith, John Prof. MA` middle by #531, decisions/S5; the detail mismatch by #626, decisions/A1; Q3's unconditional H2 in part by #761, decisions/H2, C13 winning after a family comma; the bare title word's name-word reading in part by 2026-10-24 #835, decisions/H8, for a listed salutation ending the given part after a family comma) — a trailing run of period-marked title words is a title. `John Smith Prof.` read family `Prof.` and lost the surname while `Smith, Prof.` read title `Prof.`; the two comma paths disagreed and #316 is that question. They agree now. - **The doctrine, stated rather than left to be inferred.** A period-marked word is claimed by SHAPE at the front and by VOCABULARY at the back. At the front an unlisted abbreviation is a title (#H2) and the shape outranks the vocabulary, which is why `Esq. Smith` reads title though `esq` is post-nominal vocabulary. At the back there is no shape rule, so only a listed title word chains into the title and `John Smith Xyz.` keeps family `Xyz.`. A BARE title word at the back is a name word — `John Smith Sir` and `Mary Jane King` are untouched — TITLES holding ordinary surnames being exactly why, which is the Excluded (TITLES) block in decisions/vocabulary-exclusions ("the trailing-position rule that must NOT be adopted"). The asymmetry is honest rather than an oversight: #109 shipped the leading inference on purpose, and no blanket trailing vocabulary rule is safe while king, judge and bishop are titles. - **Q3 declined: H2 stays unconditional.** #316's open question 2 asked for the symmetric leading rule, `Esq. Smith` → suffix. Declined: it reverses rules/S2's "a suffix never opens the string" for period-marked words for the sake of one word, and the family comma already carries the credential case (`Smith, Esq.` → suffix). Recorded at #H2, where the `Open: #316` hook was. - **Q2 decided: a RUN, mirroring #H3.** Successive single words that wear the abbreviation shape AND are title vocabulary chain into the title from the end, floor one name piece, so `John Smith Prof. Dr.` reads title `Prof. Dr.` rather than making `Prof.` a name word. Not chosen: one word only, which would have left `Prof.` standing as a name word in exactly the input the rule is about. The title field joins in INPUT order and the rendering needed no change — the title view joins TITLE tokens in token order already, so `Dr. John Smith Prof.` reads title `Dr. Prof.` for free. diff --git a/docs/design/decisions/H8-salutation-ending-the-given-part.md b/docs/design/decisions/H8-salutation-ending-the-given-part.md new file mode 100644 index 00000000..814e1d82 --- /dev/null +++ b/docs/design/decisions/H8-salutation-ending-the-given-part.md @@ -0,0 +1,22 @@ +# H8 — a salutation ending the given part + +## Standing + +- A salutation ending the given part after a family comma, written bare or with a stop at its edge that H5's abbreviation shape does not take (`Mr..`), is the title, read as its period-marked spelling reads in every comma shape and under every name order: `Doe, Jane Mrs` → title 'Mrs', given 'Jane'; `DOE, JOHN Q MR` → title 'MR', middle 'Q' — 2026-10-24 #835. +- Only the salutations marked `after_given` in the vocabulary qualify, and a salutation borne as a given name or a surname is left out by an exclusion carrying its counts, since a middle name may be either (`Doe, John Goodman` keeps middle 'Goodman') — 2026-10-24 #835. +- No other title kind qualifies (`Doe, John King` and `Doe, Jane Dr` keep their middle names), a title/suffix dual stays a suffix (`Doe, Jane Ms`), and the comma-less writing is H5's (`Jane Doe Mrs` keeps family 'Mrs') — 2026-10-24 #835. +- A salutation that is someone's middle name reads as the title after a family comma, an accepted cost (`Doe, Jane Miss`); so does the empty head's one given word moving to the family as the period-marked spelling moves it (`, Jane Mrs` → family 'Jane') — 2026-10-24 #835. +- The rule reads the comma, not a declared order: `Doe Jane Mrs` under FAMILY_FIRST keeps middle 'Mrs', so the family-comma invariant of decisions/two-input-invariants does not hold for a trailing salutation — 2026-10-24 #835. + +Open: [#877](https://github.com/derek73/python-nameparser/issues/877) (whether a bare `I` or `V` the salutation leaves ending the given part, read as the suffix by rules/C9, should report `suffix-or-name`). + +## Log + +- 2026-10-24, #835 — A BARE SALUTATION ENDING A FAMILY COMMA'S GIVEN PART IS THE TITLE. `Doe, Jane Mrs` read middle 'Mrs' while `Doe, Jane Mrs.` (H5) and `Doe, John Jr Mr` (the salutation behind a post-nominal) read title, and `Doe, John Mr Jr` read middle 'Mr'; 1.4.0 and 2.5.0 read the first as middle 'Mrs' too (wheels checked 2026-10-24). The FEC's itemized contributions write `FAMILY, GIVEN MIDDLE SALUTATION` in capitals, the period often left off: on #834's `fec-individuals` corpus (file dated 2026-10-18, measured 2026-10-22, figures in #835) 13,243 names ended in a bare title word read as a middle name, `MR` 8,483 of them. + THE REASON H5's BARE-WORD CLAUSE DOES NOT REACH HERE. H5 keeps a bare title word at the end of a name as a name word because TITLES holds ordinary surnames (`Mary Jane King`). After a family comma the surname's place is marked, so only the middle-name reading is left against the title, and of the title kinds (#750) the salutation is the one that is a form of address and not something a person is named after. So the rule is keyed on that kind, and narrower than the kind: the property marks the salutations not borne as names. + THE VOCABULARY. A behavioral title property, `after_given = true`, allowed on a `salutation` reading only (a build check), compiles into `AFTER_GIVEN_TITLES` and the new `Lexicon.after_given_titles`; v1's `Constants` has no attribute for it, and `_snapshot()` intersects it with the titles, so deleting the title is the v1 knob. Considered and not taken: keying the rule on the `salutation` kind itself, with exclusions for the borne words — it makes a kind compile into a behavior, where every behavior so far is a property (decisions/vocabulary-model). + THE VETTING (`uv run python tools/name_usage.py --json` over every salutation and the romanizations of the non-Latin ones, 2026-10-24). A middle name may be a given name or a surname, so a word borne as either is excluded. Excluded, each with its counts on the word: goodman (US Census 2010 surname 73,136), herr (11,980), herren (4,466), master (2,735; SSA given 476), mister (1,211; SSA 746), señora (SSA given 1,065, as SENORA), signor (999), signore (895), herra (638), paní (403, as PANI), señor (282, as SENOR), härra (237, as HARRA), fru and frú (a Cameroonian surname, about 10,800 bearers by the note on sv.toml's entry, and 132 in the Census), пан (Wikidata family 743), the Bengali মোঃ and মো (they abbreviate Mohammad, which #853 reads as a given name where it leads), श्री/শ্রী (also an element of given names, Sri or Shri; one Wikidata given-name bearer, excluded on the doubt rather than the count) and बाबू (romanized Babu, Census surname 1,496, ranked a given name in India and Sri Lanka by nam_dict, and an element of compound given names such as Ram Babu). A diacritic-free count stands for an accented word, the Census and SSA writing none. A romanization counts for a word in another script only where its bearers write that script: बाबू's do, while the bearers of the romanizations of सौ (Sau, Vietnamese), মি (Mi, Korean and Chinese), κα (Kyria, US given names) and пані (Pani, an Indian surname) do not, so those are kept. Kept with small counts, recorded on each word as a name reading: miss (Census surname 164), heren (123), sra (Census surname 149, SSA given 27), signora (SSA given 90), señorita, bwana and mr (SSA under 10 each), frau (Wikidata family 42), and gospodin (a Bulgarian given name in nam_dict, band 5, kept because a Bulgarian middle name is a patronymic, Gospodinov, rather than the given name itself). The line at roughly 200 US bearers is a judgment, not a measured threshold. ms and sr carry no flag, being title/suffix duals the suffix reading takes first. `hr` was excluded on the FEC read below. Recompute the set: `uv run python tools/vocab/query.py --kind salutation`, and the refusals with `--excluded after_given`. + THE MECHANISM. H5's trailing title chain, in the given part after a family comma, admits a word the vocabulary marks, matched after its edge stops are folded, so a doubled stop (`Mr..`, which is not H5's abbreviation shape) is admitted as the bare word is. Everything H5's chain already does there — transparency to the suffix reading, the floor leaving one name piece, the title joining the field in input order — holds for it unchanged. The group-side predictors of that part (the particle chain's bound and the bound given-name join's decline, which read the part as `tail_reading` reads it) admit the same words, so they cannot disagree with the walk (the Gotcha on group-side predictors in AGENTS.md). Measured: over a grid of nine family-comma heads (an empty head, a nickname-only head, particle and connective surnames, a suffix and a title before the comma), 21 given parts (initials, a bound given name, particles, a title-particle, an ambiguous acronym, numerals V and VI, a post-nominal, a one-case part), seven salutation spellings and six tails (a second comma, a trailing suffix, acronym or period-marked title), under all three name orders, every one of the 23,814 bare rows reads exactly as its period-marked twin reads, fields and ambiguity kinds both, and none of the period-marked rows moved against master `3f8e29b2`. + THE CORPORA (measured 2026-10-24 against master `3f8e29b2`, py3.11). No committed corpus name moved: the snapshot gains only this rule's own examples, and the CLDR, GND and Wikidata agreement views match their records. `snapshot.py --local fec-individuals`, base recorded with `--tree`: 12,381 of 1,893,964 names move, each a salutation leaving the middle name for the title, some with a credential or a generational word behind it: `MR` 9,339, `MRS` 1,908, `MISS` 1,108, `MR..` 24, and two more joining a title already read. Three had read `HR` as the Danish and German abbreviation of Herr, one of them behind a woman's given name: in US data it is more likely two initials run together, so `hr` was excluded on that evidence and those three are not among the moves. 385 of the moves lose a `CONJUNCTION_OR_INITIAL` report (`DOE, JOHN E MR`): with the salutation gone the letter has no name word behind it to connect, which is what P8 asks. In 361 of them the letter stays a middle initial; the other 24 are among the 59 below. `ukec-individuals` does not move. + A CONSEQUENCE NOT DECIDED HERE: 59 of the FEC moves end the given part in a lone `V` or `I` once the salutation is the title (`DOE, JANE I MRS`), which then reads as the generational suffix, as `DOE, JANE I` and `DOE, JANE I MRS.` already read on master (#144's lenient tail test after a family comma). In these names it is a middle initial. The bare spelling now reads as its twins read; whether #144's reading should change is its own question. + THE FAMILY-FIRST WRITING (the design-docs review). A declared FAMILY_FIRST order fixes the surname as a comma does, but the rule reads only the comma: `Doe Jane Mrs` under FAMILY_FIRST keeps middle 'Mrs' (FAMILY_FIRST_GIVEN_LAST: given 'Mrs'). decisions/two-input-invariants pairs those two writings; over the 82 flagged words written without a stop, an apostrophe or a hyphen, each after three family and three given names, they agree on 738 of 738 pairs on master and on none now. Recorded as an Accepted clause of rules/H8 rather than extended: the declared order says where the surname is, not where the given part ends, and the trailing slot of a comma-less name is H5's. diff --git a/docs/design/decisions/README.md b/docs/design/decisions/README.md index 4f447a02..b962220d 100644 --- a/docs/design/decisions/README.md +++ b/docs/design/decisions/README.md @@ -30,6 +30,7 @@ Entry conventions: - [H5 — the trailing title run](H5-trailing-title-run.md) - [H6 — a title before a family comma](H6-title-before-a-family-comma.md) - [H7 — a joined name unit carrying title vocabulary](H7-joined-name-unit-carrying-title-vocabulary.md) +- [H8 — a salutation ending the given part](H8-salutation-ending-the-given-part.md) ## Particles & surname prefixes (P) diff --git a/docs/design/decisions/vocabulary-exclusions.md b/docs/design/decisions/vocabulary-exclusions.md index e4d7b89a..11bc1e7e 100644 --- a/docs/design/decisions/vocabulary-exclusions.md +++ b/docs/design/decisions/vocabulary-exclusions.md @@ -62,7 +62,7 @@ Excluded (TITLES): - The per-word refusals are data since #750 (2026-10-19): each word refused a title reading — the Indic ঠাকুর/ठाकुर, कुमारी/কুমারী, बेगम/বেগম, शेख/শেখ, आचार्य and राजा/रानी (#343, #344), and #606's ông/bà, thiru, puan, pan, rodina, gróf, cik, paron, tikin, Latin janab, marshal and justice, #662's malam (Derek's decline) and bay, #660's shehu, and #789's wm (2026-10-22) — is an `excluded` entry in its language's file with its reason, and with its counts as name readings where they were taken. `uv run python tools/vocab/query.py --excluded title` lists them. A contributor who adds one back meets the reason as a build error. - shehu's counts do not settle a graf-class weighing, and must not be read as if they did: its one leading-position figure, the Wikidata given-name count on its entry, is a lower bound (Shagari's own entry has no given-name statement), and its other counts are family-name bearers, a position C-i does not ask about. The evidence that Shehu leads a name is not a count. It is Shagari himself, and CLDR's Hausa sample data, whose six renderings carrying the word all file it as the given name (`grep '"given": "Shehu"' tools/differential/cldr_fields.jsonl`). It arrived with an early bulk import (1a51555f) and was never put to C-i before #660. Whoever re-argues it counts Hausa given-name use first; the accepted cost is the honorific use, `Shehu Abubakar` reading given Shehu as any untitled name does, and the reading belongs in an opt-in pack if anyone asks for it. malam is the opposite case: CLDR's Hausa data files Malam as the title (`grep '"title": "Malam"' tools/differential/cldr_fields.jsonl`), so its refusal is a decline on the counts, not a shehu-style removal. - The class-level half of the Indic refusals stays here: the divergence on शेख/শেখ is deliberate and is the sri/shri case run backwards — the Latin sheikh cluster SHIPS as given-name titles for the Arabic addressing form, while the Indic spellings name the family. shehu's removal moved three CLDR corpus names (`Shehu Muhammed`, `Muhammed, Shehu`, `Modibbo, Shehu Bello`), all three now agreeing with CLDR on every role (moves: `uv run python tools/differential/snapshot.py --since v2.4.1`). -- The trailing-position rule that must NOT be adopted, and 2026-09-08 says what the line is: TITLES holds hundreds of words in no suffix set, at least nineteen of them ordinary English surnames (king, judge, bishop, baron, sheriff, ...), so a blanket "vocabulary outranks position in the trailing slot" reading would turn "Mary Jane King" into title="King" with the family name gone. The leading half of this argument is AGENTS.md's "Dean is deliberately absent" gotcha; this is the trailing half, and it shadows the family name rather than the given (#316). What #316 settled is that the prohibition is on BARE words: "Mary Jane King" still reads family King, while "John Smith Prof." reads title Prof. **The second half of that argument is RETRACTED 2026-09-09**, in review of the docs commit: "a period-marked trailing word being one nobody writes as a surname" claimed the period keeps the collision out of the trailing slot, and it does not. Measured on the branch tree — "Mary Jane King." reads title "King.", given Mary, family Jane, the family name shadowed exactly as the bare reading would have shadowed it, and "John Smith Judge." reads title "Judge." The reach is the vocabulary rather than a handful: of the 746 titles in no suffix set, 627 read as a trailing title once a period is written behind them, and every one of the 119 that do not is kept out by the abbreviation SHAPE — a digit, a hyphen, an apostrophe, or a script whose letters carry combining marks (rules/H2 records that half) — no plain ASCII-letter title missing at all. So king, judge, bishop, baron and sheriff ARE reachable in the trailing slot; what the prohibition above buys is the bare spelling and nothing more. The period is a WRITING convention the rule can read, not evidence about the word, and reading it is an ACCEPTED COST under the premise that the input is a name (rules.md H Background): someone who ends a name with a period-marked word has written an abbreviation. rules/H5 carries it as an Accepted clause with "Mary Jane King." as the executable example. Recompute the reach with `L = Parser().lexicon; ws = [w for w in L.titles - L.suffix_acronyms - L.suffix_words if " " not in w]; len([w for w in ws if parse("Mary Jane %s." % w).title])` against `len(ws)` — 627 of 746 on 2026-09-09 — the digits move as the vocabulary grows and neither argument does. See #H5. +- The trailing-position rule that must NOT be adopted, and 2026-09-08 says what the line is: TITLES holds hundreds of words in no suffix set, at least nineteen of them ordinary English surnames (king, judge, bishop, baron, sheriff, ...), so a blanket "vocabulary outranks position in the trailing slot" reading would turn "Mary Jane King" into title="King" with the family name gone. The leading half of this argument is AGENTS.md's "Dean is deliberately absent" gotcha; this is the trailing half, and it shadows the family name rather than the given (#316). What #316 settled is that the prohibition is on BARE words: "Mary Jane King" still reads family King, while "John Smith Prof." reads title Prof. **The second half of that argument is RETRACTED 2026-09-09**, in review of the docs commit: "a period-marked trailing word being one nobody writes as a surname" claimed the period keeps the collision out of the trailing slot, and it does not. Measured on the branch tree — "Mary Jane King." reads title "King.", given Mary, family Jane, the family name shadowed exactly as the bare reading would have shadowed it, and "John Smith Judge." reads title "Judge." The reach is the vocabulary rather than a handful: of the 746 titles in no suffix set, 627 read as a trailing title once a period is written behind them, and every one of the 119 that do not is kept out by the abbreviation SHAPE — a digit, a hyphen, an apostrophe, or a script whose letters carry combining marks (rules/H2 records that half) — no plain ASCII-letter title missing at all. So king, judge, bishop, baron and sheriff ARE reachable in the trailing slot; what the prohibition above buys is the bare spelling and nothing more. The period is a WRITING convention the rule can read, not evidence about the word, and reading it is an ACCEPTED COST under the premise that the input is a name (rules.md H Background): someone who ends a name with a period-marked word has written an abbreviation. rules/H5 carries it as an Accepted clause with "Mary Jane King." as the executable example. Recompute the reach with `L = Parser().lexicon; ws = [w for w in L.titles - L.suffix_acronyms - L.suffix_words if " " not in w]; len([w for w in ws if parse("Mary Jane %s." % w).title])` against `len(ws)` — 627 of 746 on 2026-09-09 — the digits move as the vocabulary grows and neither argument does. See #H5. The bare prohibition has one exception since #835 (decisions/H8): a listed salutation ending the given part after a family comma is the title, the comma having marked the surname's place and the salutations borne as names being excluded. Excluded (Policy.script_orders defaults): Script.KATAKANA is deliberately absent — a pure-katakana token is predominantly a transcribed foreign name kept in its source order, so nothing defaults on it (rule W4's boundary). Noted 2026-08-15: of the three Script-keyed axes, this is the one with no force-a-decision guard (mechanisms.md#FORCE-A-DECISION-TABLE), so a new Script member silently gets no order. diff --git a/docs/design/decisions/vocabulary-model.md b/docs/design/decisions/vocabulary-model.md index cb78c0f3..8369d462 100644 --- a/docs/design/decisions/vocabulary-model.md +++ b/docs/design/decisions/vocabulary-model.md @@ -18,7 +18,7 @@ The vocabulary was a set of base sets plus marker subsets (`particles` and `part - Order and shape are separate: a language supplies only a default order, and the shapes (where titles, suffixes, particles and commas sit) belong to each order and live in the core; vocabulary readings are to settle a clash and name readings only to keep the gate on — 2026-10-15/16. - `cased` replaces `capitalization_exceptions` and, with `same_as`, a canonical spelling, and a reading's `native` spelling is for surfacing; `acronym` is a reading property replacing the acronym/word split, one suffix kind per word; `expands` records the meaning an entry was added for; `addresses` takes `family` and `given`, `full` and `any` being refused until a later phase — 2026-10-15/16; 2026-10-18 #737. - The vocabulary is per-word TOML in `nameparser/vocab/.toml`, one file per BCP 47 tag plus `und` and `mul`, at `schema = 1` (a compiler refuses a newer one), every reading a table and no `language` key; `tools/vocab/build.py` compiles it into a committed `nameparser/config/_compiled.py`, the parser never reads the TOML, and parsing it at import was declined — 2026-10-18 #737. -- The marker sets are explicit flags on readings (`may_be_given`, `may_be_name`, `may_be_initial`, `attached`, `addresses`, `cased`), not yet derived from name readings; `may_be_given` is stated on every particle, never defaulted, and two kinds of one word must agree on it — 2026-10-18 #737; 2026-10-19 #749. +- The marker sets are explicit flags on readings (`may_be_given`, `may_be_name`, `may_be_initial`, `attached`, `addresses`, `after_given`, `cased`), not yet derived from name readings; `may_be_given` is stated on every particle, never defaulted, and two kinds of one word must agree on it; `after_given` stands on a salutation reading only (decisions/H8) — 2026-10-18 #737; 2026-10-19 #749; 2026-10-24 #835. - Exclusions are data: an `excluded` key refuses a kind, a category or a behavioral property, in the file of the language whose reading it refuses, and the build refuses the excluded thing in any file, quoting the reason — 2026-10-18 #737. - A per-word fact that moves into the TOML is replaced in decisions/ by a query that finds it (decisions/README.md, edit kind 4) — 2026-10-18 #737. - The script is read from the word, not the file: a script subtag is only for a difference the text cannot show — 2026-10-18 #737. @@ -127,3 +127,6 @@ Open: #752 (phase 2, derived ambiguity), #789 (whether the dotted `Wm.`, `Chas.` - **2026-10-22 #789 (Derek) — `wm` LEAVES THE TITLES.** SUPERSEDES: the `wm` bullet of 2026-10-19 #750, the rest of the titles. #68 asked that `Wm. Scott Magargee III` not read given name `Wm.`, and `wm` was listed as a title for it in 0.5.5. The request was mistaken: `Wm` is William written short, and a name it opens is a man named William, which a reader of the name would not hesitate over. The case came back from the FEC donor list (`tools/differential/local_corpora.py`, fec-individuals), where bare `WM` opened the given part of 69 names (re-measured 2026-10-24 on 08d130ed), every one read as a title with the given name taken from the next word or left empty. The title reading is refused in en.toml with that reason, so a contributor who adds it back meets it as a build error. - **The dotted spelling is split by position, not by spelling.** rules/H2's shape reads an unlisted period-marked abbreviation opening the given part as a title, so `Wm. Smith` keeps title 'Wm.' and `Smith, Wm.` too; making `Wm.`, `Chas.`, `Geo.`, `Ma.` or `Jo.` a given name there is a reading that outranks a shape rule, phase 2's question (#752, D14(b) of its spec), and tests/v2/cases.py pins today's `Wm. Smith` so that change shows. Where VOCABULARY reads the word, the dotted spelling moves with the bare one: a trailing `Wm.` was rules/H5's title (`John Smith Wm.` → family 'Wm.', `Smith, John Wm.` → middle 'Wm.', the readings of 1.4.0 through 2.2.0), and a `Wm.` opening the part before a family comma was rules/H6's (`Wm. Smith, John` → family 'Wm. Smith', the reading of 1.4.0 through 2.4.0). The 2026-10-19 bullet said removal moves only the bare spelling, and is corrected in place. - **Measured** against 08d130ed (`snapshot.py --local` with the base recorded by `--tree`): no committed corpus holds the token `wm`, so the snapshot and the CLDR, GND and Wikidata agreement views could not move and did not; psc-individuals' field agreements do not move either. fec-individuals moves 77 of 1,893,964 names: 69 with bare `WM` opening the given part, 3 with bare `WM` behind another title in it, 3 with a trailing `WM.` and 2 with `WM.` before the family comma. ctgov-officials moves 1 of 352,104, `Wm` behind a title before a family comma. + +- **2026-10-24 #835 — `after_given`, the salutations that may end a given part.** A behavioral title property, allowed on a `salutation` reading only (a build check, since an office or rank word is borne as a middle name), compiles into `AFTER_GIVEN_TITLES` and `Lexicon.after_given_titles`, read by rules/H8. A salutation borne as a name refuses it with an exclusion carrying its counts. The rule is keyed on a property rather than on the kind, which would make a kind compile into a behavior where every other behavior is a property; the vetting and the measurement are decisions/H8's. Adding the field changes `Lexicon`'s pickle layout, as 2.1 and 2.3 did. + - **Recompute** rather than quoting: `uv run python tools/vocab/query.py --kind salutation`, `--excluded after_given`. diff --git a/docs/design/rules/H-titles-and-honorifics.md b/docs/design/rules/H-titles-and-honorifics.md index 32aff09c..a214ef9c 100644 --- a/docs/design/rules/H-titles-and-honorifics.md +++ b/docs/design/rules/H-titles-and-honorifics.md @@ -1,5 +1,5 @@ ## Titles & honorifics (H) -Background: an honorific title precedes a name and is not itself part of it; it addresses or ranks the person. Most titles address by surname ("Mr. Johnson"), but a few — knighthoods, some clerical and courtesy titles — address by given name ("Sir John"). The library keeps a vocabulary of titles and, separately, of these given-name titles. A period-marked word is claimed by SHAPE at the front and by VOCABULARY at the back. At the front an unlisted abbreviation is a title (H2), the shape outranking the vocabulary; at the back there is no shape rule, so only a listed title word chains into the title (H5) and an unlisted abbreviation stays a name word. A BARE title word at the back is a name word too, TITLES holding ordinary surnames — king, judge, bishop — which is what bars the blanket vocabulary-wins answer (#316, decided 2026-09-08). An input the title peel eats down to one last title-vocabulary word is H4's, and what it does with that word is a convention rather than a reading of the vocabulary. Two criteria govern two different questions here. Membership in the given-name-title list follows HOW THE TITLE ADDRESSES: a title that precedes and addresses by the given name belongs (Sir, Sheikh, the Arabic honorifics الدكتور/الشيخ — which qualify even though those traditions fully retain family names). Whether an EMPTY FAMILY is correct output is the separate question, governed by surname retention: renunciation abolishes the surname, so for Swami, Guru, Baba or Lama family="" is right (#346), while rabbi and imam traditions keep surnames — "Rabbi Cohen" addresses by title and keeps family "Cohen". Conflating the two criteria either ejects the Arabic entries or sweeps in titles that break +Background: an honorific title precedes a name and is not itself part of it; it addresses or ranks the person. Most titles address by surname ("Mr. Johnson"), but a few — knighthoods, some clerical and courtesy titles — address by given name ("Sir John"). The library keeps a vocabulary of titles and, separately, of these given-name titles. A period-marked word is claimed by SHAPE at the front and by VOCABULARY at the back. At the front an unlisted abbreviation is a title (H2), the shape outranking the vocabulary; at the back there is no shape rule, so only a listed title word chains into the title (H5) and an unlisted abbreviation stays a name word. A BARE title word at the back is a name word too, TITLES holding ordinary surnames — king, judge, bishop — which is what bars the blanket vocabulary-wins answer (#316, decided 2026-09-08); the one exception is a listed salutation ending the given part after a family comma, the comma having marked the surname's place (H8). An input the title peel eats down to one last title-vocabulary word is H4's, and what it does with that word is a convention rather than a reading of the vocabulary. Two criteria govern two different questions here. Membership in the given-name-title list follows HOW THE TITLE ADDRESSES: a title that precedes and addresses by the given name belongs (Sir, Sheikh, the Arabic honorifics الدكتور/الشيخ — which qualify even though those traditions fully retain family names). Whether an EMPTY FAMILY is correct output is the separate question, governed by surname retention: renunciation abolishes the surname, so for Swami, Guru, Baba or Lama family="" is right (#346), while rabbi and imam traditions keep surnames — "Rabbi Cohen" addresses by title and keeps family "Cohen". Conflating the two criteria either ejects the Arabic entries or sweeps in titles that break "Rabbi Cohen". diff --git a/docs/design/rules/H5-trailing-title-run.md b/docs/design/rules/H5-trailing-title-run.md index 26cac408..d04f6059 100644 --- a/docs/design/rules/H5-trailing-title-run.md +++ b/docs/design/rules/H5-trailing-title-run.md @@ -42,7 +42,7 @@ "John van der Berg Prof." → title="Prof." "John van der Berg Prof." → family="van der Berg" ``` -- **After a family comma** ("Smith, John van der Berg Prof."): the given part's particle chain stops in front of the trailing run of suffixes and titles, read as the comma-less name reads it, so there too a trailing title is the title and P6's attachment reads the particle in front of it as though the title were absent. +- **After a family comma** ("Smith, John van der Berg Prof."): the given part's particle chain stops in front of the trailing run of suffixes and titles, read as the comma-less name reads it with the bare salutations H8 adds, so there too a trailing title is the title and P6's attachment reads the particle in front of it as though the title were absent. ``` "Smith, John van der Berg Prof." → title="Prof." "Smith, John van der Berg Prof." → middle="van der Berg" @@ -52,6 +52,10 @@ ``` "Mary Smith née Jones Prof." → title="Prof." ``` +- **A bare salutation after a family comma** (H8): the bare title word this rule leaves a name word is the title where it is a listed salutation ending the given part after a family comma, the family being fixed, and the chain takes it there as it takes a period-marked title word. + ``` + "Smith, Jane Mrs" → title="Mrs" + ``` - **A bound given-name word beside a trailing title** (P5): what the title chain leaves is also what counts as a name word to spare (P5). A trailing title word is not one, so a bound given-name word behind one joins exactly as it joins with the title absent — and lands in the same field, the run in FRONT of the joined pair being the run H1 asks about. Said of the comma-less writing, which is where the title chain runs before the reserve is read. After a family comma the reserve reads no peel, but the join declines a trailing title word the title chain takes (P5), so the title is read there too, save the shapes P5's Accepted clause records, as the segment written with an ordinary given name reads it. ``` "Berg, abdul Prof." → title="Prof." diff --git a/docs/design/rules/H8-salutation-ending-the-given-part.md b/docs/design/rules/H8-salutation-ending-the-given-part.md new file mode 100644 index 00000000..4143ea84 --- /dev/null +++ b/docs/design/rules/H8-salutation-ending-the-given-part.md @@ -0,0 +1,80 @@ +# H8 — a salutation ending the given part + +**When:** a family comma divides the name, and the given part after it ends in a word whose title reading is a salutation, written bare or with a stop at its edge that H5's abbreviation shape does not take (a doubled stop, as in "Mr.."), with a name word in front of it, as in "Doe, Jane Mrs". + +**Then:** the word joins the title field, and the given part reads as it would without it. + +- **It reads as the period-marked spelling reads.** The salutation chains into the trailing title run (H5) as a period-marked title word does, so the bare and the period-marked spellings read alike in every comma shape and under every name order: behind a post-nominal or in front of one, and behind another title. + ``` + "Doe, John Mr Jr" → title="Mr" + "Doe, John Mr Jr" → suffix="Jr" + "Doe, John Jr Mr" → title="Mr" + "Doe, Jane Mrs, PhD" → title="Mrs" + "Doe, Jane Mr Mrs" → title="Mr Mrs" + "Doe, Jane Mr.." → title="Mr.." + ``` + +**Because:** H5 keeps a bare title word ending a name as a name word because the title vocabulary holds ordinary surnames, and after a family comma that reason is gone, the comma having marked the surname's place. What is left is the middle-name reading, and a salutation is a form of address, not a rank or office someone could be named after. Filers' software writes this order in volume (`DOE, JOHN Q MR`), the period as often left off as written. + +``` +"Doe, Jane Mrs" → title="Mrs" +"Doe, Jane Mrs" → given="Jane" +"DOE, JOHN Q MR" → middle="Q" +"van Doe, Jane Mrs" → family="van Doe" +"Doe, Mrs" → family="Doe" · boundary +"Jane Doe Mrs" → family="Mrs" · boundary +``` + +**Except:** + +- **A salutation that is also borne as a name stays a name word.** A middle name may be a given name or a surname, so the vocabulary marks only the salutations borne as neither, and goodman, master, herr and fru are left out with their counts, as is a non-Latin salutation whose romanization is borne where that script is written (बाबू, Babu). + ``` + "Doe, John Goodman" → middle="Goodman" + "Doe, John Master" → middle="Master" + "Doe, Jane Fru" → middle="Fru" + "शर्मा, राम बाबू" → middle="बाबू" + ``` +- **A title of any other kind stays a name word.** Office, rank and religious words are borne as middle names, and an academic title is not a salutation, so Dr written bare there stays one too. + ``` + "Doe, John King" → middle="King" + "Doe, Jane Dr" → middle="Dr" + ``` +- **A title/suffix dual stays with the suffix run** (S2): Ms and Sr are post-nominals as well, and the suffix reading takes them before any title reading is asked. + ``` + "Doe, Jane Ms" → suffix="Ms" + ``` + +**Precedence:** + +- **A bare title word ending the name** (H5): H5 reads one as a name word, and after a family comma a salutation is this rule's, so H5's chain takes it as it takes a period-marked title. + ``` + "Doe, Jane Mrs" → middle="" + ``` +- **A bound given-name word in front of the salutation** (P5): the join declines a salutation this rule reads, as it declines a trailing title H5 takes, so the bound word stands alone as the given name. + ``` + "Doe, abdul Mrs" → given="abdul" + ``` +- **A particle in front of the salutation** (P6): the attachment reads the particle as though the salutation were absent, as it does in front of a period-marked title. + ``` + "Doe, Jane van Mrs" → family="van Doe" + ``` + +**Accepted:** + +- A salutation that is someone's middle name, after a family comma, reads as the title. Miss is the nearest case the vocabulary keeps, borne as a surname by about 160 people in the US Census 2010. + ``` + "Doe, Jane Miss" → title="Miss" + ``` +- With nothing before the comma, the one given word the salutation leaves reads as a title's lone name word reads (H1), the family, as the period-marked spelling has read it since H5 (2.3.0, #316); written alone, that word is the given name. + ``` + ", Jane Mrs" → family="Jane" + ", Jane" → given="Jane" + ``` +- The comma-less writing keeps the salutation a name word under a declared family-first order, though the declared order fixes the surname as a comma does: the rule reads the comma written in the name, not the order declared for it, so the two writings decisions/two-input-invariants pairs read this salutation differently. + ``` + "Doe Jane Mrs" family-first → middle="Mrs" + ``` + +--- +**Signal:** comma: a family comma dividing the name, with listed salutation vocabulary ending the given part +**history:** decisions/H8 · **implemented:** nameparser/_pipeline/_assign.py, nameparser/_pipeline/_classify.py, nameparser/_pipeline/_pieces.py diff --git a/docs/design/rules/P5-bound-given-names.md b/docs/design/rules/P5-bound-given-names.md index 20a8f9be..3d33b3b2 100644 --- a/docs/design/rules/P5-bound-given-names.md +++ b/docs/design/rules/P5-bound-given-names.md @@ -51,7 +51,7 @@ **Except:** -- **The join never absorbs a marker standing as a word of its own** — a marker is not a name word (M2) — nor a word of the unambiguous suffix vocabulary (S2), wherever position will then place it, nor a title word a trailing rule reads as a title: one the trailing title run (H5) takes from the given part after a family comma, read as the comma-less name reads it, or any title word in a part past the second comma (C2). +- **The join never absorbs a marker standing as a word of its own** — a marker is not a name word (M2) — nor a word of the unambiguous suffix vocabulary (S2), wherever position will then place it, nor a title word a trailing rule reads as a title: one the trailing title run (H5) takes from the given part after a family comma, read as the comma-less name reads it with the bare salutations H8 adds, or any title word in a part past the second comma (C2). ``` "abdul Jr Smith Berg" → given="abdul" "abdul Jr Smith Berg" → middle="Jr Smith" diff --git a/docs/design/rules/README.md b/docs/design/rules/README.md index bcbca97a..8475dd32 100644 --- a/docs/design/rules/README.md +++ b/docs/design/rules/README.md @@ -122,6 +122,7 @@ Terms the rules use without defining them each time, each with the rule that def - [H5 — the trailing title run](H5-trailing-title-run.md) - [H6 — a title before a family comma](H6-title-before-a-family-comma.md) - [H7 — a joined name unit carrying title vocabulary](H7-joined-name-unit-carrying-title-vocabulary.md) +- [H8 — a salutation ending the given part](H8-salutation-ending-the-given-part.md) ## [Particles & surname prefixes (P)](P-particles-and-surname-prefixes.md) diff --git a/docs/locales.rst b/docs/locales.rst index 74913ccd..e23e1868 100644 --- a/docs/locales.rst +++ b/docs/locales.rst @@ -313,8 +313,8 @@ right and is validated on its own, before it is unioned onto the base in the set that field marks (``particles``, ``suffix_acronyms``, ``suffix_words``). A fragment holding ``particles_ambiguous={"van"}`` without ``particles={"van"}`` raises ``ValueError`` before any union -happens. ``given_name_titles`` and ``conjunctions_ambiguous`` are not -checked this way, and can mark a word only the base holds. ``zh`` is the +happens. ``given_name_titles``, ``conjunctions_ambiguous`` and +``after_given_titles`` are not checked this way, and can mark a word only the base holds. ``zh`` is the shipped worked example: its ``Lexicon(surnames=...)`` has to satisfy every ``Lexicon`` rule standing alone, before anything unions it onto the base. diff --git a/docs/migrate.rst b/docs/migrate.rst index edc49840..89112d18 100644 --- a/docs/migrate.rst +++ b/docs/migrate.rst @@ -263,8 +263,9 @@ fields: ``md`` to ``MD`` Some ``Lexicon`` fields have no ``CONSTANTS`` attribute at all — -``conjunctions_ambiguous``, ``maiden_markers``, ``surnames`` and -``honorific_tails`` — so they are reached through the 2.0 API only. +``conjunctions_ambiguous``, ``maiden_markers``, ``surnames``, +``honorific_tails`` and ``after_given_titles`` — so they are reached +through the 2.0 API only. .. warning:: diff --git a/docs/release_log.rst b/docs/release_log.rst index 4bf56a12..e096e78b 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -22,6 +22,7 @@ Release Log - **Read a leading Md as the given name it abbreviates, Muhammad, rather than a title.** ``Mr Md Karim Uddin`` gives title ``Mr``, first ``Md``, middle ``Karim``, where 1.4.0 through 2.5.0 gave title ``Mr Md``, first ``Karim``; ``John Smith, MD Jones`` gives last ``Smith``, suffix ``MD Jones`` where it gave title ``MD``, last ``John Smith``. ``MD`` after a name is still the degree, and ``Md.`` with a full stop is still a title. One limit: ``Md`` opening the part after a family comma reads as a credential, so ``Uddin, Md Karim`` gives suffix ``Md Karim`` where it gave title ``Md``, first ``Karim``. :issue:`853` - **Read Wm as the given name it abbreviates, William, rather than a title.** ``Wm Smith`` gives first ``Wm``, last ``Smith``, where 1.4.0 through 2.5.0 gave title ``Wm`` with no first name. ``Wm.`` with a full stop opening the name is still a title. :issue:`789` - **Fix a surname written as two dotted initials reading as a credential.** ``Ram Bahadur B.K.`` gives middle ``Bahadur``, last ``B.K.``, where 2.4.0 through 2.5.0 gave last ``Bahadur``, suffix ``B.K.``, restoring 2.3.0's reading, and ``Hari G.C.`` gives last ``G.C.``, where 2.0.0 through 2.5.0 gave suffix ``G.C.`` with no last name. Three or more dotted letters after a full name are still a credential (``John Smith X.Y.Z.``), and so is a listed one (``John Smith M.A.``). :issue:`864` + - **Read a salutation ending the first name after a family comma as the title.** ``Doe, Jane Mrs`` gives title ``Mrs``, first ``Jane``, last ``Doe``, where 1.4.0 through 2.5.0 gave middle ``Mrs``; ``Doe, Jane Mrs.`` has read as the title since 2.3.0. A salutation also borne as a name, such as ``Goodman`` or ``Herr``, is still read as a middle name, and so is any other title, such as ``King`` or ``Dr``; ``Lexicon.after_given_titles`` lists the salutations. A ``Lexicon`` or ``Parser`` pickled by an earlier release no longer loads; re-pickle after upgrading. :issue:`835` - **Fix a zero-width space or byte-order mark sticking to the word beside it.** ``Dr. John Smith`` with an invisible zero-width space in front gives title ``Dr.``, first ``John``, where 1.4.0 through 2.5.0 gave first ``Dr.`` (the character kept), middle ``John``. The two now divide the words of a name as a space does, with no setting to turn that off, though not inside quotes or a ``(née ...)`` clause; the zero-width joiner and non-joiner still stay inside the word. :issue:`851` * 2.5.0 - October 19, 2026 diff --git a/nameparser/_config_shim.py b/nameparser/_config_shim.py index 1083d7bc..8a970ba1 100644 --- a/nameparser/_config_shim.py +++ b/nameparser/_config_shim.py @@ -1048,6 +1048,7 @@ def _build_snapshot(self) -> tuple[Lexicon, Policy, _RenderDefaults]: from nameparser.config.maiden_markers import MAIDEN_MARKERS from nameparser.config.suffixes import GLUED_HONORIFICS from nameparser.config.surnames import KOREAN_SURNAMES + from nameparser.config.titles import AFTER_GIVEN_TITLES # Every set field and every capitalization_exceptions key passes # _v1_matchable BEFORE any set algebra below: an entry v1 could # never match, or one Lexicon folds to empty (#582), is dropped @@ -1207,6 +1208,14 @@ def matchable(field: str) -> frozenset[str]: # stops mattering, the same rule ambiguous_acronyms gets # against suffix_acronyms above. honorific_tails=GLUED_HONORIFICS & suffix_words, + # likewise no v1 manager: the salutations ending a given + # part are 2.6 behavior (#835), so they ride in the + # snapshot only, and the one v1 knob that reaches them is + # deleting the title. Lexicon does not check this pair + # (classify reads a member only where it is a title too), + # so the intersection is kept for `_snapshot() == + # Lexicon.default()` legibility, as conjunctions_ambiguous'. + after_given_titles=AFTER_GIVEN_TITLES & titles, # TupleManager is dict[str, object] (v1 parity: values were # never statically str-typed); every real entry is a str, # same assumption _DelimiterManager's sentinel lookup makes. diff --git a/nameparser/_lexicon.py b/nameparser/_lexicon.py index f18d5007..949d4d03 100644 --- a/nameparser/_lexicon.py +++ b/nameparser/_lexicon.py @@ -25,7 +25,7 @@ "titles", "given_name_titles", "suffix_acronyms", "suffix_words", "suffix_acronyms_ambiguous", "particles", "particles_ambiguous", "conjunctions", "conjunctions_ambiguous", "bound_given_names", - "maiden_markers", "surnames", "honorific_tails", + "maiden_markers", "surnames", "honorific_tails", "after_given_titles", ) #: (marker, base, why) triples. Each marker QUALIFIES how entries of @@ -60,13 +60,14 @@ #: property on a word reading (tools/vocab/build.py). The check here #: is the only thing that holds a caller's own. #: -#: Two fields are deliberately NOT here and have no check of their own. +#: Three fields are deliberately NOT here and have no check of their own. #: given_name_titles -- see the note in __post_init__ for why every #: attempt at one rejected working configurations. conjunctions_ambiguous #: -- an orphan is never consulted because both the classify fork and #: its ambiguity emitter require the base entry as well, so an orphan #: decides nothing, and AGENTS.md's invariants rule guards harm, not -#: no-ops. +#: no-ops. after_given_titles -- the same: classify tags a word from it +#: only where the word is in titles too, so an orphan is never read. _SUBSET_FIELDS = ( ("particles_ambiguous", "particles", "an orphan emits a spurious particle-or-given ambiguity"), @@ -689,6 +690,15 @@ class Lexicon: #: vocabulary is. Full default list: #: :data:`~nameparser.config.suffixes.GLUED_HONORIFICS`. honorific_tails: frozenset[str] = frozenset() + #: Salutations that read as a title where one ENDS the given part + #: after a family comma, written bare: "Doe, Jane Mrs" reads title + #: "Mrs" where an office or rank word there stays a middle name + #: ("Doe, John King"), rules/H8. A period-marked title word reads as + #: a title there whether listed here or not (rules/H5). Entries need + #: not also be in ``titles``; one that is not is never consulted. + #: Full default list: + #: :data:`~nameparser.config.titles.AFTER_GIVEN_TITLES`. + after_given_titles: frozenset[str] = frozenset() #: Lowercase word -> case mask used by capitalized(): the word's #: own letters and digits, each in the case it takes ("phd" -> #: "PhD"), laid over the word as written, so the one entry @@ -945,7 +955,8 @@ def _default_lexicon() -> Lexicon: SUFFIX_WORDS, ) from nameparser.config.surnames import KOREAN_SURNAMES - from nameparser.config.titles import GIVEN_NAME_TITLES, TITLES + from nameparser.config.titles import ( + AFTER_GIVEN_TITLES, GIVEN_NAME_TITLES, TITLES) # every vocabulary constant is a frozenset since #293, so each one # feeds its strictly-typed frozenset[str] field as it stands -- and @@ -973,6 +984,7 @@ def _default_lexicon() -> Lexicon: maiden_markers=MAIDEN_MARKERS, surnames=KOREAN_SURNAMES, honorific_tails=GLUED_HONORIFICS, + after_given_titles=AFTER_GIVEN_TITLES, # pass canonical pair-tuples so this strictly-typed call site never # feeds a Mapping to the tuple-annotated field; __post_init__ # still tolerates a Mapping at runtime for interactive use diff --git a/nameparser/_pipeline/_assign.py b/nameparser/_pipeline/_assign.py index 354a8cb5..1fdbb38d 100644 --- a/nameparser/_pipeline/_assign.py +++ b/nameparser/_pipeline/_assign.py @@ -1259,7 +1259,11 @@ def reads_as_a_suffix(m: int, titled: frozenset[int]) -> bool: candidates = [k for k in range(n, len(pieces)) if k == n or not reads_as_a_suffix(k, frozenset())] - kept = trailing_titles(candidates, pieces, ptags, tokens) + # rules/H8: "the word joins the title field, and the given + # part reads as it would without it" -- a bare salutation + # chains here as a period-marked title does + kept = trailing_titles(candidates, pieces, ptags, tokens, + salutations=True) walkable = frozenset(candidates) titled_idx = frozenset(candidates[kept:]) for k in titled_idx: diff --git a/nameparser/_pipeline/_classify.py b/nameparser/_pipeline/_classify.py index ff7e733e..f4ae3cd6 100644 --- a/nameparser/_pipeline/_classify.py +++ b/nameparser/_pipeline/_classify.py @@ -30,6 +30,7 @@ Tags emitted -- stable (API): "particle", "conjunction", "initial"; namespaced (unstable): "vocab:title", "vocab:given-title", +"vocab:after-given-title", "vocab:suffix", "vocab:suffix-word", "vocab:suffix-ambiguous", "vocab:particle-ambiguous", "vocab:bound-given", "vocab:maiden-marker", "vocab:maiden-marker-cont"; and, in a namespace of its own, @@ -128,6 +129,10 @@ def _tags_for(token: WorkToken, n: str, state: ParseState, tags.add("vocab:title") if n in lex.given_name_titles: tags.add("vocab:given-title") + # rules/H8: "the word joins the title field" -- tagged only where the + # word is a title too, so an entry outside `titles` is inert + if n in lex.after_given_titles and n in lex.titles: + tags.add("vocab:after-given-title") if suffix_as_written(n, token.text, lex): tags.add("vocab:suffix") if n in lex.suffix_words: diff --git a/nameparser/_pipeline/_group.py b/nameparser/_pipeline/_group.py index 333136e2..1a51dae1 100644 --- a/nameparser/_pipeline/_group.py +++ b/nameparser/_pipeline/_group.py @@ -1074,6 +1074,8 @@ def chain(tail: int) -> None: # rules/P5: "nor a title word a trailing rule reads as a # title" -- after a family comma one H5's trailing run takes # from the given part, read as the comma-less name reads it, + # plus the bare salutations H8 adds after a family comma + # (`salutations=True`), # which the LENIENT join reads no peel for and so asks here, # only where a title word follows ('Berg, abdul Prof.', # #637; where the given part's own walk then reads the words @@ -1092,7 +1094,8 @@ def chain(tail: int) -> None: and (site is ClauseSite.TAIL or fk + 1 in tail_reading( peel_walk(fk, ptags), pieces, ptags, - tokens, one_case)[1]))): + tokens, one_case, + salutations=True)[1]))): pass elif bound_join is BoundJoin.LENIENT: # post-comma the family is fixed and the pair is the diff --git a/nameparser/_pipeline/_pieces.py b/nameparser/_pipeline/_pieces.py index 3a9d9944..65e20c7c 100644 --- a/nameparser/_pipeline/_pieces.py +++ b/nameparser/_pipeline/_pieces.py @@ -1431,17 +1431,21 @@ def credential_run(rest: Sequence[int], peel: Peel, p: int, def trailing_titles(rest: Sequence[int], pieces: Sequence[Sequence[int]], ptags: Sequence[Set[str]], tokens: Sequence[WorkToken], - end: int | None = None) -> int: + end: int | None = None, + salutations: bool = False) -> int: """How many pieces of `rest` the trailing title chain LEAVES standing: `rest[:kept]` are the name pieces and `rest[kept:]` the - period-marked title words the chain took, in piece order. Counted + title words the chain took (period-marked, or under `salutations` + a bare salutation), in piece order. Counted the way `peel_trailing` counts, so the two answers compose without arithmetic at the call site. `rest` is the caller's NAME pieces: on the no-comma path what the S2 peel left, after a family comma the segment's pieces that the segment's own suffix reading does not claim, and in `tail_reading` the leftovers of whichever peel is current. `end`, where given, reads `rest[:end]` without the - copy, for `tail_reading`'s passes. + copy, for `tail_reading`'s passes. `salutations` is the given + part after a family comma asking, where a bare salutation the + vocabulary marks `after_given` chains as well (rules/H8). Floor: the first position of `rest` is never taken, so one name piece stands and a name is never all title. An empty `rest` returns 0, which is what leaves assign's bare-suffix @@ -1474,9 +1478,16 @@ def trailing_titles(rest: Sequence[int], pieces: Sequence[Sequence[int]], # the shape is ANDed with is_title_piece, so the word is listed # vocabulary, and a listed CJK title wearing a stop should read # as a title. - if (len(piece) == 1 - and _PERIOD_ABBREV.match(tokens[piece[0]].text) - and is_title_piece(piece, ptags[idx], tokens)): + if len(piece) == 1 and ( + (_PERIOD_ABBREV.match(tokens[piece[0]].text) + and is_title_piece(piece, ptags[idx], tokens)) + # rules/H8: "the given part after it ends in a word whose + # title reading is a salutation" -- the salutation chains + # as a period-marked title does ("It reads as the + # period-marked spelling reads"); a tag test, so a name + # word pays no frame + or (salutations and "vocab:after-given-title" + in tokens[piece[0]].tags)): k -= 1 continue break @@ -1575,6 +1586,7 @@ def tail_reading(rest: list[int], pieces: Sequence[Sequence[int]], one_case: bool | None, units: Sequence[int] | None = None, count_numeral: bool = True, + salutations: bool = False, ) -> tuple[list[int], tuple[int, ...], Peel]: """The S2 peel and the H5 chain read together to a FIXED POINT: peel, chain, splice the chained pieces out, peel again over what @@ -1649,7 +1661,8 @@ def tail_reading(rest: list[int], pieces: Sequence[Sequence[int]], spliced pieces. The fuzz that checked this against the re-peeling loop, the three conditions each shown load-bearing by a weakened copy failing it, is recorded on the PR closing #558. - `count_numeral` is `peel_trailing`'s, passed to every pass. + `count_numeral` is `peel_trailing`'s, passed to every pass, and + `salutations` is `trailing_titles`', passed to every chain. The splice itself is never materialized in between: the runs are collected back to front and joined once. """ @@ -1663,7 +1676,7 @@ def tail_reading(rest: list[int], pieces: Sequence[Sequence[int]], units=units, second=second, count_numeral=count_numeral) kept = trailing_titles(rest, pieces, ptags, tokens, - end=peeled.names) + end=peeled.names, salutations=salutations) if kept == peeled.names: p = run_start(rest, peeled.names, pieces, ptags, tokens, units) return rest, (), (peeled if p == peeled.names else credential_run( @@ -1713,7 +1726,7 @@ def tail_reading(rest: list[int], pieces: Sequence[Sequence[int]], picks = list(peeled.picks) numeral = peeled.numeral kept = trailing_titles(rest, pieces, ptags, tokens, - end=peeled.names) + end=peeled.names, salutations=salutations) if behind: rest = rest[:hi] + [j for run in reversed(behind) for j in run] final = Peel(peeled.names, numeral, tuple(picks), None) @@ -1959,7 +1972,8 @@ def trailing_run_start(start: int, pieces: Sequence[Sequence[int]], bound for a reading that takes the trailing titles as well as the suffixes, as assign's walk does in the given part after a family comma (#623), where `trailing_start` sees only the suffixes and - `trailing_start_past_titles` leaves the titles inside the bound.""" + `trailing_start_past_titles` leaves the titles inside the bound. + The chain takes a bare salutation too, as that walk does (rules/H8).""" # rules/S4: "the given part's trailing slot after a family comma is # read without it (S5)" -- the numeral half of the count, which # assign's walk there never asks either, so the chain this bounds @@ -1967,7 +1981,7 @@ def trailing_run_start(start: int, pieces: Sequence[Sequence[int]], # acronym half is still counted here, as before #718 rest, titled, peel = tail_reading(peel_walk(start, ptags), pieces, ptags, tokens, one_case, - count_numeral=False) + count_numeral=False, salutations=True) first = len(pieces) for k in rest[peel.names:]: first = min(first, k) diff --git a/nameparser/config/_compiled.py b/nameparser/config/_compiled.py index 35adb150..52c3fd0e 100644 --- a/nameparser/config/_compiled.py +++ b/nameparser/config/_compiled.py @@ -876,6 +876,97 @@ 'স্বামী', }) +AFTER_GIVEN_TITLES: frozenset[str] = frozenset({ + 'bapak', + 'batoni', + 'binibini', + 'bwana', + 'dhr', + 'doamna', + 'doamnă', + 'domnișoara', + 'domnul', + 'encik', + 'familie', + 'frau', + 'frk', + 'fräulein', + 'fröken', + 'frøken', + 'ginang', + 'ginoo', + 'goodwife', + 'gospodin', + 'gospođa', + 'gospođica', + 'heren', + 'herrn', + 'juffrouw', + 'kalbatoni', + 'madam', + 'madame', + 'mademoiselle', + 'meneer', + 'mevr', + 'mevrouw', + 'mevrouwe', + 'miss', + 'misses', + 'mlle', + 'mme', + 'monsieur', + 'mr', + 'mrs', + 'mx', + 'neiti', + 'owura', + 'ponas', + 'ponia', + 'preili', + 'proua', + 'rouva', + 'senhor', + 'senhora', + 'senhorita', + 'señores', + 'señorita', + 'señoritas', + 'sig.na', + 'sig.ra', + 'signora', + 'signorina', + 'slečna', + 'sra', + 'sres', + 'srta', + 'srtas', + 'ungfrú', + 'κα', + 'κος', + 'г-жа', + 'г-н', + 'пані', + "גב'", + 'גברת', + 'גב׳', + 'מר', + 'जनाब', + 'श्रीमती', + 'श्रीमान', + 'श्रीयुत', + 'सुश्री', + 'सौ', + 'জনাব', + 'মি', + 'মিঃ', + 'মিসেস', + 'মোছা', + 'মোছাঃ', + 'মোসা', + 'মোসাঃ', + 'শ্রীমতী', +}) + NON_GIVEN_NAME_PARTICLES: frozenset[str] = frozenset({ "'t", 'ad-', diff --git a/nameparser/config/titles.py b/nameparser/config/titles.py index b459850b..c5652db1 100644 --- a/nameparser/config/titles.py +++ b/nameparser/config/titles.py @@ -3,7 +3,8 @@ # kind. tools/vocab/build.py compiles them into _compiled.py and runs the # checks that were import-time asserts here: a title reading with # addresses = "given" is what puts a word in GIVEN_NAME_TITLES, so the -# subset relation holds by construction (decisions/vocabulary-model). +# subset relation holds by construction (decisions/vocabulary-model), as +# a salutation reading with after_given = true does for AFTER_GIVEN_TITLES. from nameparser.config import _compiled GIVEN_NAME_TITLES = _compiled.GIVEN_NAME_TITLES @@ -18,6 +19,14 @@ #: recognition titles like "Deputy Secretary of State". TITLES = _compiled.TITLES +AFTER_GIVEN_TITLES = _compiled.AFTER_GIVEN_TITLES +""" +Salutations read as a title where one ends the given part after a family +comma, written without a period: "Doe, Jane Mrs". A subset of TITLES. A +salutation that is also borne as a name (goodman, master, herr) is left +out, so "Doe, John Goodman" keeps middle "Goodman". +""" + # 1.x name, deprecated in 2.2 and removed in 3.0 (#293). Unlike the # module moves in #293, the constant did not change module, so this @@ -47,4 +56,5 @@ # retired name silently; with a PARTIAL __all__ it would bind the # retired name and drop the live ones instead. # Source order, not alphabetical -- see the note in suffixes.py. -__all__ = ["GIVEN_NAME_TITLES", "TITLES", "FIRST_NAME_TITLES"] +__all__ = ["GIVEN_NAME_TITLES", "TITLES", "AFTER_GIVEN_TITLES", + "FIRST_NAME_TITLES"] diff --git a/nameparser/vocab/README.md b/nameparser/vocab/README.md index dd970371..3b57734f 100644 --- a/nameparser/vocab/README.md +++ b/nameparser/vocab/README.md @@ -173,6 +173,7 @@ A property is **behavioral** when it compiles into a set and so changes how name | Property | Type | Applies to | Unwritten | Effect | |---|---|---|---|---| | `addresses` | behavioral | `title` | `"family"` | `"given"` adds the word to `GIVEN_NAME_TITLES`: `Sir John` reads `John` as the given name ([rules/H1](../../docs/design/rules/H1-title-and-one-name-word.md), a title and one name word) | +| `after_given` | behavioral | `title` | false | `AFTER_GIVEN_TITLES`, on a `salutation` reading only: written bare at the end of the given part after a family comma, the word is the title, as in `Doe, Jane Mrs` ([rules/H8](../../docs/design/rules/H8-salutation-ending-the-given-part.md), a salutation ending the given part). A salutation borne as a given name or a surname is left out with an exclusion of the property, its counts beside it (`goodman`, `herr`), since a middle name may be either | | `acronym` | behavioral | `suffix` | true for `credential`, false for every other kind | true: `SUFFIX_ACRONYMS`, matched with every period removed (`M.D.` is `md`); false: `SUFFIX_WORDS` | | `may_be_name` | behavioral | `suffix` | false | `SUFFIX_ACRONYMS_AMBIGUOUS`, on an acronym reading only: an acronym that is also a name (`ma`) is read as a suffix only where its spelling or the name around it says so: `John Smith MA` and `John Smith M.A.` end in a suffix, `John Smith Ma` in a family name ([rules/S4](../../docs/design/rules/S4-ambiguous-acronym-needs-words-to-spare.md), the ambiguous acronym needs words to spare; [rules/S6](../../docs/design/rules/S6-case-signal-at-the-suffix-slot.md), the case signal at the suffix slot) | | `attached` | behavioral | `suffix` | absent | `GLUED_HONORIFICS`, on a word reading only: `{ side = "after" }` splits the word off the end of a name token, as in `田中さん` ([rules/W2](../../docs/design/rules/W2-glued-honorific-peel.md), the glued-honorific peel) | diff --git a/nameparser/vocab/ak.toml b/nameparser/vocab/ak.toml index f486aeb4..c0c4cba2 100644 --- a/nameparser/vocab/ak.toml +++ b/nameparser/vocab/ak.toml @@ -2,4 +2,4 @@ schema = 1 [[word]] text = "owura" -salutation = { issue = 662, gender = "m", note = "under 100 US Census bearers and none found on Wikidata (tools/name_usage.py, 2026-10-17)" } +salutation = { issue = 662, gender = "m", note = "under 100 US Census bearers and none found on Wikidata (tools/name_usage.py, 2026-10-17)", after_given = true } diff --git a/nameparser/vocab/bg.toml b/nameparser/vocab/bg.toml index 54e02641..4032fb4c 100644 --- a/nameparser/vocab/bg.toml +++ b/nameparser/vocab/bg.toml @@ -2,11 +2,11 @@ schema = 1 [[word]] text = "г-жа" -salutation = { gender = "f", issue = 269, expands = "госпожа, Mrs." } +salutation = { gender = "f", issue = 269, expands = "госпожа, Mrs.", after_given = true } [[word]] text = "г-н" -salutation = { gender = "m", issue = 269, expands = "господин, Mr." } +salutation = { gender = "m", issue = 269, expands = "господин, Mr.", after_given = true } [[word]] text = "акад" diff --git a/nameparser/vocab/bn.toml b/nameparser/vocab/bn.toml index 952eb819..09fdf5d9 100644 --- a/nameparser/vocab/bn.toml +++ b/nameparser/vocab/bn.toml @@ -10,7 +10,7 @@ excluded = { salutation = { reason = "Kumari is a given name and a family name i [[word]] text = "জনাব" -salutation = { gender = "m", issue = 343, expands = "Janab, Mr." } +salutation = { gender = "m", issue = 343, expands = "Janab, Mr.", after_given = true } [[word]] text = "বেগম" @@ -18,47 +18,48 @@ excluded = { salutation = { reason = "Begum is a trailing name element far more [[word]] text = "মিঃ" -salutation = { gender = "m", issue = 343, expands = "Mr., borrowed" } +salutation = { gender = "m", issue = 343, expands = "Mr., borrowed", after_given = true } [[word]] text = "মি" -salutation = { gender = "m", issue = 343, same_as = "মিঃ", expands = "the period spelling মি." } +salutation = { gender = "m", issue = 343, same_as = "মিঃ", expands = "the period spelling মি.", after_given = true } [[word]] text = "মিসেস" -salutation = { gender = "f", issue = 343, expands = "Mrs., borrowed" } +salutation = { gender = "f", issue = 343, expands = "Mrs., borrowed", after_given = true } [[word]] text = "মোঃ" -salutation = { gender = "m", issue = 343, expands = "Md., Mohammad", note = "a name PREFIX, not a title semantically; TITLES is the functional home because `given` must stay the name the person is addressed by (decisions/indic-honorifics)" } +salutation = { gender = "m", issue = 343, expands = "Md., Mohammad", note = "a name PREFIX, not a title semantically; TITLES is the functional home because `given` must stay the name the person is addressed by (decisions/indic-honorifics)", excluded = { after_given = "abbreviates Mohammad, a given name, and is read as one where it leads (#853)" } } +given = { wikidata = 37, as_of = 2026-10-24 } [[word]] text = "মো" -salutation = { gender = "m", issue = 343, same_as = "মোঃ", expands = "the period spelling মো." } +salutation = { gender = "m", issue = 343, same_as = "মোঃ", expands = "the period spelling মো.", excluded = { after_given = "abbreviates Mohammad, a given name, as মোঃ does (#853)" } } [[word]] text = "মোসাঃ" -salutation = { gender = "f", issue = 343, expands = "Mst., Mosammat, the women's counterpart of মোঃ" } +salutation = { gender = "f", issue = 343, expands = "Mst., Mosammat, the women's counterpart of মোঃ", after_given = true } [[word]] text = "মোছাঃ" -salutation = { gender = "f", issue = 343, same_as = "মোসাঃ", expands = "the ছ spelling" } +salutation = { gender = "f", issue = 343, same_as = "মোসাঃ", expands = "the ছ spelling", after_given = true } [[word]] text = "মোছা" -salutation = { gender = "f", issue = 343, same_as = "মোছাঃ", expands = "the period spelling মোছা." } +salutation = { gender = "f", issue = 343, same_as = "মোছাঃ", expands = "the period spelling মোছা.", after_given = true } [[word]] text = "মোসা" -salutation = { gender = "f", issue = 343, same_as = "মোসাঃ", expands = "the period spelling মোসা." } +salutation = { gender = "f", issue = 343, same_as = "মোসাঃ", expands = "the period spelling মোসা.", after_given = true } [[word]] text = "শ্রী" -salutation = { gender = "m", issue = 343, expands = "Shri, Mr." } +salutation = { gender = "m", issue = 343, expands = "Shri, Mr.", excluded = { after_given = "also an element of given names (Sri, Shri), so where it ends the given part it may be one" } } [[word]] text = "শ্রীমতী" -salutation = { gender = "f", issue = 343, expands = "Shrimati, Mrs." } +salutation = { gender = "f", issue = 343, expands = "Shrimati, Mrs.", after_given = true } [[word]] text = "আলহাজ্ব" diff --git a/nameparser/vocab/bs.toml b/nameparser/vocab/bs.toml index 3ec5eb8e..ce12b09e 100644 --- a/nameparser/vocab/bs.toml +++ b/nameparser/vocab/bs.toml @@ -2,15 +2,15 @@ schema = 1 [[word]] text = "gospodin" -salutation = { issue = 606, gender = "m" } +salutation = { issue = 606, gender = "m", after_given = true } [[word]] text = "gospođa" -salutation = { issue = 606, gender = "f" } +salutation = { issue = 606, gender = "f", after_given = true } [[word]] text = "gospođica" -salutation = { issue = 606, gender = "f" } +salutation = { issue = 606, gender = "f", after_given = true } [[word]] text = "mlađa" diff --git a/nameparser/vocab/cs.toml b/nameparser/vocab/cs.toml index 57abcea5..05900b56 100644 --- a/nameparser/vocab/cs.toml +++ b/nameparser/vocab/cs.toml @@ -6,7 +6,8 @@ excluded = { salutation = { reason = "Czech and Polish Mr., and the Chinese surn [[word]] text = "paní" -salutation = { issue = 606, gender = "f", note = "keeps its accent, so the name Pani (~76,000 surnames and ~65,000 given names, India and Indonesia) is untouched, and the unaccented pani is not shipped; the lexicon folds no accent (decisions/salutation-titles)" } +salutation = { issue = 606, gender = "f", note = "keeps its accent, so the name Pani (~76,000 surnames and ~65,000 given names, India and Indonesia) is untouched, and the unaccented pani is not shipped; the lexicon folds no accent (decisions/salutation-titles)", excluded = { after_given = "spelled PANI, a surname (Census 2010), and middle names are drawn from surnames" } } +family = { us_census_2010 = 403, as_of = 2026-10-24, note = "counted as PANI, the Census writing no diacritics" } [[word]] text = "rodina" @@ -14,7 +15,7 @@ excluded = { salutation = { reason = "Czech 'family', and a Russian surname, oft [[word]] text = "slečna" -salutation = { issue = 606, gender = "f" } +salutation = { issue = 606, gender = "f", after_given = true } [[word]] text = "ml" diff --git a/nameparser/vocab/da.toml b/nameparser/vocab/da.toml index 8c225aed..b6000e72 100644 --- a/nameparser/vocab/da.toml +++ b/nameparser/vocab/da.toml @@ -2,7 +2,7 @@ schema = 1 [[word]] text = "familie" -salutation = { issue = 606, expands = "the family addressed, 'Familie Hansen'" } +salutation = { issue = 606, expands = "the family addressed, 'Familie Hansen'", after_given = true } [[word]] text = "fru" @@ -10,11 +10,11 @@ salutation = { gender = "f" } [[word]] text = "frøken" -salutation = { gender = "f" } +salutation = { gender = "f", after_given = true } [[word]] text = "frk" -salutation = { gender = "f", same_as = "frøken" } +salutation = { gender = "f", same_as = "frøken", after_given = true } [[word]] text = "herr" diff --git a/nameparser/vocab/de.toml b/nameparser/vocab/de.toml index f5be4bd8..2b197702 100644 --- a/nameparser/vocab/de.toml +++ b/nameparser/vocab/de.toml @@ -2,31 +2,35 @@ schema = 1 [[word]] text = "familie" -salutation = { issue = 606, expands = "the family addressed, 'Familie Hansen'" } +salutation = { issue = 606, expands = "the family addressed, 'Familie Hansen'", after_given = true } [[word]] text = "frau" -salutation = { gender = "f" } +salutation = { gender = "f", after_given = true } +family = { wikidata = 42, as_of = 2026-10-24, note = "kept for after_given (decisions/H8)" } [[word]] text = "fräulein" -salutation = { gender = "f" } +salutation = { gender = "f", after_given = true } [[word]] text = "herr" -salutation = { gender = "m" } +salutation = { gender = "m", excluded = { after_given = "a surname (Census 2010 HERR), and middle names are drawn from surnames" } } +given = { wikidata = 3, as_of = 2026-10-24 } +family = { us_census_2010 = 11980, wikidata = 135, as_of = 2026-10-24 } [[word]] text = "herrn" -salutation = { gender = "m", same_as = "herr", expands = "Herr, inflected (an Herrn Müller)" } +salutation = { gender = "m", same_as = "herr", expands = "Herr, inflected (an Herrn Müller)", after_given = true } [[word]] text = "hr" -salutation = { gender = "m", same_as = "herr" } +salutation = { gender = "m", same_as = "herr", excluded = { after_given = "ending a given part in US data, more likely two initials run together than the abbreviation of Herr: one of three such fec-individuals names has a woman's given name (2026-10-24)" } } [[word]] text = "herren" -salutation = { gender = "m", expands = "the plural of Herr" } +salutation = { gender = "m", expands = "the plural of Herr", excluded = { after_given = "a surname (Census 2010 HERREN), and middle names are drawn from surnames" } } +family = { us_census_2010 = 4466, wikidata = 37, as_of = 2026-10-24 } [[word]] text = "dipl.-ing" diff --git a/nameparser/vocab/el.toml b/nameparser/vocab/el.toml index df8d47d7..2804af38 100644 --- a/nameparser/vocab/el.toml +++ b/nameparser/vocab/el.toml @@ -6,11 +6,11 @@ excluded = { salutation = { reason = "κ. (Mr.) has the initial-and-surname shap [[word]] text = "κα" -salutation = { gender = "f", issue = 269, expands = "κυρία, Mrs." } +salutation = { gender = "f", issue = 269, expands = "κυρία, Mrs.", after_given = true } [[word]] text = "κος" -salutation = { gender = "m", issue = 269, expands = "κύριος, Mr." } +salutation = { gender = "m", issue = 269, expands = "κύριος, Mr.", after_given = true } [[word]] text = "δρ" diff --git a/nameparser/vocab/en.toml b/nameparser/vocab/en.toml index d9829856..055f25b4 100644 --- a/nameparser/vocab/en.toml +++ b/nameparser/vocab/en.toml @@ -6,43 +6,51 @@ excluded = { title = { reason = "William, abbreviated, and no title of any kind: [[word]] text = "goodman" -salutation = { gender = "m", expands = "the archaic Mr., 'Goodman Brown'" } +salutation = { gender = "m", expands = "the archaic Mr., 'Goodman Brown'", excluded = { after_given = "a common surname (Benny Goodman), and middle names are drawn from surnames" } } +given = { us_ssa = 22, wikidata = 11, as_of = 2026-10-24 } +family = { us_census_2010 = 73136, wikidata = 743, as_of = 2026-10-24 } [[word]] text = "goodwife" -salutation = { gender = "f", expands = "the archaic Mrs., 'Goodwife Proctor'" } +salutation = { gender = "f", expands = "the archaic Mrs., 'Goodwife Proctor'", after_given = true } [[word]] text = "madam" -salutation = { gender = "f" } +salutation = { gender = "f", after_given = true } [[word]] text = "madame" -salutation = { gender = "f" } +salutation = { gender = "f", after_given = true } [[word]] text = "master" -salutation = { addresses = "given", gender = "m", expands = "the style for a boy, 'Master Harry'" } +salutation = { addresses = "given", gender = "m", expands = "the style for a boy, 'Master Harry'", excluded = { after_given = "a surname and a given name, borne as a middle name either way" } } +given = { us_ssa = 476, wikidata = 2, as_of = 2026-10-24 } +family = { us_census_2010 = 2735, wikidata = 61, as_of = 2026-10-24 } [[word]] text = "miss" -salutation = { gender = "f" } +salutation = { gender = "f", after_given = true } +family = { us_census_2010 = 164, wikidata = 4, as_of = 2026-10-24, note = "kept for after_given (decisions/H8): a middle name of this spelling is the accepted cost" } [[word]] text = "misses" -salutation = { gender = "f", expands = "the plural of miss, 'the Misses Smith'" } +salutation = { gender = "f", expands = "the plural of miss, 'the Misses Smith'", after_given = true } [[word]] text = "mr" -salutation = { gender = "m" } +salutation = { gender = "m", after_given = true } +given = { us_ssa = 5, as_of = 2026-10-24, note = "kept for after_given (decisions/H8)" } [[word]] text = "mister" -salutation = { gender = "m", same_as = "mr" } +salutation = { gender = "m", same_as = "mr", excluded = { after_given = "a given name and a surname, borne as a middle name either way" } } +given = { us_ssa = 746, as_of = 2026-10-24 } +family = { us_census_2010 = 1211, wikidata = 2, as_of = 2026-10-24 } [[word]] text = "mrs" -salutation = { gender = "f" } +salutation = { gender = "f", after_given = true } [[word]] text = "ms" @@ -52,7 +60,7 @@ note = "NOT may_be_name, though #296's audit table put it there for the leading- [[word]] text = "mx" -salutation = { gender = "n" } +salutation = { gender = "n", after_given = true } [[word]] text = "doyen" diff --git a/nameparser/vocab/es.toml b/nameparser/vocab/es.toml index 62c4c332..ea5c96c4 100644 --- a/nameparser/vocab/es.toml +++ b/nameparser/vocab/es.toml @@ -2,7 +2,8 @@ schema = 1 [[word]] text = "señor" -salutation = { gender = "m" } +salutation = { gender = "m", excluded = { after_given = "spelled SENOR, a surname (Census 2010), and middle names are drawn from surnames" } } +family = { us_census_2010 = 282, wikidata = 2, as_of = 2026-10-24, note = "the Census count is of SENOR, which writes no diacritics" } [[word]] text = "sr" @@ -10,35 +11,39 @@ salutation = { gender = "m", same_as = "señor" } [[word]] text = "señora" -salutation = { gender = "f" } +salutation = { gender = "f", excluded = { after_given = "a given name (US SSA, as Senora), borne as a middle name" } } +given = { us_ssa = 1065, as_of = 2026-10-24, note = "counted as SENORA, the SSA writing no diacritics" } [[word]] text = "sra" -salutation = { gender = "f", same_as = "señora" } +salutation = { gender = "f", same_as = "señora", after_given = true } +given = { us_ssa = 27, as_of = 2026-10-24 } +family = { us_census_2010 = 149, as_of = 2026-10-24, note = "kept for after_given (decisions/H8)" } [[word]] text = "señores" -salutation = { expands = "the plural of señor" } +salutation = { expands = "the plural of señor", after_given = true } [[word]] text = "sres" -salutation = { same_as = "señores" } +salutation = { same_as = "señores", after_given = true } [[word]] text = "señorita" -salutation = { gender = "f" } +salutation = { gender = "f", after_given = true } +given = { us_ssa = 6, as_of = 2026-10-24, note = "counted as SENORITA; kept for after_given (decisions/H8)" } [[word]] text = "srta" -salutation = { gender = "f", same_as = "señorita" } +salutation = { gender = "f", same_as = "señorita", after_given = true } [[word]] text = "señoritas" -salutation = { gender = "f" } +salutation = { gender = "f", after_given = true } [[word]] text = "srtas" -salutation = { gender = "f", same_as = "señoritas" } +salutation = { gender = "f", same_as = "señoritas", after_given = true } [[word]] text = "dra" diff --git a/nameparser/vocab/et.toml b/nameparser/vocab/et.toml index f85d7901..027a7b35 100644 --- a/nameparser/vocab/et.toml +++ b/nameparser/vocab/et.toml @@ -2,15 +2,16 @@ schema = 1 [[word]] text = "härra" -salutation = { issue = 606, gender = "m", note = "keeps its accent, so the surname Harra (~1,100) is untouched; the lexicon folds no accent (decisions/salutation-titles)" } +salutation = { issue = 606, gender = "m", note = "keeps its accent, so the surname Harra (~1,100) is untouched; the lexicon folds no accent (decisions/salutation-titles)", excluded = { after_given = "spelled HARRA, a surname (Census 2010), and middle names are drawn from surnames" } } +family = { us_census_2010 = 237, as_of = 2026-10-24, note = "counted as HARRA, the Census writing no diacritics" } [[word]] text = "preili" -salutation = { issue = 606, gender = "f" } +salutation = { issue = 606, gender = "f", after_given = true } [[word]] text = "proua" -salutation = { issue = 606, gender = "f" } +salutation = { issue = 606, gender = "f", after_given = true } [[word]] text = "krahv" diff --git a/nameparser/vocab/fi.toml b/nameparser/vocab/fi.toml index fbad874b..47428161 100644 --- a/nameparser/vocab/fi.toml +++ b/nameparser/vocab/fi.toml @@ -2,15 +2,16 @@ schema = 1 [[word]] text = "herra" -salutation = { issue = 606, gender = "m", note = "borne: ~3,000 surnames (Costa Rica, Dominican Republic) and ~670 given-name uses (Indonesia, Philippines), forebears.io, worldwide, rounded, 2026-10-04; shipped under the graf class (decisions/salutation-titles), so 'Herra Wijaya' reads title Herra" } +salutation = { issue = 606, gender = "m", note = "borne: ~3,000 surnames (Costa Rica, Dominican Republic) and ~670 given-name uses (Indonesia, Philippines), forebears.io, worldwide, rounded, 2026-10-04; shipped under the graf class (decisions/salutation-titles), so 'Herra Wijaya' reads title Herra", excluded = { after_given = "a surname (Census 2010 HERRA), and middle names are drawn from surnames" } } +family = { us_census_2010 = 638, as_of = 2026-10-24 } [[word]] text = "neiti" -salutation = { issue = 606, gender = "f" } +salutation = { issue = 606, gender = "f", after_given = true } [[word]] text = "rouva" -salutation = { issue = 606, gender = "f" } +salutation = { issue = 606, gender = "f", after_given = true } [[word]] text = "kreivi" diff --git a/nameparser/vocab/fil.toml b/nameparser/vocab/fil.toml index 352445ce..3d0592ae 100644 --- a/nameparser/vocab/fil.toml +++ b/nameparser/vocab/fil.toml @@ -2,15 +2,15 @@ schema = 1 [[word]] text = "binibini" -salutation = { issue = 606, gender = "f", note = "borne: ~500 surnames, in Philippine lists writing Surname, Given with a comma, forebears.io, worldwide, rounded, 2026-10-04; shipped under the graf class (decisions/salutation-titles)" } +salutation = { issue = 606, gender = "f", note = "borne: ~500 surnames, in Philippine lists writing Surname, Given with a comma, forebears.io, worldwide, rounded, 2026-10-04; shipped under the graf class (decisions/salutation-titles)", after_given = true } [[word]] text = "ginang" -salutation = { issue = 606, gender = "f", note = "borne: ~500 surnames, in Philippine lists writing Surname, Given with a comma, forebears.io, worldwide, rounded, 2026-10-04; shipped under the graf class (decisions/salutation-titles)" } +salutation = { issue = 606, gender = "f", note = "borne: ~500 surnames, in Philippine lists writing Surname, Given with a comma, forebears.io, worldwide, rounded, 2026-10-04; shipped under the graf class (decisions/salutation-titles)", after_given = true } [[word]] text = "ginoo" -salutation = { issue = 606, gender = "m", note = "borne: ~500 surnames, in Philippine lists writing Surname, Given with a comma, forebears.io, worldwide, rounded, 2026-10-04; shipped under the graf class (decisions/salutation-titles)" } +salutation = { issue = 606, gender = "m", note = "borne: ~500 surnames, in Philippine lists writing Surname, Given with a comma, forebears.io, worldwide, rounded, 2026-10-04; shipped under the graf class (decisions/salutation-titles)", after_given = true } [[word]] text = "dela" diff --git a/nameparser/vocab/fr.toml b/nameparser/vocab/fr.toml index 6fa25fc6..55d375ae 100644 --- a/nameparser/vocab/fr.toml +++ b/nameparser/vocab/fr.toml @@ -2,23 +2,23 @@ schema = 1 [[word]] text = "madame" -salutation = { gender = "f" } +salutation = { gender = "f", after_given = true } [[word]] text = "mme" -salutation = { gender = "f", same_as = "madame" } +salutation = { gender = "f", same_as = "madame", after_given = true } [[word]] text = "mademoiselle" -salutation = { gender = "f" } +salutation = { gender = "f", after_given = true } [[word]] text = "mlle" -salutation = { gender = "f", same_as = "mademoiselle" } +salutation = { gender = "f", same_as = "mademoiselle", after_given = true } [[word]] text = "monsieur" -salutation = { gender = "m" } +salutation = { gender = "m", after_given = true } [[word]] text = "cheikh" diff --git a/nameparser/vocab/he.toml b/nameparser/vocab/he.toml index b3e424ef..68f96b7c 100644 --- a/nameparser/vocab/he.toml +++ b/nameparser/vocab/he.toml @@ -2,19 +2,19 @@ schema = 1 [[word]] text = "גב'" -salutation = { gender = "f", issue = 269, expands = "Mrs., the abbreviation", note = "the ASCII-quote spelling: the quote has no whitespace before it, so it opens no delimited span, and with no opener before it a dangling straight apostrophe is literal text (WORD_INTERNAL_DELIMITERS in _pipeline/_extract.py; probed 2026-07-17)" } +salutation = { gender = "f", issue = 269, expands = "Mrs., the abbreviation", note = "the ASCII-quote spelling: the quote has no whitespace before it, so it opens no delimited span, and with no opener before it a dangling straight apostrophe is literal text (WORD_INTERNAL_DELIMITERS in _pipeline/_extract.py; probed 2026-07-17)", after_given = true } [[word]] text = "גב׳" -salutation = { gender = "f", issue = 269, same_as = "גב'", expands = "the U+05F3 geresh spelling" } +salutation = { gender = "f", issue = 269, same_as = "גב'", expands = "the U+05F3 geresh spelling", after_given = true } [[word]] text = "גברת" -salutation = { gender = "f", issue = 269, expands = "Mrs., Ms., the full form" } +salutation = { gender = "f", issue = 269, expands = "Mrs., Ms., the full form", after_given = true } [[word]] text = "מר" -salutation = { gender = "m", issue = 269, expands = "Mr." } +salutation = { gender = "m", issue = 269, expands = "Mr.", after_given = true } [[word]] text = "ד\"ר" diff --git a/nameparser/vocab/hi.toml b/nameparser/vocab/hi.toml index c8d2cfb8..0a2146a8 100644 --- a/nameparser/vocab/hi.toml +++ b/nameparser/vocab/hi.toml @@ -14,11 +14,12 @@ excluded = { salutation = { reason = "Kumari is a given name and a family name i [[word]] text = "जनाब" -salutation = { gender = "m", issue = 344, expands = "Janab, Mr." } +salutation = { gender = "m", issue = 344, expands = "Janab, Mr.", after_given = true } [[word]] text = "बाबू" -salutation = { gender = "m", issue = 344, expands = "Babu", note = "leading in Hindi; the Bengali বাবু is trailing, a suffix word in bn.toml, a different codepoint sequence" } +salutation = { gender = "m", issue = 344, expands = "Babu", note = "leading in Hindi; the Bengali বাবু is trailing, a suffix word in bn.toml, a different codepoint sequence", excluded = { after_given = "romanized Babu, an element of compound given names in India (Ram Babu) and a surname" } } +family = { us_census_2010 = 1496, wikidata = 37, as_of = 2026-10-24, note = "counted as BABU, the romanization; nam_dict ranks Babu a given name in India and Sri Lanka, band 3" } [[word]] text = "बेगम" @@ -26,23 +27,24 @@ excluded = { salutation = { reason = "Begum is a trailing name element far more [[word]] text = "श्री" -salutation = { gender = "m", issue = 269, expands = "Shri, Mr.", note = "no Latin twin: Latin sri and shri are given names (their exclusions in hi.toml)" } +salutation = { gender = "m", issue = 269, expands = "Shri, Mr.", note = "no Latin twin: Latin sri and shri are given names (their exclusions in hi.toml)", excluded = { after_given = "also an element of given names (Sri, Shri), so where it ends the given part it may be one" } } +given = { wikidata = 1, as_of = 2026-10-24 } [[word]] text = "श्रीमती" -salutation = { gender = "f", issue = 269, expands = "Shrimati, Mrs." } +salutation = { gender = "f", issue = 269, expands = "Shrimati, Mrs.", after_given = true } [[word]] text = "श्रीमान" -salutation = { gender = "m", issue = 344, expands = "Shriman, Mr." } +salutation = { gender = "m", issue = 344, expands = "Shriman, Mr.", after_given = true } [[word]] text = "श्रीयुत" -salutation = { gender = "m", issue = 344, expands = "Shriyut, Mr." } +salutation = { gender = "m", issue = 344, expands = "Shriyut, Mr.", after_given = true } [[word]] text = "सुश्री" -salutation = { gender = "f", issue = 344, expands = "Sushri, Ms." } +salutation = { gender = "f", issue = 344, expands = "Sushri, Ms.", after_given = true } [[word]] text = "ठाकुर" diff --git a/nameparser/vocab/hr.toml b/nameparser/vocab/hr.toml index 592ca8fa..33fbae34 100644 --- a/nameparser/vocab/hr.toml +++ b/nameparser/vocab/hr.toml @@ -2,15 +2,15 @@ schema = 1 [[word]] text = "gospodin" -salutation = { issue = 606, gender = "m" } +salutation = { issue = 606, gender = "m", after_given = true } [[word]] text = "gospođa" -salutation = { issue = 606, gender = "f" } +salutation = { issue = 606, gender = "f", after_given = true } [[word]] text = "gospođica" -salutation = { issue = 606, gender = "f" } +salutation = { issue = 606, gender = "f", after_given = true } [[word]] text = "ml" diff --git a/nameparser/vocab/id.toml b/nameparser/vocab/id.toml index 7eb3e7f7..16db04a5 100644 --- a/nameparser/vocab/id.toml +++ b/nameparser/vocab/id.toml @@ -2,4 +2,4 @@ schema = 1 [[word]] text = "bapak" -salutation = { issue = 662, gender = "m", note = "under 100 US Census bearers and none found on Wikidata (tools/name_usage.py, 2026-10-17)" } +salutation = { issue = 662, gender = "m", note = "under 100 US Census bearers and none found on Wikidata (tools/name_usage.py, 2026-10-17)", after_given = true } diff --git a/nameparser/vocab/is.toml b/nameparser/vocab/is.toml index 4cb45798..b629a252 100644 --- a/nameparser/vocab/is.toml +++ b/nameparser/vocab/is.toml @@ -2,7 +2,7 @@ schema = 1 [[word]] text = "frú" -salutation = { issue = 606, gender = "f" } +salutation = { issue = 606, gender = "f", excluded = { after_given = "spelled FRU, a Cameroonian surname, as fru is (sv.toml), and middle names are drawn from surnames" } } [[word]] text = "herra" @@ -10,7 +10,7 @@ salutation = { issue = 606, gender = "m" } [[word]] text = "ungfrú" -salutation = { issue = 606, gender = "f" } +salutation = { issue = 606, gender = "f", after_given = true } [[word]] text = "greifi" diff --git a/nameparser/vocab/it.toml b/nameparser/vocab/it.toml index 8474ff41..664aedd0 100644 --- a/nameparser/vocab/it.toml +++ b/nameparser/vocab/it.toml @@ -6,27 +6,30 @@ excluded = { salutation = { reason = "Sig is borne as a given name (US SSA 50+ m [[word]] text = "signora" -salutation = { gender = "f" } +salutation = { gender = "f", after_given = true } +given = { us_ssa = 90, as_of = 2026-10-24, note = "kept for after_given (decisions/H8)" } [[word]] text = "sig.ra" -salutation = { gender = "f", same_as = "signora", cased = "Sig.ra", issue = 840 } +salutation = { gender = "f", same_as = "signora", cased = "Sig.ra", issue = 840, after_given = true } [[word]] text = "signore" -salutation = { gender = "m" } +salutation = { gender = "m", excluded = { after_given = "a surname (Census 2010 SIGNORE), and middle names are drawn from surnames" } } +family = { us_census_2010 = 895, wikidata = 17, as_of = 2026-10-24 } [[word]] text = "signor" -salutation = { gender = "m", same_as = "signore" } +salutation = { gender = "m", same_as = "signore", excluded = { after_given = "a surname (Census 2010 SIGNOR), and middle names are drawn from surnames" } } +family = { us_census_2010 = 999, wikidata = 4, as_of = 2026-10-24 } [[word]] text = "signorina" -salutation = { gender = "f" } +salutation = { gender = "f", after_given = true } [[word]] text = "sig.na" -salutation = { gender = "f", same_as = "signorina", cased = "Sig.na", issue = 840 } +salutation = { gender = "f", same_as = "signorina", cased = "Sig.na", issue = 840, after_given = true } [[word]] text = "ch.ma" diff --git a/nameparser/vocab/ka.toml b/nameparser/vocab/ka.toml index 9faf5bb2..5896c7e3 100644 --- a/nameparser/vocab/ka.toml +++ b/nameparser/vocab/ka.toml @@ -2,11 +2,11 @@ schema = 1 [[word]] text = "batoni" -salutation = { issue = 606, gender = "m", note = "borne: ~1,100 surnames (the painter Pompeo Batoni) and ~300 given-name uses (Tanzania), forebears.io, worldwide, rounded, 2026-10-04; shipped under the graf class (decisions/salutation-titles)" } +salutation = { issue = 606, gender = "m", note = "borne: ~1,100 surnames (the painter Pompeo Batoni) and ~300 given-name uses (Tanzania), forebears.io, worldwide, rounded, 2026-10-04; shipped under the graf class (decisions/salutation-titles)", after_given = true } [[word]] text = "kalbatoni" -salutation = { issue = 606, gender = "f" } +salutation = { issue = 606, gender = "f", after_given = true } [[word]] text = "grafi" diff --git a/nameparser/vocab/lt.toml b/nameparser/vocab/lt.toml index 07009fa5..80c97ba5 100644 --- a/nameparser/vocab/lt.toml +++ b/nameparser/vocab/lt.toml @@ -2,11 +2,11 @@ schema = 1 [[word]] text = "ponas" -salutation = { issue = 606, gender = "m", note = "borne: ~4,000 surnames (Iran), forebears.io, worldwide, rounded, 2026-10-04; shipped under the graf class (decisions/salutation-titles)" } +salutation = { issue = 606, gender = "m", note = "borne: ~4,000 surnames (Iran), forebears.io, worldwide, rounded, 2026-10-04; shipped under the graf class (decisions/salutation-titles)", after_given = true } [[word]] text = "ponia" -salutation = { issue = 606, gender = "f", note = "borne: ~560 surnames, forebears.io, worldwide, rounded, 2026-10-04; shipped under the graf class (decisions/salutation-titles)" } +salutation = { issue = 606, gender = "f", note = "borne: ~560 surnames, forebears.io, worldwide, rounded, 2026-10-04; shipped under the graf class (decisions/salutation-titles)", after_given = true } [[word]] text = "grafas" diff --git a/nameparser/vocab/mr.toml b/nameparser/vocab/mr.toml index 57de2cb9..aa2871d7 100644 --- a/nameparser/vocab/mr.toml +++ b/nameparser/vocab/mr.toml @@ -6,11 +6,11 @@ salutation = { gender = "m", issue = 269, expands = "Shri, Mr." } [[word]] text = "श्रीमती" -salutation = { gender = "f", issue = 269, expands = "Shrimati, Mrs." } +salutation = { gender = "f", issue = 269, expands = "Shrimati, Mrs.", after_given = true } [[word]] text = "सौ" -salutation = { gender = "f", issue = 344, expands = "Sau., Mrs." } +salutation = { gender = "f", issue = 344, expands = "Sau., Mrs.", after_given = true } [[word]] text = "महात्मा" diff --git a/nameparser/vocab/ms.toml b/nameparser/vocab/ms.toml index 80706209..e4b55d62 100644 --- a/nameparser/vocab/ms.toml +++ b/nameparser/vocab/ms.toml @@ -6,7 +6,7 @@ excluded = { salutation = { reason = "Malay Miss: unverified, so not shipped", i [[word]] text = "encik" -salutation = { issue = 606, gender = "m" } +salutation = { issue = 606, gender = "m", after_given = true } [[word]] text = "puan" diff --git a/nameparser/vocab/nb.toml b/nameparser/vocab/nb.toml index 7fabbc37..b4919387 100644 --- a/nameparser/vocab/nb.toml +++ b/nameparser/vocab/nb.toml @@ -2,7 +2,7 @@ schema = 1 [[word]] text = "familie" -salutation = { issue = 606, expands = "the family addressed, 'Familie Hansen'" } +salutation = { issue = 606, expands = "the family addressed, 'Familie Hansen'", after_given = true } [[word]] text = "fru" @@ -10,11 +10,11 @@ salutation = { gender = "f" } [[word]] text = "frøken" -salutation = { gender = "f" } +salutation = { gender = "f", after_given = true } [[word]] text = "frk" -salutation = { gender = "f", same_as = "frøken" } +salutation = { gender = "f", same_as = "frøken", after_given = true } [[word]] text = "herr" diff --git a/nameparser/vocab/nl.toml b/nameparser/vocab/nl.toml index fe1cffa0..b8f94215 100644 --- a/nameparser/vocab/nl.toml +++ b/nameparser/vocab/nl.toml @@ -2,31 +2,32 @@ schema = 1 [[word]] text = "dhr" -salutation = { gender = "m", expands = "de heer, Mr." } +salutation = { gender = "m", expands = "de heer, Mr.", after_given = true } [[word]] text = "heren" -salutation = { gender = "m", expands = "the plural of heer, 'de heren'" } +salutation = { gender = "m", expands = "the plural of heer, 'de heren'", after_given = true } +family = { us_census_2010 = 123, wikidata = 1, as_of = 2026-10-24, note = "kept for after_given (decisions/H8)" } [[word]] text = "juffrouw" -salutation = { issue = 606, gender = "f" } +salutation = { issue = 606, gender = "f", after_given = true } [[word]] text = "meneer" -salutation = { issue = 606, gender = "m" } +salutation = { issue = 606, gender = "m", after_given = true } [[word]] text = "mevrouw" -salutation = { gender = "f" } +salutation = { gender = "f", after_given = true } [[word]] text = "mevr" -salutation = { gender = "f", same_as = "mevrouw" } +salutation = { gender = "f", same_as = "mevrouw", after_given = true } [[word]] text = "mevrouwe" -salutation = { gender = "f", same_as = "mevrouw", expands = "the archaic spelling of mevrouw" } +salutation = { gender = "f", same_as = "mevrouw", expands = "the archaic spelling of mevrouw", after_given = true } [[word]] text = "baron" diff --git a/nameparser/vocab/pt.toml b/nameparser/vocab/pt.toml index f4fc016d..16b5674e 100644 --- a/nameparser/vocab/pt.toml +++ b/nameparser/vocab/pt.toml @@ -2,7 +2,7 @@ schema = 1 [[word]] text = "senhor" -salutation = { gender = "m" } +salutation = { gender = "m", after_given = true } [[word]] text = "sr" @@ -10,15 +10,15 @@ salutation = { gender = "m", same_as = "senhor" } [[word]] text = "senhora" -salutation = { gender = "f" } +salutation = { gender = "f", after_given = true } [[word]] text = "sra" -salutation = { gender = "f", same_as = "senhora" } +salutation = { gender = "f", same_as = "senhora", after_given = true } [[word]] text = "senhorita" -salutation = { gender = "f" } +salutation = { gender = "f", after_given = true } [[word]] text = "dra" diff --git a/nameparser/vocab/ro.toml b/nameparser/vocab/ro.toml index b8e1c16d..aba14f30 100644 --- a/nameparser/vocab/ro.toml +++ b/nameparser/vocab/ro.toml @@ -2,16 +2,16 @@ schema = 1 [[word]] text = "doamnă" -salutation = { issue = 606, gender = "f" } +salutation = { issue = 606, gender = "f", after_given = true } [[word]] text = "doamna" -salutation = { issue = 606, gender = "f", same_as = "doamnă", expands = "doamnă without the breve" } +salutation = { issue = 606, gender = "f", same_as = "doamnă", expands = "doamnă without the breve", after_given = true } [[word]] text = "domnișoara" -salutation = { issue = 606, gender = "f" } +salutation = { issue = 606, gender = "f", after_given = true } [[word]] text = "domnul" -salutation = { issue = 606, gender = "m" } +salutation = { issue = 606, gender = "m", after_given = true } diff --git a/nameparser/vocab/ru.toml b/nameparser/vocab/ru.toml index 7ef0ac4d..fb7390ec 100644 --- a/nameparser/vocab/ru.toml +++ b/nameparser/vocab/ru.toml @@ -2,11 +2,11 @@ schema = 1 [[word]] text = "г-жа" -salutation = { gender = "f", issue = 269, expands = "госпожа, Mrs." } +salutation = { gender = "f", issue = 269, expands = "госпожа, Mrs.", after_given = true } [[word]] text = "г-н" -salutation = { gender = "m", issue = 269, expands = "господин, Mr." } +salutation = { gender = "m", issue = 269, expands = "господин, Mr.", after_given = true } [[word]] text = "акад" diff --git a/nameparser/vocab/sr.toml b/nameparser/vocab/sr.toml index a626f3ff..5c478449 100644 --- a/nameparser/vocab/sr.toml +++ b/nameparser/vocab/sr.toml @@ -2,15 +2,16 @@ schema = 1 [[word]] text = "gospodin" -salutation = { issue = 606, gender = "m" } +salutation = { issue = 606, gender = "m", after_given = true } +given = { note = "nam_dict ranks Gospodin a Bulgarian given name, band 5 (2026-10-24); kept for after_given (decisions/H8): a Bulgarian middle name is a patronymic, Gospodinov" } [[word]] text = "gospođa" -salutation = { issue = 606, gender = "f" } +salutation = { issue = 606, gender = "f", after_given = true } [[word]] text = "gospođica" -salutation = { issue = 606, gender = "f" } +salutation = { issue = 606, gender = "f", after_given = true } [[word]] text = "mlađa" diff --git a/nameparser/vocab/sv.toml b/nameparser/vocab/sv.toml index e2b12522..5a02d82e 100644 --- a/nameparser/vocab/sv.toml +++ b/nameparser/vocab/sv.toml @@ -2,15 +2,16 @@ schema = 1 [[word]] text = "fru" -salutation = { gender = "f", note = "Fru is also a Cameroonian surname (~10,800, often written surname-first), so 'Fru Anna' reads title Fru; a collision older than #606, recorded in decisions/salutation-titles rather than argued" } +salutation = { gender = "f", note = "Fru is also a Cameroonian surname (~10,800, often written surname-first), so 'Fru Anna' reads title Fru; a collision older than #606, recorded in decisions/salutation-titles rather than argued", excluded = { after_given = "a Cameroonian surname (~10,800, the note above), and middle names are drawn from surnames" } } +family = { us_census_2010 = 132, as_of = 2026-10-24 } [[word]] text = "fröken" -salutation = { issue = 606, gender = "f" } +salutation = { issue = 606, gender = "f", after_given = true } [[word]] text = "frk" -salutation = { gender = "f", same_as = "fröken" } +salutation = { gender = "f", same_as = "fröken", after_given = true } [[word]] text = "herr" diff --git a/nameparser/vocab/sw.toml b/nameparser/vocab/sw.toml index 5eb0500b..4a461517 100644 --- a/nameparser/vocab/sw.toml +++ b/nameparser/vocab/sw.toml @@ -2,4 +2,6 @@ schema = 1 [[word]] text = "bwana" -salutation = { gender = "m", expands = "Mr., sir" } +salutation = { gender = "m", expands = "Mr., sir", after_given = true } +given = { us_ssa = 5, as_of = 2026-10-24 } +family = { wikidata = 1, as_of = 2026-10-24, note = "kept for after_given (decisions/H8)" } diff --git a/nameparser/vocab/uk.toml b/nameparser/vocab/uk.toml index adad8735..682b61b4 100644 --- a/nameparser/vocab/uk.toml +++ b/nameparser/vocab/uk.toml @@ -2,11 +2,12 @@ schema = 1 [[word]] text = "пан" -salutation = { gender = "m", issue = 269, expands = "Mr." } +salutation = { gender = "m", issue = 269, expands = "Mr.", excluded = { after_given = "a surname in Cyrillic (Wikidata), and middle names are drawn from surnames" } } +family = { wikidata = 743, as_of = 2026-10-24 } [[word]] text = "пані" -salutation = { gender = "f", issue = 269, expands = "Mrs." } +salutation = { gender = "f", issue = 269, expands = "Mrs.", after_given = true } [[word]] text = "акад" diff --git a/tests/v2/cases.py b/tests/v2/cases.py index c9268c49..a52dd23e 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -11220,6 +11220,69 @@ def _check_cjk_shape_purity(self) -> None: notes="the boundary of the row above, kept: #432 reads the " "period as the abbreviation mark it is, so 'V.' is a " "middle initial -- 'Smith, John V.' plus a title"), + Case("bare_salutation_ending_a_family_comma_given_part", + "Doe, Jane Mrs", + {"title": "Mrs", "given": "Jane", "family": "Doe"}, + classification="fix(#835)", + notes="H8's reading (#835): the family comma has fixed the surname, so " + "the one reading a bare salutation there competes with " + "is a middle name, and H5's chain takes it as it takes " + "'Doe, Jane Mrs.'"), + Case("bare_salutation_ending_a_one_case_given_part", + "DOE, JOHN Q MR", + {"title": "MR", "given": "JOHN", "middle": "Q", "family": "DOE"}, + classification="fix(#835)", + notes="the FEC's spelling of the shape (#834), the period " + "left off"), + Case("bare_salutation_ahead_of_a_suffix_after_a_family_comma", + "Doe, John Mr Jr", + {"title": "Mr", "given": "John", "family": "Doe", + "suffix": "Jr"}, classification="fix(#835)", + notes="the walk's candidates leave the suffix out, so the " + "salutation is the last of them, as 'Smith, John Prof. " + "Jr.' reaches 'Prof.'"), + Case("bare_salutation_before_a_bound_given_name_join", + "Doe, abdul Mrs", + {"title": "Mrs", "given": "abdul", "family": "Doe"}, + classification="fix(#835)", + notes="rules/P5's join declines a word H8 reads as the title, " + "as it declines 'Berg, abdul Prof.': the group-side " + "predictor admits the same bare salutations the walk " + "does, or the join read given 'abdul Mrs'"), + Case("bare_salutations_around_a_suffix_before_a_bound_given_join", + "Doe, abdul Mr Jr Mrs", + {"title": "Mr Mrs", "given": "abdul", "family": "Doe", + "suffix": "Jr"}, classification="fix(#835)", + notes="title, suffix, title behind a bound given name: the " + "join's decline reads the part through tail_reading, " + "whose second chain pass takes 'Mr' only if the " + "salutation flag reaches it, as 'Doe, abdul Mr. Jr Mrs.' " + "reads"), + Case("bare_salutation_behind_a_trailing_particle", + "Doe, Jane van Mrs", + {"title": "Mrs", "given": "Jane", "family": "van Doe"}, + ambiguities=("particle-or-given",), + classification="fix(#835)", + notes="P6 attaches the particle as though the salutation were " + "absent ('Doe, Jane van' reads the same), and reports " + "the fork it decides, as 'Doe, Jane van Mrs.' does"), + Case("bare_salutation_borne_as_a_name_stays_a_middle_name", + "Doe, John Goodman", + {"given": "John", "middle": "Goodman", "family": "Doe"}, + notes="rules/H8's Except: goodman is a salutation the " + "vocabulary refuses after_given, a surname (the counts " + "are on en.toml's entry), since a middle name may be a " + "surname"), + Case("bare_office_word_ending_a_family_comma_given_part", + "Doe, John King", + {"given": "John", "middle": "King", "family": "Doe"}, + notes="rules/H8's Except: only a salutation qualifies, and an " + "office or rank word is borne as a middle name"), + Case("bare_salutation_without_a_comma_is_a_name_word", + "Jane Doe Mrs", + {"given": "Jane", "middle": "Doe", "family": "Mrs"}, + notes="rules/H5's bare title word: without the comma the " + "surname is not fixed, so the word stays a name word"), Case("title_word_trailing_ahead_of_a_reserved_acronym", "John Smith Prof. MA", {"title": "Prof.", "given": "John", "family": "Smith", diff --git a/tests/v2/pipeline/test_post_rules.py b/tests/v2/pipeline/test_post_rules.py index 784f04a2..6de9994e 100644 --- a/tests/v2/pipeline/test_post_rules.py +++ b/tests/v2/pipeline/test_post_rules.py @@ -26,6 +26,7 @@ _LEX = Lexicon( titles=frozenset({"mr", "sir", "dr"}), given_name_titles=frozenset({"sir"}), + after_given_titles=frozenset({"mr"}), particles=frozenset({"de", "der", "ibn", "la", "van", "vd"}), particles_ambiguous=frozenset({"la", "van"}), # `씨` is a ONE-LETTER suffix word, which starts no credential run diff --git a/tests/v2/rules_budget.json b/tests/v2/rules_budget.json index b43cea9c..204e8262 100644 --- a/tests/v2/rules_budget.json +++ b/tests/v2/rules_budget.json @@ -21,9 +21,10 @@ "H2": {"words": 316, "then_words": 166, "because_sentences": 1, "except": 1, "precedence": 1, "accepted": 3, "accepted_words": 203}, "H3": {"words": 142, "then_words": 72, "because_sentences": 2, "except": 0, "precedence": 0, "accepted": 3, "accepted_words": 113}, "H4": {"words": 390, "then_words": 183, "because_sentences": 3, "except": 1, "precedence": 1, "accepted": 1, "accepted_words": 127}, - "H5": {"words": 699, "then_words": 289, "because_sentences": 2, "except": 0, "precedence": 4, "accepted": 1, "accepted_words": 117}, + "H5": {"words": 757, "then_words": 289, "because_sentences": 2, "except": 0, "precedence": 5, "accepted": 1, "accepted_words": 117}, "H6": {"words": 280, "then_words": 79, "because_sentences": 3, "except": 3, "precedence": 0, "accepted": 2, "accepted_words": 100}, "H7": {"words": 343, "then_words": 274, "because_sentences": 2, "except": 0, "precedence": 0, "accepted": 1, "accepted_words": 54}, + "H8": {"words": 431, "then_words": 68, "because_sentences": 3, "except": 3, "precedence": 3, "accepted": 3, "accepted_words": 128}, "M1": {"words": 231, "then_words": 120, "because_sentences": 2, "except": 1, "precedence": 2, "accepted": 0, "accepted_words": 0}, "M2": {"words": 701, "then_words": 422, "because_sentences": 1, "except": 4, "precedence": 1, "accepted": 6, "accepted_words": 209}, "M3": {"words": 234, "then_words": 84, "because_sentences": 2, "except": 2, "precedence": 2, "accepted": 1, "accepted_words": 80}, @@ -41,7 +42,7 @@ "P2": {"words": 270, "then_words": 234, "because_sentences": 1, "except": 0, "precedence": 0, "accepted": 3, "accepted_words": 167}, "P3": {"words": 639, "then_words": 598, "because_sentences": 1, "except": 0, "precedence": 0, "accepted": 0, "accepted_words": 0}, "P4": {"words": 190, "then_words": 82, "because_sentences": 1, "except": 1, "precedence": 0, "accepted": 0, "accepted_words": 0}, - "P5": {"words": 662, "then_words": 399, "because_sentences": 1, "except": 2, "precedence": 3, "accepted": 3, "accepted_words": 198}, + "P5": {"words": 668, "then_words": 399, "because_sentences": 1, "except": 2, "precedence": 3, "accepted": 3, "accepted_words": 198}, "P6": {"words": 974, "then_words": 644, "because_sentences": 2, "except": 2, "precedence": 4, "accepted": 6, "accepted_words": 459}, "P7": {"words": 138, "then_words": 64, "because_sentences": 3, "except": 0, "precedence": 0, "accepted": 1, "accepted_words": 28}, "P8": {"words": 515, "then_words": 450, "because_sentences": 2, "except": 0, "precedence": 0, "accepted": 2, "accepted_words": 309}, diff --git a/tests/v2/test_cases.py b/tests/v2/test_cases.py index 0a5e64f3..1bdfa654 100644 --- a/tests/v2/test_cases.py +++ b/tests/v2/test_cases.py @@ -228,9 +228,11 @@ def test_the_field_sweep_sees_the_claims_it_checks() -> None: # which carries its own policy, under one; 1396 -> 1399 the same # day: the review's 'Jane Doe MA R.T.' row, one claim under each; # 1399 -> 1400 the same day: #754's 'Beethoven Ludwig Maria van' - # row, which carries its own policy, one particle-or-given claim. - assert claims == 1400, ( - f"the field sweep checks {claims} claims, recorded as 1400 on " + # row, which carries its own policy, one particle-or-given claim; + # 1400 -> 1403 the same day: #835's particle row 'Doe, Jane van + # Mrs', one particle-or-given claim under each order. + assert claims == 1403, ( + f"the field sweep checks {claims} claims, recorded as 1403 on " f"2026-10-24") diff --git a/tests/v2/test_config_shim.py b/tests/v2/test_config_shim.py index b0645996..9f4cebc3 100644 --- a/tests/v2/test_config_shim.py +++ b/tests/v2/test_config_shim.py @@ -552,6 +552,24 @@ def test_snapshot_removing_a_conjunction_turns_the_marker_off() -> None: assert name.initials() == "j. e. s." +def test_snapshot_removing_a_title_turns_its_after_given_reading_off() -> None: + # The deciding case for after_given_titles (#835, rules/H8), which has + # no v1 manager of its own: the snapshot intersects AFTER_GIVEN_TITLES + # with the v1 title set, so the marker drops the deleted title. The + # parse check below is held by classify too, which tags a marker + # entry only where it is a title, so the intersection is kept for + # `_snapshot() == Lexicon.default()` legibility, as + # conjunctions_ambiguous' is; the set assertion is what pins it + c = Constants() + assert HumanName("Doe, Jane Mrs", constants=c).title == "Mrs" # baseline + c.titles.remove("mrs") + lexicon, _, _ = c._snapshot() + assert "mrs" not in lexicon.after_given_titles + name = HumanName("Doe, Jane Mrs", constants=c) + assert (name.title, name.first, name.middle, name.last) == ( + "", "Jane", "Mrs", "Doe") + + def test_snapshot_field_translation() -> None: c = Constants() lexicon, policy, defaults = c._snapshot() diff --git a/tests/v2/test_lexicon.py b/tests/v2/test_lexicon.py index 9b2cbf21..cbbfc494 100644 --- a/tests/v2/test_lexicon.py +++ b/tests/v2/test_lexicon.py @@ -978,6 +978,66 @@ def test_removing_a_honorific_tail_is_the_peels_off_switch() -> None: assert (on.family, on.suffix) == ("田中", "さん") +def test_after_given_titles_is_a_vocab_field() -> None: + # #835: add, remove and | reach it, and it is normalized like every + # other field, so the period-marked entry matches the bare word + lex = Lexicon.empty().add(after_given_titles={"Mrs.", "mr"}) + assert lex.after_given_titles == frozenset({"mrs", "mr"}) + respelled = (Lexicon.default().remove(after_given_titles={"mrs"}) + .add(after_given_titles={"Mrs."})) + assert Parser(lexicon=respelled).parse("Doe, Jane Mrs").title == "Mrs" + assert lex.remove(after_given_titles={"mr"}).after_given_titles == \ + frozenset({"mrs"}) + merged = (Lexicon(after_given_titles=frozenset({"mrs"})) + | Lexicon(after_given_titles=frozenset({"miss"}))) + assert merged.after_given_titles == frozenset({"mrs", "miss"}) + with pytest.raises(TypeError): + Lexicon(after_given_titles="mrs") # type: ignore[arg-type] + edited = Lexicon.default().add(after_given_titles={"zzq"}) + assert pickle.loads(pickle.dumps(edited)) == edited + assert "after_given_titles: +1" in repr(edited) + + +def test_removing_an_after_given_title_is_rule_h8s_off_switch() -> None: + # Dropping the marker leaves the bare salutation a middle name after + # a family comma, while the period-marked spelling still reads as + # the title by H5's shape, which never consults the marker + lex = Lexicon.default().remove(after_given_titles={"mrs"}) + p = Parser(lexicon=lex) + assert (p.parse("Doe, Jane Mrs").title, + p.parse("Doe, Jane Mrs").middle) == ("", "Mrs") + assert p.parse("Doe, Jane Mrs.").title == "Mrs." + # the baseline the removal is against + assert Parser().parse("Doe, Jane Mrs").title == "Mrs" + + +def test_every_shipped_after_given_title_is_reachable() -> None: + # a property of the set, not its contents: each entry must reach + # rules/H8 at the slot it is listed for, so a member that another + # reading claims first (a suffix dual, a particle, a bound given + # name) shows up here as silently inert + p = Parser() + unreached = [w for w in sorted(p.lexicon.after_given_titles) + if p.parse(f"Doe, Jane {w.title()}").title != w.title()] + assert not unreached, unreached + + +@pytest.mark.parametrize(("lexicon", "text", "word"), [ + # in the marker, removed from the titles + (Lexicon.default().remove(titles={"mrs"}), "Doe, Jane Mrs", "Mrs"), + # added to the marker only + (Lexicon.default().add(after_given_titles={"zzq"}), "Doe, Jane Zzq", + "Zzq"), +], ids=["removed-from-titles", "marker-only"]) +def test_an_after_given_title_that_is_no_title_is_never_read( + lexicon: Lexicon, text: str, word: str) -> None: + # the orphan contract docs/customize.rst states: an entry is read + # only for a word that is a title, so an orphan is inert rather than + # an error (classify tags the marker only where the word is a title) + name = Parser(lexicon=lexicon).parse(text) + assert (name.title, name.middle) == ("", word) + + @pytest.mark.parametrize("field", _PHRASE_FIELDS) def test_a_shipped_phrase_entry_is_stored_as_written(field: str) -> None: """The half of a phrase entry's storage rule config's own diff --git a/tests/v2/test_properties.py b/tests/v2/test_properties.py index 4b2ad983..b8372875 100644 --- a/tests/v2/test_properties.py +++ b/tests/v2/test_properties.py @@ -3521,3 +3521,51 @@ def test_the_one_case_grid_sees_the_letter_in_front( _vocab.is_trailing_numeral_suffix(text, preceding, None)) assert (len(_one_case_disagreements()) == _ONE_CASE_LETTER_IN_FRONT_UNGUARDED) + + +#: H8's grid (#835): family-comma heads, given parts and tails +#: holding a bare salutation, each paired with its period-marked twin. +#: 704 texts, each parsed with its twin under each of the three +#: orders: 2,112 pairs, about 0.5s on py3.11 (measured 2026-10-24). +_SALUTATION_HEADS = ("Doe, ", "van Doe, ", ", ", "Doe Jr., ") +_SALUTATION_GIVENS = ("Jane", "abdul", "Jane van", "Jane van der", + "abdul van", "J.", "Jane Q", "St", "Freiherr", + "jane", "JANE") +_SALUTATION_TAILS = ("Mrs", "Mr Mrs", "Mrs Jr", "Mr Jr Mrs", "Mrs Jr Mr", + "Mrs PhD Mr", "Mrs, PhD", "Mrs V", "Mr VI", "Mrs MA", + "Mrs MA Mr", "Mrs Jr MA Mr", "Mrs Sr.", "Mrs X.Y.Z.", + "Mrs III", "Mrs, Jr., PhD") + + +def test_a_bare_salutation_reads_as_its_period_marked_twin() -> None: + """H8's first Then bullet (#835), the bare spelling reading as the + period-marked one reads. After a family comma a listed salutation + written bare chains into H5's trailing title run as the + period-marked spelling does, so the two spellings read alike, fields (stops aside) and ambiguity kinds, in + every comma shape here and under every order. The grid crosses a + bound given name and a trailing particle, whose group-side + predictors read the part through `tail_reading`, with title, + suffix, title tails, which reach that function's second chain pass. + + Mutation-checked 2026-10-24: dropping the `salutations` hand-off on + `tail_reading`'s second chain call fails this on 192 of its 2,112 + pairs, and the suite had no other test of that call.""" + def dotted(text: str) -> str: + return re.sub(r"\b(Mrs|Mr)\b", r"\1.", text) + + def fields(name: ParsedName) -> tuple[object, ...]: + stops_aside = sorted((k, v.replace(".", "")) + for k, v in name.as_dict().items()) + return (tuple(stops_aside), + tuple(sorted(a.kind for a in name.ambiguities))) + + failures = [] + for order in (GIVEN_FIRST, FAMILY_FIRST, FAMILY_FIRST_GIVEN_LAST): + parser = Parser(policy=Policy(name_order=order)) + for head, given_part, tail in itertools.product( + _SALUTATION_HEADS, _SALUTATION_GIVENS, _SALUTATION_TAILS): + text = f"{head}{given_part} {tail}" + if fields(parser.parse(text)) != fields(parser.parse(dotted(text))): + failures.append(f"{order!r}: {text!r}") + assert not failures, (f"{len(failures)} bare salutation(s) read unlike " + f"the period-marked twin:\n" + "\n".join(failures[:10])) diff --git a/tests/v2/test_vocab.py b/tests/v2/test_vocab.py index d8fb6706..88996758 100644 --- a/tests/v2/test_vocab.py +++ b/tests/v2/test_vocab.py @@ -87,6 +87,7 @@ def test_the_compiled_sets_are_the_default_lexicon(shipped: dict[str, str]) -> N bound_given_names=c["BOUND_GIVEN_NAMES"], maiden_markers=c["MAIDEN_MARKERS"], surnames=lex.surnames, honorific_tails=c["GLUED_HONORIFICS"], + after_given_titles=c["AFTER_GIVEN_TITLES"], capitalization_exceptions=tuple(c["CAPITALIZATION_EXCEPTIONS"].items()), ) assert rebuilt == lex @@ -204,6 +205,11 @@ def _w(body: str) -> str: ("may-be-name-on-a-word", {"en.toml": _w('[[word]]\ntext = "jr"\ngeneration = { may_be_name = true }')}, "may_be_name marks an acronym"), ("may-be-initial-on-a-word", {"und.toml": _w('[[word]]\ntext = "and"\nconjunction = { may_be_initial = true }')}, "CASED single letter"), ("may-be-initial-caseless", {"ar.toml": _w('[[word]]\ntext = "و"\nconjunction = { may_be_initial = true }')}, "CASED single letter"), + ("after-given-on-an-office", {"en.toml": _w('[[word]]\ntext = "king"\nnobility = { after_given = true }')}, "after_given is allowed on a salutation reading only"), + ("after-given-on-a-generic-title", {"und.toml": _w('[[word]]\ntext = "zzq"\ntitle = { after_given = true }')}, "after_given is allowed on a salutation reading only"), + # herr's shape: refused in de.toml, flagged in da.toml + ("excluded-after-given-in-another-file", {"de.toml": _w('[[word]]\ntext = "herr"\nsalutation = { excluded = { after_given = "a surname" } }'), "da.toml": _w('[[word]]\ntext = "herr"\nsalutation = { after_given = true }')}, "after_given = true is excluded.*a surname"), + ("after-given-not-a-flag", {"en.toml": _w('[[word]]\ntext = "mr"\nsalutation = { after_given = "yes" }')}, "after_given must be true or false"), ("particle-without-may-be-given", {"und.toml": _w('[[word]]\ntext = "van"\nparticle = {}')}, "states may_be_given"), # every particle kind, not only the generic one (#749): unwritten, # the flag would read false and put the word in the never-given half @@ -789,6 +795,7 @@ def _sets_fed(kind: str, props: str = "{}", text: str = "zzq") -> set[str]: "may_be_given": ("particle", "{ may_be_given = false }", "zzq"), "may_be_initial": ("conjunction", "{ may_be_initial = true }", "q"), "cased": ("title", '{ cased = "ZzQ" }', "zzq"), + "after_given": ("salutation", "{ after_given = true }", "zzq"), } #: The behavioral properties the build refuses to leave unwritten. _REQUIRED = {"may_be_given"} diff --git a/tools/differential/corpus_rules.jsonl b/tools/differential/corpus_rules.jsonl index 22930246..b2181f9f 100644 --- a/tools/differential/corpus_rules.jsonl +++ b/tools/differential/corpus_rules.jsonl @@ -5,6 +5,8 @@ "'Smitty' Sir John" "(" ", Ana García López" +", Jane" +", Jane Mrs" ".," "1. née Smith" "A.B. Vajpayee" @@ -68,6 +70,7 @@ "DOE, JANE MA VD PHD" "DOE, JANE PHD VD" "DOE, JANE PHD VD MA" +"DOE, JOHN Q MR" "DR. NÉE JUAN y LOPEZ" "DUT.RA MARIA SILVA" "Davis Royce, Ed" @@ -75,25 +78,42 @@ "De La Cruz, M.J. K.L." "De La Cruz, M.J. PhD" "Del Toro" +"Doe Jane Mrs" "Doe nee Smith Jr. Prof., Jane" "Doe nee Smith, Jane" +"Doe, Jane Dr" +"Doe, Jane Fru" +"Doe, Jane Miss" +"Doe, Jane Mr Mrs" +"Doe, Jane Mr.." +"Doe, Jane Mrs" +"Doe, Jane Mrs, PhD" +"Doe, Jane Ms" "Doe, Jane PhD MEng" "Doe, Jane nee Smith" +"Doe, Jane van Mrs" "Doe, Jane, and Jr." "Doe, John DO" "Doe, John DO Ed" "Doe, John Do" +"Doe, John Goodman" +"Doe, John Jr Mr" +"Doe, John King" "Doe, John MA" "Doe, John MA JD" "Doe, John MA Ma" "Doe, John MA Smith" "Doe, John Ma" +"Doe, John Master" +"Doe, John Mr Jr" "Doe, John Msc.Ed." "Doe, John X.Y.Z." "Doe, John van DO" "Doe, John van Ma" "Doe, MA PhD" +"Doe, Mrs" "Doe, PhD Jones" +"Doe, abdul Mrs" "Dr Jr" "Dr King Jr" "Dr Smith V" @@ -209,6 +229,7 @@ "Jane Doe (nee Smith) MA" "Jane Doe Jr. nee Smith" "Jane Doe MA R.T." +"Jane Doe Mrs" "Jane Doe PhD nee Smith" "Jane Doe nee King." "Jane Doe nee MA" @@ -548,6 +569,7 @@ "Smith, Esq. John" "Smith, Insp. John" "Smith, Jan van Prof." +"Smith, Jane Mrs" "Smith, John" "Smith, John PhD I." "Smith, John PhD Jones" @@ -725,6 +747,7 @@ "st.john smith" "tran lac" "van Berg Jan de" +"van Doe, Jane Mrs" "van Gogh" "van Gogh Ana María" "van Gogh, Sir" @@ -750,6 +773,7 @@ "جورج بوش الابن" "سلمان، محمد" "राम शर्मा साहब मोहन" +"शर्मा, राम बाबू" "প্রকৌশলী. Sen" "สมชาย ใจดี พ.บ." "김. 민준" diff --git a/tools/differential/snapshot.jsonl b/tools/differential/snapshot.jsonl index 630c39c7..4cb8808f 100644 --- a/tools/differential/snapshot.jsonl +++ b/tools/differential/snapshot.jsonl @@ -10,6 +10,8 @@ {"name": "(김민준.) John Smith", "given": "김", "middle": "민준. John", "family": "Smith", "initials": "김. 민. J. S."} {"name": ","} {"name": ", Ana García López", "given": "Ana", "middle": "García López", "initials": "A. G. L."} +{"name": ", Jane", "given": "Jane", "initials": "J."} +{"name": ", Jane Mrs", "title": "Mrs", "family": "Jane", "initials": "J."} {"name": ", John", "given": "John", "initials": "J."} {"name": ".,"} {"name": "1 & 2, 3 4 5, Mr.", "title": "Mr.", "given": "3", "middle": "4 5", "family": "1 & 2", "initials": "3. 4. 5. 1. 2."} @@ -548,6 +550,7 @@ {"name": "DOE, JANE PHD VD", "given": "JANE", "family": "VD DOE", "suffix": "PHD", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. D.", "capitalized": "Jane vd Doe PhD"} {"name": "DOE, JANE PHD VD MA", "given": "JANE", "family": "DOE", "suffix": "PHD VD MA", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. D.", "capitalized": "Jane Doe PhD VD MA"} {"name": "DOE, JOHN MA", "given": "JOHN", "family": "DOE", "suffix": "MA", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. D.", "capitalized": "John Doe MA"} +{"name": "DOE, JOHN Q MR", "title": "MR", "given": "JOHN", "middle": "Q", "family": "DOE", "initials": "J. Q. D.", "capitalized": "Mr John Q Doe"} {"name": "DOE, MARY JO MA", "given": "MARY", "middle": "JO", "family": "DOE", "suffix": "MA", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "M. J. D.", "capitalized": "Mary Jo Doe MA"} {"name": "DR DOE", "title": "DR", "family": "DOE", "initials": "D.", "capitalized": "Dr Doe"} {"name": "DR. NÉE JUAN y LOPEZ", "title": "DR.", "given": "NÉE", "family": "JUAN y LOPEZ", "initials": "N. J. L.", "capitalized_force": "Dr. Née Juan y Lopez"} @@ -606,6 +609,7 @@ {"name": "Dl Dr. Daniel Ionescu Dragoș Ioan Ardelean Jr.", "given": "Dl", "middle": "Dr. Daniel Ionescu Dragoș Ioan", "family": "Ardelean", "suffix": "Jr.", "initials": "D. D. D. I. D. I. A."} {"name": "Do Quang Minh", "given": "Do", "middle": "Quang", "family": "Minh", "ambiguities": ["PARTICLE_OR_GIVEN"], "initials": "D. Q. M."} {"name": "Doctor, Jane E.", "given": "Jane", "middle": "E.", "family": "Doctor", "initials": "J. E. D."} +{"name": "Doe Jane Mrs", "given": "Doe", "middle": "Jane", "family": "Mrs", "initials": "D. J. M."} {"name": "Doe nee Smith Jr. Prof., Jane", "given": "Jane", "family": "Doe Prof.", "suffix": "Jr.", "maiden": "Smith", "initials": "J. D. P."} {"name": "Doe nee Smith, Jane", "given": "Jane", "family": "Doe", "maiden": "Smith", "initials": "J. D."} {"name": "Doe, Dr. John", "title": "Dr.", "given": "John", "family": "Doe", "initials": "J. D."} @@ -625,6 +629,14 @@ {"name": "Doe, J. MA", "given": "J.", "family": "Doe", "suffix": "MA", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. D."} {"name": "Doe, J. ba", "given": "J.", "family": "Doe", "suffix": "ba", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. D.", "capitalized_force": "J. Doe BA"} {"name": "Doe, J. nee MA ba", "given": "J.", "middle": "nee", "family": "Doe", "suffix": "MA ba", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. n. D.", "capitalized_force": "J. Nee Doe MA BA"} +{"name": "Doe, Jane Dr", "given": "Jane", "middle": "Dr", "family": "Doe", "initials": "J. D. D."} +{"name": "Doe, Jane Fru", "given": "Jane", "middle": "Fru", "family": "Doe", "initials": "J. F. D."} +{"name": "Doe, Jane Miss", "title": "Miss", "given": "Jane", "family": "Doe", "initials": "J. D."} +{"name": "Doe, Jane Mr Mrs", "title": "Mr Mrs", "given": "Jane", "family": "Doe", "initials": "J. D."} +{"name": "Doe, Jane Mr..", "title": "Mr..", "given": "Jane", "family": "Doe", "initials": "J. D."} +{"name": "Doe, Jane Mrs", "title": "Mrs", "given": "Jane", "family": "Doe", "initials": "J. D."} +{"name": "Doe, Jane Mrs, PhD", "title": "Mrs", "given": "Jane", "family": "Doe", "suffix": "PhD", "initials": "J. D."} +{"name": "Doe, Jane Ms", "given": "Jane", "family": "Doe", "suffix": "Ms", "initials": "J. D.", "capitalized_force": "Jane Doe MS"} {"name": "Doe, Jane PhD MEng", "given": "Jane", "family": "Doe", "suffix": "PhD MEng", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. D.", "capitalized_force": "Jane Doe PhD MENG"} {"name": "Doe, Jane Q. nee Smith MA", "given": "Jane", "middle": "Q. nee Smith", "family": "Doe", "suffix": "MA", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. Q. n. S. D.", "capitalized_force": "Jane Q. Nee Smith Doe MA"} {"name": "Doe, Jane nee Puig i Soler", "given": "Jane", "middle": "nee Puig i Soler", "family": "Doe", "initials": "J. n. P. S. D.", "capitalized_force": "Jane Nee Puig i Soler Doe"} @@ -638,6 +650,7 @@ {"name": "Doe, Jane nee Smith PhD MEng", "given": "Jane", "middle": "nee Smith", "family": "Doe", "suffix": "PhD MEng", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. n. S. D.", "capitalized_force": "Jane Nee Smith Doe PhD MENG"} {"name": "Doe, Jane nee Smith do", "given": "Jane", "middle": "nee Smith", "family": "do Doe", "ambiguities": ["PARTICLE_OR_GIVEN"], "initials": "J. n. S. D.", "capitalized_force": "Jane Nee Smith do Doe"} {"name": "Doe, Jane nee Smith ma", "given": "Jane", "middle": "nee Smith", "family": "Doe", "suffix": "ma", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. n. S. D.", "capitalized_force": "Jane Nee Smith Doe MA"} +{"name": "Doe, Jane van Mrs", "title": "Mrs", "given": "Jane", "family": "van Doe", "ambiguities": ["PARTICLE_OR_GIVEN"], "initials": "J. D."} {"name": "Doe, Jane, and Jr.", "given": "Jane", "family": "Doe", "suffix": "and Jr.", "ambiguities": ["COMMA_STRUCTURE"], "initials": "J. D."} {"name": "Doe, John", "given": "John", "family": "Doe", "initials": "J. D."} {"name": "Doe, John A.", "given": "John", "middle": "A.", "family": "Doe", "initials": "J. A. D."} @@ -652,7 +665,10 @@ {"name": "Doe, John DO Ed", "given": "John", "middle": "DO Ed", "family": "Doe", "initials": "J. E. D.", "capitalized_force": "John do Ed Doe"} {"name": "Doe, John Do", "given": "John", "family": "Do Doe", "ambiguities": ["PARTICLE_OR_GIVEN"], "initials": "J. D.", "capitalized_force": "John do Doe"} {"name": "Doe, John Ed", "given": "John", "middle": "Ed", "family": "Doe", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. E. D."} +{"name": "Doe, John Goodman", "given": "John", "middle": "Goodman", "family": "Doe", "initials": "J. G. D."} {"name": "Doe, John III", "given": "John", "family": "Doe", "suffix": "III", "initials": "J. D."} +{"name": "Doe, John Jr Mr", "title": "Mr", "given": "John", "family": "Doe", "suffix": "Jr", "initials": "J. D."} +{"name": "Doe, John King", "given": "John", "middle": "King", "family": "Doe", "initials": "J. K. D."} {"name": "Doe, John MA", "given": "John", "family": "Doe", "suffix": "MA", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. D."} {"name": "Doe, John MA JD", "given": "John", "family": "Doe", "suffix": "MA JD", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. D."} {"name": "Doe, John MA Jr", "given": "John", "family": "Doe", "suffix": "MA Jr", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. D."} @@ -662,6 +678,8 @@ {"name": "Doe, John MA do", "given": "John", "middle": "MA", "family": "do Doe", "ambiguities": ["PARTICLE_OR_GIVEN"], "initials": "J. M. D.", "capitalized_force": "John Ma do Doe"} {"name": "Doe, John MA y", "given": "John", "middle": "MA y", "family": "Doe", "initials": "J. M. D.", "capitalized_force": "John Ma y Doe"} {"name": "Doe, John Ma", "given": "John", "middle": "Ma", "family": "Doe", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. M. D."} +{"name": "Doe, John Master", "given": "John", "middle": "Master", "family": "Doe", "initials": "J. M. D."} +{"name": "Doe, John Mr Jr", "title": "Mr", "given": "John", "family": "Doe", "suffix": "Jr", "initials": "J. D."} {"name": "Doe, John Msc.Ed.", "given": "John", "family": "Doe", "suffix": "Msc.Ed.", "initials": "J. D."} {"name": "Doe, John PhD MA", "given": "John", "family": "Doe", "suffix": "PhD MA", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. D."} {"name": "Doe, John Q. MA", "given": "John", "middle": "Q.", "family": "Doe", "suffix": "MA", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. Q. D."} @@ -684,10 +702,12 @@ {"name": "Doe, Mary - Kate, RN", "given": "Mary", "middle": "- Kate", "family": "Doe", "suffix": "RN", "initials": "M. -. K. D."} {"name": "Doe, Mary Jo Ma", "given": "Mary", "middle": "Jo Ma", "family": "Doe", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "M. J. M. D."} {"name": "Doe, Mr. MA PhD", "title": "Mr.", "family": "Doe", "suffix": "MA PhD", "initials": "D."} +{"name": "Doe, Mrs", "title": "Mrs", "family": "Doe", "initials": "D."} {"name": "Doe, PhD Jones", "family": "Doe", "suffix": "PhD Jones", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "D."} {"name": "Doe, Rev. John A. Jr.", "title": "Rev.", "given": "John", "middle": "A.", "family": "Doe", "suffix": "Jr.", "initials": "J. A. D."} {"name": "Doe, Rev. John A., V, Jr.", "title": "Rev.", "given": "John", "middle": "A.", "family": "Doe", "suffix": "V, Jr.", "initials": "J. A. D."} {"name": "Doe, Rev. John V, Jr.", "title": "Rev.", "given": "John", "middle": "V", "family": "Doe", "suffix": "Jr.", "initials": "J. V. D."} +{"name": "Doe, abdul Mrs", "title": "Mrs", "given": "abdul", "family": "Doe", "initials": "a. D.", "capitalized_force": "Mrs Abdul Doe"} {"name": "Doe,, Jr.", "family": "Doe", "suffix": "Jr.", "initials": "D."} {"name": "Doe-Ray, Dr. John P., CLU, CFP, LUTC", "title": "Dr.", "given": "John", "middle": "P.", "family": "Doe-Ray", "suffix": "CLU, CFP, LUTC", "ambiguities": ["COMMA_STRUCTURE"], "initials": "J. P. D.", "capitalized_force": "Dr. John P. Doe-Ray CLU, CFP, Lutc"} {"name": "Doe-Ray, Hon. Barrington P. Jr.", "title": "Hon.", "given": "Barrington", "middle": "P.", "family": "Doe-Ray", "suffix": "Jr.", "initials": "B. P. D."} @@ -1380,6 +1400,7 @@ {"name": "Jane Doe Jr. nee Smith", "given": "Jane", "family": "Doe", "suffix": "Jr. nee Smith", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. D.", "capitalized_force": "Jane Doe Jr. Nee Smith"} {"name": "Jane Doe Jr. nee Smith Ma", "given": "Jane", "family": "Doe", "suffix": "Jr. nee Smith Ma", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. D.", "capitalized_force": "Jane Doe Jr. Nee Smith MA"} {"name": "Jane Doe MA R.T.", "given": "Jane", "middle": "Doe MA", "family": "R.T.", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. D. M. R.", "capitalized_force": "Jane Doe Ma R.T."} +{"name": "Jane Doe Mrs", "given": "Jane", "middle": "Doe", "family": "Mrs", "initials": "J. D. M."} {"name": "Jane Doe PhD nee Smith", "given": "Jane", "family": "Doe", "suffix": "PhD nee Smith", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. D.", "capitalized_force": "Jane Doe PhD Nee Smith"} {"name": "Jane Doe geb. Smith MA", "given": "Jane", "family": "Doe", "suffix": "MA", "maiden": "Smith", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. D."} {"name": "Jane Doe nee King.", "given": "Jane", "family": "Doe", "maiden": "King.", "initials": "J. D."} @@ -2964,6 +2985,7 @@ {"name": "Smith, J. R.", "given": "J.", "middle": "R.", "family": "Smith", "initials": "J. R. S."} {"name": "Smith, J.R.", "given": "J.R.", "family": "Smith", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. S."} {"name": "Smith, Jan van Prof.", "title": "Prof.", "given": "Jan", "family": "van Smith", "ambiguities": ["PARTICLE_OR_GIVEN"], "initials": "J. S."} +{"name": "Smith, Jane Mrs", "title": "Mrs", "given": "Jane", "family": "Smith", "initials": "J. S."} {"name": "Smith, John", "given": "John", "family": "Smith", "initials": "J. S."} {"name": "Smith, John ,", "given": "John", "family": "Smith", "initials": "J. S."} {"name": "Smith, John E, III, Jr", "given": "John", "middle": "E", "family": "Smith", "suffix": "III, Jr", "initials": "J. E. S."} @@ -3673,6 +3695,7 @@ {"name": "u-van den u-Wold u-Prof. Dr. u-Ada Cornelia u-Eva Sophia isizukulwane, M.D. Ph.D.", "order": "FAMILY_FIRST", "given": "den u-Wold u-Prof. Dr. u-Ada Cornelia u-Eva Sophia isizukulwane", "family": "u-van", "suffix": "M.D. Ph.D.", "initials": "d. u. u. D. u. C. u. S. i. u.", "capitalized_force": "Den U-Wold U-Prof. Dr. U-Ada Cornelia U-Eva Sophia Isizukulwane U-van M.D. Ph.D."} {"name": "vai la", "given": "vai", "family": "la", "initials": "v. l.", "capitalized": "Vai La"} {"name": "van Berg Jan de", "given": "van", "middle": "Berg Jan", "family": "de", "ambiguities": ["PARTICLE_OR_GIVEN"], "initials": "v. B. J. d.", "capitalized_force": "Van Berg Jan De"} +{"name": "van Doe, Jane Mrs", "title": "Mrs", "given": "Jane", "family": "van Doe", "initials": "J. D."} {"name": "van Gogh", "given": "van", "family": "Gogh", "ambiguities": ["PARTICLE_OR_GIVEN"], "initials": "v. G.", "capitalized_force": "Van Gogh"} {"name": "van Gogh Ana María", "given": "van", "middle": "Gogh Ana", "family": "María", "ambiguities": ["PARTICLE_OR_GIVEN"], "initials": "v. G. A. M.", "capitalized_force": "Van Gogh Ana María"} {"name": "van Gogh, Sir", "title": "Sir", "family": "van Gogh", "initials": "G."} @@ -5133,6 +5156,7 @@ {"name": "शरद मल्होत्रा", "given": "शरद", "family": "मल्होत्रा", "initials": "श. म."} {"name": "शर्मा ललित", "order": "FAMILY_FIRST", "given": "ललित", "family": "शर्मा", "initials": "ल. श."} {"name": "शर्मा ललित मोहन", "order": "FAMILY_FIRST", "given": "ललित", "middle": "मोहन", "family": "शर्मा", "initials": "ल. म. श."} +{"name": "शर्मा, राम बाबू", "given": "राम", "middle": "बाबू", "family": "शर्मा", "initials": "र. ब. श."} {"name": "शर्मा, ललित", "given": "ललित", "family": "शर्मा", "initials": "ल. श."} {"name": "शर्मा, ललित कुमार एम॰डी॰", "given": "ललित", "middle": "कुमार एम॰डी॰", "family": "शर्मा", "initials": "ल. क. ए. श."} {"name": "शर्मा, ललित मोहन", "given": "ललित", "middle": "मोहन", "family": "शर्मा", "initials": "ल. म. श."} diff --git a/tools/vocab/build.py b/tools/vocab/build.py index f29847cb..173e706c 100644 --- a/tools/vocab/build.py +++ b/tools/vocab/build.py @@ -62,10 +62,12 @@ in check_pool, 6 first and 10 last; plus the data invariants that were asserts: may_be_initial only on a -cased single letter, and an elided particle in both apostrophes (#700). +cased single letter, an elided particle in both apostrophes (#700), and +after_given only on a salutation reading (#835). No post-nominal is both an acronym and a word, by checks 4 and 5; a final check in compile_pool is the backstop. The subset relations -(GIVEN_NAME_TITLES in TITLES, each marker set in its base) need no +(GIVEN_NAME_TITLES and AFTER_GIVEN_TITLES in TITLES, each marker set in +its base) need no check: a flag cannot exist without its reading. The one soft finding, a name reading on a word with no vocabulary reading and no exclusion, is a VocabWarning, which the shipped-data test turns into an error. @@ -208,7 +210,7 @@ def _alnum(text: str) -> str: #: Behavioral properties per category: each compiles into a field, and #: its allowed values are exactly what today's pipeline does. BEHAVIORAL: dict[str, frozenset[str]] = { - "title": frozenset({"addresses", "cased"}), + "title": frozenset({"addresses", "after_given", "cased"}), "suffix": frozenset({"acronym", "may_be_name", "attached", "cased"}), "particle": frozenset({"may_be_given", "cased"}), "conjunction": frozenset({"may_be_initial", "cased"}), @@ -263,7 +265,7 @@ def acronym(self) -> bool: return bool(self.props.get("acronym", self.kind == "credential")) def flag(self, prop: str) -> bool: - """A `may_be_*` flag: unwritten, false.""" + """A `may_be_*` flag or `after_given`: unwritten, false.""" return bool(self.props.get(prop, False)) @property @@ -723,7 +725,8 @@ def _check_reading(r: Reading, pool: Pool) -> None: if k not in allowed: raise _reserved_or_unknown(where, k, allowed) _check_descriptive(where, p) - for flag in ("acronym", "may_be_name", "may_be_given", "may_be_initial"): + for flag in ("acronym", "may_be_name", "may_be_given", "may_be_initial", + "after_given"): if flag in p and not isinstance(p[flag], bool): raise VocabError(f"{where}: {flag} must be true or false") if r.category == "title": @@ -733,6 +736,13 @@ def _check_reading(r: Reading, pool: Pool) -> None: f"for a later phase; phase 1 has 'family' and 'given'") if addresses not in ("family", "given"): raise VocabError(f"{where}: addresses must be 'family' or 'given'") + if r.flag("after_given") and r.kind != "salutation": + # a salutation is a form of address, where an office or a + # rank is borne as a middle name: the reason the given + # part's end may read one as a title (decisions/H8, #835) + raise VocabError(f"{where}: after_given is allowed on a " + f"salutation reading only, not on {r.kind!r} " + f"(rules/H8)") if r.category == "suffix": acronym = r.acronym if acronym and "." in r.text: @@ -968,7 +978,7 @@ def _behavior(r: Reading, prop: str) -> object: if prop == "attached": att = r.props.get("attached") return None if att is None else att.get("side") - if prop.startswith("may_be"): + if prop.startswith("may_be") or prop == "after_given": return r.flag(prop) return r.props.get(prop) @@ -978,6 +988,7 @@ def _behavior(r: Reading, prop: str) -> object: CONSTANTS = ( ("GIVEN_NAME_TITLES", "titles"), ("TITLES", "titles"), + ("AFTER_GIVEN_TITLES", "titles"), ("NON_GIVEN_NAME_PARTICLES", "particles"), ("PARTICLES", "particles"), ("CONJUNCTIONS", "conjunctions"), @@ -997,6 +1008,7 @@ class Compiled(TypedDict): order, typed as _compiled.py declares it.""" GIVEN_NAME_TITLES: frozenset[str] TITLES: frozenset[str] + AFTER_GIVEN_TITLES: frozenset[str] NON_GIVEN_NAME_PARTICLES: frozenset[str] PARTICLES: frozenset[str] CONJUNCTIONS: frozenset[str] @@ -1021,6 +1033,7 @@ def compile_pool(pool: Pool) -> Compiled: check_pool(pool) titles: set[str] = set() given_titles: set[str] = set() + after_given: set[str] = set() particles: set[str] = set() never_given: set[str] = set() conjunctions: set[str] = set() @@ -1040,6 +1053,8 @@ def compile_pool(pool: Pool) -> Compiled: titles.add(t) if r.addresses == "given": given_titles.add(t) + if r.flag("after_given"): + after_given.add(t) elif r.category == "suffix": if r.acronym: acronyms.add(t) @@ -1067,6 +1082,7 @@ def compile_pool(pool: Pool) -> Compiled: out = Compiled( GIVEN_NAME_TITLES=frozenset(given_titles), TITLES=frozenset(titles), + AFTER_GIVEN_TITLES=frozenset(after_given), NON_GIVEN_NAME_PARTICLES=frozenset(never_given), PARTICLES=frozenset(particles), CONJUNCTIONS=frozenset(conjunctions),