From e5c3712491c9e244de23a9c852b59607bf039d64 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sun, 25 Oct 2026 01:03:02 -0700 Subject: [PATCH 1/3] Derive the two name-clash markers from name readings (#752) Phase 2a. SUFFIX_ACRONYMS_AMBIGUOUS is now the words with an acronym suffix reading and a given or family reading, and the particles outside NON_GIVEN_NAME_PARTICLES the words with a particle reading and a given reading, wherever in the files the readings sit (build check 11). The 44 flagged words that lacked one get a name reading, 49 in all, with counts from tools/name_usage.py (2026-10-25); eight particles that no source counts carry a required note instead. ld's given count moves into its credential note, so it stays unmarked. The build refuses may_be_name and may_be_given = true, naming the reading to write, and still requires may_be_given = false on a particle whose word has no given reading. may_be_initial stays a property. No compiled set moves: _compiled.py rebuilds byte for byte, and the snapshot, every agreement record, the ctgov, fec and tse local snapshots and psc-individuals' agreement all match. Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 12 +- docs/design/decisions/vocabulary-model.md | 18 ++- nameparser/config/particles.py | 15 +- nameparser/config/suffixes.py | 4 +- nameparser/vocab/README.md | 37 +++-- nameparser/vocab/ar.toml | 10 +- nameparser/vocab/de.toml | 12 +- nameparser/vocab/en.toml | 106 ++++++------- nameparser/vocab/es.toml | 9 +- nameparser/vocab/fil.toml | 2 +- nameparser/vocab/fr.toml | 8 +- nameparser/vocab/ga.toml | 4 +- nameparser/vocab/gd.toml | 2 +- nameparser/vocab/he.toml | 5 +- nameparser/vocab/it.toml | 18 ++- nameparser/vocab/ms.toml | 6 +- nameparser/vocab/mul.toml | 2 +- nameparser/vocab/nl.toml | 27 ++-- nameparser/vocab/pl.toml | 3 +- nameparser/vocab/pt.toml | 5 +- nameparser/vocab/und.toml | 4 +- tests/v2/test_vocab.py | 172 ++++++++++++++++------ tools/vocab/build.py | 117 ++++++++++----- 23 files changed, 387 insertions(+), 211 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index e0f65aa8..d030417e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -192,14 +192,14 @@ The library has two layers: `nameparser/config/` (data) and `nameparser/parser.p ### Vocabulary (`nameparser/vocab/`) and the configuration layer (`nameparser/config/`) -**The default vocabulary is written per word, in TOML, and compiled.** `nameparser/vocab/.toml` holds one file per BCP 47 language tag, each `[[word]]` entry a `text` and its readings keyed by kind. What a file means, how an entry is built, the kinds and properties with the set each compiles into, exclusions, bearer counts and the steps for adding a word are in `nameparser/vocab/README.md`, the reference for anyone editing the TOML; `tests/v2/test_vocab.py` holds its kind, property and reserved-name tables to the names `build.py` accepts and the sets it compiles, so read it rather than restating it here. The parser never reads the TOML: `tools/vocab/build.py` compiles it into the committed `nameparser/config/_compiled.py`, frozenset literals one entry per line, and runs the build checks that replaced the modules' import-time asserts (its docstring numbers them). Each `config/` module re-exports its constants from there (`TITLES = _compiled.TITLES`) under its autodoc docstring, so the public names, the docs and the deprecated-alias bridge did not move. `tests/v2/test_vocab.py` fails when `_compiled.py` is stale, naming the refresh command, and gives every build check a mutation. The model, the phasing and what was declined: decisions/vocabulary-model. +**The default vocabulary is written per word, in TOML, and compiled.** `nameparser/vocab/.toml` holds one file per BCP 47 language tag, each `[[word]]` entry a `text` and its readings keyed by kind. What a file means, how an entry is built, the kinds and properties with the set each compiles into, exclusions, bearer counts and the steps for adding a word are in `nameparser/vocab/README.md`, the reference for anyone editing the TOML; `tests/v2/test_vocab.py` holds its kind, property, derived-marker and reserved-name tables to the names `build.py` accepts and the sets it compiles, so read it rather than restating it here. The parser never reads the TOML: `tools/vocab/build.py` compiles it into the committed `nameparser/config/_compiled.py`, frozenset literals one entry per line, and runs the build checks that replaced the modules' import-time asserts (its docstring numbers them). Each `config/` module re-exports its constants from there (`TITLES = _compiled.TITLES`) under its autodoc docstring, so the public names, the docs and the deprecated-alias bridge did not move. `tests/v2/test_vocab.py` fails when `_compiled.py` is stale, naming the refresh command, and gives every build check a mutation. The model, the phasing and what was declined: decisions/vocabulary-model. The `config/` SETS are frozen since 2.2 (#293): there is no `.add()`/`.remove()` on any of them, so a default word list is changed at runtime by configuring an object — a private `Constants` for `HumanName`, a `Lexicon` for the 2.0 API — never by editing the constant, and the shipped default is changed by editing the TOML and rebuilding. `CONSTANTS`/`Constants` still hand out mutable `SetManager`s; the freeze is on the module set constants they copy from. **Neither dict was frozen, and `CAPITALIZATION_EXCEPTIONS` is not covered by anything else either.** `REGEXES` is a compiled-pattern table rather than vocabulary and was never in #293's scope; `CAPITALIZATION_EXCEPTIONS` is vocabulary-shaped and is a decided, in-scope exemption. So the split-default hazard the freeze closes is still live for it, measured on 2.2 and re-measured 2026-09-23 with the key below once `'PhD'` became the shipped value: `CAPITALIZATION_EXCEPTIONS['dphil'] = 'DPhil'` reaches a freshly built `Constants` and neither the cached `Lexicon.default()` nor the shared `CONSTANTS`. Same advice — configure the object (`constants.capitalization_exceptions[...]`, or `dataclasses.replace(lexicon, capitalization_exceptions=...)`). `tests/v2/test_contracts.py::test_every_vocabulary_constant_is_frozen` names both dicts as explicit exemptions rather than letting its `isinstance` filter drop them. - `titles.py` — `TITLES` (every title reading; prenominals), `GIVEN_NAME_TITLES` (the readings with `addresses = "given"`, e.g. "Sir", which treat the following name as given, not family — a subset by construction) and `AFTER_GIVEN_TITLES` (the salutation readings with `after_given = true`, read as the title where one ends the given part after a family comma, rules/H8, #835 — a subset by construction, with no v1 `Constants` attribute) -- `suffixes.py` — `SUFFIX_ACRONYMS` (suffix readings with `acronym = true`, matched with every period removed, e.g. "M.D.") and `SUFFIX_WORDS` (the rest, e.g. "Jr."), disjoint because a word has one suffix kind; `SUFFIX_ACRONYMS_AMBIGUOUS` (`may_be_name`); and `GLUED_HONORIFICS` (#308, `attached`), the subset of `SUFFIX_WORDS` the peel may split off the END of a name token — a separate, harsher bar, since the glued position has no writer-drawn boundary to lean on, and each word refused it carries `excluded = { attached = … }` -- `particles.py` — `PARTICLES` (family-name particles, e.g. "de", "van") and `NON_GIVEN_NAME_PARTICLES`, the curated subset that is *never* a standalone given name (the particle readings with `may_be_given = false`). What the subset licenses is rules/P1's fold in `post_rules` (the "rule 1b" of older issue comments — decisions/legacy-rule-numbers maps the old numbers), and the fold has ONE site: the piece that OPENS the name. A lone never-given particle there, with at least one more name word behind it, starts the family. Titles do not move the opening position (rules/P4), so `Sir de Mesnil` reads family `de Mesnil` under every `name_order`, as `de Mesnil` does. How far the fold reaches depends on the order the name was read under: by default it takes the rest of the name (`de Mesnil Jean` → family `de Mesnil Jean`); under either family-first order it takes the particle run plus ONE name unit and lays the rest out by the order (`de Mesnil Jean` → family `de Mesnil`, given `Jean`); after a family comma it takes the rest of that part whatever the order (`Smith, de Mesnil Jean` → family `Smith de Mesnil Jean`). The leading particle stays a lone piece for the fold to find — `group` does not chain it forward, and the fold walks the particles after it itself — so `de la Vega` reads family `de la Vega` under all three orders. The GIVEN position is not a site. It was a second one until #467, which removed it because under a family-first order that slot holds what the caller DECLARED to be the given name: `Mesnil de` under `FAMILY_FIRST` or `FAMILY_FIRST_GIVEN_LAST` reads given `de`, family `Mesnil`. Nor is a particle inside the name: `Juan de la Vega` under a family-first order reads family `Juan`, given `de la Vega`, with no ambiguity recorded — what the vocabulary forbids is the bare particle as a given name, not a name part that begins with one. A bare `de` has nothing to fold into and is placed by position like any one-word name: given `de` by default, family `de` under both family-first orders. A leading particle OUTSIDE the set is left to position — `van Gogh` is given `van`, family `Gogh` by default and family `van`, given `Gogh` under both family-first orders — and records a `PARTICLE_OR_GIVEN` ambiguity under all three; a never-given particle opening the name records none (`de Mesnil`), and neither does an ambiguous one inside it (`Vincent van Gogh`). `Lexicon.particles_ambiguous` is its complement within `PARTICLES`, so the two mark OPPOSITE sets — see the flip warning in `docs/migrate.rst` before translating either -- `bound_given_names.py` — `BOUND_GIVEN_NAMES` (`bound_given` readings, e.g. "abdul", "abu"; one that is also a particle needs `may_be_given = true`, a build check); a group-stage rule joins the first non-title piece to its following piece before roles are assigned, reserving a family word unless a family comma has already fixed it or a given-name title stands ahead (#369, rules/P5) (v1's `_join_bound_first_name`, ported into `_pipeline/_group.py` and gone from the tree — the v1 descriptions further down are history, not current code) +- `suffixes.py` — `SUFFIX_ACRONYMS` (suffix readings with `acronym = true`, matched with every period removed, e.g. "M.D.") and `SUFFIX_WORDS` (the rest, e.g. "Jr."), disjoint because a word has one suffix kind; `SUFFIX_ACRONYMS_AMBIGUOUS` (an acronym reading on a word with a `given` or `family` reading, derived since 2026-10-25: the build refuses the phase 1 flag `may_be_name`); and `GLUED_HONORIFICS` (#308, `attached`), the subset of `SUFFIX_WORDS` the peel may split off the END of a name token — a separate, harsher bar, since the glued position has no writer-drawn boundary to lean on, and each word refused it carries `excluded = { attached = … }` +- `particles.py` — `PARTICLES` (family-name particles, e.g. "de", "van") and `NON_GIVEN_NAME_PARTICLES`, the curated subset that is *never* a standalone given name (the particles whose word has no `given` reading, each stating `may_be_given = false`; a `given` reading on the word is what makes a particle possibly given, the phase 1 flag `may_be_given = true` refused since 2026-10-25). What the subset licenses is rules/P1's fold in `post_rules` (the "rule 1b" of older issue comments — decisions/legacy-rule-numbers maps the old numbers), and the fold has ONE site: the piece that OPENS the name. A lone never-given particle there, with at least one more name word behind it, starts the family. Titles do not move the opening position (rules/P4), so `Sir de Mesnil` reads family `de Mesnil` under every `name_order`, as `de Mesnil` does. How far the fold reaches depends on the order the name was read under: by default it takes the rest of the name (`de Mesnil Jean` → family `de Mesnil Jean`); under either family-first order it takes the particle run plus ONE name unit and lays the rest out by the order (`de Mesnil Jean` → family `de Mesnil`, given `Jean`); after a family comma it takes the rest of that part whatever the order (`Smith, de Mesnil Jean` → family `Smith de Mesnil Jean`). The leading particle stays a lone piece for the fold to find — `group` does not chain it forward, and the fold walks the particles after it itself — so `de la Vega` reads family `de la Vega` under all three orders. The GIVEN position is not a site. It was a second one until #467, which removed it because under a family-first order that slot holds what the caller DECLARED to be the given name: `Mesnil de` under `FAMILY_FIRST` or `FAMILY_FIRST_GIVEN_LAST` reads given `de`, family `Mesnil`. Nor is a particle inside the name: `Juan de la Vega` under a family-first order reads family `Juan`, given `de la Vega`, with no ambiguity recorded — what the vocabulary forbids is the bare particle as a given name, not a name part that begins with one. A bare `de` has nothing to fold into and is placed by position like any one-word name: given `de` by default, family `de` under both family-first orders. A leading particle OUTSIDE the set is left to position — `van Gogh` is given `van`, family `Gogh` by default and family `van`, given `Gogh` under both family-first orders — and records a `PARTICLE_OR_GIVEN` ambiguity under all three; a never-given particle opening the name records none (`de Mesnil`), and neither does an ambiguous one inside it (`Vincent van Gogh`). `Lexicon.particles_ambiguous` is its complement within `PARTICLES`, so the two mark OPPOSITE sets — see the flip warning in `docs/migrate.rst` before translating either +- `bound_given_names.py` — `BOUND_GIVEN_NAMES` (`bound_given` readings, e.g. "abdul", "abu"; one that is also a particle needs a `given` reading, a build check); a group-stage rule joins the first non-title piece to its following piece before roles are assigned, reserving a family word unless a family comma has already fixed it or a given-name title stands ahead (#369, rules/P5) (v1's `_join_bound_first_name`, ported into `_pipeline/_group.py` and gone from the tree — the v1 descriptions further down are history, not current code) - `conjunctions.py` — `CONJUNCTIONS` (e.g. "and", "of") used to chain multi-word titles and to join name parts (rules/P3), plus `CONJUNCTIONS_AMBIGUOUS` (#383/#479, rules/P8, `may_be_initial`, refused by the build on anything but a cased single letter), the subset of single cased letters that read as an INITIAL in a name written wholly in one case — a marker set with no v1 `Constants` attribute of its own, so the only v1 knob that reaches it is deleting the conjunction - `maiden_markers.py` — `MAIDEN_MARKERS` (`birth_marker` readings, e.g. "née", "geb.") routing the following name to `maiden`; with an `addresses = "given"` title, the one kind a phrase may be (`z domu`) - `surnames.py` — `KOREAN_SURNAMES`, the census list the 2.0 API splits unspaced hangul on (#271), still written in Python with an import-time `assert_normalized`: a segmentation list, outside the per-word model, like the locale packs' lists. With `maiden_markers.py` it is one of the two data modules `Constants` has **no** attribute for: both reach the parse only through `Constants._snapshot()` → `Lexicon`, so the v1 surface stays frozen and there is no v1 knob to turn either off (the opt-out is the 2.0 `Policy`) @@ -270,11 +270,11 @@ Add a dedicated `copy.deepcopy()` round-trip test for it too (see `test_regexes_ **Unknown-key attribute access on `TupleManager`/`RegexTupleManager` warns (1.4, #256) — 2.0 note: the warning became a hard `AttributeError` naming the known keys (shim `TupleManager`); the paragraph below describes the deleted v1 machinery and applies only when reading v1 history.** — a key not currently in the dict emits `DeprecationWarning` naming the miss and the known keys (`_warn_unknown_key`), before falling back to the same `None`/`EMPTY_REGEX` default as before; `.get()` stays silent. Dunder probes (`__deepcopy__`) still raise `AttributeError` outright, and single-underscore probes (`_repr_html_`, IPython's `_ipython_canary_method_should_not_exist_`, etc.) are excluded from the warning too — no real config key starts with `_`, so both guards just keep protocol/introspection probes from misfiring as "typo" warnings. This means internal parser code that reads `self.C.regexes.` unconditionally (e.g. `squash_bidi`'s `bidi`) now warns if a caller's custom `regexes` dict omits that key — a previously-silent partial-override pattern is on the same deprecation path as an actual typo. -**Adding a word to the vocabulary** — the steps are in `nameparser/vocab/README.md`; this paragraph is the traps they have run into. First check the word's OTHER readings: `uv run python tools/vocab/query.py --word ` lists every reading and exclusion of it in every file, which is where the overlaps between categories now show. Real overlaps exist: `st`/`freiherr` ∈ `PARTICLES` ∩ `TITLES`, and `do`/`mc`/`vd` ∈ `PARTICLES` ∩ `SUFFIX_ACRONYMS`; `abd` = "ABD" ∈ `SUFFIX_ACRONYMS`; `abu` ∈ `PARTICLES` ∩ `BOUND_GIVEN_NAMES` (position-dependent: leading token → given-name join, mid-name → family-name join). Usually position-dependent and harmless, but can force a guard or an exclusion (the `last_base` all-particles guard; `abd`, kept out of `BOUND_GIVEN_NAMES` until decisions/P5 found the credential collision cost nothing at either end). Then check the word against real names, not memory: `uv run python tools/name_usage.py ` counts its bearers as a given name and as a family name (US Census surnames, SSA given names, Wikidata with countries of citizenship), and gives its given-name frequency band per country from Jörg Michael's `nam_dict.txt` (54 countries and an "other countries" column, GFDL, downloaded into the cache and never committed) — the one source that sees a given name outside the US and Wikidata's lower bound, such as Korean `In` (band A in Korea) or Vietnamese `Bay` (6 in Vietnam). A band is a frequency class, not a count, and the dictionary is a selection, so a miss there (`Shehu`) proves nothing; and because a truncated source reads as a run of false zeros, a pinned download is checked against its sha256 on every load from the cache, never against a floor on what it holds — #775's first fix refused fewer than 40,000 names and passed a `nam_dict.txt` missing only its last line (`Zyta`). Pass a name as it is written (`McDonald`): Wikidata labels match exactly, and no rule rebuilds an internal capital from `mcdonald`. A word in a non-Latin script is counted by its romanization too, since the Census, SSA and nam_dict hold only Latin names, and that count weighs only where its bearers write the script: #835 vetted its salutations by their own spellings and its review found `बाबू` flagged while `babu` is a US surname (1,496) and an Indian given-name element (Ram Babu), where the romanization of `মি`, `Mi`, counts Korean and Chinese bearers who never write Bengali. It reports and does not decide — read the counts against the position the set's claim acts on. For a suffix it also has the other side: `--fetch-nppes` counts the US healthcare providers who write the word as a credential (NPPES, the 4.8 million credentialed providers in the pinned August 2026 file, so blind to law, accounting and honours), and `--rank N --unlisted` lists the commonest credentials the vocabulary misses, with where each lands in `John Smith, X` -- the list #797 came from. Where a local corpus is fetched (`tools/differential/local_corpora.py`, mechanisms.md#NAME-DATA-JOBS), a row says how many of its records (for ClinicalTrials.gov, officials) write the word, and how many after a comma: ClinicalTrials.gov's study officials show where a credential actually stands in a name as typed. A word that would peel GLUED (`GLUED_HONORIFICS`) asks a different question, whether any name ENDS in it, and for a CJK word the tool answers that too: an ends-with row from a cached index of Wikidata's CJK name labels (built on first use; 33 minutes on 2026-10-17), plus the names the word ends across the family/given join (阿部 + 長 for 部長). Read the tool's hits by item before counting them, because labels include transcriptions and mislabels: #707's one hit for 二世 was 厄德二世, the given name Odo with a regnal numeral in its Chinese label, borne by 109 Western people named Odo. Then ask a third question the tool cannot: does a common longer word end in it? The peel matches the end of a token, spaced or glued, so a listed tail splits every unlisted compound that ends in it — #707 vetted 議員, 社長 and 医学博士 against names and its review found 衆議院議員, 副社長 and 獣医学博士 cut at the tail (`田中 副社長` → given 副); no corpus held a compound office word until #707's rules examples added two, so the snapshot was silent about the class. List the common compounds whole or record the split as a cost. And check the predicates that test a member's SPELLING rather than its membership: they encode what the set's members happened to look like when they were written. `_pieces.starts_a_credential_run` implemented the credential run's "Latin" (rules/S8 now, part of S2 until #682 split it) as `isascii()`, exact until #661 added `mlađi`, which would have read `mlađi Jones` differently from `stariji Jones`; the design-docs review caught it, after a first draft had credited the test to the wrong clause of S2 and proposed deleting it on a mutation that moved nothing because nothing pinned it. (#679 then dropped the script test on purpose, after measuring what it protected.) Before adding the first member of a new script, or the first accented one, grep the set's consumers for `isascii`, `isalpha` and script tests, and read the rule each one implements. The same goes for the first member holding PUNCTUATION: #700 added `d'` and `az-`, the parse read them right and every snapshot move was an intended one, and `capitalized()` still wrote `D' Ivoi`, because case repair splits a token at its apostrophe or hyphen and asks the vocabulary of the stem. The snapshot records both case views since #826, but only for the names the corpora hold, so parse such a member's names through `capitalized()` and `capitalized(force=True)` in each case before calling it done. The user docs use real words as examples of an UNLISTED one, and listing the word breaks the example: #797 listed `dphil`, and docs/customize.rst's capitalization example (`parse("jane smith dphil").capitalized()` giving `Dphil`) failed in all five CI build jobs while `uv run pytest` passed, because `sphinx-build -b doctest docs` is a separate step. Run it after adding a word, and grep `docs/*.rst` for the word; the example now uses `dclinpsy`, a credential left unlisted on purpose, so list it only after moving the example. Record what the vetting found ON the word, not in a comment or a decisions sentence: `note`, `issue`, a `given`/`family` reading with dated counts, and for a word you decided to keep out an `excluded` entry with its reason, so the next contributor who adds it meets the reason as a build error (decisions/vocabulary-model). +**Adding a word to the vocabulary** — the steps are in `nameparser/vocab/README.md`; this paragraph is the traps they have run into. First check the word's OTHER readings: `uv run python tools/vocab/query.py --word ` lists every reading and exclusion of it in every file, which is where the overlaps between categories now show. Real overlaps exist: `st`/`freiherr` ∈ `PARTICLES` ∩ `TITLES`, and `do`/`mc`/`vd` ∈ `PARTICLES` ∩ `SUFFIX_ACRONYMS`; `abd` = "ABD" ∈ `SUFFIX_ACRONYMS`; `abu` ∈ `PARTICLES` ∩ `BOUND_GIVEN_NAMES` (position-dependent: leading token → given-name join, mid-name → family-name join). Usually position-dependent and harmless, but can force a guard or an exclusion (the `last_base` all-particles guard; `abd`, kept out of `BOUND_GIVEN_NAMES` until decisions/P5 found the credential collision cost nothing at either end). Then check the word against real names, not memory: `uv run python tools/name_usage.py ` counts its bearers as a given name and as a family name (US Census surnames, SSA given names, Wikidata with countries of citizenship), and gives its given-name frequency band per country from Jörg Michael's `nam_dict.txt` (54 countries and an "other countries" column, GFDL, downloaded into the cache and never committed) — the one source that sees a given name outside the US and Wikidata's lower bound, such as Korean `In` (band A in Korea) or Vietnamese `Bay` (6 in Vietnam). A band is a frequency class, not a count, and the dictionary is a selection, so a miss there (`Shehu`) proves nothing; and because a truncated source reads as a run of false zeros, a pinned download is checked against its sha256 on every load from the cache, never against a floor on what it holds — #775's first fix refused fewer than 40,000 names and passed a `nam_dict.txt` missing only its last line (`Zyta`). Pass a name as it is written (`McDonald`): Wikidata labels match exactly, and no rule rebuilds an internal capital from `mcdonald`. A word in a non-Latin script is counted by its romanization too, since the Census, SSA and nam_dict hold only Latin names, and that count weighs only where its bearers write the script: #835 vetted its salutations by their own spellings and its review found `बाबू` flagged while `babu` is a US surname (1,496) and an Indian given-name element (Ram Babu), where the romanization of `মি`, `Mi`, counts Korean and Chinese bearers who never write Bengali. It reports and does not decide — read the counts against the position the set's claim acts on. For a suffix it also has the other side: `--fetch-nppes` counts the US healthcare providers who write the word as a credential (NPPES, the 4.8 million credentialed providers in the pinned August 2026 file, so blind to law, accounting and honours), and `--rank N --unlisted` lists the commonest credentials the vocabulary misses, with where each lands in `John Smith, X` -- the list #797 came from. Where a local corpus is fetched (`tools/differential/local_corpora.py`, mechanisms.md#NAME-DATA-JOBS), a row says how many of its records (for ClinicalTrials.gov, officials) write the word, and how many after a comma: ClinicalTrials.gov's study officials show where a credential actually stands in a name as typed. A word that would peel GLUED (`GLUED_HONORIFICS`) asks a different question, whether any name ENDS in it, and for a CJK word the tool answers that too: an ends-with row from a cached index of Wikidata's CJK name labels (built on first use; 33 minutes on 2026-10-17), plus the names the word ends across the family/given join (阿部 + 長 for 部長). Read the tool's hits by item before counting them, because labels include transcriptions and mislabels: #707's one hit for 二世 was 厄德二世, the given name Odo with a regnal numeral in its Chinese label, borne by 109 Western people named Odo. Then ask a third question the tool cannot: does a common longer word end in it? The peel matches the end of a token, spaced or glued, so a listed tail splits every unlisted compound that ends in it — #707 vetted 議員, 社長 and 医学博士 against names and its review found 衆議院議員, 副社長 and 獣医学博士 cut at the tail (`田中 副社長` → given 副); no corpus held a compound office word until #707's rules examples added two, so the snapshot was silent about the class. List the common compounds whole or record the split as a cost. And check the predicates that test a member's SPELLING rather than its membership: they encode what the set's members happened to look like when they were written. `_pieces.starts_a_credential_run` implemented the credential run's "Latin" (rules/S8 now, part of S2 until #682 split it) as `isascii()`, exact until #661 added `mlađi`, which would have read `mlađi Jones` differently from `stariji Jones`; the design-docs review caught it, after a first draft had credited the test to the wrong clause of S2 and proposed deleting it on a mutation that moved nothing because nothing pinned it. (#679 then dropped the script test on purpose, after measuring what it protected.) Before adding the first member of a new script, or the first accented one, grep the set's consumers for `isascii`, `isalpha` and script tests, and read the rule each one implements. The same goes for the first member holding PUNCTUATION: #700 added `d'` and `az-`, the parse read them right and every snapshot move was an intended one, and `capitalized()` still wrote `D' Ivoi`, because case repair splits a token at its apostrophe or hyphen and asks the vocabulary of the stem. The snapshot records both case views since #826, but only for the names the corpora hold, so parse such a member's names through `capitalized()` and `capitalized(force=True)` in each case before calling it done. The user docs use real words as examples of an UNLISTED one, and listing the word breaks the example: #797 listed `dphil`, and docs/customize.rst's capitalization example (`parse("jane smith dphil").capitalized()` giving `Dphil`) failed in all five CI build jobs while `uv run pytest` passed, because `sphinx-build -b doctest docs` is a separate step. Run it after adding a word, and grep `docs/*.rst` for the word; the example now uses `dclinpsy`, a credential left unlisted on purpose, so list it only after moving the example. Record what the vetting found ON the word, not in a comment or a decisions sentence: `note`, `issue`, a `given`/`family` reading with dated counts, and for a word you decided to keep out an `excluded` entry with its reason, so the next contributor who adds it meets the reason as a build error (decisions/vocabulary-model). A name reading is not only provenance on a word that has a credential or particle reading: it IS the clash, and derives the marker (an ambiguous acronym, a particle that may be given, since 2026-10-25), so counts kept as evidence AGAINST marking such a word go in the clashing reading's `note` instead, as `ld`'s do. **Before adding a short/common word to `PARTICLES` globally**, test it mid-string against realistic 3-token names, not just check for English-word collisions: Korean/Vietnamese given names put a short syllable in the middle slot (`Park In Hwan`, `Nguyen To Nga`), and Western names put a bare initial there (`John V. Smith`). A word that looks safe ("nobody is named 'to'") can still swallow a real middle name/initial into the last name once it's a global prefix — confirmed regressions for `to`/`in`/`an`/`ten`/`then` and bare `v` this way (PR #191). -**Adding a curated sub-set of an existing vocabulary set** (a marker over a base set, as `GIVEN_NAME_TITLES` is over `TITLES`) — it is a behavioral PROPERTY of a reading, not a second word list: a row in `tools/vocab/build.py`'s `BEHAVIORAL` table, a branch in `compile_pool` adding the words whose reading carries it, a constant in its `CONSTANTS` roster and a key of the same name in its `Compiled` TypedDict (an import-time assert holds the two to one list), and the `NAME = _compiled.NAME` assignment in the config module. The subset relation then holds by construction, since the property cannot exist without its reading, so it needs no check. A relation construction does not give (`∩ == ∅` with a set the sub-set cannot logically overlap; a sub-set member that is also a title is silently inert, title handling consuming it first) goes in `check_pool`, with a mutation in `tests/v2/test_vocab.py`'s table. Do **not** forbid `titles ∩ particles` — that overlap is intentional (`freiherr`, `st`). On the v1 side the sub-set is a `_SetManagerAttribute` on `Constants` where v1 had one (like `first_name_titles`; a sub-set new in 2.x, such as `honorific_tails`, `conjunctions_ambiguous` or `after_given_titles`, gets none and rides in `_snapshot()` as an intersection with its base, the shim roster under the 2.0 conventions listing each — the v1 ATTRIBUTE names did not move in #293, only the module constants did), **not** a `_CachedUnionMember` — only `prefixes`/`suffix_acronyms`/`suffix_not_acronyms`/`titles` feed the `_pst` hot-path cache, so a sub-set costs nothing at runtime and stays out of `is_rootname`. A runtime `.add()` to the sub-set's `SetManager` does **not** propagate to the parent's (same as `first_name_titles`→`titles`), so a caller adding a brand-new word adds it to both. +**Adding a curated sub-set of an existing vocabulary set** (a marker over a base set, as `GIVEN_NAME_TITLES` is over `TITLES`) — it is a behavioral PROPERTY of a reading, not a second word list: a row in `tools/vocab/build.py`'s `BEHAVIORAL` table, a branch in `compile_pool` adding the words whose reading carries it, a constant in its `CONSTANTS` roster and a key of the same name in its `Compiled` TypedDict (an import-time assert holds the two to one list), and the `NAME = _compiled.NAME` assignment in the config module. The subset relation then holds by construction, since the property cannot exist without its reading, so it needs no check. A marker that records a clash with a NAME is the other shape: derived from the word's name reading beside the clashing one (`SUFFIX_ACRONYMS_AMBIGUOUS`, and the complement `NON_GIVEN_NAME_PARTICLES` works from), not written as a property, so it too holds by construction (decisions/vocabulary-model, 2026-10-25). A relation construction does not give (`∩ == ∅` with a set the sub-set cannot logically overlap; a sub-set member that is also a title is silently inert, title handling consuming it first) goes in `check_pool`, with a mutation in `tests/v2/test_vocab.py`'s table. Do **not** forbid `titles ∩ particles` — that overlap is intentional (`freiherr`, `st`). On the v1 side the sub-set is a `_SetManagerAttribute` on `Constants` where v1 had one (like `first_name_titles`; a sub-set new in 2.x, such as `honorific_tails`, `conjunctions_ambiguous` or `after_given_titles`, gets none and rides in `_snapshot()` as an intersection with its base, the shim roster under the 2.0 conventions listing each — the v1 ATTRIBUTE names did not move in #293, only the module constants did), **not** a `_CachedUnionMember` — only `prefixes`/`suffix_acronyms`/`suffix_not_acronyms`/`titles` feed the `_pst` hot-path cache, so a sub-set costs nothing at runtime and stays out of `is_rootname`. A runtime `.add()` to the sub-set's `SetManager` does **not** propagate to the parent's (same as `first_name_titles`→`titles`), so a caller adding a brand-new word adds it to both. **Adding a flag-gated post-parse transform** (reorder/adjust) — add a `Constants` boolean (default `False`), implement a `handle_*()` method, and call it in `post_process()` after `handle_firstnames()` and before `handle_capitalization()`, gated on the flag. Default-off keeps existing parses byte-for-byte unchanged. Two shipped examples: `patronymic_name_order` gates both `handle_east_slavic_patronymic_name_order()` (#85) and `handle_turkic_patronymic_name_order()` (#185) — one flag driving two independent handlers, added in the same `post_process()` slot; `middle_name_as_last` gates `handle_middle_name_as_last()` (#133), which folds `middle_list` into `last_list`. diff --git a/docs/design/decisions/vocabulary-model.md b/docs/design/decisions/vocabulary-model.md index 8369d462..4b843243 100644 --- a/docs/design/decisions/vocabulary-model.md +++ b/docs/design/decisions/vocabulary-model.md @@ -18,7 +18,8 @@ The vocabulary was a set of base sets plus marker subsets (`particles` and `part - Order and shape are separate: a language supplies only a default order, and the shapes (where titles, suffixes, particles and commas sit) belong to each order and live in the core; vocabulary readings are to settle a clash and name readings only to keep the gate on — 2026-10-15/16. - `cased` replaces `capitalization_exceptions` and, with `same_as`, a canonical spelling, and a reading's `native` spelling is for surfacing; `acronym` is a reading property replacing the acronym/word split, one suffix kind per word; `expands` records the meaning an entry was added for; `addresses` takes `family` and `given`, `full` and `any` being refused until a later phase — 2026-10-15/16; 2026-10-18 #737. - The vocabulary is per-word TOML in `nameparser/vocab/.toml`, one file per BCP 47 tag plus `und` and `mul`, at `schema = 1` (a compiler refuses a newer one), every reading a table and no `language` key; `tools/vocab/build.py` compiles it into a committed `nameparser/config/_compiled.py`, the parser never reads the TOML, and parsing it at import was declined — 2026-10-18 #737. -- The marker sets are explicit flags on readings (`may_be_given`, `may_be_name`, `may_be_initial`, `attached`, `addresses`, `after_given`, `cased`), not yet derived from name readings; `may_be_given` is stated on every particle, never defaulted, and two kinds of one word must agree on it; `after_given` stands on a salutation reading only (decisions/H8) — 2026-10-18 #737; 2026-10-19 #749; 2026-10-24 #835. +- The marker sets are explicit properties on readings (`may_be_initial`, `attached`, `addresses`, `after_given`, `cased`), and `after_given` stands on a salutation reading only (decisions/H8) — 2026-10-18 #737; 2026-10-24 #835. +- The two markers that record a clash with a name are derived from name readings: an acronym suffix reading beside a `given` or `family` reading is an ambiguous acronym, a particle reading beside a `given` reading a particle that may be given, wherever in the files each reading sits. The build refuses `may_be_name` and `may_be_given = true`, naming the reading to write; a particle whose word has no `given` reading still states `may_be_given = false`, never defaulted; a name reading may stand without a count, with a `note` saying why; and a count kept as evidence against marking such a word goes in the clashing reading's `note` (`ld`). `may_be_initial` is not derived — 2026-10-25 #752. - Exclusions are data: an `excluded` key refuses a kind, a category or a behavioral property, in the file of the language whose reading it refuses, and the build refuses the excluded thing in any file, quoting the reason — 2026-10-18 #737. - A per-word fact that moves into the TOML is replaced in decisions/ by a query that finds it (decisions/README.md, edit kind 4) — 2026-10-18 #737. - The script is read from the word, not the file: a script subtag is only for a difference the text cannot show — 2026-10-18 #737. @@ -32,7 +33,7 @@ The vocabulary was a set of base sets plus marker subsets (`particles` and `part Not standing: the plans the classifications carried out, phase 1's mechanical conversion of the unclassified categories into `und.toml` (2026-10-18 #737) and the shared readings left in place until their categories were classified (2026-10-19 #749), both done by 2026-10-20 #751, the conjunctions; and #747's reason for exempting `und.toml`, its entries keeping the Python lists' order for its vetting comments to point along (2026-10-19 #747), overtaken once it held two entries and no such comment. -Open: #752 (phase 2, derived ambiguity), #789 (whether the dotted `Wm.`, `Chas.`, `Geo.`, `Ma.` and `Jo.` opening the given part read given, outranking rules/H2's shape; phase 2's), #753 (phase 3, language packs), #754 (phase 4, order, shapes and the output model), #766 (whether `bon`, `dí` and three other words stay in the vocabulary). +Open: #752 (phase 2, derived ambiguity: the gate on every reading kind), #789 (whether the dotted `Wm.`, `Chas.`, `Geo.`, `Ma.` and `Jo.` opening the given part read given, outranking rules/H2's shape; phase 2's), #753 (phase 3, language packs), #754 (phase 4, order, shapes and the output model), #766 (whether `bon`, `dí` and three other words stay in the vocabulary). ## Log @@ -62,7 +63,7 @@ Open: #752 (phase 2, derived ambiguity), #789 (whether the dotted `Wm.`, `Chas.` Phasing, fixed then: (1) the data format and a compiler to today's sets, snapshot-neutral; (2) derived ambiguity, with the gate extended to every reading kind — the step that changes behavior, with its own design; (3) pack semantics, needing 1 and 2; (4) the rules menu, order and shapes, and the output model. -- **2026-10-18 #737 — phase 1 shipped: per-word TOML compiled to today's sets (approved by Derek, 2026-10-18, all eight of the spec's recommendations).** No parse, render or public name moves; it is a re-encoding plus the build checks the encoding makes possible. +- **2026-10-18 #737 — phase 1 shipped: per-word TOML compiled to today's sets (approved by Derek, 2026-10-18, all eight of the spec's recommendations).** (`may_be_name` and `may_be_given` as written flags superseded by 2026-10-25 #752, which derives them as this entry's flags bullet planned.) No parse, render or public name moves; it is a re-encoding plus the build checks the encoding makes possible. - **The format.** `nameparser/vocab/.toml`, one file per BCP 47 tag in canonical case (`pt-BR`, `zh-Hant`), plus `und` for words not yet assigned a language and `mul` for words shared on purpose (the Roman numerals). `schema = 1` is the only other top-level key: a new kind or property is additive and does not bump it, a change in what a key means does, and a compiler refuses a newer schema. Each `[[word]]` has a `text`, already in `_normalize` form, and zero or more readings keyed by KIND, each a TABLE (`generation = {}`, never `= true`) so a property can be added to any reading later. The file name is the language; there is no `language` key to drift from it. - **Kinds as keys, with generic category keys.** Each kind belongs to one category, and the category decides which compiled field a reading feeds. A category key is itself a kind (`title = {}`, a title not yet classified), so classifying a word later is a data edit that moves no compiled set. Kind names are unique across categories. Phase 1 classifies the SUFFIX category fully — `generation`, `credential`, `honorific`, `professional_honorific`, `status`, `courtesy` — and converts every other category mechanically into `und.toml` with its generic kind and today's flags, so the TOML is the single source from the first PR. One suffix kind per word replaces the import-time assert that the acronym and word sets are disjoint. - **Flags now, derivation in phase 2.** The marker sets became explicit properties — `may_be_given` on particles, `may_be_name` on acronym credentials, `may_be_initial` on conjunctions, `attached = { side = "after" }` for the glued peel, `addresses = "given"`, `cased` — rather than being derived from declared name readings, which would have meant declaring a given reading for `te`, `tho` and `du` that no count supports. Measured for the later swap: deriving both ambiguity markers from name readings reproduces them exactly if the readings are declared on exactly the flagged words, because no word sits in one marker set while needing to be absent from the other (`do` is in both; `mc` and `vd`, particles and credentials, in neither, consistent with #454). Recompute: `L = Lexicon.default(); L.particles_ambiguous & (L.suffix_acronyms - L.suffix_acronyms_ambiguous)` and `L.suffix_acronyms_ambiguous & (L.particles - L.particles_ambiguous)`, both empty. @@ -78,7 +79,7 @@ Open: #752 (phase 2, derived ambiguity), #789 (whether the dotted `Wm.`, `Chas.` - **2026-10-19 — decision 11 widened to words a file refuses (Derek).** A name reading may also sit on a word whose vocabulary reading some file refuses: the reading is the evidence the refusal rests on, as `fils`'s `family` reading in fr.toml backs its excluded `generation`. The build already allowed this (its soft warning fires on a name reading only where the word has neither a vocabulary reading nor an exclusion), so the decision now matches the shipped data; #758's review found the mismatch. -- **2026-10-19 #749 — the particles classified by language and kind.** No compiled set moves: `tools/vocab/build.py --check` reports the committed module current, and `tools/differential/snapshot.py` matches. Each particle reading left `und.toml` for the file of every language whose names it occurs in (decision 4), so one word may sit in several (`de` in five); the readings it shares with other categories (`do`, `mc` and `vd` as credentials, `abu` and the Arabic أبو/ابو as bound given names, `freiherr` and `st` as titles) stay where they were until those categories are classified. Eight particle kinds join the generic one: `preposition`, `article`, `patronymic`, `teknonymic`, `clan`, `rank`, `saint` and `alias`, defined in nameparser/vocab/README.md. The `clan`/`patronymic` line is drawn for decision 8's patronymic field: a patronymic is followed by a parent's name, a clan prefix by a family name the lineage shares, so the Irish Ó and Ní and the Gaelic Mac are clans although each began as "descendant/son/daughter of". +- **2026-10-19 #749 — the particles classified by language and kind.** (Check 5's agreement on `may_be_given` and the unexamined-word convention keyed on `may_be_given = true` superseded by 2026-10-25 #752, which derives the marker from a `given` reading and keys the convention on that reading.) No compiled set moves: `tools/vocab/build.py --check` reports the committed module current, and `tools/differential/snapshot.py` matches. Each particle reading left `und.toml` for the file of every language whose names it occurs in (decision 4), so one word may sit in several (`de` in five); the readings it shares with other categories (`do`, `mc` and `vd` as credentials, `abu` and the Arabic أبو/ابو as bound given names, `freiherr` and `st` as titles) stay where they were until those categories are classified. Eight particle kinds join the generic one: `preposition`, `article`, `patronymic`, `teknonymic`, `clan`, `rank`, `saint` and `alias`, defined in nameparser/vocab/README.md. The `clan`/`patronymic` line is drawn for decision 8's patronymic field: a patronymic is followed by a parent's name, a clan prefix by a family name the lineage shares, so the Irish Ó and Ní and the Gaelic Mac are clans although each began as "descendant/son/daughter of". - **One word, a kind per language, one flag.** A particle may take a different kind in each file (`de` is a preposition in French and an article in Dutch), so check 5 now compares the readings of a CATEGORY rather than of a kind: two kinds of one word disagreeing on `may_be_given` would otherwise have compiled silently to the false. The suffix category is unaffected, check 4 allowing it one kind per word. - **`expands` carries the meaning, `note` the vetting.** A particle's gloss in its language ("of the, fem. pl.") is its `expands`, so `note` keeps the meaning the comments gave it: a `may_be_given = true` word whose particle readings carry no `note` in any file, or whose note says so, has not been examined (the "bare entry" convention of the old module comments). A word's note sits on one of its files, not repeated in each, so the test is per word: `mac` is examined though only en.toml holds its note. Descriptive `gender` is recorded on the markers that name the bearer's sex (decision 9), among them `bin`/`binti`, `ibn`/`bint`, `ó`/`ní` and `freiherr`/`freiherrin` (`uv run python tools/vocab/query.py --kind particle` shows each). - **The keep-outs became exclusions**, in the file of the language whose reading they refuse: `ben` and `ال` in ar.toml, `בר` in he.toml, `nic` and unaccented `o` in ga.toml, `bt` and `bte` in ms.toml (`uv run python tools/vocab/query.py --excluded particle`). decisions/vocabulary-exclusions' particle block keeps its class-level bullets and points at the query for the rest. @@ -130,3 +131,12 @@ Open: #752 (phase 2, derived ambiguity), #789 (whether the dotted `Wm.`, `Chas.` - **2026-10-24 #835 — `after_given`, the salutations that may end a given part.** A behavioral title property, allowed on a `salutation` reading only (a build check, since an office or rank word is borne as a middle name), compiles into `AFTER_GIVEN_TITLES` and `Lexicon.after_given_titles`, read by rules/H8. A salutation borne as a name refuses it with an exclusion carrying its counts. The rule is keyed on a property rather than on the kind, which would make a kind compile into a behavior where every other behavior is a property; the vetting and the measurement are decisions/H8's. Adding the field changes `Lexicon`'s pickle layout, as 2.1 and 2.3 did. - **Recompute** rather than quoting: `uv run python tools/vocab/query.py --kind salutation`, `--excluded after_given`. + +- **2026-10-25 #752 — A CLASH WITH A NAME IS A NAME READING: phase 2a derives the two name-clash markers (the three decisions below agreed by Derek on 2026-10-23).** SUPERSEDES: the flags bullet of 2026-10-18 #737 and the `may_be_given` agreement of 2026-10-19 #749. No compiled set moves: `tools/vocab/build.py` rebuilds `nameparser/config/_compiled.py` byte for byte. `SUFFIX_ACRONYMS_AMBIGUOUS` is now the words with an acronym suffix reading and a `given` or `family` reading, and the particles outside `NON_GIVEN_NAME_PARTICLES` the words with a particle reading and a `given` reading, each wherever in the files the two readings sit (build check 11). The model's reason (decision 15, name readings keep the gate on) is unchanged; what changes is that the reading, not a flag beside it, is what the build reads, so a clash cannot be marked without the evidence that it is one, nor carry evidence and stay unmarked. + - **The readings written.** Every flagged word that lacked one got a name reading: the seven credentials `ba`, `do`, `ed`, `jd`, `lac`, `ma` and `meng`, and the 38 flagged particles other than `bon` and `dí`, `do` among them, so 44 words and 49 readings (a credential with both a Census and an SSA count got both). The counts are `tools/name_usage.py`'s, taken 2026-10-25 and recorded as each reading's `as_of`. + - **A name reading may be declared without a count, with a required `note` saying why.** Derivation needs a `given` reading on every flagged particle, and some are borne as given names in no source the tool reads; refusing a count-less reading would have forced each of those to the never-given side, against decisions/vocabulary-collisions C-i's "under uncertainty, ambiguous". Eight particle words have no count in any source (`aen`, `bat`, `binte`, `freiherr`, `freiherrin`, `thoe`, `ua`, `vande`); each keeps its may-be-given reading on decisions/vocabulary-collisions C-i's default, with a required `note` saying so. The build refuses a count-less name reading with no `note`. The unexamined-word test of nameparser/vocab/README.md keeps its shape, keyed on the given reading: a word whose particle readings carry no `note` is unexamined, whatever counts its given reading carries, since those were taken for the derivation and not read against the particle position. + - **`ld` keeps its count in its credential reading's `note` and has no name reading, so it stays unmarked.** Its `given` reading recorded the evidence for NOT marking it (45 SSA births against 11,703 NPPES providers), which derivation would have read as a mark. The count moved into the credential reading's `note`, so a name reading that keeps the gate off, contradicting decision 15, never exists. Declined: deriving it literally, which moves 3 ClinicalTrials.gov and 5 FEC names and no PSC record (measured 2026-10-25 by writing the given reading back on this change). + - **A particle with no given reading still states `may_be_given = false`, required,** so a new particle never reaches the never-given side unremarked, the reason the property had no default in phase 1; `may_be_given = true` and `may_be_name` (either value) are refused with a message naming the reading to write, and `false` beside a `given` reading is refused as contradicting it. Check 9 now asks for a `given` reading on a bound given name that is also a particle. `may_be_given` is a STATED property in the README's table: held to the readings, compiled to nothing by itself. + - **`may_be_initial` stays a property.** Its clash is with the initial, a shape every cased single letter has, and which letters a tradition writes both ways is a judgment about the letter, not a name anyone bears. + - **Measured (2026-10-25, against 632685c8):** `tools/differential/snapshot.py` matches; `cldr_agreement.py` matches every record; `snapshot.py --local` for ctgov-officials, fec-individuals and tse-donors, recorded from the base with `--tree`, matches; `local_agreement.py psc-individuals` matches the record. Negative control: dropping `jd`'s given reading takes it out of `SUFFIX_ACRONYMS_AMBIGUOUS`, and dropping `van`'s, with `may_be_given = false` written, puts it in `NON_GIVEN_NAME_PARTICLES` (tests/v2/test_vocab.py). + - **Recompute** rather than quoting: `uv run python tools/vocab/query.py --kind given --count`; the markers' sizes, `uv run python -c "from nameparser.config import suffixes, particles; print(len(suffixes.SUFFIX_ACRONYMS_AMBIGUOUS), len(particles.PARTICLES - particles.NON_GIVEN_NAME_PARTICLES))"`. diff --git a/nameparser/config/particles.py b/nameparser/config/particles.py index 9a4878bf..09a36e32 100644 --- a/nameparser/config/particles.py +++ b/nameparser/config/particles.py @@ -1,10 +1,11 @@ # The words, and the vetting notes that stood beside them here, live in -# nameparser/vocab/*.toml: every particle reading states may_be_given, -# true for the ambiguous half and false for the never-given half, so the two -# partition PARTICLES by construction. tools/vocab/build.py compiles them -# into _compiled.py and runs the checks that were import-time asserts -# here -- a bound given name that is also a particle must be one that may -# be given, and an elided particle ships in both apostrophes (#700) +# nameparser/vocab/*.toml: a particle whose word carries a given reading +# is the ambiguous half, and every other particle reading states +# may_be_given = false, the never-given half, so the two partition +# PARTICLES by construction. tools/vocab/build.py compiles them into +# _compiled.py and runs the checks that were import-time asserts here -- +# a bound given name that is also a particle must be one that may be +# given, and an elided particle ships in both apostrophes (#700) # (decisions/vocabulary-model). from nameparser.config import _compiled @@ -84,7 +85,7 @@ #: ambiguity for the reading not taken, and one inside it records none. #: #: Both sets are compiled from the same particle readings, the never-given -#: ones being those with ``may_be_given = false``, which is what guarantees every +#: ones being those whose word has no given reading, which is what guarantees every #: :py:data:`NON_GIVEN_NAME_PARTICLES` member is also a particle (and #: still joins forward), with no drift. PARTICLES = _compiled.PARTICLES diff --git a/nameparser/config/suffixes.py b/nameparser/config/suffixes.py index 1874cedc..c030b5ad 100644 --- a/nameparser/config/suffixes.py +++ b/nameparser/config/suffixes.py @@ -2,8 +2,8 @@ # nameparser/vocab/*.toml, classified by kind (generation, credential, # honorific, ...) and by language. A suffix reading is an acronym or a # word (`acronym`, true by default for a credential), so the two sets -# below are disjoint by construction; `may_be_name` on an acronym reading -# and `attached` on a word reading make the two marker subsets. +# below are disjoint by construction; a name reading beside an acronym +# reading and `attached` on a word reading make the two marker subsets. # tools/vocab/build.py compiles them into _compiled.py and runs the # checks that were import-time asserts here (decisions/vocabulary-model). from nameparser.config import _compiled diff --git a/nameparser/vocab/README.md b/nameparser/vocab/README.md index 3b57734f..22b493e9 100644 --- a/nameparser/vocab/README.md +++ b/nameparser/vocab/README.md @@ -26,30 +26,31 @@ Each file has `schema = 1` at the top, then one `[[word]]` table per word. An en ```toml [[word]] text = "van" +given = { us_ssa = 31194, wikidata = 282, as_of = 2026-10-25, note = "US SSA births summed over both sexes; Wikidata is a lower bound; nam_dict band 13 in Vietnam, 2 in U.S.A." } [word.preposition] -may_be_given = true expands = "of, from" -note = "Vietnamese Văn, and Van Johnson -- the canonical particle-or-given ambiguity this whole flag exists for" +note = "Vietnamese Văn, and Van Johnson -- the canonical particle-or-given ambiguity this whole marker exists for" ``` -`van`, in `nl.toml`, has one reading, a particle of the kind `preposition`, with one behavioral property (`may_be_given`: the word is also a given name, so `van Gogh` keeps `van` as the given name and reports the fork) and two descriptive ones (`expands` and `note`, compiled to nothing). +`van`, in `nl.toml`, has two readings: a particle of the kind `preposition`, with two descriptive properties (`expands` and `note`, compiled to nothing), and a `given` name reading with the counts it was declared on. The two together make `van` a particle that may be a given name, a [derived marker](#derived-markers): `van Gogh` keeps `van` as the given name and reports the fork. ```toml [[word]] text = "do" -credential = { may_be_name = true, expands = "Doctor of Osteopathic Medicine" } +family = { us_census_2010 = 30301, wikidata = 100, as_of = 2026-10-25 } +given = { us_ssa = 11, wikidata = 9, as_of = 2026-10-25, note = "male births only; Wikidata is a lower bound; nam_dict band 8 in Korea, 6 in Vietnam, 1 in the Netherlands" } +credential = { expands = "Doctor of Osteopathic Medicine" } ``` ```toml [[word]] text = "do" [word.preposition] -may_be_given = true expands = "of the, masc. sg." note = "Vietnamese Đỗ leads a great many surnames" ``` -`do` has two readings in two categories, a credential (an American degree, in `en.toml`) and a particle (a Portuguese preposition, in `pt.toml`), so it is in both `SUFFIX_ACRONYMS` and `PARTICLES`; which one applies is decided by where the word stands. A word's readings are where clashes between categories are recorded: before this format, such a clash was visible only as two sets that happened to overlap. +`do` has readings in three categories: a credential (an American degree, in `en.toml`), a particle (a Portuguese preposition, in `pt.toml`) and two name readings, so it is in both `SUFFIX_ACRONYMS` and `PARTICLES`; which one applies is decided by where the word stands. A word's readings are where clashes between categories are recorded: before this format, such a clash was visible only as two sets that happened to overlap. Its name readings make both clashes with a name: the credential may be a name and the particle may be given, each a [derived marker](#derived-markers), though the name readings sit in only one of its files. ```toml [[word]] @@ -160,7 +161,7 @@ The particle kinds say what the word does in the names of the file's language: - `saint`: "Saint" opening a surname (`san`, `santa`, `st`, `ste`); - `alias`: a word joining two surnames of one person (`vel`). -Unlike the suffix kinds, one word may take a different particle kind in each language: `de` is a preposition in `fr.toml` and an article in `nl.toml`. Check 5 holds every particle reading of a word to the same `may_be_given` whatever its kind. On a particle reading, `expands` is the word's meaning in that language, and `note` is reserved for vetting. A never-given particle may rest on its kind and `expands` alone: one attested bearer settles the may-be-given side, but no amount of searching settles absence, so what is recorded is what the word IS (an article, a preposition) rather than a failed search. And a word with `may_be_given = true` whose particle readings carry no `note` in any file sits on the conservative default and nobody has examined it, and so does one whose note says so, as `heer`'s does. The test is per word, because a word's note sits in one of its files, not in each: `mac` is examined though only `en.toml` holds its note. +Unlike the suffix kinds, one word may take a different particle kind in each language: `de` is a preposition in `fr.toml` and an article in `nl.toml`. Whether a particle may be given is the word's, not the reading's: its `given` reading, in any file, decides it for every particle reading (check 11). On a particle reading, `expands` is the word's meaning in that language, and `note` is reserved for vetting. A never-given particle may rest on its kind and `expands` alone: one attested bearer settles the may-be-given side, but no amount of searching settles absence, so what is recorded is what the word IS (an article, a preposition) rather than a failed search. And a word with a `given` reading whose particle readings carry no `note` in any file sits on the conservative default and nobody has examined it, and so does one whose note says so, as `heer`'s does: counts on the given reading were taken for phase 2's derivation (2026-10-25), not read against the particle position. The test is per word, because a word's note sits in one of its files, not in each: `mac` is examined though only `en.toml` holds its note. **Classifying a category adds kinds to every check that tests it.** While a category has only its generic kind, a check that asks for the category and one that asks for the generic kind give the same answer, so a test row written in the generic kind cannot tell them apart. When a category gets its first fine kinds, give every check on that category a row in `tests/v2/test_vocab.py` written in a fine kind, and run each row against the check keyed on the kind to see it fail. #749's review found four particle checks pinned only on the generic `particle` kind once the shipped data used it for two words: the required `may_be_given`, an exclusion refusing a sibling kind, the bound given name that is a never-given particle, and the elided particle's two spellings. Each passed every test when keyed on the kind, which would have let a `preposition = {}` with no `may_be_given` compile silently into the never-given set. @@ -168,16 +169,15 @@ The `name` kinds compile to nothing. They record that a word is borne as a given ## Properties -A property is **behavioral** when it compiles into a set and so changes how names are parsed, and **descriptive** when it is checked and recorded but compiled to nothing. Each is allowed only on the categories listed; a `name` reading takes its own keys (see [Name readings](#name-readings)). +A property is **behavioral** when it compiles into a set and so changes how names are parsed, **stated** when it compiles to nothing but the build holds it to the word's readings, and **descriptive** when it is checked and recorded but compiled to nothing. Each is allowed only on the categories listed; a `name` reading takes its own keys (see [Name readings](#name-readings)). Two of the sets are not properties at all but [derived markers](#derived-markers), read off a word's name readings. | Property | Type | Applies to | Unwritten | Effect | |---|---|---|---|---| | `addresses` | behavioral | `title` | `"family"` | `"given"` adds the word to `GIVEN_NAME_TITLES`: `Sir John` reads `John` as the given name ([rules/H1](../../docs/design/rules/H1-title-and-one-name-word.md), a title and one name word) | | `after_given` | behavioral | `title` | false | `AFTER_GIVEN_TITLES`, on a `salutation` reading only: written bare at the end of the given part after a family comma, the word is the title, as in `Doe, Jane Mrs` ([rules/H8](../../docs/design/rules/H8-salutation-ending-the-given-part.md), a salutation ending the given part). A salutation borne as a given name or a surname is left out with an exclusion of the property, its counts beside it (`goodman`, `herr`), since a middle name may be either | | `acronym` | behavioral | `suffix` | true for `credential`, false for every other kind | true: `SUFFIX_ACRONYMS`, matched with every period removed (`M.D.` is `md`); false: `SUFFIX_WORDS` | -| `may_be_name` | behavioral | `suffix` | false | `SUFFIX_ACRONYMS_AMBIGUOUS`, on an acronym reading only: an acronym that is also a name (`ma`) is read as a suffix only where its spelling or the name around it says so: `John Smith MA` and `John Smith M.A.` end in a suffix, `John Smith Ma` in a family name ([rules/S4](../../docs/design/rules/S4-ambiguous-acronym-needs-words-to-spare.md), the ambiguous acronym needs words to spare; [rules/S6](../../docs/design/rules/S6-case-signal-at-the-suffix-slot.md), the case signal at the suffix slot) | | `attached` | behavioral | `suffix` | absent | `GLUED_HONORIFICS`, on a word reading only: `{ side = "after" }` splits the word off the end of a name token, as in `田中さん` ([rules/W2](../../docs/design/rules/W2-glued-honorific-peel.md), the glued-honorific peel) | -| `may_be_given` | behavioral | `particle` | none: required | `NON_GIVEN_NAME_PARTICLES` when false: a particle opening the name starts the family name ([rules/P1](../../docs/design/rules/P1-lone-particle-fold.md), the lone particle fold) | +| `may_be_given` | stated | `particle` | none: required where the word has no `given` reading | `false` states that the word is not a given name, which is what puts it in `NON_GIVEN_NAME_PARTICLES`: a particle opening the name starts the family name ([rules/P1](../../docs/design/rules/P1-lone-particle-fold.md), the lone particle fold). It is written exactly where the word has no `given` reading in any file; `true` is refused, naming the `given` reading to write instead (check 11) | | `may_be_initial` | behavioral | `conjunction` | false | `CONJUNCTIONS_AMBIGUOUS`, on a cased single letter only: read as an initial in a name written in one case, as in `JOHN E SMITH` ([rules/P8](../../docs/design/rules/P8-single-letter-connective-against-the-initial-it-spells.md), a single-letter connective against the initial it spells) | | `cased` | behavioral | `title`, `suffix`, `particle`, `conjunction`, `bound_given`, `relation` | the word as written | `CAPITALIZATION_EXCEPTIONS`: the case `capitalized()` gives the word (`PhD`), a mask over its letters and digits, never a replacement (#459) | | `note` | descriptive | every category, a `name` reading included | | free text: why the reading is as it is | @@ -190,9 +190,20 @@ A property is **behavioral** when it compiles into a set and so changes how name Two of these need more than their row: -- **`may_be_given` has no default.** The format before this one listed the never-given particles and treated every other particle as possibly a given name, so a new particle was ambiguous unless argued otherwise ([decisions/vocabulary-collisions](../../docs/design/decisions/vocabulary-collisions.md), C-i). A default here would pick one side of that silently, so the build refuses a particle reading that does not state it. When unsure, write `true`. +- **`may_be_given` has no default.** The format before this one listed the never-given particles and treated every other particle as possibly a given name, so a new particle was ambiguous unless argued otherwise ([decisions/vocabulary-collisions](../../docs/design/decisions/vocabulary-collisions.md), C-i). A default here would pick one side of that silently, so the build refuses a particle reading that neither states `may_be_given = false` nor has a `given` reading on its word. When unsure, write the `given` reading, with a `note` if no source counts it. - **`cased` belongs to the word**, not the reading: one mask per word, whichever reading carries it. +### Derived markers + +Two sets mark a clash with a name, and both are derived: a word is in one when it carries the clashing reading and a name reading, in any of its files (check 11, since 2026-10-25, [decisions/vocabulary-model](../../docs/design/decisions/vocabulary-model.md)). Phase 1 wrote them as flags, `may_be_name` and `may_be_given = true`, which the build now refuses, naming the reading to write. + +| Marker | Readings | Compiled into | Effect | +|---|---|---|---| +| possibly a name | an acronym `suffix` reading and a `given` or `family` reading | `SUFFIX_ACRONYMS_AMBIGUOUS` | an acronym that is also a name (`ma`) is read as a suffix only where its spelling or the name around it says so: `John Smith MA` and `John Smith M.A.` end in a suffix, `John Smith Ma` in a family name ([rules/S4](../../docs/design/rules/S4-ambiguous-acronym-needs-words-to-spare.md), the ambiguous acronym needs words to spare; [rules/S6](../../docs/design/rules/S6-case-signal-at-the-suffix-slot.md), the case signal at the suffix slot) | +| possibly given | a `particle` reading and a `given` reading | kept out of `NON_GIVEN_NAME_PARTICLES` | a particle opening the name stays a given name there (`van Gogh`, given `van`) and the fork is reported, rather than starting the family name ([rules/P1](../../docs/design/rules/P1-lone-particle-fold.md), the lone particle fold) | + +A word reading of the suffix category (`jr`, `generation`) beside a name reading is no clash the parser weighs and marks nothing. The marker is the clash, not the counts: a count recorded for a word that should stay unmarked goes in the clashing reading's `note`, as `ld`'s does, since a name reading would mark it. `may_be_initial` is not derived: its clash is with the initial, a shape every cased single letter has, and which letters a tradition writes both ways is a judgment about the letter, not a name anyone bears. + ### Name readings A `given` or `family` reading takes its own keys: @@ -207,6 +218,8 @@ A `given` or `family` reading takes its own keys: | `note` | free text | | `gender` | `"f"`, `"m"` or `"n"` | +A name reading may stand without a count, as a particle kept possibly given under C-i's default with no source counting it does, but it then needs a `note` saying why it is declared (check 3). + ## Exclusions An exclusion records that a reading was considered and refused, with the reason, so that the next person to propose it meets the reason as a build error instead of rediscovering the evidence. @@ -227,7 +240,7 @@ Record a count on the word as a `given` or `family` reading, with the date you t ## Adding a word 1. **Look up the word's other readings.** `uv run python tools/vocab/query.py --word ` lists every reading and exclusion of it in every file. An exclusion there is a decision already made; a reading in another category is a clash you are about to create, which is fine when position settles it (`do`) and needs an argument when it does not. -2. **Check it against real names, not memory.** `uv run python tools/name_usage.py ` gives the counts (add `--fetch-nppes` for a suffix, to see how often it is written as a credential), and [decisions/vocabulary-collisions](../../docs/design/decisions/vocabulary-collisions.md) is the criterion they are read against: a word that is a real name in the position the reading acts on needs the matching flag (`may_be_given`, `may_be_name`) or should stay out. The root `AGENTS.md`'s "Adding a word to the vocabulary" lists the traps that have bitten before. +2. **Check it against real names, not memory.** `uv run python tools/name_usage.py ` gives the counts (add `--fetch-nppes` for a suffix, to see how often it is written as a credential), and [decisions/vocabulary-collisions](../../docs/design/decisions/vocabulary-collisions.md) is the criterion they are read against: a word that is a real name in the position the reading acts on needs a name reading with those counts, which marks the clash (see [Derived markers](#derived-markers)), or should stay out. The root `AGENTS.md`'s "Adding a word to the vocabulary" lists the traps that have bitten before. 3. **Check how it is written.** The entry's `text` is the folded spelling (`mbchb`), and `capitalized()` writes an acronym credential in capitals unless the word has a `cased` mask, so one that people write in mixed case (`MBChB`, `DPhil`, `ScD`) needs the mask: `cased = "MBChB"`. Nothing in the build can see this, since no source it reads records case. The ClinicalTrials.gov row of `name_usage.py` lists the spellings as typed ("written MBChB 1,442, MBCHB 72, MbChB 28"), once that local corpus is fetched (`uv run python tools/differential/local_corpora.py fetch ctgov-officials`). Where a mask would cost a name its rendering, refuse it instead, with the reason, as `edd` does (`'edd smith'` would render `'EdD Smith'`). #806 added twelve mixed-case credentials without their masks, and review caught it. 4. **Write the entry** in the file of the language whose names it occurs in (`und.toml` when that is not known), in its place in the [word order](#word-order), with what the checks found on it: `note`, `issue`, a dated name reading, a `cased` mask, or an `excluded` entry if you decided against it. 5. **Rebuild**: `uv run python tools/vocab/build.py`. It refuses what it can detect, naming the word and, where one file is at fault, the file, and rewrites `nameparser/config/_compiled.py`. Commit both. diff --git a/nameparser/vocab/ar.toml b/nameparser/vocab/ar.toml index 9eb25eab..3cc74dd0 100644 --- a/nameparser/vocab/ar.toml +++ b/nameparser/vocab/ar.toml @@ -158,8 +158,8 @@ generation = { issue = 679 } [[word]] text = "al" +given = { us_ssa = 17334, wikidata = 1602, as_of = 2026-10-25, note = "US SSA births summed over both sexes; Wikidata is a lower bound; nam_dict band 2 in Great Britain, 2 in U.S.A., 2 in Arabia/Persia, 1 in the Netherlands" } [word.article] -may_be_given = true expands = "the" note = "Al Gore: 'Al' is an ordinary English given name. Also the Arabic definite article, which is the particle reading" @@ -396,8 +396,8 @@ reason = "Ben is a common given name standing exactly where a forward-joining pa [[word]] text = "bin" +given = { us_ssa = 11, wikidata = 541, as_of = 2026-10-25, note = "male births only; Wikidata is a lower bound; nam_dict band 8 in China, 6 in Korea, 1 in Japan" } [word.patronymic] -may_be_given = true expands = "son of" gender = "m" note = "Kept ambiguous deliberately: #269 judged the Latin transliteration separately from the native-script بن, which IS never-given" @@ -439,8 +439,8 @@ note = "never a bare given name, like بن; mirrors Latin 'bint'" [[word]] text = "abu" +given = { us_ssa = 468, wikidata = 51, as_of = 2026-10-25, note = "male births only; Wikidata is a lower bound; nam_dict band 3 in Arabia/Persia" } [word.teknonymic] -may_be_given = true issue = 269 expands = "father of" gender = "m" @@ -449,8 +449,8 @@ note = "\"Abu Bakr\" -- Abu reads as the given name; also a bound given-name wor [[word]] text = "أبو" +given = { wikidata = 48, as_of = 2026-10-25, note = "Wikidata is a lower bound" } [word.teknonymic] -may_be_given = true issue = 269 expands = "father of: abu, hamza spelling" gender = "m" @@ -460,8 +460,8 @@ issue = 269 [[word]] text = "ابو" +given = { wikidata = 33, as_of = 2026-10-25, note = "Wikidata is a lower bound" } [word.teknonymic] -may_be_given = true issue = 269 expands = "father of: abu, hamza-less spelling" same_as = "أبو" diff --git a/nameparser/vocab/de.toml b/nameparser/vocab/de.toml index 2b197702..b58a7276 100644 --- a/nameparser/vocab/de.toml +++ b/nameparser/vocab/de.toml @@ -82,9 +82,9 @@ nobility = { gender = "f", expands = "baroness" } [[word]] text = "freiherr" +given = { note = "carries the particle's may-be-given reading, the C-i default (the particle's note argues the chain, not a given name); no source counts it as a given name (2026-10-25)" } nobility = { gender = "m", expands = "baron" } [word.rank] -may_be_given = true issue = 18 expands = "baron" gender = "m" @@ -156,7 +156,8 @@ preposition = { may_be_given = false, issue = 18, expands = "upon: \"auf der Hei [[word]] text = "tho" -preposition = { may_be_given = true, issue = 18, expands = "to, Low German zu" } +given = { us_ssa = 52, as_of = 2026-10-25, note = "US SSA births summed over both sexes; nam_dict band 8 in Vietnam" } +preposition = { issue = 18, expands = "to, Low German zu" } [[word]] text = "vom" @@ -164,8 +165,8 @@ preposition = { may_be_given = false, expands = "from the, von dem contracted" } [[word]] text = "von" +given = { us_ssa = 6011, wikidata = 46, as_of = 2026-10-25, note = "US SSA births summed over both sexes; Wikidata is a lower bound" } [word.preposition] -may_be_given = true expands = "of, from" note = "Von Miller: a real given name, so this stays ambiguous even though \"von Braun\" is misparsed today. The case that shows a word can look like a pure particle, BE misparsed as one, and still have a bearer" @@ -179,7 +180,8 @@ article = { may_be_given = false, issue = 18, expands = "the, dative" } [[word]] text = "den" -article = { may_be_given = true, expands = "the: \"von den Steinen\"" } +given = { us_ssa = 17, wikidata = 5, as_of = 2026-10-25, note = "male births only; Wikidata is a lower bound" } +article = { expands = "the: \"von den Steinen\"" } [[word]] text = "der" @@ -187,8 +189,8 @@ article = { may_be_given = false, expands = "the: \"von der Leyen\"" } [[word]] text = "freiherrin" +given = { note = "carries the particle's may-be-given reading, as freiherr does; no source counts it as a given name (2026-10-25)" } [word.rank] -may_be_given = true issue = 18 expands = "baroness" gender = "f" diff --git a/nameparser/vocab/en.toml b/nameparser/vocab/en.toml index f6d3f8d9..c328da1c 100644 --- a/nameparser/vocab/en.toml +++ b/nameparser/vocab/en.toml @@ -56,7 +56,7 @@ salutation = { gender = "f", after_given = true } text = "ms" salutation = { gender = "f" } [word.credential] -note = "NOT may_be_name, though #296's audit table put it there for the leading-title collision: the gate is position-blind and the collision is not (gated, 'John Smith, MS' read title 'MS'); a title/credential dual that position decides (decisions/C8)" +note = "NOT marked as possibly a name (no name reading), though #296's audit table put it there for the leading-title collision: the gate is position-blind and the collision is not (gated, 'John Smith, MS' read title 'MS'); a title/credential dual that position decides (decisions/C8)" [[word]] text = "mx" @@ -300,9 +300,9 @@ religious = { addresses = "given", gender = "f", expands = "a nun or religious s [[word]] text = "st" +given = { us_ssa = 24, as_of = 2026-10-25, note = "male births only" } religious = { expands = "Saint, abbreviated" } [word.saint] -may_be_given = true expands = "Saint, abbreviated" note = "Also in TITLES and consumed there before particle logic runs, so moving it between the may-be-given and never-given halves changes nothing; kept for the load-bearing TITLES intersection" @@ -2295,7 +2295,7 @@ credential = {} text = "aba" family = { us_census_2010 = 231, wikidata = 18, as_of = 2026-10-21 } given = { us_ssa = 32, wikidata = 40, as_of = 2026-10-21, note = "US SSA births summed over both sexes; Wikidata is a lower bound" } -credential = { may_be_name = true, note = "NPPES 183 US healthcare providers write it (healthcare only, August 2026 file). Marked on C-i: the surname reading is substantial (Census 2010 or Wikidata).", issue = 796 } +credential = { note = "NPPES 183 US healthcare providers write it (healthcare only, August 2026 file). Marked on C-i: the surname reading is substantial (Census 2010 or Wikidata).", issue = 796 } [[word]] text = "abc" @@ -2318,7 +2318,7 @@ credential = {} [[word]] text = "aca" family = { us_census_2010 = 375, as_of = 2026-10-21 } -credential = { may_be_name = true, note = "NPPES 141 US healthcare providers write it (healthcare only, August 2026 file). Marked on C-i: the surname reading is substantial (Census 2010 or Wikidata).", issue = 796 } +credential = { note = "NPPES 141 US healthcare providers write it (healthcare only, August 2026 file). Marked on C-i: the surname reading is substantial (Census 2010 or Wikidata).", issue = 796 } [[word]] text = "acas" @@ -2328,12 +2328,12 @@ credential = {} text = "ace" family = { us_census_2010 = 1199, wikidata = 8, as_of = 2026-10-21 } given = { us_ssa = 26483, wikidata = 45, as_of = 2026-10-21, note = "US SSA births summed over both sexes; Wikidata is a lower bound" } -credential = { may_be_name = true, note = "NPPES 9 US healthcare providers write it (healthcare only, August 2026 file). Marked on C-i: the surname reading is substantial (Census 2010 or Wikidata).", issue = 796 } +credential = { note = "NPPES 9 US healthcare providers write it (healthcare only, August 2026 file). Marked on C-i: the surname reading is substantial (Census 2010 or Wikidata).", issue = 796 } [[word]] text = "acha" family = { us_census_2010 = 477, wikidata = 26, as_of = 2026-10-21 } -credential = { may_be_name = true, note = "NPPES 1 US healthcare provider writes it (healthcare only, August 2026 file). Marked on C-i: the surname reading is substantial (Census 2010 or Wikidata).", issue = 796 } +credential = { note = "NPPES 1 US healthcare provider writes it (healthcare only, August 2026 file). Marked on C-i: the surname reading is substantial (Census 2010 or Wikidata).", issue = 796 } [[word]] text = "acp" @@ -2379,7 +2379,7 @@ credential = {} text = "ala" family = { us_census_2010 = 743, wikidata = 14, as_of = 2026-10-21 } given = { us_ssa = 943, wikidata = 163, as_of = 2026-10-21, note = "US SSA births summed over both sexes; Wikidata is a lower bound" } -credential = { may_be_name = true, note = "NPPES 3 US healthcare providers write it (healthcare only, August 2026 file). Marked on C-i: the surname reading is substantial (Census 2010 or Wikidata).", issue = 796 } +credential = { note = "NPPES 3 US healthcare providers write it (healthcare only, August 2026 file). Marked on C-i: the surname reading is substantial (Census 2010 or Wikidata).", issue = 796 } [[word]] text = "alc" @@ -2389,7 +2389,7 @@ credential = {} text = "alp" family = { us_census_2010 = 116, wikidata = 55, as_of = 2026-10-21 } given = { us_ssa = 467, wikidata = 75, as_of = 2026-10-21, note = "US SSA births summed over both sexes; Wikidata is a lower bound" } -credential = { may_be_name = true, note = "NPPES 0 US healthcare providers write it (healthcare only, August 2026 file). Marked on C-i: the surname reading is substantial (Census 2010 or Wikidata).", issue = 796 } +credential = { note = "NPPES 0 US healthcare providers write it (healthcare only, August 2026 file). Marked on C-i: the surname reading is substantial (Census 2010 or Wikidata).", issue = 796 } [[word]] text = "am" @@ -2410,7 +2410,7 @@ credential = {} [[word]] text = "ams" family = { us_census_2010 = 135, wikidata = 2, as_of = 2026-10-21 } -credential = { may_be_name = true, note = "NPPES 5 US healthcare providers write it (healthcare only, August 2026 file). Surname reading is the larger of the two small counts (#796's revised rule).", issue = 796 } +credential = { note = "NPPES 5 US healthcare providers write it (healthcare only, August 2026 file). Surname reading is the larger of the two small counts (#796's revised rule).", issue = 796 } [[word]] text = "aphr" @@ -2443,7 +2443,7 @@ credential = {} [[word]] text = "arm" family = { us_census_2010 = 283, wikidata = 8, as_of = 2026-10-21 } -credential = { may_be_name = true, note = "NPPES 1 US healthcare provider writes it (healthcare only, August 2026 file). Marked on C-i: the surname reading is substantial (Census 2010 or Wikidata).", issue = 796 } +credential = { note = "NPPES 1 US healthcare provider writes it (healthcare only, August 2026 file). Marked on C-i: the surname reading is substantial (Census 2010 or Wikidata).", issue = 796 } [[word]] text = "arrc" @@ -2453,7 +2453,7 @@ credential = {} text = "asa" family = { us_census_2010 = 508, wikidata = 5, as_of = 2026-10-21 } given = { us_ssa = 23067, wikidata = 283, as_of = 2026-10-21, note = "US SSA births summed over both sexes; Wikidata is a lower bound" } -credential = { may_be_name = true, note = "NPPES 3 US healthcare providers write it (healthcare only, August 2026 file). Marked on C-i: the surname reading is substantial (Census 2010 or Wikidata).", issue = 796 } +credential = { note = "NPPES 3 US healthcare providers write it (healthcare only, August 2026 file). Marked on C-i: the surname reading is substantial (Census 2010 or Wikidata).", issue = 796 } [[word]] text = "asc" @@ -2470,7 +2470,7 @@ credential = {} [[word]] text = "asp" family = { us_census_2010 = 733, wikidata = 63, as_of = 2026-10-21 } -credential = { may_be_name = true, note = "NPPES 1 US healthcare provider writes it (healthcare only, August 2026 file). Marked on C-i: the surname reading is substantial (Census 2010 or Wikidata).", issue = 796 } +credential = { note = "NPPES 1 US healthcare provider writes it (healthcare only, August 2026 file). Marked on C-i: the surname reading is substantial (Census 2010 or Wikidata).", issue = 796 } [[word]] text = "atc" @@ -2482,8 +2482,9 @@ credential = {} [[word]] text = "ba" +family = { us_census_2010 = 2451, wikidata = 117, as_of = 2026-10-25 } +given = { us_ssa = 5, wikidata = 59, as_of = 2026-10-25, note = "male births only; Wikidata is a lower bound; nam_dict band 9 in Vietnam" } [word.credential] -may_be_name = true expands = "Bachelor of Arts" issue = 342 note = "Ba is a real surname (Vietnamese; Senegalese Fula) about as common as BA is a credential, the ma/Ma shape" @@ -2507,7 +2508,7 @@ credential = {} [[word]] text = "bem" family = { us_census_2010 = 339, wikidata = 29, as_of = 2026-10-21 } -credential = { may_be_name = true, note = "NPPES 0 US healthcare providers write it (healthcare only, August 2026 file). Marked on C-i: the surname reading is substantial (Census 2010 or Wikidata).", issue = 796 } +credential = { note = "NPPES 0 US healthcare providers write it (healthcare only, August 2026 file). Marked on C-i: the surname reading is substantial (Census 2010 or Wikidata).", issue = 796 } [[word]] text = "bls-i" @@ -2560,7 +2561,7 @@ credential = {} [[word]] text = "caha" family = { us_census_2010 = 116, wikidata = 28, as_of = 2026-10-21 } -credential = { may_be_name = true, note = "NPPES 1 US healthcare provider writes it (healthcare only, August 2026 file). Marked on C-i: the surname reading is substantial (Census 2010 or Wikidata).", issue = 796 } +credential = { note = "NPPES 1 US healthcare provider writes it (healthcare only, August 2026 file). Marked on C-i: the surname reading is substantial (Census 2010 or Wikidata).", issue = 796 } [[word]] text = "caia" @@ -2577,7 +2578,7 @@ credential = {} [[word]] text = "capa" family = { us_census_2010 = 175, wikidata = 7, as_of = 2026-10-21 } -credential = { may_be_name = true, note = "NPPES 11 US healthcare providers write it (healthcare only, August 2026 file). Surname reading is the larger of the two small counts (#796's revised rule).", issue = 796 } +credential = { note = "NPPES 11 US healthcare providers write it (healthcare only, August 2026 file). Surname reading is the larger of the two small counts (#796's revised rule).", issue = 796 } [[word]] text = "capm" @@ -2587,7 +2588,7 @@ credential = {} text = "capp" family = { us_census_2010 = 1588, wikidata = 13, as_of = 2026-10-21 } given = { us_ssa = 5, as_of = 2026-10-21, note = "US SSA births summed over both sexes; Wikidata is a lower bound" } -credential = { may_be_name = true, note = "NPPES 7 US healthcare providers write it (healthcare only, August 2026 file). Marked on C-i: the surname reading is substantial (Census 2010 or Wikidata).", issue = 796 } +credential = { note = "NPPES 7 US healthcare providers write it (healthcare only, August 2026 file). Marked on C-i: the surname reading is substantial (Census 2010 or Wikidata).", issue = 796 } [[word]] text = "caps" @@ -2597,7 +2598,7 @@ credential = {} text = "caro" family = { us_census_2010 = 10064, wikidata = 236, as_of = 2026-10-21 } given = { us_ssa = 274, wikidata = 50, as_of = 2026-10-21, note = "US SSA births summed over both sexes; Wikidata is a lower bound" } -credential = { may_be_name = true, note = "NPPES 0 US healthcare providers write it (healthcare only, August 2026 file). Marked on C-i: the surname reading is substantial (Census 2010 or Wikidata).", issue = 796 } +credential = { note = "NPPES 0 US healthcare providers write it (healthcare only, August 2026 file). Marked on C-i: the surname reading is substantial (Census 2010 or Wikidata).", issue = 796 } [[word]] text = "cas" @@ -2766,7 +2767,7 @@ credential = {} [[word]] text = "cea" family = { us_census_2010 = 1275, wikidata = 21, as_of = 2026-10-21 } -credential = { may_be_name = true, note = "NPPES 13 US healthcare providers write it (healthcare only, August 2026 file). Marked on C-i: the surname reading is substantial (Census 2010 or Wikidata).", issue = 796 } +credential = { note = "NPPES 13 US healthcare providers write it (healthcare only, August 2026 file). Marked on C-i: the surname reading is substantial (Census 2010 or Wikidata).", issue = 796 } [[word]] text = "ceas" @@ -2788,7 +2789,7 @@ credential = {} text = "cela" family = { us_census_2010 = 637, wikidata = 18, as_of = 2026-10-21 } given = { us_ssa = 90, as_of = 2026-10-21, note = "US SSA births summed over both sexes; Wikidata is a lower bound" } -credential = { may_be_name = true, note = "NPPES 1 US healthcare provider writes it (healthcare only, August 2026 file). Marked on C-i: the surname reading is substantial (Census 2010 or Wikidata).", issue = 796 } +credential = { note = "NPPES 1 US healthcare provider writes it (healthcare only, August 2026 file). Marked on C-i: the surname reading is substantial (Census 2010 or Wikidata).", issue = 796 } [[word]] text = "cem" @@ -2802,7 +2803,7 @@ credential = {} text = "cera" family = { us_census_2010 = 1224, wikidata = 18, as_of = 2026-10-21 } given = { us_ssa = 1410, as_of = 2026-10-21, note = "US SSA births summed over both sexes; Wikidata is a lower bound" } -credential = { may_be_name = true, note = "NPPES 0 US healthcare providers write it (healthcare only, August 2026 file). Marked on C-i: the surname reading is substantial (Census 2010 or Wikidata).", issue = 796 } +credential = { note = "NPPES 0 US healthcare providers write it (healthcare only, August 2026 file). Marked on C-i: the surname reading is substantial (Census 2010 or Wikidata).", issue = 796 } [[word]] text = "cet" @@ -2930,7 +2931,7 @@ credential = {} text = "che" family = { us_census_2010 = 2393, wikidata = 5936, as_of = 2026-10-21 } given = { us_ssa = 2469, wikidata = 48, as_of = 2026-10-21, note = "US SSA births summed over both sexes; Wikidata is a lower bound" } -credential = { may_be_name = true, note = "NPPES 16 US healthcare providers write it (healthcare only, August 2026 file). Marked on C-i: the surname reading is substantial (Census 2010 or Wikidata).", issue = 796 } +credential = { note = "NPPES 16 US healthcare providers write it (healthcare only, August 2026 file). Marked on C-i: the surname reading is substantial (Census 2010 or Wikidata).", issue = 796 } [[word]] text = "ches" @@ -2945,7 +2946,6 @@ text = "chi" family = { us_census_2010 = 7273, wikidata = 219, as_of = 2026-10-21 } given = { us_ssa = 1326, wikidata = 123, as_of = 2026-10-21, note = "US SSA births summed over both sexes; Wikidata is a lower bound" } [word.credential] -may_be_name = true issue = 791 note = "Chi is a common surname (Census 2010: 7,273 family-name bearers, rank 4,843; Wikidata 219+ family) and a Vietnamese given name; measured 2026-10-21 by tools/name_usage.py. No expansion is recorded: the 2019 bulk import (af5bdab, #93) carried none" @@ -3037,7 +3037,7 @@ credential = {} text = "ciro" family = { us_census_2010 = 265, wikidata = 6, as_of = 2026-10-21 } given = { us_ssa = 2587, wikidata = 409, as_of = 2026-10-21, note = "US SSA births summed over both sexes; Wikidata is a lower bound" } -credential = { may_be_name = true, note = "NPPES 0 US healthcare providers write it (healthcare only, August 2026 file). Marked on C-i: the surname reading is substantial (Census 2010 or Wikidata).", issue = 796 } +credential = { note = "NPPES 0 US healthcare providers write it (healthcare only, August 2026 file). Marked on C-i: the surname reading is substantial (Census 2010 or Wikidata).", issue = 796 } [[word]] text = "cisa" @@ -3290,7 +3290,7 @@ credential = {} [[word]] text = "csi" family = { wikidata = 590, as_of = 2026-10-21 } -credential = { may_be_name = true, note = "NPPES 95 US healthcare providers write it (healthcare only, August 2026 file). Surname evidence is Wikidata only (no Census bearers); marked on C-i.", issue = 796 } +credential = { note = "NPPES 95 US healthcare providers write it (healthcare only, August 2026 file). Surname evidence is Wikidata only (no Census bearers); marked on C-i.", issue = 796 } [[word]] text = "csp" @@ -3347,7 +3347,7 @@ credential = {} [[word]] text = "cua" family = { us_census_2010 = 1082, wikidata = 3, as_of = 2026-10-21 } -credential = { may_be_name = true, note = "NPPES 2 US healthcare providers write it (healthcare only, August 2026 file). Marked on C-i: the surname reading is substantial (Census 2010 or Wikidata).", issue = 796 } +credential = { note = "NPPES 2 US healthcare providers write it (healthcare only, August 2026 file). Marked on C-i: the surname reading is substantial (Census 2010 or Wikidata).", issue = 796 } [[word]] text = "cusp" @@ -3517,7 +3517,9 @@ credential = {} [[word]] text = "do" -credential = { may_be_name = true, expands = "Doctor of Osteopathic Medicine" } +family = { us_census_2010 = 30301, wikidata = 100, as_of = 2026-10-25 } +given = { us_ssa = 11, wikidata = 9, as_of = 2026-10-25, note = "male births only; Wikidata is a lower bound; nam_dict band 8 in Korea, 6 in Vietnam, 1 in the Netherlands" } +credential = { expands = "Doctor of Osteopathic Medicine" } [[word]] text = "dphil" @@ -3575,11 +3577,13 @@ credential = {} text = "ea" family = { us_census_2010 = 496, wikidata = 2, as_of = 2026-10-21 } given = { us_ssa = 11, wikidata = 11, as_of = 2026-10-21, note = "US SSA births summed over both sexes; Wikidata is a lower bound" } -credential = { may_be_name = true, note = "NPPES 2 US healthcare providers write it (healthcare only, August 2026 file). Marked on C-i: the surname reading is substantial (Census 2010 or Wikidata).", issue = 796 } +credential = { note = "NPPES 2 US healthcare providers write it (healthcare only, August 2026 file). Marked on C-i: the surname reading is substantial (Census 2010 or Wikidata).", issue = 796 } [[word]] text = "ed" -credential = { may_be_name = true, expands = "Doctor of Education" } +family = { us_census_2010 = 262, wikidata = 7, as_of = 2026-10-25 } +given = { us_ssa = 26370, wikidata = 2924, as_of = 2026-10-25, note = "US SSA births summed over both sexes; Wikidata is a lower bound; nam_dict band 6 in the Netherlands, 3 in U.S.A., 2 in Great Britain" } +credential = { expands = "Doctor of Education" } [[word]] text = "edd" @@ -3591,7 +3595,7 @@ cased = { reason = "a mask applies in every role, and Edd is borne as a given na text = "ei" family = { us_census_2010 = 127, wikidata = 3, as_of = 2026-10-21 } given = { wikidata = 4, as_of = 2026-10-21 } -credential = { may_be_name = true, note = "NPPES 12 US healthcare providers write it (healthcare only, August 2026 file). Not a healthcare credential (usually Engineer Intern): NPPES says nothing about it; marked under C-i's default under uncertainty.", issue = 796 } +credential = { note = "NPPES 12 US healthcare providers write it (healthcare only, August 2026 file). Not a healthcare credential (usually Engineer Intern): NPPES says nothing about it; marked under C-i's default under uncertainty.", issue = 796 } [[word]] text = "eit" @@ -3601,7 +3605,7 @@ credential = {} text = "els" family = { us_census_2010 = 362, wikidata = 9, as_of = 2026-10-21 } given = { wikidata = 322, as_of = 2026-10-21 } -credential = { may_be_name = true, note = "NPPES 1 US healthcare provider writes it (healthcare only, August 2026 file). Marked on C-i: the surname reading is substantial (Census 2010 or Wikidata).", issue = 796 } +credential = { note = "NPPES 1 US healthcare provider writes it (healthcare only, August 2026 file). Marked on C-i: the surname reading is substantial (Census 2010 or Wikidata).", issue = 796 } [[word]] text = "emd" @@ -3630,7 +3634,7 @@ credential = {} [[word]] text = "erd" family = { us_census_2010 = 222, wikidata = 1, as_of = 2026-10-21 } -credential = { may_be_name = true, note = "NPPES 0 US healthcare providers write it (healthcare only, August 2026 file). Marked on C-i: the surname reading is substantial (Census 2010 or Wikidata).", issue = 796 } +credential = { note = "NPPES 0 US healthcare providers write it (healthcare only, August 2026 file). Marked on C-i: the surname reading is substantial (Census 2010 or Wikidata).", issue = 796 } [[word]] text = "evp" @@ -3695,7 +3699,7 @@ credential = {} [[word]] text = "faia" family = { us_census_2010 = 271, wikidata = 7, as_of = 2026-10-21 } -credential = { may_be_name = true, note = "NPPES 0 US healthcare providers write it (healthcare only, August 2026 file). Marked on C-i: the surname reading is substantial (Census 2010 or Wikidata).", issue = 796 } +credential = { note = "NPPES 0 US healthcare providers write it (healthcare only, August 2026 file). Marked on C-i: the surname reading is substantial (Census 2010 or Wikidata).", issue = 796 } [[word]] text = "faicp" @@ -3752,7 +3756,7 @@ credential = {} [[word]] text = "fic" family = { wikidata = 22, as_of = 2026-10-21 } -credential = { may_be_name = true, note = "NPPES 1 US healthcare provider writes it (healthcare only, August 2026 file). Surname evidence is Wikidata only (no Census bearers); marked on C-i.", issue = 796 } +credential = { note = "NPPES 1 US healthcare provider writes it (healthcare only, August 2026 file). Surname evidence is Wikidata only (no Census bearers); marked on C-i.", issue = 796 } [[word]] text = "ficf" @@ -3946,7 +3950,8 @@ credential = {} [[word]] text = "jd" -credential = { may_be_name = true, expands = "Juris Doctor" } +given = { us_ssa = 1449, as_of = 2026-10-25, note = "US SSA births summed over both sexes" } +credential = { expands = "Juris Doctor" } [[word]] text = "jp" @@ -4002,8 +4007,8 @@ credential = { cased = "Kt" } [[word]] text = "lac" +family = { us_census_2010 = 696, wikidata = 3, as_of = 2026-10-25 } [word.credential] -may_be_name = true expands = "Licensed Acupuncturist" issue = 540 note = "Lac is a Vietnamese given name (the trailing word in native order, Nguyen Van Lac) and a French surname" @@ -4032,8 +4037,7 @@ credential = { note = "NPPES 3,023 US healthcare providers (August 2026 file); 0 [[word]] text = "ld" -given = { us_ssa = 45, as_of = 2026-10-21, note = "US SSA births summed over both sexes; Wikidata is a lower bound" } -credential = { note = "NPPES 11,703 US healthcare providers (August 2026 file); 45 ClinicalTrials.gov study officials after a comma, as typed (data 2026-10-21) write it. Not marked: 45 SSA births against 11,703 providers, the credential reading far the larger (C-i, 2026-10-21 rule)", issue = 797 } +credential = { note = "NPPES 11,703 US healthcare providers (August 2026 file); 45 ClinicalTrials.gov study officials after a comma, as typed (data 2026-10-21) write it. Not marked: 45 US SSA births (both sexes summed, 2026-10-21) against 11,703 providers, the credential reading far the larger (C-i, 2026-10-21 rule). The given count stands here rather than as a given reading, which would mark the word (decisions/vocabulary-model, 2026-10-25)", issue = 797 } [[word]] text = "lg" @@ -4113,7 +4117,9 @@ credential = {} [[word]] text = "ma" -credential = { may_be_name = true, expands = "Master of Arts" } +family = { us_census_2010 = 28452, wikidata = 4996, as_of = 2026-10-25 } +given = { us_ssa = 631, wikidata = 17, as_of = 2026-10-25, note = "female births only; Wikidata is a lower bound; nam_dict band 1 in the Netherlands, 1 in China, 1 in Korea, 1 in Vietnam" } +credential = { expands = "Master of Arts" } [[word]] text = "maaa" @@ -4158,7 +4164,7 @@ expands = "son of, Mac contracted" same_as = "mac" note = "Unlike 'mac' (Mac Miller) it is not a standalone given name. Measured misparse: \"Mc Donald\" gave given='Mc'. Also in SUFFIX_ACRONYMS as Master of Ceremonies, but that is trailing position and unaffected -- \"John Smith MC\" still reads suffix (#360)." [word.credential] -note = "NOT may_be_name though also a particle: #454 closed by design, neither mc nor vd is a borne name (rules/P6's Accepted clause)" +note = "NOT marked as possibly a name (no name reading) though also a particle: #454 closed by design, neither mc nor vd is a borne name (rules/P6's Accepted clause)" [[word]] text = "mcct" @@ -4219,8 +4225,9 @@ credential = {} [[word]] text = "meng" +family = { us_census_2010 = 5729, wikidata = 1319, as_of = 2026-10-25 } +given = { us_ssa = 568, wikidata = 23, as_of = 2026-10-25, note = "US SSA births summed over both sexes; Wikidata is a lower bound; nam_dict band 6 in China" } [word.credential] -may_be_name = true expands = "Master of Engineering" issue = 540 note = "Meng is a Chinese surname and given name" @@ -4447,13 +4454,13 @@ credential = {} text = "obi" family = { us_census_2010 = 1403, wikidata = 28, as_of = 2026-10-21 } given = { us_ssa = 299, wikidata = 17, as_of = 2026-10-21, note = "US SSA births summed over both sexes; Wikidata is a lower bound" } -credential = { may_be_name = true, note = "NPPES 0 US healthcare providers write it (healthcare only, August 2026 file). Marked on C-i: the surname reading is substantial (Census 2010 or Wikidata).", issue = 796 } +credential = { note = "NPPES 0 US healthcare providers write it (healthcare only, August 2026 file). Marked on C-i: the surname reading is substantial (Census 2010 or Wikidata).", issue = 796 } [[word]] text = "oca" family = { us_census_2010 = 344, wikidata = 2, as_of = 2026-10-21 } given = { us_ssa = 16, as_of = 2026-10-21, note = "US SSA births summed over both sexes; Wikidata is a lower bound" } -credential = { may_be_name = true, note = "NPPES 0 US healthcare providers write it (healthcare only, August 2026 file). Marked on C-i: the surname reading is substantial (Census 2010 or Wikidata).", issue = 796 } +credential = { note = "NPPES 0 US healthcare providers write it (healthcare only, August 2026 file). Marked on C-i: the surname reading is substantial (Census 2010 or Wikidata).", issue = 796 } [[word]] text = "ocm" @@ -4471,7 +4478,7 @@ credential = {} text = "om" family = { us_census_2010 = 784, wikidata = 135, as_of = 2026-10-21 } given = { us_ssa = 2321, wikidata = 10, as_of = 2026-10-21, note = "US SSA births summed over both sexes; Wikidata is a lower bound" } -credential = { may_be_name = true, note = "NPPES 66 US healthcare providers write it (healthcare only, August 2026 file). Marked on C-i: the surname reading is substantial (Census 2010 or Wikidata).", issue = 796 } +credential = { note = "NPPES 66 US healthcare providers write it (healthcare only, August 2026 file). Marked on C-i: the surname reading is substantial (Census 2010 or Wikidata).", issue = 796 } [[word]] text = "oscp" @@ -4489,7 +4496,7 @@ credential = { expands = "Occupational Therapist Registered, Licensed", note = " text = "pa" family = { us_census_2010 = 739, as_of = 2026-10-21 } given = { us_ssa = 1343, wikidata = 36, as_of = 2026-10-21, note = "US SSA births summed over both sexes, almost all female; Wikidata is a lower bound" } -credential = { may_be_name = true, expands = "Physician Assistant", note = "NPPES 54,128 US healthcare providers (August 2026 file); 23 ClinicalTrials.gov study officials after a comma, as typed (data 2026-10-21) write it. Marked on C-i: the surname reading is substantial (Census 2010 bearers 739, over the 200 threshold)", issue = 797 } +credential = { expands = "Physician Assistant", note = "NPPES 54,128 US healthcare providers (August 2026 file); 23 ClinicalTrials.gov study officials after a comma, as typed (data 2026-10-21) write it. Marked on C-i: the surname reading is substantial (Census 2010 bearers 739, over the 200 threshold)", issue = 797 } [[word]] text = "pa-c" @@ -4506,7 +4513,7 @@ credential = {} [[word]] text = "pe" family = { us_census_2010 = 582, wikidata = 305, as_of = 2026-10-21 } -credential = { may_be_name = true, note = "NPPES 44 US healthcare providers write it (healthcare only, August 2026 file). Marked on C-i: the surname reading is substantial (Census 2010 or Wikidata).", issue = 796 } +credential = { note = "NPPES 44 US healthcare providers write it (healthcare only, August 2026 file). Marked on C-i: the surname reading is substantial (Census 2010 or Wikidata).", issue = 796 } [[word]] text = "pfmp" @@ -4785,7 +4792,7 @@ credential = {} text = "se" family = { us_census_2010 = 384, wikidata = 1, as_of = 2026-10-21 } given = { us_ssa = 18, wikidata = 1, as_of = 2026-10-21, note = "US SSA births summed over both sexes; Wikidata is a lower bound" } -credential = { may_be_name = true, note = "NPPES 18 US healthcare providers write it (healthcare only, August 2026 file). Marked on C-i: the surname reading is substantial (Census 2010 or Wikidata).", issue = 796 } +credential = { note = "NPPES 18 US healthcare providers write it (healthcare only, August 2026 file). Marked on C-i: the surname reading is substantial (Census 2010 or Wikidata).", issue = 796 } [[word]] text = "secb" @@ -4808,7 +4815,6 @@ text = "si" family = { us_census_2010 = 1166, wikidata = 275, as_of = 2026-10-21 } given = { us_ssa = 185, wikidata = 124, as_of = 2026-10-21, note = "US SSA births summed over both sexes; Wikidata is a lower bound" } [word.credential] -may_be_name = true issue = 791 note = "Si is a common surname (Census 2010: 1,166 family-name bearers; Wikidata 275+ family) and a Vietnamese and Chinese given name; measured 2026-10-21 by tools/name_usage.py. No expansion is recorded: the 2019 bulk import (af5bdab, #93) carried none" @@ -4895,7 +4901,7 @@ credential = {} [[word]] text = "vd" [word.credential] -note = "NOT may_be_name though also a particle: #454 closed by design, neither mc nor vd is a borne name (rules/P6's Accepted clause)" +note = "NOT marked as possibly a name (no name reading) though also a particle: #454 closed by design, neither mc nor vd is a borne name (rules/P6's Accepted clause)" [[word]] text = "vrd" @@ -4925,8 +4931,8 @@ courtesy = { same_as = "esq" } [[word]] text = "mac" +given = { us_ssa = 11455, wikidata = 178, as_of = 2026-10-25, note = "US SSA births summed over both sexes; Wikidata is a lower bound; nam_dict band 1 in Great Britain" } [word.clan] -may_be_given = true expands = "son of" note = "Mac Miller: a real given name. This is the case that shows #360's vowel heuristic was not the criterion -- 'mac' and 'mc' differ by attestation, not by shape" diff --git a/nameparser/vocab/es.toml b/nameparser/vocab/es.toml index ea5c96c4..c231cbc0 100644 --- a/nameparser/vocab/es.toml +++ b/nameparser/vocab/es.toml @@ -55,15 +55,15 @@ preposition = { may_be_given = false, expands = "of" } [[word]] text = "del" +given = { us_ssa = 4970, wikidata = 150, as_of = 2026-10-25, note = "US SSA births summed over both sexes; Wikidata is a lower bound; nam_dict band 1 in Great Britain, 1 in U.S.A." } [word.preposition] -may_be_given = true expands = "of the" note = "BOTH readings attested: Del Shannon (given name) and del Toro (particle). The ambiguity flag is the answer" [[word]] text = "la" +given = { us_ssa = 1756, wikidata = 37, as_of = 2026-10-25, note = "US SSA births summed over both sexes; Wikidata is a lower bound" } [word.article] -may_be_given = true expands = "the, fem. sg." note = "La Shawn, La Toya: a given-name element. The Romance article is the particle reading; its PLURALS ('las', 'los') are never-given, being no one's name" @@ -85,14 +85,15 @@ note = "Same as 'las' -- and the pair is why the #390 fold narrowing regressed \ [[word]] text = "san" +given = { us_ssa = 2502, wikidata = 54, as_of = 2026-10-25, note = "US SSA births summed over both sexes; Wikidata is a lower bound; nam_dict band 5 in Korea" } [word.saint] -may_be_given = true expands = "Saint: \"San Martín\"" note = "San San Chiou: a given-name element in Burmese and Chinese naming" [[word]] text = "santa" -saint = { may_be_given = true, expands = "Saint, fem." } +given = { us_ssa = 4410, wikidata = 38, as_of = 2026-10-25, note = "US SSA births summed over both sexes; Wikidata is a lower bound; nam_dict band 5 in Italy, 5 in Latvia, 1 in Swiss, 1 in Bulgaria" } +saint = { expands = "Saint, fem." } [[word]] text = "y" diff --git a/nameparser/vocab/fil.toml b/nameparser/vocab/fil.toml index 3d0592ae..5bc5eda3 100644 --- a/nameparser/vocab/fil.toml +++ b/nameparser/vocab/fil.toml @@ -14,7 +14,7 @@ salutation = { issue = 606, gender = "m", note = "borne: ~500 surnames, in Phili [[word]] text = "dela" +given = { us_ssa = 565, as_of = 2026-10-25, note = "US SSA births summed over both sexes; nam_dict band 1 in Germany, 1 in Bulgaria" } [word.preposition] -may_be_given = true expands = "of the, de la contracted: \"Dela Cruz\"" note = "Dela is a short form of Adela, and an African name meaning saviour/redeemer -- so the Filipino surname particle is not the only reading" diff --git a/nameparser/vocab/fr.toml b/nameparser/vocab/fr.toml index 55d375ae..751ab981 100644 --- a/nameparser/vocab/fr.toml +++ b/nameparser/vocab/fr.toml @@ -47,7 +47,7 @@ religious = { gender = "f" } [[word]] text = "st" religious = { expands = "Saint, abbreviated" } -saint = { may_be_given = true, expands = "Saint, abbreviated" } +saint = { expands = "Saint, abbreviated" } [[word]] text = "baron" @@ -130,8 +130,8 @@ note = "\"De\" IS a borne Bengali and Odia surname -- trailing, so it never reac [[word]] text = "du" +given = { us_ssa = 27, wikidata = 12, as_of = 2026-10-25, note = "male births only; Wikidata is a lower bound; nam_dict band 5 in Vietnam, 4 in Korea, 1 in China" } [word.preposition] -may_be_given = true expands = "of the, de le contracted" note = "Du is a Chinese surname (Du Fu), leading under a family-first reading" @@ -149,12 +149,12 @@ same_as = "l'" [[word]] text = "la" -article = { may_be_given = true, expands = "the, fem. sg." } +article = { expands = "the, fem. sg." } [[word]] text = "le" +given = { us_ssa = 1653, wikidata = 72, as_of = 2026-10-25, note = "US SSA births summed over both sexes; Wikidata is a lower bound; nam_dict band 9 in Vietnam, 4 in China" } [word.article] -may_be_given = true expands = "the, masc. sg." note = "Lê is among the most common Vietnamese surnames and leads under a family-first reading" diff --git a/nameparser/vocab/ga.toml b/nameparser/vocab/ga.toml index 4508375e..5964b3ce 100644 --- a/nameparser/vocab/ga.toml +++ b/nameparser/vocab/ga.toml @@ -14,7 +14,7 @@ issue = 661 [[word]] text = "mac" -clan = { may_be_given = true, expands = "son of" } +clan = { expands = "son of" } [[word]] text = "mc" @@ -54,8 +54,8 @@ note = "Never a given name (#604). The one ONE-LETTER particle, so the letter wr [[word]] text = "ua" +given = { note = "C-i's default under uncertainty, as the particle's note says; no source counts it as a given name (2026-10-25)" } [word.clan] -may_be_given = true issue = 604 expands = "descendant of, the older spelling: \"Seán Ua Buachalla\"" same_as = "ó" diff --git a/nameparser/vocab/gd.toml b/nameparser/vocab/gd.toml index 543af32a..4590e5a5 100644 --- a/nameparser/vocab/gd.toml +++ b/nameparser/vocab/gd.toml @@ -2,7 +2,7 @@ schema = 1 [[word]] text = "mac" -clan = { may_be_given = true, expands = "son of" } +clan = { expands = "son of" } [[word]] text = "mc" diff --git a/nameparser/vocab/he.toml b/nameparser/vocab/he.toml index 68f96b7c..5889b73c 100644 --- a/nameparser/vocab/he.toml +++ b/nameparser/vocab/he.toml @@ -98,15 +98,16 @@ courtesy = { issue = 269, note = "the same, U+05F4 gershayim", same_as = "שלי [[word]] text = "bar" +given = { us_ssa = 50, wikidata = 8, as_of = 2026-10-25, note = "US SSA births summed over both sexes; Wikidata is a lower bound" } [word.patronymic] -may_be_given = true expands = "son of, Aramaic: \"Bar-Lev\"" gender = "m" note = "Bar Refaeli; a common modern Israeli given name. The native-script בר is excluded, in this file" [[word]] text = "bat" -patronymic = { may_be_given = true, expands = "daughter of; native-script בת", gender = "f" } +given = { note = "declared on the conservative default (decisions/vocabulary-collisions C-i): not examined, and no source counts it as a given name (2026-10-25)" } +patronymic = { expands = "daughter of; native-script בת", gender = "f" } [[word]] text = "בן" diff --git a/nameparser/vocab/it.toml b/nameparser/vocab/it.toml index 664aedd0..50ad188f 100644 --- a/nameparser/vocab/it.toml +++ b/nameparser/vocab/it.toml @@ -93,11 +93,13 @@ same_as = "d'" [[word]] text = "da" -preposition = { may_be_given = true, expands = "from, of" } +given = { us_ssa = 302, wikidata = 331, as_of = 2026-10-25, note = "US SSA births summed over both sexes; Wikidata is a lower bound; nam_dict band 5 in Korea, 3 in China" } +preposition = { expands = "from, of" } [[word]] text = "dal" -preposition = { may_be_given = true, expands = "from the" } +given = { us_ssa = 628, wikidata = 2, as_of = 2026-10-25, note = "male births only; Wikidata is a lower bound; nam_dict band 4 in Korea" } +preposition = { expands = "from the" } [[word]] text = "de" @@ -125,7 +127,7 @@ preposition = { may_be_given = false, expands = "of the, masc. pl." } [[word]] text = "del" -preposition = { may_be_given = true, expands = "of the" } +preposition = { expands = "of the" } [[word]] text = "dell'" @@ -141,8 +143,8 @@ same_as = "dell'" [[word]] text = "della" +given = { us_ssa = 70949, wikidata = 91, as_of = 2026-10-25, note = "US SSA births summed over both sexes; Wikidata is a lower bound; nam_dict band 3 in U.S.A., 1 in Great Britain" } [word.preposition] -may_be_given = true expands = "of the, fem. sg." note = "Della Reese: an ordinary given name" @@ -160,8 +162,8 @@ preposition = { may_be_given = false, expands = "of the, masc. sg." } [[word]] text = "di" +given = { us_ssa = 70, wikidata = 177, as_of = 2026-10-25, note = "US SSA births summed over both sexes; Wikidata is a lower bound; nam_dict band 5 in China, 1 in Great Britain, 1 in U.S.A." } [word.preposition] -may_be_given = true expands = "of" note = "BOTH: Di as a short form of Diana, and the Italian particle (DiCaprio)" @@ -179,15 +181,15 @@ same_as = "l'" [[word]] text = "la" -article = { may_be_given = true, expands = "the, fem. sg." } +article = { expands = "the, fem. sg." } [[word]] text = "san" -saint = { may_be_given = true, expands = "Saint" } +saint = { expands = "Saint" } [[word]] text = "santa" -saint = { may_be_given = true, expands = "Saint, fem." } +saint = { expands = "Saint, fem." } [[word]] text = "e" diff --git a/nameparser/vocab/ms.toml b/nameparser/vocab/ms.toml index e4b55d62..4ddae46c 100644 --- a/nameparser/vocab/ms.toml +++ b/nameparser/vocab/ms.toml @@ -18,12 +18,12 @@ nobility = {} [[word]] text = "bin" -patronymic = { may_be_given = true, expands = "son of: \"Ahmad bin Ismail\"", gender = "m" } +patronymic = { expands = "son of: \"Ahmad bin Ismail\"", gender = "m" } [[word]] text = "binti" +given = { us_ssa = 54, as_of = 2026-10-25, note = "female births only" } [word.patronymic] -may_be_given = true issue = 604 expands = "daughter of: \"Ina binti Navalamar\"" gender = "f" @@ -31,8 +31,8 @@ note = "The female partner of 'bin'. Binti is also a given name (Swahili, \"daug [[word]] text = "binte" +given = { note = "declared beside binti, as the particle reading is kept beside binti's (#604); no source counts it as a given name (2026-10-25)" } [word.patronymic] -may_be_given = true issue = 604 expands = "daughter of, Singapore spelling" same_as = "binti" diff --git a/nameparser/vocab/mul.toml b/nameparser/vocab/mul.toml index b26eda1b..8401c128 100644 --- a/nameparser/vocab/mul.toml +++ b/nameparser/vocab/mul.toml @@ -50,7 +50,7 @@ generation = {} text = "ing" family = { us_census_2010 = 1814, wikidata = 13, as_of = 2026-10-23 } given = { wikidata = 10, as_of = 2026-10-23, note = "nam_dict band 5 in Sweden; Wikidata is a lower bound" } -credential = { may_be_name = true, cased = "Ing", expands = "Ingenieur, inženýr: the engineering degree of Czech, Slovak, Austrian and Romanian names", note = "52 of 536,981 ClinicalTrials.gov study officials write it, 43 after a comma (data 2026-10-21); NPPES 0, a credential outside US healthcare. Marked on C-i: the surname reading is substantial (Census 2010 bearers 1,814, over the 200 threshold)", issue = 848 } +credential = { cased = "Ing", expands = "Ingenieur, inženýr: the engineering degree of Czech, Slovak, Austrian and Romanian names", note = "52 of 536,981 ClinicalTrials.gov study officials write it, 43 after a comma (data 2026-10-21); NPPES 0, a credential outside US healthcare. Marked on C-i: the surname reading is substantial (Census 2010 bearers 1,814, over the 200 threshold)", issue = 848 } excluded = { academic = { reason = "Ing is borne as a given name (nam_dict band 5 in Sweden, Wikidata 10+ given) and as a surname (US Census 1,814), measured 2026-10-22 with tools/name_usage.py, and a title claims the word in front of a name ('Ing Smith'); with its period, 'Ing.' (Ingegnere, Ingenieur) already reads as a title by its shape (rules/H2). After a full name and a comma it reads as the ambiguous credential instead, #848", issue = 840 } } [[word]] diff --git a/nameparser/vocab/nl.toml b/nameparser/vocab/nl.toml index b8f94215..41da2204 100644 --- a/nameparser/vocab/nl.toml +++ b/nameparser/vocab/nl.toml @@ -47,12 +47,13 @@ occupation = {} [[word]] text = "aan" -preposition = { may_be_given = true, issue = 18, expands = "on, at: \"aan den Toorn\"" } +given = { wikidata = 1, as_of = 2026-10-25, note = "Wikidata is a lower bound" } +preposition = { issue = 18, expands = "on, at: \"aan den Toorn\"" } [[word]] text = "aen" +given = { note = "declared on the conservative default (decisions/vocabulary-collisions C-i): not examined, and no source counts it as a given name (2026-10-25)" } [word.preposition] -may_be_given = true issue = 18 expands = "aan, archaic spelling: \"aen de Landerwer\"" @@ -62,8 +63,8 @@ preposition = { may_be_given = false, issue = 18, expands = "at, on: \"op den Be [[word]] text = "te" +given = { us_ssa = 10, wikidata = 7, as_of = 2026-10-25, note = "female births only; Wikidata is a lower bound; nam_dict band 3 in Korea" } [word.preposition] -may_be_given = true issue = 18 expands = "at: \"te Riele\"" note = "Te Awanui-a-Rangi: a Māori given-name element, and in the corpus. Looks like the Dutch preposition 'te', which is why attestation rather than etymology decides" @@ -75,28 +76,30 @@ preposition = { may_be_given = false, expands = "at the, te der contracted: \"te [[word]] text = "tho" [word.preposition] -may_be_given = true issue = 18 expands = "to, Low German zu: \"tho Wopenreijss\"" [[word]] text = "thoe" -preposition = { may_be_given = true, issue = 18, expands = "to, archaic: \"thoe Holthues\"" } +given = { note = "declared on the conservative default (decisions/vocabulary-collisions C-i): not examined, and no source counts it as a given name (2026-10-25)" } +preposition = { issue = 18, expands = "to, archaic: \"thoe Holthues\"" } [[word]] text = "van" +given = { us_ssa = 31194, wikidata = 282, as_of = 2026-10-25, note = "US SSA births summed over both sexes; Wikidata is a lower bound; nam_dict band 13 in Vietnam, 2 in U.S.A." } [word.preposition] -may_be_given = true expands = "of, from" -note = "Vietnamese Văn, and Van Johnson -- the canonical particle-or-given ambiguity this whole flag exists for" +note = "Vietnamese Văn, and Van Johnson -- the canonical particle-or-given ambiguity this whole marker exists for" [[word]] text = "vande" -preposition = { may_be_given = true, issue = 18, expands = "van de contracted: \"Vande Casteele\"" } +given = { note = "declared on the conservative default (decisions/vocabulary-collisions C-i): not examined, and no source counts it as a given name (2026-10-25)" } +preposition = { issue = 18, expands = "van de contracted: \"Vande Casteele\"" } [[word]] text = "vander" -preposition = { may_be_given = true, expands = "van der contracted" } +given = { us_ssa = 1828, wikidata = 18, as_of = 2026-10-25, note = "male births only; Wikidata is a lower bound" } +preposition = { expands = "van der contracted" } [[word]] text = "vd" @@ -124,7 +127,7 @@ article = { may_be_given = false, expands = "the: \"de Vries\"" } [[word]] text = "den" -article = { may_be_given = true, expands = "the: \"van den Berg\"" } +article = { expands = "the: \"van den Berg\"" } [[word]] text = "der" @@ -136,11 +139,11 @@ article = { may_be_given = false, issue = 18, expands = "the" } [[word]] text = "heer" +given = { us_ssa = 272, as_of = 2026-10-25, note = "female births only" } [word.rank] -may_be_given = true issue = 18 expands = "lord, the nobiliary 'heer van'" -note = "added from #18's test \"Johan de heer Van Kampen\" (family 'de heer Van Kampen'), which appears to be two real names run together (Johannes de Heer, a Dutch Protestant figure, and the surname Van Kampen): the string occurs elsewhere only in another name parser's test data (hex.pm full_name_splitter 1.0.3). No real name supports the rank reading, and its may_be_given = true is the unexamined default: nobody has checked Heer as a given name. Removing it moves PARTICLES, so it is left to #766" +note = "added from #18's test \"Johan de heer Van Kampen\" (family 'de heer Van Kampen'), which appears to be two real names run together (Johannes de Heer, a Dutch Protestant figure, and the surname Van Kampen): the string occurs elsewhere only in another name parser's test data (hex.pm full_name_splitter 1.0.3). No real name supports the rank reading, and its may-be-given reading was the unexamined default: nobody had checked Heer as a given name before the counts on its given reading, taken for the derivation (2026-10-25) and not read by item. Removing it moves PARTICLES, so it is left to #766" [[word]] text = "geboren" diff --git a/nameparser/vocab/pl.toml b/nameparser/vocab/pl.toml index 8a88ae7c..eb4d903d 100644 --- a/nameparser/vocab/pl.toml +++ b/nameparser/vocab/pl.toml @@ -10,7 +10,8 @@ nobility = { gender = "m", expands = "count", issue = 606, note = "borne: ~750 s [[word]] text = "vel" -alias = { may_be_given = true, expands = "or, between two surnames: \"Kowalski vel Nowak\"" } +given = { us_ssa = 24, wikidata = 6, as_of = 2026-10-25, note = "US SSA births summed over both sexes; Wikidata is a lower bound" } +alias = { expands = "or, between two surnames: \"Kowalski vel Nowak\"" } [[word]] text = "i" diff --git a/nameparser/vocab/pt.toml b/nameparser/vocab/pt.toml index 16b5674e..07a7c599 100644 --- a/nameparser/vocab/pt.toml +++ b/nameparser/vocab/pt.toml @@ -33,7 +33,7 @@ note = "the standard Portuguese spelling of Junior: tse-donors (2026-10-22) has [[word]] text = "da" -preposition = { may_be_given = true, expands = "of the, fem. sg." } +preposition = { expands = "of the, fem. sg." } [[word]] text = "das" @@ -50,7 +50,6 @@ preposition = { may_be_given = false, expands = "of" } [[word]] text = "do" [word.preposition] -may_be_given = true expands = "of the, masc. sg." note = "Vietnamese Đỗ leads a great many surnames" @@ -60,7 +59,7 @@ preposition = { may_be_given = false, expands = "of the, masc. pl." } [[word]] text = "santa" -saint = { may_be_given = true, expands = "Saint, fem." } +saint = { expands = "Saint, fem." } [[word]] text = "e" diff --git a/nameparser/vocab/und.toml b/nameparser/vocab/und.toml index 61e561d9..87d11419 100644 --- a/nameparser/vocab/und.toml +++ b/nameparser/vocab/und.toml @@ -11,11 +11,11 @@ schema = 1 [[word]] text = "bon" -particle = { may_be_given = true, note = "no particle use found; borne as a given name and as a surname (the counts below), so removal is proposed" } +particle = { note = "no particle use found; borne as a given name and as a surname (the counts below), so removal is proposed" } given = { us_ssa = 274, wikidata = 9, as_of = 2026-10-19, note = "male births only" } family = { us_census_2010 = 1123, wikidata = 102, as_of = 2026-10-19, note = "Wikidata bearers mostly French and Italian" } [[word]] text = "dí" -particle = { may_be_given = true, note = "no particle use found; the Wikidata bearers are Chinese given names (romanized Dí), so removal is proposed" } +particle = { note = "no particle use found; the Wikidata bearers are Chinese given names (romanized Dí), so removal is proposed" } given = { wikidata = 7, as_of = 2026-10-19, note = "5 of 7 PRC citizens; the US sources fold the accent to Di, so their counts are not this spelling's" } diff --git a/tests/v2/test_vocab.py b/tests/v2/test_vocab.py index 88996758..26d4e7dd 100644 --- a/tests/v2/test_vocab.py +++ b/tests/v2/test_vocab.py @@ -113,12 +113,21 @@ def _edit(shipped: dict[str, str], file: str, old: str, new: str) -> dict[str, s return files -def test_dropping_a_flag_changes_the_compiled_module(shipped: dict[str, str]) -> None: - files = _edit(shipped, "en.toml", 'text = "jd"\ncredential = { may_be_name = true, ', - 'text = "jd"\ncredential = { ') - c = _compile(files) +def test_dropping_a_name_reading_changes_the_compiled_module(shipped: dict[str, str]) -> None: + # the markers are derived from the name readings (check 11), so the + # reading is what the compiled module answers to + jd = re.search(r'text = "jd"\n(given = \{[^\n]*\}\n)', shipped["en.toml"]) + assert jd + c = _compile(_edit(shipped, "en.toml", jd.group(1), "")) assert "jd" not in c["SUFFIX_ACRONYMS_AMBIGUOUS"] assert build.render(c) != build.OUTPUT.read_text(encoding="utf-8") + van = re.search(r'text = "van"\n(given = \{[^\n]*\}\n)\[word.preposition\]\n', + shipped["nl.toml"]) + assert van + files = _edit(shipped, "nl.toml", van.group(0), + 'text = "van"\n[word.preposition]\nmay_be_given = false\n') + assert "van" in _compile(files)["NON_GIVEN_NAME_PARTICLES"] + def test_moving_a_word_between_files_changes_nothing(shipped: dict[str, str]) -> None: @@ -188,7 +197,7 @@ def _w(body: str) -> str: ("excluded-defaulted-property-overridden", {"und.toml": _w('[[word]]\ntext = "mba"\ncredential = { acronym = false, excluded = { acronym = "x" } }')}, "acronym = false is excluded"), ("excluded-property-set-in-another-file", {"ja.toml": _w('[[word]]\ntext = "名"\nhonorific = { excluded = { attached = "x" } }'), "zh.toml": _w('[[word]]\ntext = "名"\nhonorific = { attached = { side = "after" } }')}, "attached = .* is excluded"), # `cased` compiles per word, so its exclusion holds over every kind - ("excluded-cased-set-by-another-kind", {"und.toml": _w('[[word]]\ntext = "mc"\ntitle = { excluded = { cased = "keep as written" } }'), "en.toml": _w('[[word]]\ntext = "mc"\nparticle = { may_be_given = true, cased = "MC" }')}, "keep as written"), + ("excluded-cased-set-by-another-kind", {"und.toml": _w('[[word]]\ntext = "mc"\ntitle = { excluded = { cased = "keep as written" } }'), "en.toml": _w('[[word]]\ntext = "mc"\nparticle = { may_be_given = false, cased = "MC" }')}, "keep as written"), # a property exclusion's baseline is the EXCLUDING kind's unwritten # value: a credential keeps its acronym unwritten, a generic suffix # does not, so the word reading is the refused one, and says so @@ -202,7 +211,6 @@ def _w(body: str) -> str: ("addresses-reserved-value-on-a-title-kind", {"en.toml": _w('[[word]]\ntext = "sir"\nnobility = { addresses = "full" }')}, "reserved for a later phase"), ("attached-on-an-acronym", {"und.toml": _w('[[word]]\ntext = "mba"\ncredential = { attached = { side = "after" } }')}, "needs a word reading"), ("attached-before", {"ja.toml": _w('[[word]]\ntext = "さん"\nhonorific = { attached = { side = "before" } }')}, "side = 'after'"), - ("may-be-name-on-a-word", {"en.toml": _w('[[word]]\ntext = "jr"\ngeneration = { may_be_name = true }')}, "may_be_name marks an acronym"), ("may-be-initial-on-a-word", {"und.toml": _w('[[word]]\ntext = "and"\nconjunction = { may_be_initial = true }')}, "CASED single letter"), ("may-be-initial-caseless", {"ar.toml": _w('[[word]]\ntext = "و"\nconjunction = { may_be_initial = true }')}, "CASED single letter"), ("after-given-on-an-office", {"en.toml": _w('[[word]]\ntext = "king"\nnobility = { after_given = true }')}, "after_given is allowed on a salutation reading only"), @@ -210,7 +218,21 @@ def _w(body: str) -> str: # herr's shape: refused in de.toml, flagged in da.toml ("excluded-after-given-in-another-file", {"de.toml": _w('[[word]]\ntext = "herr"\nsalutation = { excluded = { after_given = "a surname" } }'), "da.toml": _w('[[word]]\ntext = "herr"\nsalutation = { after_given = true }')}, "after_given = true is excluded.*a surname"), ("after-given-not-a-flag", {"en.toml": _w('[[word]]\ntext = "mr"\nsalutation = { after_given = "yes" }')}, "after_given must be true or false"), + # 11: a clash with a name is a name reading; the two markers that + # were flags are refused, naming the reading to write + ("may-be-name-refused", {"en.toml": _w('[[word]]\ntext = "ma"\ncredential = { may_be_name = true }')}, "may_be_name = true is derived.*`given` or `family` reading"), + ("may-be-name-false-refused", {"en.toml": _w('[[word]]\ntext = "ma"\ncredential = { may_be_name = false }')}, "may_be_name = false is derived"), + ("may-be-name-on-a-word", {"en.toml": _w('[[word]]\ntext = "jr"\ngeneration = { may_be_name = true }')}, "may_be_name = true is derived"), + ("may-be-given-true-refused", {"und.toml": _w('[[word]]\ntext = "van"\nparticle = { may_be_given = true }')}, "may_be_given = true is derived.*a `given` reading"), + ("may-be-given-true-refused-on-a-particle-kind", {"nl.toml": _w('[[word]]\ntext = "van"\npreposition = { may_be_given = true }')}, "may_be_given = true is derived"), + ("may-be-given-on-a-title", {"en.toml": _w('[[word]]\ntext = "dr"\ntitle = { may_be_given = false }')}, "may_be_given = false is derived"), + ("may-be-given-false-beside-a-given-reading", {"nl.toml": _w('[[word]]\ntext = "van"\ngiven = { note = "x" }\npreposition = { may_be_given = false }')}, "contradicts the word's given reading"), + ("may-be-given-false-beside-a-given-reading-in-another-file", {"nl.toml": _w('[[word]]\ntext = "van"\npreposition = { may_be_given = false }'), "en.toml": _w('[[word]]\ntext = "van"\ngiven = { note = "x" }\nexcluded = { title = "x" }')}, "contradicts the word's given reading"), ("particle-without-may-be-given", {"und.toml": _w('[[word]]\ntext = "van"\nparticle = {}')}, "states may_be_given"), + # a family reading is not a given reading: the particle still states + ("particle-beside-a-family-reading-without-may-be-given", {"nl.toml": _w('[[word]]\ntext = "van"\nfamily = { note = "x" }\npreposition = {}')}, "states may_be_given"), + # a name reading may stand without a count, never without a reason + ("name-reading-without-a-count-or-a-note", {"nl.toml": _w('[[word]]\ntext = "van"\ngiven = {}\npreposition = {}')}, "with no count needs a `note`"), # every particle kind, not only the generic one (#749): unwritten, # the flag would read false and put the word in the never-given half ("particle-kind-without-may-be-given", {"nl.toml": _w('[[word]]\ntext = "van"\npreposition = {}')}, "states may_be_given"), @@ -231,21 +253,21 @@ def _w(body: str) -> str: ("two-suffix-kinds", {"en.toml": OK, "und.toml": _w('[[word]]\ntext = "jr"\nstatus = {}')}, "one suffix kind per word"), # 5: one word reads one way in every file ("acronym-disagrees-across-files", {"ja.toml": _w('[[word]]\ntext = "博士"\ncredential = {}'), "zh.toml": _w('[[word]]\ntext = "博士"\ncredential = { acronym = false }')}, "acronym is"), - ("may-be-given-disagrees-across-files", {"und.toml": _w('[[word]]\ntext = "van"\nparticle = { may_be_given = true }'), "nl.toml": _w('[[word]]\ntext = "van"\nparticle = { may_be_given = false }')}, "may_be_given is"), - # keyed on the category: a word may take another particle kind in - # each language, and the kinds must still agree (#749 widened this - # from the kind; the narrower test let both may_be_given rows through) - ("may-be-given-disagrees-across-particle-kinds", {"fr.toml": _w('[[word]]\ntext = "de"\npreposition = { may_be_given = false }'), "nl.toml": _w('[[word]]\ntext = "de"\narticle = { may_be_given = true }')}, "may_be_given is"), + # may_be_given is stated per reading, so each file's reading states + # it (check 11): one file stating false does not cover another's, + # across files and across particle kinds (#749's rows, rewritten when + # the marker was derived) + ("may-be-given-disagrees-across-files", {"und.toml": _w('[[word]]\ntext = "van"\nparticle = {}'), "nl.toml": _w('[[word]]\ntext = "van"\nparticle = { may_be_given = false }')}, "und.toml: 'van' particle: a particle states may_be_given"), + ("may-be-given-disagrees-across-particle-kinds", {"fr.toml": _w('[[word]]\ntext = "de"\npreposition = { may_be_given = false }'), "nl.toml": _w('[[word]]\ntext = "de"\narticle = {}')}, "nl.toml: 'de' article: a particle states may_be_given"), # `cased` too: the narrower check let two kinds through to the # per-word mask check, which refused them with its own message ("cased-disagrees-across-particle-kinds", {"und.toml": _w('[[word]]\ntext = "ó"\nparticle = { may_be_given = false, cased = "Ó" }'), "ga.toml": _w('[[word]]\ntext = "ó"\nclan = { may_be_given = false, cased = "ó" }')}, "cased is"), - ("may-be-given-disagrees-with-the-generic-kind", {"und.toml": _w('[[word]]\ntext = "van"\nparticle = { may_be_given = true }'), "nl.toml": _w('[[word]]\ntext = "van"\npreposition = { may_be_given = false }')}, "may_be_given is"), + ("may-be-given-disagrees-with-the-generic-kind", {"und.toml": _w('[[word]]\ntext = "van"\nparticle = {}'), "nl.toml": _w('[[word]]\ntext = "van"\npreposition = { may_be_given = false }')}, "und.toml: 'van' particle: a particle states may_be_given"), ("addresses-disagrees-across-files", {"und.toml": _w('[[word]]\ntext = "sir"\ntitle = { addresses = "given" }'), "en.toml": _w('[[word]]\ntext = "sir"\ntitle = {}')}, "addresses is"), # keyed on the category: a title may take another kind in each file # (#750), and the kinds must still agree on which name they address ("addresses-disagrees-across-title-kinds", {"en.toml": _w('[[word]]\ntext = "sir"\nnobility = { addresses = "given" }'), "fr.toml": _w('[[word]]\ntext = "sir"\nsalutation = {}')}, "addresses is"), ("addresses-disagrees-with-the-generic-title", {"und.toml": _w('[[word]]\ntext = "sir"\ntitle = {}'), "en.toml": _w('[[word]]\ntext = "sir"\nnobility = { addresses = "given" }')}, "addresses is"), - ("may-be-name-disagrees-across-files", {"en.toml": _w('[[word]]\ntext = "ma"\ncredential = {}'), "und.toml": _w('[[word]]\ntext = "ma"\ncredential = { may_be_name = true }')}, "may_be_name is"), # 6: an exclusion holds across files ("excluded-kind-in-another-file", {"fr.toml": _w('[[word]]\ntext = "fils"\nexcluded = { generation = "Marie Fils" }'), "en.toml": _w('[[word]]\ntext = "fils"\ngeneration = {}')}, "Marie Fils"), # the excluded kind ITSELF, with a property moving it to the other @@ -259,7 +281,7 @@ def _w(body: str) -> str: # a particle kind's exclusion refuses every other particle kind: # one particle kind per file made this impossible before #749 ("excluded-particle-kind-sidestepped-by-a-sibling-kind", {"ar.toml": _w('[[word]]\ntext = "ben"\nexcluded = { patronymic = "Ben is a given name" }'), "pl.toml": _w('[[word]]\ntext = "ben"\npreposition = { may_be_given = false }')}, "Ben is a given name"), - ("excluded-particle-kind-sidestepped-by-the-generic-kind", {"ar.toml": _w('[[word]]\ntext = "ben"\nexcluded = { patronymic = "Ben is a given name" }'), "und.toml": _w('[[word]]\ntext = "ben"\nparticle = { may_be_given = true }')}, "Ben is a given name"), + ("excluded-particle-kind-sidestepped-by-the-generic-kind", {"ar.toml": _w('[[word]]\ntext = "ben"\nexcluded = { patronymic = "Ben is a given name" }'), "und.toml": _w('[[word]]\ntext = "ben"\nparticle = { may_be_given = false }')}, "Ben is a given name"), ("excluded-particle-category-refuses-a-kind", {"ga.toml": _w('[[word]]\ntext = "o"\nexcluded = { particle = "a periodless initial" }\nclan = { may_be_given = false }')}, "a periodless initial"), # a title kind's exclusion refuses every other title kind, and the # title category every kind: one title kind made both impossible @@ -287,9 +309,11 @@ def _w(body: str) -> str: ("phrase-title-kind-addressing-the-family", {"en.toml": _w('[[word]]\ntext = "deputy secretary"\noffice = {}')}, "phrase can never match"), ("phrase-with-cased", {"und.toml": _w('[[word]]\ntext = "sir and dame"\ntitle = { addresses = "given", cased = "Sir and Dame" }')}, "looked up per word"), ("phrase-particle", {"und.toml": _w('[[word]]\ntext = "de la"\nparticle = { may_be_given = false }')}, "phrase can never match"), - # 9: a bound given name that is a particle may be given - ("bound-given-particle-never-given", {"und.toml": _w('[[word]]\ntext = "abu"\nparticle = { may_be_given = false }\nbound_given = {}')}, "may_be_given = true"), - ("bound-given-particle-kind-never-given", {"und.toml": _w('[[word]]\ntext = "abu"\nbound_given = {}'), "ar.toml": _w('[[word]]\ntext = "abu"\nteknonymic = { may_be_given = false }')}, "may_be_given = true"), + # 9: a bound given name that is a particle has a given reading + ("bound-given-particle-never-given", {"und.toml": _w('[[word]]\ntext = "abu"\nparticle = { may_be_given = false }\nbound_given = {}')}, "needs a given reading"), + ("bound-given-particle-kind-never-given", {"und.toml": _w('[[word]]\ntext = "abu"\nbound_given = {}'), "ar.toml": _w('[[word]]\ntext = "abu"\nteknonymic = { may_be_given = false }')}, "needs a given reading"), + # a family reading does not answer it + ("bound-given-particle-with-a-family-reading", {"und.toml": _w('[[word]]\ntext = "abu"\nfamily = { note = "x" }\nparticle = { may_be_given = false }\nbound_given = {}')}, "needs a given reading"), # the data invariants that were import-time asserts ("elided-particle-without-its-twin", {"und.toml": _w('[[word]]\ntext = "d\'"\nparticle = { may_be_given = false }')}, "U\\+2019 spelling"), ("elided-particle-kind-without-its-twin", {"fr.toml": _w('[[word]]\ntext = "l\'"\narticle = { may_be_given = false }')}, "U\\+2019 spelling"), @@ -332,7 +356,7 @@ def _w(body: str) -> str: # second kind of a category ("same-as-cycle-across-files", {"en.toml": _w('[[word]]\ntext = "jr"\ngeneration = {}\n\n[[word]]\ntext = "jnr"\ngeneration = { same_as = "jr" }'), "fr.toml": _w('[[word]]\ntext = "jr"\ngeneration = { same_as = "jnr" }')}, "same_as loops"), ("same-as-cycle-behind-another-files-head", {"de.toml": _w('[[word]]\ntext = "x"\ngeneration = {}\n\n[[word]]\ntext = "jr"\ngeneration = { same_as = "x" }'), "en.toml": _w('[[word]]\ntext = "jnr"\ngeneration = { same_as = "jr" }\n\n[[word]]\ntext = "jr"\ngeneration = { same_as = "jnr" }')}, "same_as loops, jnr -> jr -> jnr;"), - ("same-as-cycle-through-a-second-kind", {"nl.toml": _w('[[word]]\ntext = "c"\npreposition = { may_be_given = true }\n\n[[word]]\ntext = "a"\npreposition = { may_be_given = true, same_as = "c" }\narticle = { may_be_given = true, same_as = "b" }\n\n[[word]]\ntext = "b"\narticle = { may_be_given = true, same_as = "a" }')}, "same_as loops, a -> b -> a;"), + ("same-as-cycle-through-a-second-kind", {"nl.toml": _w('[[word]]\ntext = "c"\npreposition = { may_be_given = false }\n\n[[word]]\ntext = "a"\npreposition = { may_be_given = false, same_as = "c" }\narticle = { may_be_given = false, same_as = "b" }\n\n[[word]]\ntext = "b"\narticle = { may_be_given = false, same_as = "a" }')}, "same_as loops, a -> b -> a;"), # a chain running INTO a loop: the loop is named, not the chain ("same-as-chain-into-a-loop", {"en.toml": _w('[[word]]\ntext = "xa"\ngeneration = { same_as = "xb" }\n\n[[word]]\ntext = "xb"\ngeneration = { same_as = "xc" }\n\n[[word]]\ntext = "xc"\ngeneration = { same_as = "xb" }')}, "same_as loops, xb -> xc -> xb;"), ] @@ -354,7 +378,11 @@ def test_every_mutation_starts_from_a_pool_the_checks_accept() -> None: "zh.toml": _w('[[word]]\ntext = "博士"\ncredential = { acronym = false }')}, {"en.toml": _w('[[word]]\ntext = "jr"\ngeneration = {}\n\n' '[[word]]\ntext = "jnr"\ngeneration = { same_as = "jr" }')}, - {"und.toml": _w('[[word]]\ntext = "abu"\nparticle = { may_be_given = true }\nbound_given = {}')}, + {"und.toml": _w('[[word]]\ntext = "abu"\ngiven = { note = "x" }\nparticle = {}\nbound_given = {}')}, + # check 11 met: a particle beside a given reading states nothing, + # in its own file or another, and a count-less one carries a note + {"nl.toml": _w('[[word]]\ntext = "van"\ngiven = { note = "x" }\npreposition = {}'), + "de.toml": _w('[[word]]\ntext = "van"\npreposition = {}')}, {"und.toml": _w('[[word]]\ntext = "d\'"\nparticle = { may_be_given = false }\n\n' '[[word]]\ntext = "d’"\nparticle = { may_be_given = false }')}, {"und.toml": _w('[[word]]\ntext = "sir and dame"\ntitle = { addresses = "given" }')}, @@ -366,7 +394,7 @@ def test_every_mutation_starts_from_a_pool_the_checks_accept() -> None: # a title kind's exclusion refuses the title set, not the word: # its particle reading stays free {"cs.toml": _w('[[word]]\ntext = "pan"\nexcluded = { salutation = "the Chinese surname Pan" }'), - "und.toml": _w('[[word]]\ntext = "pan"\nparticle = { may_be_given = true }')}, + "und.toml": _w('[[word]]\ntext = "pan"\nparticle = { may_be_given = false }')}, # one title in a different kind per file, agreeing on addresses {"en.toml": _w('[[word]]\ntext = "sir"\nnobility = { addresses = "given" }'), "fr.toml": _w('[[word]]\ntext = "sir"\nsalutation = { addresses = "given" }')}, @@ -386,9 +414,8 @@ def test_every_mutation_starts_from_a_pool_the_checks_accept() -> None: {"fr.toml": _w('[[word]]\ntext = "fils"\nfamily = { us_census_2010 = 767, as_of = 2026-10-16 }\n' 'excluded = { given = "x", generation = "Marie Fils" }')}, # a property exclusion refuses a value other than the default: - # writing the default, as a particle must for may_be_given, passes - {"und.toml": _w('[[word]]\ntext = "zz"\nparticle = { may_be_given = false, excluded = { may_be_given = "x" } }\n\n' - '[[word]]\ntext = "yy"\ncredential = { may_be_name = false, excluded = { may_be_name = "x" } }\n\n' + # writing the default passes + {"und.toml": _w('[[word]]\ntext = "zz"\nsalutation = { after_given = false, excluded = { after_given = "x" } }\n\n' '[[word]]\ntext = "xx"\ncredential = { excluded = { acronym = "x" } }')}, # the defaults agree with the written value across files {"und.toml": _w('[[word]]\ntext = "dr"\ntitle = {}'), @@ -441,6 +468,44 @@ def test_every_mutation_starts_from_a_pool_the_checks_accept() -> None: _compile(files) +def _derived(files: dict[str, str]) -> tuple[bool, bool]: + """Whether zzq compiles as an ambiguous acronym and as a particle + that may be given.""" + c = _compile(files) + return ("zzq" in c["SUFFIX_ACRONYMS_AMBIGUOUS"], + "zzq" in c["PARTICLES"] and "zzq" not in c["NON_GIVEN_NAME_PARTICLES"]) + + +@pytest.mark.parametrize(("files", "expected"), [ + # an acronym and a name reading of either kind, in one file or two + ({"en.toml": _w('[[word]]\ntext = "zzq"\ngiven = { note = "x" }\ncredential = {}')}, (True, False)), + ({"en.toml": _w('[[word]]\ntext = "zzq"\nfamily = { note = "x" }\ncredential = {}')}, (True, False)), + ({"en.toml": _w('[[word]]\ntext = "zzq"\ncredential = {}'), + "fr.toml": _w('[[word]]\ntext = "zzq"\nfamily = { note = "x" }\nexcluded = { title = "x" }')}, (True, False)), + # an acronym reading of another suffix kind is marked too: the set is + # the acronyms', not the credential kind's + ({"en.toml": _w('[[word]]\ntext = "zzq"\ngiven = { note = "x" }\ncourtesy = { acronym = true }')}, (True, False)), + # a word reading beside a name is no acronym, and is not marked + ({"en.toml": _w('[[word]]\ntext = "zzq"\ngiven = { note = "x" }\ngeneration = {}')}, (False, False)), + ({"en.toml": _w('[[word]]\ntext = "zzq"\ncredential = {}')}, (False, False)), + # a particle and a given reading, in one file or two, any kind + ({"nl.toml": _w('[[word]]\ntext = "zzq"\ngiven = { note = "x" }\npreposition = {}')}, (False, True)), + ({"nl.toml": _w('[[word]]\ntext = "zzq"\narticle = {}'), + "en.toml": _w('[[word]]\ntext = "zzq"\ngiven = { note = "x" }\nexcluded = { title = "x" }')}, (False, True)), + # a family reading does not make a particle given: it states false + ({"nl.toml": _w('[[word]]\ntext = "zzq"\nfamily = { note = "x" }\npreposition = { may_be_given = false }')}, (False, False)), + # one word holding both clashes derives both markers, as do does + ({"en.toml": _w('[[word]]\ntext = "zzq"\ngiven = { note = "x" }\ncredential = {}'), + "pt.toml": _w('[[word]]\ntext = "zzq"\npreposition = {}')}, (True, True)), +], ids=["acronym-given", "acronym-family", "acronym-name-in-another-file", + "acronym-of-another-kind", "word-reading-given", "acronym-alone", + "particle-given", "particle-given-in-another-file", + "particle-family-only", "both-clashes"]) +def test_the_markers_derive_from_a_name_reading(files: dict[str, str], + expected: tuple[bool, bool]) -> None: + assert _derived(files) == expected + + def _apply_order_messages(texts: list[str]) -> int: """Move entries as check 10's messages say until it passes; the moves.""" moves = 0 @@ -774,10 +839,16 @@ def _code(cell: str) -> list[str]: def _sets_fed(kind: str, props: str = "{}", text: str = "zzq") -> set[str]: """The compiled constants a one-word file holding a `kind` reading - with `props` puts the word into (a particle states may_be_given).""" + with `props` puts the word into. A particle that does not state + may_be_given gets a given reading, as check 11 requires, so the bare + kind feeds only the sets its category does, and a bare name reading + gets the note a count-less one carries.""" + if build.CATEGORY.get(kind) == "name" and props == "{}": + props = '{ note = "probe" }' + entry = f'{kind} = {props}' if build.CATEGORY.get(kind) == "particle" and "may_be_given" not in props: - props = "{ may_be_given = true }" - files = {"en.toml": _w(f'[[word]]\ntext = "{text}"\n{kind} = {props}')} + entry = 'given = { note = "probe" }\n' + entry + files = {"en.toml": _w(f'[[word]]\ntext = "{text}"\n{entry}')} with warnings.catch_warnings(): warnings.simplefilter("ignore", build.VocabWarning) # a name reading is inert compiled = build.compile_files(files) @@ -790,15 +861,11 @@ def _sets_fed(kind: str, props: str = "{}", text: str = "zzq") -> set[str]: _PROPERTY_PROBES = { "addresses": ("title", '{ addresses = "given" }', "zzq"), "acronym": ("generation", "{ acronym = true }", "zzq"), - "may_be_name": ("credential", "{ may_be_name = true }", "zzq"), "attached": ("honorific", '{ attached = { side = "after" } }', "zzq"), - "may_be_given": ("particle", "{ may_be_given = false }", "zzq"), "may_be_initial": ("conjunction", "{ may_be_initial = true }", "q"), "cased": ("title", '{ cased = "ZzQ" }', "zzq"), "after_given": ("salutation", "{ after_given = true }", "zzq"), } -#: The behavioral properties the build refuses to leave unwritten. -_REQUIRED = {"may_be_given"} @pytest.mark.parametrize("kind", [k for k, c in build.CATEGORY.items() if c == "title"]) @@ -823,14 +890,17 @@ def test_the_readme_lists_every_kind_with_its_category_and_set() -> None: def test_the_readme_lists_every_property_with_its_type_and_categories() -> None: rows = _readme_table("## Properties") behavioral = {p for ps in build.BEHAVIORAL.values() for p in ps} + stated = {p for ps in build.STATED.values() for p in ps} assert {_code(r[0])[0]: r[1] for r in rows} == ( - {p: "behavioral" for p in behavioral} | {p: "descriptive" for p in build.DESCRIPTIVE}) - assert len(rows) == len(behavioral | build.DESCRIPTIVE) + {p: "behavioral" for p in behavioral} | {p: "stated" for p in stated} + | {p: "descriptive" for p in build.DESCRIPTIVE}) + assert len(rows) == len(behavioral | stated | build.DESCRIPTIVE) for prop, kind_of, applies, *_ in rows: name = _code(prop)[0] - if kind_of == "behavioral": + if kind_of in ("behavioral", "stated"): + table = build.BEHAVIORAL if kind_of == "behavioral" else build.STATED assert set(_code(applies)) == { - c for c, ps in build.BEHAVIORAL.items() if name in ps}, name + c for c, ps in table.items() if name in ps}, name else: # a name reading takes only the descriptive keys in NAME_KEYS assert ("but `name`" in applies) == (name not in build.NAME_KEYS), name @@ -840,8 +910,7 @@ def test_the_readme_states_each_behavioral_propertys_effect() -> None: rows = {_code(r[0])[0]: r[4] for r in _readme_table("## Properties") if r[1] == "behavioral"} assert set(_PROPERTY_PROBES) == set(rows) for prop, (kind, props, text) in _PROPERTY_PROBES.items(): - bare = "{ may_be_given = true }" if kind == "particle" else "{}" - moved = _sets_fed(kind, props, text) ^ _sets_fed(kind, bare, text) + moved = _sets_fed(kind, props, text) ^ _sets_fed(kind, "{}", text) assert moved, prop # a probe that moves nothing tests nothing assert set(_code(rows[prop])) & {n for n, _ in build.CONSTANTS} == moved, prop @@ -849,10 +918,10 @@ def test_the_readme_states_each_behavioral_propertys_effect() -> None: def test_the_readme_states_each_behavioral_propertys_unwritten_value() -> None: for row in _readme_table("## Properties"): prop, kind_of, _, unwritten = _code(row[0])[0], row[1], row[2], row[3] - if kind_of != "behavioral": - continue - if prop in _REQUIRED: + if kind_of == "stated": + # check 11 requires each stated property somewhere assert "required" in unwritten, prop + if kind_of != "behavioral": continue defaults = {build._behavior(build.Reading("en.toml", "zzq", kind, {}), prop) for kind, cat in build.CATEGORY.items() @@ -866,9 +935,30 @@ def test_the_readme_states_each_behavioral_propertys_unwritten_value() -> None: assert value is None and not {"true", "false"} & set(unwritten.split()), prop +#: Each derived marker's probe: a reading of the clashing category +#: (kind, props without the name reading, props with it), compiled with +#: and without a given reading -- what moves is the marker's effect. +_MARKER_PROBES = { + "possibly a name": ("credential", "{}", "{}"), + "possibly given": ("preposition", "{ may_be_given = false }", "{}"), +} + + +def test_the_readme_states_each_derived_markers_effect() -> None: + rows = {r[0]: r[2] for r in _readme_table("### Derived markers")} + assert set(rows) == set(_MARKER_PROBES) + for marker, (kind, without, with_) in _MARKER_PROBES.items(): + files = {"en.toml": _w(f'[[word]]\ntext = "zzq"\n{kind} = {without}')} + named = {"en.toml": _w(f'[[word]]\ntext = "zzq"\ngiven = {{ note = "x" }}\n{kind} = {with_}')} + a, b = build.compile_files(files), build.compile_files(named) + moved = {n for n, _ in build.CONSTANTS if ("zzq" in a[n]) != ("zzq" in b[n])} + assert moved, marker + assert set(_code(rows[marker])) & {n for n, _ in build.CONSTANTS} == moved, marker + + def test_the_readme_names_every_compiled_constant() -> None: - # each constant is reached by a kind or by a behavioral property - named = {name for heading in ("## Kinds", "## Properties") + # each constant is reached by a kind, a property or a derived marker + named = {name for heading in ("## Kinds", "## Properties", "### Derived markers") for row in _readme_table(heading) for cell in row[1:] for name in _code(cell)} assert {name for name, _ in build.CONSTANTS} <= named diff --git a/tools/vocab/build.py b/tools/vocab/build.py index 173e706c..a0f27dd0 100644 --- a/tools/vocab/build.py +++ b/tools/vocab/build.py @@ -38,7 +38,8 @@ 3. every kind registered, every property allowed on its kind with an allowed value (a date, never a date-time; `as_of` only beside a count), an acronym's text free of periods (it is matched with them - removed), and no entry without a reading or an exclusion; + removed), a `note` on a name reading with no count, saying why it + is declared, and no entry without a reading or an exclusion; 4. one suffix kind per word, across files; 5. the same category of the same word agrees on every behavioral property in every file, whatever kind each file gives it; @@ -52,14 +53,21 @@ its word's letters (#459), at most one `cased` per word; 8. a phrase only as a birth_marker or an addresses = "given" title, and never with a `cased`, which is looked up per word; -9. a bound given name that is also a particle has may_be_given; +9. a bound given name that is also a particle has a given reading; 10. no same_as chain loops, in any file, and the words of every file but und.toml in canonical order (_order_key): by kind, then by the word a spelling is the same_as of, then by text; - -numbered by subject, not by when they run: 1-3, 8 and 7's case mask -run per entry as each file loads, and 4-7, 9 and 10 over the whole pool -in check_pool, 6 first and 10 last; +11. a clash with a name is stated as a name reading, from which the two + markers are derived (decisions/vocabulary-model, 2026-10-25): + `may_be_name` and `may_be_given = true` are refused, each naming + the reading to write, and a particle reading states + `may_be_given = false` exactly where its word has no `given` + reading in any file; + +numbered by subject, not by when they run: 1-3, 8, 7's case mask and +11's refusals run per entry as each file loads, and 4-7, 9, 11's +particle statement and 10 over the whole pool in check_pool, 6 first +and 10 last; plus the data invariants that were asserts: may_be_initial only on a cased single letter, an elided particle in both apostrophes (#700), and @@ -68,7 +76,8 @@ final check in compile_pool is the backstop. The subset relations (GIVEN_NAME_TITLES and AFTER_GIVEN_TITLES in TITLES, each marker set in its base) need no -check: a flag cannot exist without its reading. The one soft finding, a +check: a flag cannot exist without its reading, and a derived marker is +read off the reading it marks. The one soft finding, a name reading on a word with no vocabulary reading and no exclusion, is a VocabWarning, which the shipped-data test turns into an error. """ @@ -211,12 +220,25 @@ def _alnum(text: str) -> str: #: its allowed values are exactly what today's pipeline does. BEHAVIORAL: dict[str, frozenset[str]] = { "title": frozenset({"addresses", "after_given", "cased"}), - "suffix": frozenset({"acronym", "may_be_name", "attached", "cased"}), - "particle": frozenset({"may_be_given", "cased"}), + "suffix": frozenset({"acronym", "attached", "cased"}), + "particle": frozenset({"cased"}), "conjunction": frozenset({"may_be_initial", "cased"}), "bound_given": frozenset({"cased"}), "relation": frozenset({"cased"}), } +#: Stated properties per category: compiled to nothing, but held to the +#: word's readings (check 11). A particle states `may_be_given = false` +#: where its word has no given reading, so a new particle never falls to +#: the never-given side unremarked (decisions/vocabulary-collisions C-i). +STATED: dict[str, frozenset[str]] = { + "particle": frozenset({"may_be_given"}), +} +#: The two markers phase 1 wrote as flags and phase 2 derives from a +#: name reading (check 11), each with the reading that now states it. +DERIVED = { + "may_be_name": "a `given` or `family` reading on the word", + "may_be_given": "a `given` reading on the word", +} #: Descriptive properties: recorded, validated, compiled to nothing. DESCRIPTIVE = frozenset({"note", "issue", "decided", "expands", "same_as", "native", "gender"}) @@ -717,16 +739,30 @@ def _check_reading(r: Reading, pool: Pool) -> None: if "as_of" in p and not COUNT_SOURCES & p.keys(): raise VocabError(f"{where}: as_of dates a count, and this reading " f"has none ({', '.join(sorted(COUNT_SOURCES))})") + if not COUNT_SOURCES & p.keys() and "note" not in p: + # a name reading may stand without a count -- a particle kept + # possibly given under C-i's default, with no source counting + # it -- but says why, so an unexamined word stays findable + raise VocabError(f"{where}: a name reading with no count needs a " + f"`note` saying why it is declared") _check_descriptive(where, p) pool.readings.append(r) return - allowed = BEHAVIORAL[r.category] | DESCRIPTIVE | {"excluded"} + stated = STATED.get(r.category, frozenset()) + allowed = BEHAVIORAL[r.category] | stated | DESCRIPTIVE | {"excluded"} for k in p: + if k in stated and not isinstance(p[k], bool): + raise VocabError(f"{where}: {k} must be true or false") + if k in DERIVED and not (k in stated and p[k] is False): + # build check 11: the marker is derived from a name reading + raise VocabError(f"{where}: {k} = {_shown(p[k])} is derived since " + f"phase 2; write {DERIVED[k]} instead, with dated " + f"counts from tools/name_usage.py or a `note` " + f"saying why") if k not in allowed: raise _reserved_or_unknown(where, k, allowed) _check_descriptive(where, p) - for flag in ("acronym", "may_be_name", "may_be_given", "may_be_initial", - "after_given"): + for flag in ("acronym", "may_be_initial", "after_given"): if flag in p and not isinstance(p[flag], bool): raise VocabError(f"{where}: {flag} must be true or false") if r.category == "title": @@ -753,9 +789,6 @@ def _check_reading(r: Reading, pool: Pool) -> None: f"{where}: an acronym is matched with every period removed, " f"so {r.text!r} can never match; write " f"{r.text.replace('.', '')!r} (it reads the dotted spelling too)") - if "may_be_name" in p and not acronym: - raise VocabError(f"{where}: may_be_name marks an acronym reading; " - f"this reading has acronym = false") if "attached" in p: att = p["attached"] if not isinstance(att, dict): @@ -770,13 +803,6 @@ def _check_reading(r: Reading, pool: Pool) -> None: raise VocabError(f"{where}: attached needs a word reading " f"(acronym = false); the peel splits word tails") _check_descriptive(f"{where} attached", att) - if r.category == "particle" and "may_be_given" not in p: - # stated, never defaulted: the old encoding listed the never-given - # particles, so a new one was ambiguous unless argued otherwise - # (decisions/vocabulary-collisions C-i), and a default here would - # silently pick one side of that - raise VocabError(f"{where}: a particle states may_be_given = true or " - f"false; under uncertainty, true (vocabulary-collisions C-i)") if r.category == "conjunction" and r.flag("may_be_initial"): if len(r.text) != 1 or r.text.upper() == r.text.lower(): raise VocabError( @@ -855,9 +881,8 @@ def _overrides(r: Reading, x: Exclusion) -> bool: unwritten-false for every other kind, so an exclusion of a credential's `acronym` refuses a generic `suffix = {}` (a word, where the exclusion keeps the acronym) and lets `suffix = { acronym = true }` - through. Writing the baseline (`may_be_name = false`) is the same as - omitting it, so a particle, which must state may_be_given, can still - state it false beside an exclusion of it.""" + through. Writing the baseline (`after_given = false`) is the same as + omitting it.""" assert x.prop is not None baseline = Reading(x.file, x.text, x.kind, {}) return _behavior(r, x.prop) != _behavior(baseline, x.prop) @@ -906,7 +931,7 @@ def check_pool(pool: Pool) -> None: # on the CATEGORY, not the kind: the sets are sets of words, and a # particle may take a different kind in each language (`de`, a # preposition in French and an article in Dutch), and two kinds - # disagreeing on may_be_given would compile silently to the false + # disagreeing on a property would compile silently to one of them for text, rs in by_text.items(): first: dict[str, Reading] = {} for r in rs: @@ -936,15 +961,36 @@ def check_pool(pool: Pool) -> None: if len(masks) > 1: raise VocabError(f"{text!r}: one cased mask per word, got {sorted(masks)}") + # 11, the pool's half: a particle reading states may_be_given = false + # exactly where its word has no given reading in any file. Stated, + # never defaulted: the old encoding listed the never-given particles, + # so a new one was ambiguous unless argued otherwise + # (decisions/vocabulary-collisions C-i), and a default here would + # silently pick one side of that; a false beside a given reading + # would say the opposite of the marker it derives + given = {t for t, rs in by_text.items() if any(r.kind == "given" for r in rs)} + for r in pool.readings: + if r.category != "particle": + continue + if r.text in given and "may_be_given" in r.props: + raise VocabError(f"{r.where()}: may_be_given = false contradicts " + f"the word's given reading, which derives it; " + f"drop the property") + if r.text not in given and "may_be_given" not in r.props: + raise VocabError(f"{r.where()}: a particle states may_be_given = " + f"false, or its word carries a given reading; " + f"under uncertainty, the given reading " + f"(vocabulary-collisions C-i)") + # 9: a bound given name that is also a particle must be one that may # be given -- the bound join reads it as a given name. Lexicon raises # on the same contradiction; this says which file and word for text, rs in by_text.items(): - if any(r.kind == "bound_given" for r in rs): + if any(r.kind == "bound_given" for r in rs) and text not in given: for r in rs: - if r.category == "particle" and not r.flag("may_be_given"): + if r.category == "particle": raise VocabError(f"{r.where()}: also a bound given name, so " - f"the particle needs may_be_given = true") + f"the word needs a given reading") # an elided particle reads only the apostrophe it is spelled with, # nothing folding U+2019 to U+0027, so both spellings ship (#700) @@ -978,7 +1024,7 @@ def _behavior(r: Reading, prop: str) -> object: if prop == "attached": att = r.props.get("attached") return None if att is None else att.get("side") - if prop.startswith("may_be") or prop == "after_given": + if prop == "may_be_initial" or prop == "after_given": return r.flag(prop) return r.props.get(prop) @@ -1031,6 +1077,10 @@ class Compiled(TypedDict): def compile_pool(pool: Pool) -> Compiled: """Fold the readings into today's constants.""" check_pool(pool) + # the two derived markers (check 11): a clash with a name is a word's + # name reading beside the reading it clashes with + named = {r.text for r in pool.readings if r.category == "name"} + given = {r.text for r in pool.readings if r.kind == "given"} titles: set[str] = set() given_titles: set[str] = set() after_given: set[str] = set() @@ -1058,7 +1108,7 @@ def compile_pool(pool: Pool) -> Compiled: elif r.category == "suffix": if r.acronym: acronyms.add(t) - if r.flag("may_be_name"): + if t in named: ambiguous.add(t) else: words.add(t) @@ -1066,10 +1116,7 @@ def compile_pool(pool: Pool) -> Compiled: glued.add(t) elif r.category == "particle": particles.add(t) - # never-given where any reading has may_be_given = false; - # check 5 makes every reading of one word agree, so "any" - # is "all" - if not r.flag("may_be_given"): + if t not in given: never_given.add(t) elif r.category == "conjunction": conjunctions.add(t) From 7c5f76fe71cadda855127bf8d01eabb58dd840d7 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sun, 25 Oct 2026 00:59:41 -0700 Subject: [PATCH 2/3] Fix the design-docs review findings on phase 2a Co-Authored-By: Claude Opus 5.5 --- docs/design/decisions/ma-do.md | 2 +- docs/design/decisions/vocabulary-exclusions.md | 2 +- docs/design/decisions/vocabulary-model.md | 10 +++++----- docs/design/vocabulary-model-rfc.md | 14 ++++++++------ nameparser/vocab/README.md | 6 +++--- 5 files changed, 18 insertions(+), 16 deletions(-) diff --git a/docs/design/decisions/ma-do.md b/docs/design/decisions/ma-do.md index 21013372..965e9a4e 100644 --- a/docs/design/decisions/ma-do.md +++ b/docs/design/decisions/ma-do.md @@ -2,7 +2,7 @@ ## Standing -- `ma` and `do` are members of the ambiguous acronym set (`suffix_acronyms_ambiguous`: credential readings with `may_be_name`, `uv run python tools/vocab/query.py --word ma`), kept there because both are common surnames; the set has grown past these two since (decisions/suffix-acronym-collisions), and the marking is still what lets `Jack Ma` read family 'Ma' with `suffix-or-name` reported, while `John Smith MA` reads suffix — 2026-07-17. +- `ma` and `do` are members of the ambiguous acronym set (`suffix_acronyms_ambiguous`: acronym readings on a word with a `given` or `family` reading, decisions/vocabulary-model 2026-10-25, `uv run python tools/vocab/query.py --word ma`), kept there because both are common surnames; the set has grown past these two since (decisions/suffix-acronym-collisions), and the marking is still what lets `Jack Ma` read family 'Ma' with `suffix-or-name` reported, while `John Smith MA` reads suffix — 2026-07-17. - In a name written in more than one case, a member in capitals leans credential and is taken with nothing to spare (`Jack MA`, `Jack DO` → given 'Jack', suffix 'MA'/'DO'), where `Jack Ma` keeps family 'Ma'; a deliberate break from v1, which read `Jack MA` as family 'MA' — 2026-09-15 #289. - A member written with its periods, one after each letter, counts unambiguously (`Jack M.A.`, `Jack D.O.` → suffix, `given-or-family` reported when only the given name remains), and one trailing period is no such spelling (`Jack Ma.` → family 'Ma.') — 2026-07-17. - A parenthesized bare `(MA)` or `(DO)` reads as a nickname, no longer escaping to suffix as in 1.x, and reports `suffix-or-nickname` — 2026-07-17. diff --git a/docs/design/decisions/vocabulary-exclusions.md b/docs/design/decisions/vocabulary-exclusions.md index 261861f3..5b6016ee 100644 --- a/docs/design/decisions/vocabulary-exclusions.md +++ b/docs/design/decisions/vocabulary-exclusions.md @@ -19,7 +19,7 @@ Excluded (the never-given / ambiguous particle line, a particle reading's `may_b - Only 9 of the ambiguous members were individually justified at #360's audit (2026-08); the rest sit there by the conservative default (ambiguous unless argued never-given). #604 (2026-10-04) argued three more on entry — ua, binti, binte — under vocabulary-collisions. - The audit's scope, as the module comment above the ambiguous particles recorded it until #749 turned the comments into fields: #360 examined "the 12 that appear leading in tools/differential's corpora plus a few more", and left the rest bare rather than record an absence of knowledge as a judgment. Examined is wider than the 9 justified above. - The per-word reasons are data since #749 (2026-10-19, decisions/vocabulary-model): each member's `note` on its particle reading, in the file of each language whose names it occurs in, and each word kept out (`ben`, `nic`, unaccented `o`, `bt`, `bte`, and the native-script `ال` and `בר`) an `excluded` entry with its reason. `uv run python tools/vocab/query.py --kind particle` lists the members (each word once per file it is in), `--issue 360 --kind particle` the five never-given words #360 added or moved (das, las, los, mc, ste), `--issue 604 --kind particle` #604's five (the never-given ó and ní beside the three above), and `--excluded particle` the refusals; the nine ambiguous members #360 justified carry a `note` but no `issue`, so no query isolates them; a contributor who re-adds a refused word meets the recorded reason as a build error. -- Encoding rationale (#293, predating #360's membership questions): the data layer stores the NEVER-GIVEN set and derives the ambiguous one, because that is safe-by-default for new particles — a one-place addition — and the v1 shim translates by one-directional complement. And the constants are FROZEN specifically to kill the cached-Lexicon.default()-vs-fresh- Constants desync that runtime module-constant mutation caused. The data-layer half is superseded since 2026-10-18 (decisions/vocabulary-model): every particle reading states `may_be_given`, true or false, and neither side is a default; the freeze half stands. +- Encoding rationale (#293, predating #360's membership questions): the data layer stores the NEVER-GIVEN set and derives the ambiguous one, because that is safe-by-default for new particles — a one-place addition — and the v1 shim translates by one-directional complement. And the constants are FROZEN specifically to kill the cached-Lexicon.default()-vs-fresh- Constants desync that runtime module-constant mutation caused. The data-layer half is superseded since 2026-10-18 (decisions/vocabulary-model): every particle reading states `may_be_given`, true or false, and neither side is a default, and since 2026-10-25 a particle may be given exactly where its word has a `given` reading, every other particle reading stating `may_be_given = false`; the freeze half stands. - Load-bearing dependency: TITLES ∩ ambiguous — {do, freiherr, st} until #296's audit, {freiherr, st} since — is what keeps the particle-or-given ambiguity emitter reachable at all; moving them all would leave the shipped vocabulary unable to reach it, which is why test_the_chained_emitter_is_reachable_by_construction (tests/v2/test_parser.py) pins reachability by construction rather than by shipped vocabulary — an empty intersection fails no test and does not mean the emitter is unreachable. Open (contested vocabulary memberships — the rule is right, the word's set is questioned; the issue is canonical): none. The block held three entries and all three were answered inside a fortnight — #346, #343 and #344 together on 2026-09-06 (see #indic-honorifics) and #342 on 2026-09-07 (see #suffix-acronym-collisions). The heading stays with nothing under it on purpose, the same reason an empty ledger roster section is a statement and a missing one is nobody having looked. sa, se and om (all three since decided, sa on 2026-10-25, #857) are named as the next candidates by #342's own comment and carry no issue, so they are not entries here. diff --git a/docs/design/decisions/vocabulary-model.md b/docs/design/decisions/vocabulary-model.md index 4b843243..04ce8b02 100644 --- a/docs/design/decisions/vocabulary-model.md +++ b/docs/design/decisions/vocabulary-model.md @@ -63,10 +63,10 @@ Open: #752 (phase 2, derived ambiguity: the gate on every reading kind), #789 (w Phasing, fixed then: (1) the data format and a compiler to today's sets, snapshot-neutral; (2) derived ambiguity, with the gate extended to every reading kind — the step that changes behavior, with its own design; (3) pack semantics, needing 1 and 2; (4) the rules menu, order and shapes, and the output model. -- **2026-10-18 #737 — phase 1 shipped: per-word TOML compiled to today's sets (approved by Derek, 2026-10-18, all eight of the spec's recommendations).** (`may_be_name` and `may_be_given` as written flags superseded by 2026-10-25 #752, which derives them as this entry's flags bullet planned.) No parse, render or public name moves; it is a re-encoding plus the build checks the encoding makes possible. +- **2026-10-18 #737 — phase 1 shipped: per-word TOML compiled to today's sets (approved by Derek, 2026-10-18, all eight of the spec's recommendations).** (`may_be_name` and `may_be_given = true` as written flags, `may_be_given` stated on every particle, and a property exclusion of `may_be_given` superseded by 2026-10-25 #752, which derives the two markers as this entry's flags bullet planned.) No parse, render or public name moves; it is a re-encoding plus the build checks the encoding makes possible. - **The format.** `nameparser/vocab/.toml`, one file per BCP 47 tag in canonical case (`pt-BR`, `zh-Hant`), plus `und` for words not yet assigned a language and `mul` for words shared on purpose (the Roman numerals). `schema = 1` is the only other top-level key: a new kind or property is additive and does not bump it, a change in what a key means does, and a compiler refuses a newer schema. Each `[[word]]` has a `text`, already in `_normalize` form, and zero or more readings keyed by KIND, each a TABLE (`generation = {}`, never `= true`) so a property can be added to any reading later. The file name is the language; there is no `language` key to drift from it. - **Kinds as keys, with generic category keys.** Each kind belongs to one category, and the category decides which compiled field a reading feeds. A category key is itself a kind (`title = {}`, a title not yet classified), so classifying a word later is a data edit that moves no compiled set. Kind names are unique across categories. Phase 1 classifies the SUFFIX category fully — `generation`, `credential`, `honorific`, `professional_honorific`, `status`, `courtesy` — and converts every other category mechanically into `und.toml` with its generic kind and today's flags, so the TOML is the single source from the first PR. One suffix kind per word replaces the import-time assert that the acronym and word sets are disjoint. - - **Flags now, derivation in phase 2.** The marker sets became explicit properties — `may_be_given` on particles, `may_be_name` on acronym credentials, `may_be_initial` on conjunctions, `attached = { side = "after" }` for the glued peel, `addresses = "given"`, `cased` — rather than being derived from declared name readings, which would have meant declaring a given reading for `te`, `tho` and `du` that no count supports. Measured for the later swap: deriving both ambiguity markers from name readings reproduces them exactly if the readings are declared on exactly the flagged words, because no word sits in one marker set while needing to be absent from the other (`do` is in both; `mc` and `vd`, particles and credentials, in neither, consistent with #454). Recompute: `L = Lexicon.default(); L.particles_ambiguous & (L.suffix_acronyms - L.suffix_acronyms_ambiguous)` and `L.suffix_acronyms_ambiguous & (L.particles - L.particles_ambiguous)`, both empty. + - **Flags now, derivation in phase 2.** The marker sets became explicit properties — `may_be_given` on particles, `may_be_name` on acronym credentials, `may_be_initial` on conjunctions, `attached = { side = "after" }` for the glued peel, `addresses = "given"`, `cased` — rather than being derived from declared name readings, which would have meant declaring a given reading for `te`, `tho` and `du` that no count supports. (Corrected 2026-10-25, #752: each has a US SSA given-name count, `te` 10, `tho` 52 and `du` 27 births summed over all years, so the premise was wrong when written; the figures are on their given readings.) Measured for the later swap: deriving both ambiguity markers from name readings reproduces them exactly if the readings are declared on exactly the flagged words, because no word sits in one marker set while needing to be absent from the other (`do` is in both; `mc` and `vd`, particles and credentials, in neither, consistent with #454). Recompute: `L = Lexicon.default(); L.particles_ambiguous & (L.suffix_acronyms - L.suffix_acronyms_ambiguous)` and `L.suffix_acronyms_ambiguous & (L.particles - L.particles_ambiguous)`, both empty. - **`may_be_given` is stated on every particle, never defaulted.** Found while converting, not in the spec: the old encoding stored the NEVER-GIVEN particles and derived the rest, so a new particle was ambiguous unless argued never-given — #293's safe-by-default and decisions/vocabulary-collisions C-i's "under uncertainty, ambiguous". A flag with a default would have picked one side of that silently (a bare `particle = {}` reading as never-given is the unsafe side), so the build refuses a particle reading without it. - **Exclusions are data.** `excluded` refuses a kind or category at word level, or a behavioral property at reading level (`honorific = { excluded = { attached = "…" } }`), with a reason. It lives in the file of the language whose reading it refuses, and the build refuses the excluded thing in ANY file, quoting the reason, so a contributor adding a refused reading meets the record of why. An excluded kind refuses itself whatever its properties, and every other kind compiling to the same set, so a generic `suffix = {}` cannot put back a refused `generation` (found in the PR review); the name kinds, which compile to nothing, refuse only themselves. A refused property is refused any value other than the one it takes unwritten on a reading of the excluding kind (false for the `may_be_*` flags, true for a credential's `acronym`, so a particle can still state `may_be_given = false` beside an exclusion of it), on any reading of its category — and `cased`, compiled once per word, on every reading of the word. The pilot carried the generational-words refusals, rai/cha/mai and the glued-peel refusals; the README states the exception that lets those sentences here become queries. - **Script is read from the word, not the file**, reversing the notes' premise that Serbian Latin and Cyrillic need separate files: `sr.toml` holds `mlađi` and `млађи` side by side. A script subtag is for what the text cannot tell apart, which in practice is Simplified and Traditional Han. @@ -132,10 +132,10 @@ Open: #752 (phase 2, derived ambiguity: the gate on every reading kind), #789 (w - **2026-10-24 #835 — `after_given`, the salutations that may end a given part.** A behavioral title property, allowed on a `salutation` reading only (a build check, since an office or rank word is borne as a middle name), compiles into `AFTER_GIVEN_TITLES` and `Lexicon.after_given_titles`, read by rules/H8. A salutation borne as a name refuses it with an exclusion carrying its counts. The rule is keyed on a property rather than on the kind, which would make a kind compile into a behavior where every other behavior is a property; the vetting and the measurement are decisions/H8's. Adding the field changes `Lexicon`'s pickle layout, as 2.1 and 2.3 did. - **Recompute** rather than quoting: `uv run python tools/vocab/query.py --kind salutation`, `--excluded after_given`. -- **2026-10-25 #752 — A CLASH WITH A NAME IS A NAME READING: phase 2a derives the two name-clash markers (the three decisions below agreed by Derek on 2026-10-23).** SUPERSEDES: the flags bullet of 2026-10-18 #737 and the `may_be_given` agreement of 2026-10-19 #749. No compiled set moves: `tools/vocab/build.py` rebuilds `nameparser/config/_compiled.py` byte for byte. `SUFFIX_ACRONYMS_AMBIGUOUS` is now the words with an acronym suffix reading and a `given` or `family` reading, and the particles outside `NON_GIVEN_NAME_PARTICLES` the words with a particle reading and a `given` reading, each wherever in the files the two readings sit (build check 11). The model's reason (decision 15, name readings keep the gate on) is unchanged; what changes is that the reading, not a flag beside it, is what the build reads, so a clash cannot be marked without the evidence that it is one, nor carry evidence and stay unmarked. +- **2026-10-25 #752 — A CLASH WITH A NAME IS A NAME READING: phase 2a derives the two name-clash markers (the three decisions below agreed by Derek on 2026-10-23).** SUPERSEDES: in 2026-10-18 #737, the flags bullet, the stated-on-every-particle bullet and the Exclusions bullet's `may_be_given = false` beside an exclusion of it (only behavioral properties may be excluded, and `may_be_given` is now a stated one); in 2026-10-19 #749, the `may_be_given` agreement and the unexamined-word convention keyed on `may_be_given = true`, re-keyed on the given reading. No compiled set moves: `tools/vocab/build.py` rebuilds `nameparser/config/_compiled.py` byte for byte. `SUFFIX_ACRONYMS_AMBIGUOUS` is now the words with an acronym suffix reading and a `given` or `family` reading, and the particles outside `NON_GIVEN_NAME_PARTICLES` the words with a particle reading and a `given` reading, each wherever in the files the two readings sit (build check 11). The model's reason (decision 15, name readings keep the gate on) is unchanged; what changes is that the reading, not a flag beside it, is what the build reads, so a clash cannot be marked without the evidence that it is one, nor carry evidence and stay unmarked. - **The readings written.** Every flagged word that lacked one got a name reading: the seven credentials `ba`, `do`, `ed`, `jd`, `lac`, `ma` and `meng`, and the 38 flagged particles other than `bon` and `dí`, `do` among them, so 44 words and 49 readings (a credential with both a Census and an SSA count got both). The counts are `tools/name_usage.py`'s, taken 2026-10-25 and recorded as each reading's `as_of`. - - **A name reading may be declared without a count, with a required `note` saying why.** Derivation needs a `given` reading on every flagged particle, and some are borne as given names in no source the tool reads; refusing a count-less reading would have forced each of those to the never-given side, against decisions/vocabulary-collisions C-i's "under uncertainty, ambiguous". Eight particle words have no count in any source (`aen`, `bat`, `binte`, `freiherr`, `freiherrin`, `thoe`, `ua`, `vande`); each keeps its may-be-given reading on decisions/vocabulary-collisions C-i's default, with a required `note` saying so. The build refuses a count-less name reading with no `note`. The unexamined-word test of nameparser/vocab/README.md keeps its shape, keyed on the given reading: a word whose particle readings carry no `note` is unexamined, whatever counts its given reading carries, since those were taken for the derivation and not read against the particle position. - - **`ld` keeps its count in its credential reading's `note` and has no name reading, so it stays unmarked.** Its `given` reading recorded the evidence for NOT marking it (45 SSA births against 11,703 NPPES providers), which derivation would have read as a mark. The count moved into the credential reading's `note`, so a name reading that keeps the gate off, contradicting decision 15, never exists. Declined: deriving it literally, which moves 3 ClinicalTrials.gov and 5 FEC names and no PSC record (measured 2026-10-25 by writing the given reading back on this change). + - **A name reading may be declared without a count, with a required `note` saying why.** Derivation needs a `given` reading on every flagged particle, and some are borne as given names in no source the tool reads; refusing a count-less reading would have forced each of those to the never-given side, against decisions/vocabulary-collisions C-i's "under uncertainty, ambiguous". Eight particle words have no count in any source (`aen`, `bat`, `binte`, `freiherr`, `freiherrin`, `thoe`, `ua`, `vande`); each keeps its may-be-given reading on decisions/vocabulary-collisions C-i's default or, for `binte`, beside `binti` (#604), with a required `note` saying so. The build refuses a count-less name reading with no `note`. The unexamined-word test of nameparser/vocab/README.md keeps its shape, keyed on the given reading: a word whose particle readings carry no `note` is unexamined, whatever counts its given reading carries, since those were taken for the derivation and not read against the particle position. + - **`ld` keeps its count in its credential reading's `note` and has no name reading, so it stays unmarked.** Its `given` reading recorded the evidence for NOT marking it (45 SSA births against 11,703 NPPES providers), which derivation would have read as a mark. The count moved into the credential reading's `note`, so a name reading that keeps the gate off, contradicting decision 15, never exists. Declined: deriving it literally, which moves 3 ClinicalTrials.gov and 5 FEC names, measured 2026-10-25 by writing the given reading back on this change; psc-individuals holds no record writing it, so it cannot move there. - **A particle with no given reading still states `may_be_given = false`, required,** so a new particle never reaches the never-given side unremarked, the reason the property had no default in phase 1; `may_be_given = true` and `may_be_name` (either value) are refused with a message naming the reading to write, and `false` beside a `given` reading is refused as contradicting it. Check 9 now asks for a `given` reading on a bound given name that is also a particle. `may_be_given` is a STATED property in the README's table: held to the readings, compiled to nothing by itself. - **`may_be_initial` stays a property.** Its clash is with the initial, a shape every cased single letter has, and which letters a tradition writes both ways is a judgment about the letter, not a name anyone bears. - **Measured (2026-10-25, against 632685c8):** `tools/differential/snapshot.py` matches; `cldr_agreement.py` matches every record; `snapshot.py --local` for ctgov-officials, fec-individuals and tse-donors, recorded from the base with `--tree`, matches; `local_agreement.py psc-individuals` matches the record. Negative control: dropping `jd`'s given reading takes it out of `SUFFIX_ACRONYMS_AMBIGUOUS`, and dropping `van`'s, with `may_be_given = false` written, puts it in `NON_GIVEN_NAME_PARTICLES` (tests/v2/test_vocab.py). diff --git a/docs/design/vocabulary-model-rfc.md b/docs/design/vocabulary-model-rfc.md index 6b9022f9..032054c0 100644 --- a/docs/design/vocabulary-model-rfc.md +++ b/docs/design/vocabulary-model-rfc.md @@ -11,19 +11,21 @@ nameparser recognizes words by vocabulary: titles, suffixes, particles, conjunct ```toml [[word]] text = "do" -credential = { may_be_name = true, expands = "Doctor of Osteopathic Medicine" } -[word.particle] -may_be_given = true +family = { us_census_2010 = 30301, wikidata = 100, as_of = 2026-10-25 } +given = { us_ssa = 11, wikidata = 9, as_of = 2026-10-25, note = "male births only; Wikidata is a lower bound" } +credential = { expands = "Doctor of Osteopathic Medicine" } +[word.preposition] +expands = "of the, masc. sg." ``` -`do` is a credential and a particle, and Vietnamese Đỗ makes it a surname too. Under the sets that clash was visible only as an overlap between two of them; as readings it is written on the word. Once every word says which language's names it occurs in, choosing a language can settle the clashes that exist only because the default serves every language. +`do` is a credential and a particle, and Vietnamese Đỗ makes it a surname too: its name readings, with the counts they rest on, are what mark both clashes (an entry spread over two files in the shipped data, `en.toml` and `pt.toml`). Under the sets that clash was visible only as an overlap between two of them; as readings it is written on the word. Once every word says which language's names it occurs in, choosing a language can settle the clashes that exist only because the default serves every language. Four phases, in a fixed order: | Phase | What | Changes parsing? | State | |---|---|---|---| | 1 | The per-word TOML format, compiled to today's sets | no | shipped in 2.6 (#737, #744); #749–#751 gave the words in `und.toml` their languages, leaving two particles of unrecorded origin | -| 2 | Ambiguity derived from clashing readings; the ambiguity gate extended to every reading kind | yes, the first phase that does | #752 | +| 2 | Ambiguity derived from clashing readings; the ambiguity gate extended to every reading kind | yes, the first phase that does | #752; 2a, the two name-clash markers derived, landed without moving a parse (decisions/vocabulary-model, 2026-10-25) | | 3 | Language packs that prefer or restrict their language | yes, for callers who choose one | #753, needs 1 and 2 | | 4 | Order and shape; the rules a pack can choose; the output model | yes | #754, mostly independent of 2 and 3 | @@ -47,7 +49,7 @@ Phase 1 re-encoded the vocabulary and changed no parse. One TOML file per BCP 47 Two choices in phase 1 set up phase 2 and are worth knowing before reading it: -- **The marker sets are explicit flags** (`may_be_given`, `may_be_name`, `may_be_initial`), not derived from name readings. Deriving them in phase 1 would have meant declaring a given reading for particles like `te`, `tho` and `du` that no bearer count supports. The flags record past curation, not evidence. +- **The marker sets were explicit flags** (`may_be_given`, `may_be_name`, `may_be_initial`), not derived from name readings, recording past curation rather than evidence. Phase 2a (2026-10-25) derived the two that record a clash with a name, `may_be_name` and `may_be_given`, from name readings with counts (`te`, `tho` and `du`, once thought to have none, have SSA counts), keeping `may_be_initial` as a property (decisions/vocabulary-model). - **Names a later phase will use are build errors now** (`default`, `prefer`, `strict`, `bare_only`, `leading_only`, `trailing_only`, `attaches`, `separators`, and the values `addresses = "full"` or `"any"`), so nothing reads as working before it does. The reference for the format is nameparser/vocab/README.md; the decisions behind it, with what was declined and measured, are decisions/vocabulary-model's 2026-10-18 entry. diff --git a/nameparser/vocab/README.md b/nameparser/vocab/README.md index 22b493e9..2bf7eef7 100644 --- a/nameparser/vocab/README.md +++ b/nameparser/vocab/README.md @@ -165,7 +165,7 @@ Unlike the suffix kinds, one word may take a different particle kind in each lan **Classifying a category adds kinds to every check that tests it.** While a category has only its generic kind, a check that asks for the category and one that asks for the generic kind give the same answer, so a test row written in the generic kind cannot tell them apart. When a category gets its first fine kinds, give every check on that category a row in `tests/v2/test_vocab.py` written in a fine kind, and run each row against the check keyed on the kind to see it fail. #749's review found four particle checks pinned only on the generic `particle` kind once the shipped data used it for two words: the required `may_be_given`, an exclusion refusing a sibling kind, the bound given name that is a never-given particle, and the elided particle's two spellings. Each passed every test when keyed on the kind, which would have let a `preposition = {}` with no `may_be_given` compile silently into the never-given set. -The `name` kinds compile to nothing. They record that a word is borne as a given or family name, with counts (see [Bearer counts](#bearer-counts)), as evidence for a reading or an exclusion. +The `name` kinds compile to nothing on their own. They record that a word is borne as a given or family name, with counts (see [Bearer counts](#bearer-counts)), as evidence for a reading or an exclusion, and beside an acronym or particle reading one derives a marker (see [Derived markers](#derived-markers)). ## Properties @@ -177,7 +177,7 @@ A property is **behavioral** when it compiles into a set and so changes how name | `after_given` | behavioral | `title` | false | `AFTER_GIVEN_TITLES`, on a `salutation` reading only: written bare at the end of the given part after a family comma, the word is the title, as in `Doe, Jane Mrs` ([rules/H8](../../docs/design/rules/H8-salutation-ending-the-given-part.md), a salutation ending the given part). A salutation borne as a given name or a surname is left out with an exclusion of the property, its counts beside it (`goodman`, `herr`), since a middle name may be either | | `acronym` | behavioral | `suffix` | true for `credential`, false for every other kind | true: `SUFFIX_ACRONYMS`, matched with every period removed (`M.D.` is `md`); false: `SUFFIX_WORDS` | | `attached` | behavioral | `suffix` | absent | `GLUED_HONORIFICS`, on a word reading only: `{ side = "after" }` splits the word off the end of a name token, as in `田中さん` ([rules/W2](../../docs/design/rules/W2-glued-honorific-peel.md), the glued-honorific peel) | -| `may_be_given` | stated | `particle` | none: required where the word has no `given` reading | `false` states that the word is not a given name, which is what puts it in `NON_GIVEN_NAME_PARTICLES`: a particle opening the name starts the family name ([rules/P1](../../docs/design/rules/P1-lone-particle-fold.md), the lone particle fold). It is written exactly where the word has no `given` reading in any file; `true` is refused, naming the `given` reading to write instead (check 11) | +| `may_be_given` | stated | `particle` | none: required where the word has no `given` reading | `false` is required where the word has no `given` reading in any file, and that absence, not the property, is what puts the word in `NON_GIVEN_NAME_PARTICLES`: a particle opening the name starts the family name ([rules/P1](../../docs/design/rules/P1-lone-particle-fold.md), the lone particle fold). `true` is refused, naming the `given` reading to write instead (check 11) | | `may_be_initial` | behavioral | `conjunction` | false | `CONJUNCTIONS_AMBIGUOUS`, on a cased single letter only: read as an initial in a name written in one case, as in `JOHN E SMITH` ([rules/P8](../../docs/design/rules/P8-single-letter-connective-against-the-initial-it-spells.md), a single-letter connective against the initial it spells) | | `cased` | behavioral | `title`, `suffix`, `particle`, `conjunction`, `bound_given`, `relation` | the word as written | `CAPITALIZATION_EXCEPTIONS`: the case `capitalized()` gives the word (`PhD`), a mask over its letters and digits, never a replacement (#459) | | `note` | descriptive | every category, a `name` reading included | | free text: why the reading is as it is | @@ -235,7 +235,7 @@ To see what has been refused, and why: `uv run python tools/vocab/query.py --exc Counts come from `tools/name_usage.py`, which reports how often a word is borne as a given name and as a family name in four name sources: 2010 US Census surnames (listed from 100 bearers), US Social Security given names (a name-sex pair is listed for a year from 5 births, reported per sex), Jörg Michael's nam_dict (a given-name frequency band per country, not a count) and Wikidata (live, with countries of citizenship, and a lower bound because many people have no name statement). For a word holding a Han, kana or hangul letter it adds a row of the Wikidata names that END in the word, which is the question for a glued honorific. With `--fetch-nppes` it also counts the US healthcare providers who write the word as a credential (NPPES), the other side of a suffix that is also a name; `--rank N` lists the commonest credentials with their name counts and how a comma name ending in each one parses. Its docstring says what each number can and cannot show. -Record a count on the word as a `given` or `family` reading, with the date you took it in `as_of`. A source's count is left out when it is below that source's floor, never written as 0. A suffix reading has no count field yet, so a credential count goes in the reading's `note`, with its date and source. Counts are provenance: they are the evidence a decision was made on, and nothing computes a reading from them. +Record a count on the word as a `given` or `family` reading, with the date you took it in `as_of`. A source's count is left out when it is below that source's floor, never written as 0. A suffix reading has no count field yet, so a credential count goes in the reading's `note`, with its date and source. Counts are provenance: they are the evidence a decision was made on, and nothing computes a reading from them. The reading itself is not inert on a word with an acronym or a particle reading, where it marks the clash, so counts kept there as evidence AGAINST marking go in the clashing reading's `note` (`ld`). ## Adding a word From cf8245ff0efc74cb8c1e178099349acf5163d815 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sun, 25 Oct 2026 01:02:51 -0700 Subject: [PATCH 3/3] Fix the second-round review findings on phase 2a Co-Authored-By: Claude Opus 5.5 --- .../design/decisions/C8-listing-forms-part-after-the-comma.md | 2 +- docs/design/decisions/vocabulary-exclusions.md | 4 ++-- docs/design/vocabulary-model-rfc.md | 2 +- nameparser/vocab/README.md | 4 ++-- 4 files changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/design/decisions/C8-listing-forms-part-after-the-comma.md b/docs/design/decisions/C8-listing-forms-part-after-the-comma.md index 2b1a3277..08c46669 100644 --- a/docs/design/decisions/C8-listing-forms-part-after-the-comma.md +++ b/docs/design/decisions/C8-listing-forms-part-after-the-comma.md @@ -21,4 +21,4 @@ Open: none in this file. - 2026-10-04 (Derek), #603 — A credential opening the part after the comma makes it the postnominal part, whatever follows. The record moved to decisions/C13 with that clause (2026-10-19). - 2026-10-18 #728 — SPLIT FROM decisions/C1 with the rule. The entries above were written under decisions/C1 while this rule was part of rules/C1, and moved here unchanged but for the pointers the move broke, the measurement paragraphs, each cut to one line naming its population, counts, baseline and PR, and the superseded passages, each cut to its decision and what replaced it; where one says rules/C1 or C1, it means the rule as it then stood, which held this one. - 2026-10-19 — RULE SPLIT: a credential opening the part after the comma left rules/C8 for rules/C13, one rule per piece of evidence (docs/design/AGENTS.md); C8 keeps the part with no name word and the part that is nothing but suffix words. The #603 entry moved to decisions/C13, a cross-reference standing in its place. Where the entries here or elsewhere say rules/C8 of the opened part (the #613 and #544 entries in decisions/C1 among them), that piece is rules/C13's now. -- 2026-10-25 #857 — `sa` leaves the vocabulary: its military reading (Seaman Apprentice, a title) and its credential reading are both refused on the word (`excluded`), superseding the 2026-08-23 entry's `sa` half and decisions/suffix-acronym-collisions' 2026-10-21 held-out entry. It reads as any name word does, whatever the case, because the Portuguese surname Sá written without its accent is the one bearer the corpora show. The path to it: marking `sa` ambiguous (`may_be_name`) fixed `Maria de Sa` but left three costs, and the evidence for each reading it was keeping came back empty. (1) A one-case name ending in `SA` still read a suffix (S6), so all 255 SÁ / SA pairs of tse-donors (#843) still parted. (2) The title reading took the slot in front of a name and alone after a family comma, which `ma` does not (`Smith, Sa` → title 'Sa' where `Smith, Ma` → given 'Ma'; `Smith, John PhD Sa` → title 'Sa', suffix 'PhD', unreported). (3) Every name read with `Sa` as a title in the committed and eight local corpora is a person whose given name is Sa (`Sa Liu`, `Sa Mahammed Ilyas`) or the Arabic Sa ad (`Sa ad Hariri`), or whose Portuguese surname Sá is written without its accent (`Sa Pinto`), none a rank; and every SA or S.A. suffix is a company (`Galderma S.A.`, `Novartis Biociências SA`), a South African qualifier (`FCP(SA)`), or initials typed without stops (`MANN, S.A.`, `BROWN, MARYAM SA`) — no person writes it as a credential, the 167 NPPES providers aside, whom `name_usage.py` counts and nobody has read by item. Readings now: `Maria de Sa`, `MARIA DE SA`, `maria de sa` → family 'de Sa'/'DE SA'/'de sa' (was family 'de', suffix 'Sa'/'SA'/'sa' since 1.4.0); `Joao Sa`, `ANA SA`, `ANA SILVA SA` → family 'Sa'/'SA'; `SA Smith` → given 'SA', family 'Smith'; `Smith, SA` → given 'SA' (was suffix 'SA', #296's audit row); `Mrs Sa Li` → title 'Mrs', given 'Sa' (the second title word is gone). Accepted: a credential `SA` after a mixed-case name is a name word, so `John Smith SA` reads middle 'Smith', family 'SA' (rules/S9 admits an unlisted all-caps word as a credential by shape only behind a comma, and this one has no vocabulary behind it); and so are the comma shapes, the same reading an unlisted two-letter caps word gets (`John Smith, XY`): `John Smith, SA` → given 'SA', family 'John Smith' (the name merged into the family, where 1.4.0 through 2.5.0 read given 'John', family 'Smith', suffix 'SA'), `John Smith, SA, PhD` → family 'John Smith', suffix 'PhD', `Smith, John SA` → middle 'SA', and `, SA` → given 'SA' (1.4.0 read title 'SA'; 2.5.0 suffix 'SA'). The corpora hold 15 such after-comma names in ctgov-officials and 10 in fec-individuals (`Spector, SA` went suffix → given 'SA'), and NPPES counts 167 US healthcare providers who write it, none read by item: if they turn out to be a credential, that is the evidence to restore it on. `Smith, John PhD Sa` keeps suffix 'PhD Sa' and reports SUFFIX_OR_NAME; `John Smith S.A.` and `Smith, S.A.` go through the unlisted-dotted rule and report SUFFIX_OR_NAME; a parenthesised `(SA)` after a credential, ClinicalTrials.gov's `FCP (SA)`, is a nickname 'SA' where it was inside the suffix (25 names in ctgov-officials). Local corpora, `snapshot.py --local` against the base 8c965d74 (2026-10-25): tse-donors moves 1,613 of 848,351 names, about 700 gaining the `SA` as part of the family name (`ADAO DE SA FERREIRA` → family 'DE SA FERREIRA', was family 'DE', suffix 'SA FERREIRA'; `WESLEY VITOR SA PEREIRA` → family 'PEREIRA'), and `local_corpora.py pairs tse-donors --agreement` reads 0 accent-only pairs with two surnames where the base read 255 SÁ / SA; ctgov-officials 54 of 352,104, 25 of them the `(SA)` nickname; psc-individuals 40 of 1,034,888 (three `Mrs Sa Li`-shaped names lose a title); ukec-individuals 0. Putting a reading back is cheap — add it and delete the matching `excluded` entry, which the build checks — but the form matters: a `credential` reading WITH `may_be_name` returns the costs the path above ended on (255 parting pairs, `Smith, Sa` reading a title once `military` is back), and one without it returns `Maria de Sa` reading suffix. A caller who needs `SA` as a credential adds it to their own lexicon. +- 2026-10-25 #857 — `sa` leaves the vocabulary: its military reading (Seaman Apprentice, a title) and its credential reading are both refused on the word (`excluded`), superseding the 2026-08-23 entry's `sa` half and decisions/suffix-acronym-collisions' 2026-10-21 held-out entry. It reads as any name word does, whatever the case, because the Portuguese surname Sá written without its accent is the one bearer the corpora show. The path to it: marking `sa` ambiguous (`may_be_name`) fixed `Maria de Sa` but left three costs, and the evidence for each reading it was keeping came back empty. (1) A one-case name ending in `SA` still read a suffix (S6), so all 255 SÁ / SA pairs of tse-donors (#843) still parted. (2) The title reading took the slot in front of a name and alone after a family comma, which `ma` does not (`Smith, Sa` → title 'Sa' where `Smith, Ma` → given 'Ma'; `Smith, John PhD Sa` → title 'Sa', suffix 'PhD', unreported). (3) Every name read with `Sa` as a title in the committed and eight local corpora is a person whose given name is Sa (`Sa Liu`, `Sa Mahammed Ilyas`) or the Arabic Sa ad (`Sa ad Hariri`), or whose Portuguese surname Sá is written without its accent (`Sa Pinto`), none a rank; and every SA or S.A. suffix is a company (`Galderma S.A.`, `Novartis Biociências SA`), a South African qualifier (`FCP(SA)`), or initials typed without stops (`MANN, S.A.`, `BROWN, MARYAM SA`) — no person writes it as a credential, the 167 NPPES providers aside, whom `name_usage.py` counts and nobody has read by item. Readings now: `Maria de Sa`, `MARIA DE SA`, `maria de sa` → family 'de Sa'/'DE SA'/'de sa' (was family 'de', suffix 'Sa'/'SA'/'sa' since 1.4.0); `Joao Sa`, `ANA SA`, `ANA SILVA SA` → family 'Sa'/'SA'; `SA Smith` → given 'SA', family 'Smith'; `Smith, SA` → given 'SA' (was suffix 'SA', #296's audit row); `Mrs Sa Li` → title 'Mrs', given 'Sa' (the second title word is gone). Accepted: a credential `SA` after a mixed-case name is a name word, so `John Smith SA` reads middle 'Smith', family 'SA' (rules/S9 admits an unlisted all-caps word as a credential by shape only behind a comma, and this one has no vocabulary behind it); and so are the comma shapes, the same reading an unlisted two-letter caps word gets (`John Smith, XY`): `John Smith, SA` → given 'SA', family 'John Smith' (the name merged into the family, where 1.4.0 through 2.5.0 read given 'John', family 'Smith', suffix 'SA'), `John Smith, SA, PhD` → family 'John Smith', suffix 'PhD', `Smith, John SA` → middle 'SA', and `, SA` → given 'SA' (1.4.0 read title 'SA'; 2.5.0 suffix 'SA'). The corpora hold 15 such after-comma names in ctgov-officials and 10 in fec-individuals (`Spector, SA` went suffix → given 'SA'), and NPPES counts 167 US healthcare providers who write it, none read by item: if they turn out to be a credential, that is the evidence to restore it on. `Smith, John PhD Sa` keeps suffix 'PhD Sa' and reports SUFFIX_OR_NAME; `John Smith S.A.` and `Smith, S.A.` go through the unlisted-dotted rule and report SUFFIX_OR_NAME; a parenthesised `(SA)` after a credential, ClinicalTrials.gov's `FCP (SA)`, is a nickname 'SA' where it was inside the suffix (25 names in ctgov-officials). Local corpora, `snapshot.py --local` against the base 8c965d74 (2026-10-25): tse-donors moves 1,613 of 848,351 names, about 700 gaining the `SA` as part of the family name (`ADAO DE SA FERREIRA` → family 'DE SA FERREIRA', was family 'DE', suffix 'SA FERREIRA'; `WESLEY VITOR SA PEREIRA` → family 'PEREIRA'), and `local_corpora.py pairs tse-donors --agreement` reads 0 accent-only pairs with two surnames where the base read 255 SÁ / SA; ctgov-officials 54 of 352,104, 25 of them the `(SA)` nickname; psc-individuals 40 of 1,034,888 (three `Mrs Sa Li`-shaped names lose a title); ukec-individuals 0. Putting a reading back is cheap — add it and delete the matching `excluded` entry, which the build checks — but the form matters: a `credential` reading WITH `may_be_name` returns the costs the path above ended on (255 parting pairs, `Smith, Sa` reading a title once `military` is back), and one without it returns `Maria de Sa` reading suffix. (Corrected 2026-10-25, #752: the marker is now derived, so `sa`'s `family` and `given` readings mark a `credential` reading put back, and `may_be_name` is a build error; the unmarked form would need those counts moved into the reading's `note`, as `ld`'s are.) A caller who needs `SA` as a credential adds it to their own lexicon. diff --git a/docs/design/decisions/vocabulary-exclusions.md b/docs/design/decisions/vocabulary-exclusions.md index 5b6016ee..0dcf449d 100644 --- a/docs/design/decisions/vocabulary-exclusions.md +++ b/docs/design/decisions/vocabulary-exclusions.md @@ -14,12 +14,12 @@ Excluded (Lexicon.conjunctions — the conjunction readings): Open (Lexicon.conjunctions — contested membership; the issue is canonical): none. The block held one entry, [#397](https://github.com/derek73/python-nameparser/issues/397) — whether Catalan `i` belongs in the conjunctions at all — answered 2026-09-20 by decisions/P3's bullet of that date: it does, and it joins `conjunctions_ambiguous` in the same change, which was never the contested half. The heading stays with nothing under it on purpose, the same reason an empty ledger roster section is a statement and a missing one is nobody having looked. -Excluded (the never-given / ambiguous particle line, a particle reading's `may_be_given` — #360 owns the vocabulary question): +Excluded (the never-given / ambiguous particle line, a particle's `given` reading, or `may_be_given = false` where it has none — #360 owns the vocabulary question): - Only 9 of the ambiguous members were individually justified at #360's audit (2026-08); the rest sit there by the conservative default (ambiguous unless argued never-given). #604 (2026-10-04) argued three more on entry — ua, binti, binte — under vocabulary-collisions. - The audit's scope, as the module comment above the ambiguous particles recorded it until #749 turned the comments into fields: #360 examined "the 12 that appear leading in tools/differential's corpora plus a few more", and left the rest bare rather than record an absence of knowledge as a judgment. Examined is wider than the 9 justified above. - The per-word reasons are data since #749 (2026-10-19, decisions/vocabulary-model): each member's `note` on its particle reading, in the file of each language whose names it occurs in, and each word kept out (`ben`, `nic`, unaccented `o`, `bt`, `bte`, and the native-script `ال` and `בר`) an `excluded` entry with its reason. `uv run python tools/vocab/query.py --kind particle` lists the members (each word once per file it is in), `--issue 360 --kind particle` the five never-given words #360 added or moved (das, las, los, mc, ste), `--issue 604 --kind particle` #604's five (the never-given ó and ní beside the three above), and `--excluded particle` the refusals; the nine ambiguous members #360 justified carry a `note` but no `issue`, so no query isolates them; a contributor who re-adds a refused word meets the recorded reason as a build error. -- Encoding rationale (#293, predating #360's membership questions): the data layer stores the NEVER-GIVEN set and derives the ambiguous one, because that is safe-by-default for new particles — a one-place addition — and the v1 shim translates by one-directional complement. And the constants are FROZEN specifically to kill the cached-Lexicon.default()-vs-fresh- Constants desync that runtime module-constant mutation caused. The data-layer half is superseded since 2026-10-18 (decisions/vocabulary-model): every particle reading states `may_be_given`, true or false, and neither side is a default, and since 2026-10-25 a particle may be given exactly where its word has a `given` reading, every other particle reading stating `may_be_given = false`; the freeze half stands. +- Encoding rationale (#293, predating #360's membership questions): the data layer stores the NEVER-GIVEN set and derives the ambiguous one, because that is safe-by-default for new particles — a one-place addition — and the v1 shim translates by one-directional complement. And the constants are FROZEN specifically to kill the cached-Lexicon.default()-vs-fresh- Constants desync that runtime module-constant mutation caused. The data-layer half is superseded since 2026-10-18 (decisions/vocabulary-model): from then every particle reading stated `may_be_given`, true or false, so neither side was a default, and since 2026-10-25 a particle may be given exactly where its word has a `given` reading, every other particle reading stating `may_be_given = false`, which still leaves neither side a default; the freeze half stands. - Load-bearing dependency: TITLES ∩ ambiguous — {do, freiherr, st} until #296's audit, {freiherr, st} since — is what keeps the particle-or-given ambiguity emitter reachable at all; moving them all would leave the shipped vocabulary unable to reach it, which is why test_the_chained_emitter_is_reachable_by_construction (tests/v2/test_parser.py) pins reachability by construction rather than by shipped vocabulary — an empty intersection fails no test and does not mean the emitter is unreachable. Open (contested vocabulary memberships — the rule is right, the word's set is questioned; the issue is canonical): none. The block held three entries and all three were answered inside a fortnight — #346, #343 and #344 together on 2026-09-06 (see #indic-honorifics) and #342 on 2026-09-07 (see #suffix-acronym-collisions). The heading stays with nothing under it on purpose, the same reason an empty ledger roster section is a statement and a missing one is nobody having looked. sa, se and om (all three since decided, sa on 2026-10-25, #857) are named as the next candidates by #342's own comment and carry no issue, so they are not entries here. diff --git a/docs/design/vocabulary-model-rfc.md b/docs/design/vocabulary-model-rfc.md index 032054c0..33ba46c8 100644 --- a/docs/design/vocabulary-model-rfc.md +++ b/docs/design/vocabulary-model-rfc.md @@ -49,7 +49,7 @@ Phase 1 re-encoded the vocabulary and changed no parse. One TOML file per BCP 47 Two choices in phase 1 set up phase 2 and are worth knowing before reading it: -- **The marker sets were explicit flags** (`may_be_given`, `may_be_name`, `may_be_initial`), not derived from name readings, recording past curation rather than evidence. Phase 2a (2026-10-25) derived the two that record a clash with a name, `may_be_name` and `may_be_given`, from name readings with counts (`te`, `tho` and `du`, once thought to have none, have SSA counts), keeping `may_be_initial` as a property (decisions/vocabulary-model). +- **The marker sets were explicit flags** (`may_be_given`, `may_be_name`, `may_be_initial`), not derived from name readings, recording past curation rather than evidence. Phase 2a (2026-10-25) derived the two that record a clash with a name, `may_be_name` and `may_be_given`, from name readings, each with dated counts or a `note` where no source counts the word (`te`, `tho` and `du`, once thought to have none, have SSA counts), keeping `may_be_initial` as a property (decisions/vocabulary-model). - **Names a later phase will use are build errors now** (`default`, `prefer`, `strict`, `bare_only`, `leading_only`, `trailing_only`, `attaches`, `separators`, and the values `addresses = "full"` or `"any"`), so nothing reads as working before it does. The reference for the format is nameparser/vocab/README.md; the decisions behind it, with what was declined and measured, are decisions/vocabulary-model's 2026-10-18 entry. diff --git a/nameparser/vocab/README.md b/nameparser/vocab/README.md index 2bf7eef7..99dc2342 100644 --- a/nameparser/vocab/README.md +++ b/nameparser/vocab/README.md @@ -165,7 +165,7 @@ Unlike the suffix kinds, one word may take a different particle kind in each lan **Classifying a category adds kinds to every check that tests it.** While a category has only its generic kind, a check that asks for the category and one that asks for the generic kind give the same answer, so a test row written in the generic kind cannot tell them apart. When a category gets its first fine kinds, give every check on that category a row in `tests/v2/test_vocab.py` written in a fine kind, and run each row against the check keyed on the kind to see it fail. #749's review found four particle checks pinned only on the generic `particle` kind once the shipped data used it for two words: the required `may_be_given`, an exclusion refusing a sibling kind, the bound given name that is a never-given particle, and the elided particle's two spellings. Each passed every test when keyed on the kind, which would have let a `preposition = {}` with no `may_be_given` compile silently into the never-given set. -The `name` kinds compile to nothing on their own. They record that a word is borne as a given or family name, with counts (see [Bearer counts](#bearer-counts)), as evidence for a reading or an exclusion, and beside an acronym or particle reading one derives a marker (see [Derived markers](#derived-markers)). +The `name` kinds compile to nothing on their own. They record that a word is borne as a given or family name, with counts (see [Bearer counts](#bearer-counts)), as evidence for a reading or an exclusion, and a `given` or `family` reading beside an acronym reading, or a `given` reading beside a particle reading derives a marker (see [Derived markers](#derived-markers)). ## Properties @@ -235,7 +235,7 @@ To see what has been refused, and why: `uv run python tools/vocab/query.py --exc Counts come from `tools/name_usage.py`, which reports how often a word is borne as a given name and as a family name in four name sources: 2010 US Census surnames (listed from 100 bearers), US Social Security given names (a name-sex pair is listed for a year from 5 births, reported per sex), Jörg Michael's nam_dict (a given-name frequency band per country, not a count) and Wikidata (live, with countries of citizenship, and a lower bound because many people have no name statement). For a word holding a Han, kana or hangul letter it adds a row of the Wikidata names that END in the word, which is the question for a glued honorific. With `--fetch-nppes` it also counts the US healthcare providers who write the word as a credential (NPPES), the other side of a suffix that is also a name; `--rank N` lists the commonest credentials with their name counts and how a comma name ending in each one parses. Its docstring says what each number can and cannot show. -Record a count on the word as a `given` or `family` reading, with the date you took it in `as_of`. A source's count is left out when it is below that source's floor, never written as 0. A suffix reading has no count field yet, so a credential count goes in the reading's `note`, with its date and source. Counts are provenance: they are the evidence a decision was made on, and nothing computes a reading from them. The reading itself is not inert on a word with an acronym or a particle reading, where it marks the clash, so counts kept there as evidence AGAINST marking go in the clashing reading's `note` (`ld`). +Record a count on the word as a `given` or `family` reading, with the date you took it in `as_of`. A source's count is left out when it is below that source's floor, never written as 0. A suffix reading has no count field yet, so a credential count goes in the reading's `note`, with its date and source. Counts are provenance: they are the evidence a decision was made on, and nothing computes a reading from them. The reading itself is not inert where it clashes, as a `given` or `family` reading beside an acronym reading, or a `given` reading beside a particle reading does, marking the clash, so counts kept there as evidence AGAINST marking go in the clashing reading's `note` (`ld`). ## Adding a word