diff --git a/AGENTS.md b/AGENTS.md index 3bee60b3..e0f65aa8 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -182,7 +182,7 @@ The library has two layers: `nameparser/config/` (data) and `nameparser/parser.p **Design philosophy — positional and language-agnostic.** The parser assigns parts by *position* plus small sets of words that join to neighbors; it never detects language. A name's language can't be reliably inferred from Latin-script transliteration ("Ali" is Arabic or Italian; "Van"/"Della"/"Bin" are first names in some cultures, particles in others), so language-specific rules belong in opt-in `Constants` config, never global defaults. Many "wrong for language X" reports (#133, #150, #130, #85, #103, #146, #83) are irreducible ambiguities — e.g. `de Mesnil` (want last name) vs `Van Johnson` (want first name) are the same `[prefix][word]` shape. Before adding a rule, confirm it doesn't break the opposite case (run the full suite — Portuguese and "Van Johnson" tests are the usual canaries). **The one scoped exception (2.1, #271/#272): script-conditional behavior is permitted exactly where the SCRIPT ITSELF — not statistics about it — determines the convention.** The never-detect-language rule above is about Latin *transliteration*, where the signal genuinely is destroyed; native script is a different question, and it is answered per behavior rather than per script. Five defaults fall out of it, plus a sixth that applies the same not-a-guess standard to specific WORDS rather than to a script (#308's honorific peel, below). Wholly-Han, wholly-Hangul and kana-licensed names read family-first (`Policy.script_orders`) — no language detection needed, because zh and ja both write family-first in native script, so order cannot be misread even though the language is unknowable. Unspaced hangul splits into surname + given name (`config/surnames.py` ships the Korean census list as DEFAULT vocabulary) — nothing but Korean is written in hangul and the surnames are a closed census set, and the vocabulary is self-selecting besides: a hangul entry can only ever match hangul text. Hiragana licenses Japanese (#272) — a name whose characters stay inside Han∪kana while carrying at least one kana cannot be Chinese (the kana rules it out) and is not a transcription (foreign names are transcribed in katakana ALONE, マイケル has no kanji), so 高橋みなみ and 山田 エミ read family-first too; mechanically they resolve to the HIRAGANA entry, the license's carrier key. PURE katakana — full-width or halfwidth, the halfwidth block U+FF65–FF9F being katakana in the script table since #594 — is excluded and keeps the positional default: マイケル・ジャクソン is a transcribed foreign name in its source order, and a Japanese reading written in katakana (legacy halfwidth data spells Japanese names that way too) cannot be told from one, so the script settles nothing and a caller who knows opts in through `script_orders`. And the 间隔号 U+00B7 (#298) is the transcription marker for scripts that HAVE no transcription script: a name it divides (威廉·莎士比亚 — flanked by classified characters on both sides, so Catalan's Gal·la is untouched) keeps the declared order and never segments — the orthography names the convention, with the divider carrying the signal that stands the script's family-first reading down, so the default reads it in its written order and a declared `FAMILY_FIRST` reverses it like any other name; pure katakana reaches the same reading by the opposite route, its script settling nothing so the declared order stands. And a listed CJK honorific glued to the END of a name token is split off it (#308) — 田中さん is 田中 plus さん — on the same orthography-settles-it test, narrowed for the glued position: an entry peels only where it can never end a name, so 씨/님/さん/様/先生 peel while 양/군/氏/博士/殿 stay spaced-only (김지양 and 田中博士 are names, and ~90 Japanese surnames end in 殿) and 君 is in NEITHER set (王君 is a complete Chinese name), though its kana spelling くん peels. Like the nakaguro's tokenize-level separation described next, it is reached by neither policy opt-out — but for its own reason: the vocabulary carries the license itself rather than borrowing the script's, so `segment_scripts` has nothing to say about it. Since #312 it also crosses the 间隔号, which still stops the surname split standing right beside it: it answers where a name DIVIDES into surname and given, and the peel never asks that question. Whether it also crosses the FAMILY comma is tolerated rather than settled: the 2026-09-01 demotion (rules/W3) narrowed that half from contract to best-effort, since no CJK writing system's own convention puts a comma between family and given at all — so `김, 민준씨` reads today exactly as the spaced `김 민준씨` does (family 김, given 민준, suffix 씨) while the split stands down as before, but that reading is watched in the snapshot rather than pinned as contract. Its site is accordingly the name-bearing segment runs — `segments[:2]` under a family comma, and `segments[0]` as before otherwise, the family comma being the one structure that splits the name itself across two runs, with the honorific as often glued to the given side as to the family. That is the whole reach and nothing past it (`김, 민준 지훈씨` peels; `김, 민준, 지훈씨` and `김,, 민준씨` do not, both landing in a third run), and whether `segments[1]` is name text at all is now ASKED rather than inferred from the structure — `segment` does not guarantee it, since a one-word part before the comma reads as FAMILY_COMMA even when the part after it is entirely suffix-shaped, and the peel walking into such a run took `V.` for its site, found no listed tail and abandoned (#319). The question is asked in two parts. First the comma's own decision: `_comma.decide` on a classified copy of the name as written, since #613's /simplify -- a hand copy of it here drifted from decide four times in that PR's review (the ambiguous class's count, the numeral refusal, a suffix before the comma, the split `Ph. D.`), each drift dividing a name decide read as a family comma or leaving whole one it read as a suffix comma, and asking the real decision is what ended it. Where that leaves the family comma, C1's suffix-word predicate, lifted into `_vocab.is_wholly_suffix` so the stages cannot drift: a wholly suffix-shaped second run is declined and the scan stays in `segments[0]`, so `田中さん, V.` and `田中さん, Ph. D.` give さん up as `田中さん, PhD` always did. The test is necessary but NOT sufficient, and the second condition is not decoration: every honorific tail is also a suffix word, so a glued honorific is itself part of what makes its run read as suffix-shaped, and declining a run that holds the ONLY site loses the peel outright. `segments[0]` must therefore offer a peel site of its own before the second run is declined — `이, J.씨` and `선생님, J.씨` pass the suffix test and are scanned anyway, keeping the pre-#319 reading, while `김민준씨, J.씨` has a site on both sides and peels the person's own 씨 rather than the junk one behind the comma. Uniform in the PEEL, that is — where the credential itself lands is `assign`'s question and still differs by spelling (`V.` → `given`, `PhD` and `Ph. D.` → `suffix`). Not `_is_post_nominal` pluralized: the run predicate says yes both to what the token predicate vetoes (`V.`, `V`, `I` — the class the defect was reported as) and to what the token predicate never sees at all, since `period_joined_vocab` and the delimiter routes are the run predicate's alone (`Msc.Ed.` and `J.씨` reach it that way, and `田中さん, Msc.Ed.` moves with the rest). `Policy(lenient_comma_suffixes=False)` (rules/C14) drops this call to the strict token test too — so those three read as name text again and keep the pre-#319 answer, while `Ph. D.` peels under the knob regardless, its merged `phd` passing the strict test. `田中さん, 太郎` is unchanged, and not because of its comma — the honorific there is not at the end of the name, 太郎 is. The nakaguro belongs to the same doctrine but is decided a layer down: U+30FB and its halfwidth twin U+FF65 separate tokens like whitespace, unconditionally and in tokenize, so neither policy opt-out (`script_orders={}`, `segment_scripts=()`) reaches it — the codepoints are CJK-only and appear in no other script's names, which is what licenses a tokenize-level rule where U+00B7 (also the Catalan punt volat, interior to Gal·la) needs the flanked-by-classified-script guard `_tokenize_region` gives it (#298). Han segmentation stays OPT-IN (`locales.ZH` for Chinese, `locales.JA` for Japanese) — a zh surname list corrupts Japanese kanji names, since 高 is a common Chinese surname and 高橋一郎 would split 高+橋一郎 where the correct reading is 高橋+一郎; no surname list divides a kanji name at all, so `locales.JA` activates the stage and a pluggable `Parser(segmenter=...)` does the dividing. Latin-script input is never touched by any of this: "Kim Min-jun" is genuinely order-ambiguous and stays governed by `name_order` and opt-in packs. Before adding a script-conditional rule, work out which of the three it is — certain, certain for this one behavior only, or a statistical guess wearing a script's clothes. -**A constant's membership is a question you may reopen.** Proposing that a word be ADDED, REMOVED or MOVED between vocabulary sets is ordinary design work — a shipped entry is not evidence that anyone judged it. `SUFFIX_ACRONYMS` arrived in `af5bdab` as a bulk Wikipedia import never reviewed against surname collisions: 572 of its 577 alphabetic entries leave `family` empty in `"John "` against five ambiguous-gated exceptions (recomputed 2026-09-07; the fifth is `ba`; 568 of 575 against seven, measured 2026-09-25 after #540), and `sa`, `se` and `om` are borne as surnames (measured 2026-08-23), as were `rai`, `cha`, `ba` and `mc` before `decisions/suffix-acronym-collisions` decided all four — `rai` and `cha` removed, `ba` marked ambiguous, `mc` left alone as no borne name at all. The same entry marked `meng` and `lac` ambiguous on 2026-09-25 (#540). When a fix starts to look like new machinery, check the vocabulary first. Criterion: `decisions/vocabulary-collisions`, with #360's positional qualifier. +**A constant's membership is a question you may reopen.** Proposing that a word be ADDED, REMOVED or MOVED between vocabulary sets is ordinary design work — a shipped entry is not evidence that anyone judged it. `SUFFIX_ACRONYMS` arrived in `af5bdab` as a bulk Wikipedia import never reviewed against surname collisions: 572 of its 577 alphabetic entries leave `family` empty in `"John "` against five ambiguous-gated exceptions (recomputed 2026-09-07; the fifth is `ba`; 568 of 575 against seven, measured 2026-09-25 after #540), and `sa`, `se` and `om` are borne as surnames (measured 2026-08-23; all three since decided, `sa` removed from the vocabulary on 2026-10-25, #857), as were `rai`, `cha`, `ba` and `mc` before `decisions/suffix-acronym-collisions` decided all four — `rai` and `cha` removed, `ba` marked ambiguous, `mc` left alone as no borne name at all. The same entry marked `meng` and `lac` ambiguous on 2026-09-25 (#540). When a fix starts to look like new machinery, check the vocabulary first. Criterion: `decisions/vocabulary-collisions`, with #360's positional qualifier. **Sweep the forms a change can reach: comma shapes, then `name_order`.** No comma, a FULL name before the comma, a ONE-WORD name before the comma and NOTHING before the comma are four paths, and the last two are the misses — #429, #430 and #432 are all the one-word path disagreeing with the full-name path on inputs the full-name path reads correctly, and #613's PR review found `, PhD` and `, Jr.` (an empty surname field, as CSV data writes it) read as given and title by a decision that returned early on an empty head, after two review rounds whose every grid had a name before the comma. Put an empty head and a nickname-only head (`(Bob), PhD`) in every comma grid. Count the one-word path the way C3 counts, which is neither tokens nor pieces: a particle run and the one word it attaches to are one word (#575), a connective join and a bound given-name pair count as their words (#613, so `Ortega y Gasset, Dr.` is a whole name), and a suffix beside the surname counts as no name word at all. #603's review found the suffix half (`Vega y Lopez Jr., PhD Jones` lost its family where `Vega y Lopez, PhD Jones` kept it); #613 then reversed the connective half, and both now read given 'Vega', middle 'y', family 'Lopez' — so test a particle surname, a connective surname and a surname with a suffix, each before a comma. For orders the sweep already exists (`tests/v2/test_cases.py` runs every row under all three) but carries one assertion, R2's family partition, so it checks nothing a new change moves; coverage there has been vacuous before (PR #394's review found the suite passed with `name_order` discarded from grouping). Parse your change's names in each comma shape and each order, and read the ones you did not predict. And read a generated grid's moves by SHAPE, not by corpus membership: #614's first design passed the suite, the five gates and a fingerprint of every corpus name, because its grid moves were sorted into 'corpus name moved' and 'garbage', and the review found the moves that mattered among the 'garbage' -- a bare salutation (`Mr. and Mrs.` lost its connective to the family name) and a name with one trailing word (`Freiherr von Berg X.Y.Z.`). Pull out the head-plus-one-word slice of the grid and a salutation with nothing after it, and read every move there. Two more were the second review's (#614): no grid put TWO words that are both a title and a particle at the head (`Freiherr St van Berg MA`, where an index into the pieces was applied to a merged view of them -- `TITLES ∩ particles_ambiguous` is `{freiherr, st}`, so put both in every head list, each in its bare and its period-marked spelling -- #637's grid had `St` and `Freiherr` alone and missed that `St.` and `Freiherr.` are declined by the bound join and then attached by P6), and the grids compared ambiguity KINDS, so a report telling the caller the wrong field (`Freiherr von Berg Ma`, 'read as a given name' of a word in the family) passed every pin. #701's review added one: put a title/suffix dual (`Ms`, `Sr.`, `Cpl`) SECOND in a title run, behind an ordinary title. A stop that asks only "is this word suffix vocabulary" passes every row that opens with the dual and fails `Rev. Sr. Smith, Mary`; it was found by comparing `T Smith, John` against `T John Smith` over every title word, and a comma-versus-comma-less comparison is the cheap check for any rule that moves a word between the two slots. Compare each report's `detail` against the field its tokens land in -- since #626 `test_a_report_names_the_field_its_word_lands_in` (tests/v2/test_cases.py) does it over every non-locale case row under all three orders (a row carrying its own policy only as declared), for the phrasings its `_FIELD_CLAIMS` lists -- so a row with the shape is the test only where the report's wording is listed: a new emitter's wording goes in that list, and segment's report on a part past the second comma names no field (#629). #626's own first grid then missed an emitter for want of one comma shape (`, Ma Dr.`: an empty head, an ambiguous word, a trailing title, which H1 moves to the family): a trailing title belongs in every empty-head comma grid too. And #620's: an exclusion list in code being replaced is a set of rules, not noise -- #614's merged copy excluded every word the trailing read weighs from its units, the first #620 prototype dropped the list as a symptom of the copy, and its review found the list had been keeping positional counting's rule that a member in front counts as a word (`Freiherr von Berg MA X.Y.Z.` lost its suffix). Before deleting one, mutate each entry out and read what moves. The converse holds for replacing a hand copy WITH the shared predicate: it is a behavior change wherever the two disagreed, and #735's /simplify shipped one as "no reading moves" because the suite and the snapshot were its only check (`St Sir de Mesnil` family-first, a token test replaced by P1's piece test); diff the two predicates' answers over a grid before calling the swap a refactor. @@ -348,7 +348,7 @@ Don't use the bare `python3 -m doctest .rst` CLI (no `optionflags`) to che **Prefix-join uses value-based `list.index()`** in `join_on_conjunctions` — fragile when a token value repeats (e.g. a trailing title that's also a suffix acronym, or two `van`s); constrain such lookups to start at `i + 1`. See #100. -**Title vs suffix is positional for BARE words, and the leading period-abbreviation rule overrides even that** — a word matching `TITLES` at the front of a name becomes `title`; the same word matching `SUFFIX_ACRONYMS`/`SUFFIX_WORDS` at the end becomes `suffix` (never both, regardless of the word's real-world meaning). The `TITLES`/suffix overlap was audited in #296 (2026-08-23): the pure postnominals (`jr`, `junior`, `phd`, `do`, `se`) left `TITLES`, the v1-residue `dr`/`sra` left the suffix sets, and the eleven words still in both (`ms`, `sa`, `sr`, `lt`, `ra`, `vc`, and the ranks `cpl`, `cpo`, `cpt`, `csm`, `sgm`; `md` was a twelfth until #853 took its title reading out, a leading `Md` being the given name Muhammad) are deliberate duals that position decides. External test sources (old issue gists, etc.) sometimes assert `suffix` for a leading professional abbreviation like `RA`/`PD`/`Dipl.-Ing.` — that's the source data being wrong, not a parser bug. Verify position before "fixing" it. Two qualifications the older "purely positional" wording papered over, the first measured 2026-08-01 and the second answered 2026-09-08 (#316): a PERIOD-marked leading word is claimed by the shape rule before any vocabulary is read (`"Esq. Smith"` → `title`, though `esq` is suffix-only), and a PERIOD-marked trailing word is claimed by VOCABULARY — first the suffix sets, which the peel reads before anything else (`"John Smith Esq."` → `suffix`), then the titles (`"John Smith Prof."` → `title='Prof.'`, `rules/H5`) — while an unlisted abbreviation there stays a name part (`"John Smith Xyz."` → `family='Xyz.'`) unless it is written as two or more period-separated chunks, which since 2.4 IS a trailing shape rule (`Policy.unlisted_dotted_suffixes`, default on: `"John Smith X.Y.Z."` → `suffix='X.Y.Z.'`, read by the same words-to-spare count a bare ambiguous acronym takes, `rules/S3`), except paired initials in a script with capitals, which stay a name word (`"Ram Bahadur B.K."` → `family='B.K.'`, #864). The SINGLE trailing period is still not one, deliberately — it is the abbreviation shape any word can wear. The bare word is where "positional" still holds whole -- outside a part past the second comma, which holds no name word, so C2 reads a bare title word there as a title since #603 (`Eric H. Holder, Jr., Attorney General`), and outside a listed salutation ending the given part after a family comma, which rules/H8 reads as the title since #835 (`Doe, Jane Mrs`), the comma having marked the surname's place -- and the reason it must: `TITLES` holds words that are in no suffix set — 787 of them on 2026-10-11 (746 before #606's salutations), measured with `L = Parser().lexicon; len(L.titles - L.suffix_acronyms - L.suffix_words)`, a figure that grows with the vocabulary and never shrinks the argument — and many are ordinary surnames (`king`, `bishop`, `prince`, `pope`, `judge`, `sheriff`, `baron`, `master`, ...), so a vocabulary-first rule over BARE trailing words would read `"Mary Jane King"` as `title='King'`, `family='Jane'`. The period is what separates the safe case from that one, and it separates it by being a WRITING convention rather than by making the collision go away: `"Mary Jane King"` keeps `family='King'` while `"Mary Jane King."` reads `title='King.'`, `family='Jane'`, a cost accepted under the input-is-a-name premise rather than one the rule prevents (the trailing-position bullet in decisions/vocabulary-exclusions, measured 2026-09-09). Since #316 that sentence describes the shipped rule rather than an aspiration. +**Title vs suffix is positional for BARE words, and the leading period-abbreviation rule overrides even that** — a word matching `TITLES` at the front of a name becomes `title`; the same word matching `SUFFIX_ACRONYMS`/`SUFFIX_WORDS` at the end becomes `suffix` (never both, regardless of the word's real-world meaning). The `TITLES`/suffix overlap was audited in #296 (2026-08-23): the pure postnominals (`jr`, `junior`, `phd`, `do`, `se`) left `TITLES`, the v1-residue `dr`/`sra` left the suffix sets, and the ten words still in both (`ms`, `sr`, `lt`, `ra`, `vc`, and the ranks `cpl`, `cpo`, `cpt`, `csm`, `sgm`; `md` was a twelfth until #853 took its title reading out, a leading `Md` being the given name Muhammad) are deliberate duals that position decides. External test sources (old issue gists, etc.) sometimes assert `suffix` for a leading professional abbreviation like `RA`/`PD`/`Dipl.-Ing.` — that's the source data being wrong, not a parser bug. Verify position before "fixing" it. Two qualifications the older "purely positional" wording papered over, the first measured 2026-08-01 and the second answered 2026-09-08 (#316): a PERIOD-marked leading word is claimed by the shape rule before any vocabulary is read (`"Esq. Smith"` → `title`, though `esq` is suffix-only), and a PERIOD-marked trailing word is claimed by VOCABULARY — first the suffix sets, which the peel reads before anything else (`"John Smith Esq."` → `suffix`), then the titles (`"John Smith Prof."` → `title='Prof.'`, `rules/H5`) — while an unlisted abbreviation there stays a name part (`"John Smith Xyz."` → `family='Xyz.'`) unless it is written as two or more period-separated chunks, which since 2.4 IS a trailing shape rule (`Policy.unlisted_dotted_suffixes`, default on: `"John Smith X.Y.Z."` → `suffix='X.Y.Z.'`, read by the same words-to-spare count a bare ambiguous acronym takes, `rules/S3`), except paired initials in a script with capitals, which stay a name word (`"Ram Bahadur B.K."` → `family='B.K.'`, #864). The SINGLE trailing period is still not one, deliberately — it is the abbreviation shape any word can wear. The bare word is where "positional" still holds whole -- outside a part past the second comma, which holds no name word, so C2 reads a bare title word there as a title since #603 (`Eric H. Holder, Jr., Attorney General`), and outside a listed salutation ending the given part after a family comma, which rules/H8 reads as the title since #835 (`Doe, Jane Mrs`), the comma having marked the surname's place -- and the reason it must: `TITLES` holds words that are in no suffix set — 787 of them on 2026-10-11 (746 before #606's salutations), measured with `L = Parser().lexicon; len(L.titles - L.suffix_acronyms - L.suffix_words)`, a figure that grows with the vocabulary and never shrinks the argument — and many are ordinary surnames (`king`, `bishop`, `prince`, `pope`, `judge`, `sheriff`, `baron`, `master`, ...), so a vocabulary-first rule over BARE trailing words would read `"Mary Jane King"` as `title='King'`, `family='Jane'`. The period is what separates the safe case from that one, and it separates it by being a WRITING convention rather than by making the collision go away: `"Mary Jane King"` keeps `family='King'` while `"Mary Jane King."` reads `title='King.'`, `family='Jane'`, a cost accepted under the input-is-a-name premise rather than one the rule prevents (the trailing-position bullet in decisions/vocabulary-exclusions, measured 2026-09-09). Since #316 that sentence describes the shipped rule rather than an aspiration. ### Tests (`tests/`) diff --git a/docs/design/decisions/C8-listing-forms-part-after-the-comma.md b/docs/design/decisions/C8-listing-forms-part-after-the-comma.md index 4c3985df..2b1a3277 100644 --- a/docs/design/decisions/C8-listing-forms-part-after-the-comma.md +++ b/docs/design/decisions/C8-listing-forms-part-after-the-comma.md @@ -7,7 +7,7 @@ - P1's family-first fold reaches a positionally read name before the comma as it does without the comma: `de Mesnil Jean, Dr.` → family 'de Mesnil', given 'Jean' under either family-first order — 2026-08-23 #428. - A part after the comma that is nothing but suffix pieces is the credential run, whole, read before any title reading of its words: `Smith, Ph. D. Jr.` → suffix 'Ph. D. Jr.'; `Smith, Sr.` → suffix 'Sr.' where `Sr. Garcia` reads title 'Sr.'; `Smith, Esq.` → suffix 'Esq.'; and `Smith, Dr. Jr.` → title 'Dr.', suffix 'Jr.' — 2026-08-23 #296. - #296's audit of the title/suffix overlap stands: `do`, `jr`, `junior`, `phd` and `se` left the titles, `dr` and `sra` the suffix sets, so a leading `PhD` or `Jr` is a name word (`Jr John Smith` → given 'Jr'), `Do Quang Minh` reads given 'Do' with the particle-or-given fork, and a trailing bare `Dr` is a name word (`John Smith Dr` → family 'Dr') — 2026-08-23 #296. -- `ms` and `sa` stay out of the ambiguous set, duals that position decides: `Smith, Ms.` → suffix 'Ms.'; `Smith, Ms. Jane` → title 'Ms.'; `John Smith, MS` → suffix 'MS' — 2026-08-23 #428. +- `ms` stays out of the ambiguous set, a dual that position decides (`sa` left the vocabulary altogether on 2026-10-25, #857, below): `Smith, Ms.` → suffix 'Ms.'; `Smith, Ms. Jane` → title 'Ms.'; `John Smith, MS` → suffix 'MS' — 2026-08-23 #428. - A delimiter core the policy names is a word in the listing form's part after the comma, not structure, v1's limitation kept as parity (rules/C10's Accepted) — 2026-08-23 #428. Elsewhere: `md`, which #296's audit kept a dual, left the titles — decisions/indic-honorifics' 2026-10-23 (Derek), #853 entry. A credential opening the part after the comma, which C8 held until 2026-10-19 — decisions/C13's 2026-10-04 (Derek), #603 entry, which also makes `Smith, RN - CRNA` read suffix 'RN - CRNA' with the core a word of the run; a lone name word before a part with no given name, as in `Smith, Dr.` → title 'Dr.', family 'Smith', and `Smith, Sir` → given 'Smith' — decisions/H1's 2026-10-18 #720 entry; the period gate the #428 review decided, an ambiguous acronym counting when written with a period after each letter (`Smith, Ed.` stays no credential) — decisions/S3's 2026-10-19 entry; `田中さん, Dr.` keeping its honorific glued, the #428 review's recorded limit, tolerated under rules/W3 — decisions/cjk-comma-demotion's 2026-09-01 #469 entry; the chain's report on a positionally read name before the comma (`Freiherr von Berg Ma, Mr.` reports `particle-or-given`), the #428 review's other limit — decisions/C1's 2026-10-06 (Derek), #613 entry, its decision (4). @@ -16,8 +16,9 @@ Open: none in this file. ## Log -- 2026-08-23 #296/#325 (its `ms` and `sa` joining the ambiguous set superseded by the 2026-08-23 (review of PR #428, four reviewers) entry, which keeps both out; its `md` dual superseded by #853, decisions/indic-honorifics) — the listing form's post-comma part is read for what it is. Three commits of the approved #291/#296 bundle (spec 2026-07-30, plan 2026-08-01; see decisions/comma-suffix-arc), in the plan's order. (1) A comma followed only by titles fixed no family boundary, so the pre-comma name keeps its positional read: `John Smith, Mr.` reads given 'John', family 'Smith', title 'Mr.' where 2.0 had merged the family ('Bob Jones, author' moves with it at every baseline). "Nothing but titles" is title-shaped AND not suffix vocabulary — the plan's helper said only the first, and would have read `Smith, PhD Jr.` as the title run 'PhD Jr.' and `Smith, Mr. Jr.` as 'Mr. Jr.' (both master's suffix 'Jr.'; the gate and a measurement against master caught them). (2) The TITLES/suffix overlap audit and the ordering, one commit because each alone is wrong (the plan's F1): reading the credential run first would make `Smith, Dr.` a suffix while 'dr' was SUFFIX_WORDS residue, and the audit alone leaves `Smith, Jr.` a title by the period-abbreviation inference. The audit table's dispositions ship as approved — 'do', 'jr', 'junior', 'phd', 'se' out of TITLES; 'dr' and 'sra' out of the suffix sets; 'ms' and 'sa' into the ambiguous set — with ONE deviation on evidence the table did not have: 'md' keeps dual membership, because bare 'Md' before a name is the Bengali and South Asian Muslim abbreviation of Muhammad (#343/#345's corpus rows `Md Abdul Karim`, `Md. Abdul Karim`; the drop read the first as given 'Md', middle 'Abdul') while 'MD' after the name is the degree — position decides, as for 'sr'. The ordering is wider than the plan's lone piece: a post-comma part that is nothing but suffix pieces is the credential run, whole, which is what #325 needed (`Smith, Ph. D. Jr.` had put the split credential in the given name once a second suffix made it no longer lone — a 1.4.0 regression; v1 read suffix 'Ph. D.', title 'Jr.'). Vocabulary decides which words qualify and position breaks the tie for the genuine duals: `Smith, Sr.` is Senior where `Sr. Garcia` is Señor, `Smith, Esq.` is the postnominal where H2 still reads `Esq. Smith` as a title (H2 carries that as Accepted). `Smith, Dr. Jr.` keeps the 2.0 deviation's other case — a title, then the last piece as the suffix (v1 read first 'Jr.'; 2.0 through master suffix 'Jr.') — and `Smith, John Jr.` its given name. Consequences the audit carries, all measured and pinned: a leading 'PhD' or 'Jr' is a name word; `Do Quang Minh` reads given 'Do' with the particle-or-given fork, and `Dr. Do Van Johnson, MD` given 'Do', family 'Van Johnson' (the #367 rule for that name is replaced; TITLES ∩ ambiguous is {freiherr, st} now, which keeps the chained emitter reachable); a trailing bare 'Dr' is a name word, joining every other title-only word there (#316 is the open question); `田中さん, PhD` reads suffix 'さん, PhD'; the #100 prefix-join tests use 'lt', a genuine dual, as their token. #291 (multi-word credentials) is the bundle's fourth commit and its own PR. -- 2026-08-23 (review of PR #428, four reviewers) (its `Smith, RN - CRNA` given 'RN' superseded by #603, decisions/C13; its unreported `Freiherr von Berg Ma, Mr.` superseded by #613, decisions/C1) — the repair's condition is "no name word", not "all titles": `John Smith, Mr. Jr.` (a title and a postnominal) had kept the merge the rationale argues against, and reads given 'John', family 'Smith', suffix 'Jr.' now. Its guard counts NAME pieces: the positional read peels a trailing suffix first, so `Smith Jr., Mr.` — two pieces, one name — read positionally lost its family (the code review); it keeps family 'Smith', suffix 'Jr.' at every baseline. The positional read records its ORDER, and post_rules' leading-piece scan keys on "assign records no order after a family comma" rather than on the structure, so P1's family-first fold reaches the pre-comma name as it does without the comma — `de Mesnil Jean, Dr.` under a family-first order had read family 'de' (the test review). The credential run does NOT step over a delimiter core the policy names (`Smith, RN - CRNA` reads given 'RN' under `extra_suffix_delimiters` as without it): the code review asked for it, a draft did it, and tests/test_suffixes.py's "the delimiter must not affect parsing at all" outside the suffix-comma form is v1 parity pinned on purpose — rules/C10 carries the limit as Accepted. A second deviation from the audit table: `ms` and `sa` do NOT join the ambiguous set — the gate is position-blind and the collision is not; gated, `John Smith, MS` lost its suffix-comma route and read title 'MS', and `Smith, Ms.` passed the gate on its one period and read as a credential anyway. Both are genuine duals, and position decides: `Smith, Ms.` reads suffix 'Ms.' (title at every baseline; rules/C8 carries it as Accepted, with `Smith, Ms. Jane` keeping the title) and `John Smith, MS` keeps suffix 'MS'. And the gate itself now means what S2 said (rules/S3 since 2026-10-19): an ambiguous acronym counts when written with ITS periods, one after each letter (`M.A.`, `J.D.`), not on any period — `Smith, Ed.` had become a credential on a name's trailing period, and `Jack Ma.` had read suffix 'Ma.' since 2.0 (S3 carries the boundary since 2026-10-19; 1.4.0 read family 'Ma.'). Two limits recorded rather than fixed: `田中さん, Dr.` keeps the honorific glued, joining master's `田中さん, Mr.` — the honorific peel runs in script_segment on the suffix-comma and no-comma structures, before group or assign can say the comma fixed nothing (C1 Accepted); and group's chain emitter still reports no PARTICLE_OR_GIVEN on the family-comma path, so `Freiherr von Berg Ma, Mr.` reads as its no-comma twin but without the report — group runs before assign decides which read applies, and plumbing a deferred report through the state is more than the shape is worth today. +- 2026-08-23 #296/#325 (its `ms` and `sa` joining the ambiguous set superseded by the 2026-08-23 (review of PR #428, four reviewers) entry, which keeps both out, and that entry's `sa` half by 2026-10-25 #857; its `md` dual superseded by #853, decisions/indic-honorifics) — the listing form's post-comma part is read for what it is. Three commits of the approved #291/#296 bundle (spec 2026-07-30, plan 2026-08-01; see decisions/comma-suffix-arc), in the plan's order. (1) A comma followed only by titles fixed no family boundary, so the pre-comma name keeps its positional read: `John Smith, Mr.` reads given 'John', family 'Smith', title 'Mr.' where 2.0 had merged the family ('Bob Jones, author' moves with it at every baseline). "Nothing but titles" is title-shaped AND not suffix vocabulary — the plan's helper said only the first, and would have read `Smith, PhD Jr.` as the title run 'PhD Jr.' and `Smith, Mr. Jr.` as 'Mr. Jr.' (both master's suffix 'Jr.'; the gate and a measurement against master caught them). (2) The TITLES/suffix overlap audit and the ordering, one commit because each alone is wrong (the plan's F1): reading the credential run first would make `Smith, Dr.` a suffix while 'dr' was SUFFIX_WORDS residue, and the audit alone leaves `Smith, Jr.` a title by the period-abbreviation inference. The audit table's dispositions ship as approved — 'do', 'jr', 'junior', 'phd', 'se' out of TITLES; 'dr' and 'sra' out of the suffix sets; 'ms' and 'sa' into the ambiguous set — with ONE deviation on evidence the table did not have: 'md' keeps dual membership, because bare 'Md' before a name is the Bengali and South Asian Muslim abbreviation of Muhammad (#343/#345's corpus rows `Md Abdul Karim`, `Md. Abdul Karim`; the drop read the first as given 'Md', middle 'Abdul') while 'MD' after the name is the degree — position decides, as for 'sr'. The ordering is wider than the plan's lone piece: a post-comma part that is nothing but suffix pieces is the credential run, whole, which is what #325 needed (`Smith, Ph. D. Jr.` had put the split credential in the given name once a second suffix made it no longer lone — a 1.4.0 regression; v1 read suffix 'Ph. D.', title 'Jr.'). Vocabulary decides which words qualify and position breaks the tie for the genuine duals: `Smith, Sr.` is Senior where `Sr. Garcia` is Señor, `Smith, Esq.` is the postnominal where H2 still reads `Esq. Smith` as a title (H2 carries that as Accepted). `Smith, Dr. Jr.` keeps the 2.0 deviation's other case — a title, then the last piece as the suffix (v1 read first 'Jr.'; 2.0 through master suffix 'Jr.') — and `Smith, John Jr.` its given name. Consequences the audit carries, all measured and pinned: a leading 'PhD' or 'Jr' is a name word; `Do Quang Minh` reads given 'Do' with the particle-or-given fork, and `Dr. Do Van Johnson, MD` given 'Do', family 'Van Johnson' (the #367 rule for that name is replaced; TITLES ∩ ambiguous is {freiherr, st} now, which keeps the chained emitter reachable); a trailing bare 'Dr' is a name word, joining every other title-only word there (#316 is the open question); `田中さん, PhD` reads suffix 'さん, PhD'; the #100 prefix-join tests use 'lt', a genuine dual, as their token. #291 (multi-word credentials) is the bundle's fourth commit and its own PR. +- 2026-08-23 (review of PR #428, four reviewers) (its `sa` half superseded by 2026-10-25 #857; its `Smith, RN - CRNA` given 'RN' superseded by #603, decisions/C13; its unreported `Freiherr von Berg Ma, Mr.` superseded by #613, decisions/C1) — the repair's condition is "no name word", not "all titles": `John Smith, Mr. Jr.` (a title and a postnominal) had kept the merge the rationale argues against, and reads given 'John', family 'Smith', suffix 'Jr.' now. Its guard counts NAME pieces: the positional read peels a trailing suffix first, so `Smith Jr., Mr.` — two pieces, one name — read positionally lost its family (the code review); it keeps family 'Smith', suffix 'Jr.' at every baseline. The positional read records its ORDER, and post_rules' leading-piece scan keys on "assign records no order after a family comma" rather than on the structure, so P1's family-first fold reaches the pre-comma name as it does without the comma — `de Mesnil Jean, Dr.` under a family-first order had read family 'de' (the test review). The credential run does NOT step over a delimiter core the policy names (`Smith, RN - CRNA` reads given 'RN' under `extra_suffix_delimiters` as without it): the code review asked for it, a draft did it, and tests/test_suffixes.py's "the delimiter must not affect parsing at all" outside the suffix-comma form is v1 parity pinned on purpose — rules/C10 carries the limit as Accepted. A second deviation from the audit table: `ms` and `sa` do NOT join the ambiguous set — the gate is position-blind and the collision is not; gated, `John Smith, MS` lost its suffix-comma route and read title 'MS', and `Smith, Ms.` passed the gate on its one period and read as a credential anyway. Both are genuine duals, and position decides: `Smith, Ms.` reads suffix 'Ms.' (title at every baseline; rules/C8 carries it as Accepted, with `Smith, Ms. Jane` keeping the title) and `John Smith, MS` keeps suffix 'MS'. And the gate itself now means what S2 said (rules/S3 since 2026-10-19): an ambiguous acronym counts when written with ITS periods, one after each letter (`M.A.`, `J.D.`), not on any period — `Smith, Ed.` had become a credential on a name's trailing period, and `Jack Ma.` had read suffix 'Ma.' since 2.0 (S3 carries the boundary since 2026-10-19; 1.4.0 read family 'Ma.'). Two limits recorded rather than fixed: `田中さん, Dr.` keeps the honorific glued, joining master's `田中さん, Mr.` — the honorific peel runs in script_segment on the suffix-comma and no-comma structures, before group or assign can say the comma fixed nothing (C1 Accepted); and group's chain emitter still reports no PARTICLE_OR_GIVEN on the family-comma path, so `Freiherr von Berg Ma, Mr.` reads as its no-comma twin but without the report — group runs before assign decides which read applies, and plumbing a deferred report through the state is more than the shape is worth today. - 2026-10-04 (Derek), #603 — A credential opening the part after the comma makes it the postnominal part, whatever follows. The record moved to decisions/C13 with that clause (2026-10-19). - 2026-10-18 #728 — SPLIT FROM decisions/C1 with the rule. The entries above were written under decisions/C1 while this rule was part of rules/C1, and moved here unchanged but for the pointers the move broke, the measurement paragraphs, each cut to one line naming its population, counts, baseline and PR, and the superseded passages, each cut to its decision and what replaced it; where one says rules/C1 or C1, it means the rule as it then stood, which held this one. - 2026-10-19 — RULE SPLIT: a credential opening the part after the comma left rules/C8 for rules/C13, one rule per piece of evidence (docs/design/AGENTS.md); C8 keeps the part with no name word and the part that is nothing but suffix words. The #603 entry moved to decisions/C13, a cross-reference standing in its place. Where the entries here or elsewhere say rules/C8 of the opened part (the #613 and #544 entries in decisions/C1 among them), that piece is rules/C13's now. +- 2026-10-25 #857 — `sa` leaves the vocabulary: its military reading (Seaman Apprentice, a title) and its credential reading are both refused on the word (`excluded`), superseding the 2026-08-23 entry's `sa` half and decisions/suffix-acronym-collisions' 2026-10-21 held-out entry. It reads as any name word does, whatever the case, because the Portuguese surname Sá written without its accent is the one bearer the corpora show. The path to it: marking `sa` ambiguous (`may_be_name`) fixed `Maria de Sa` but left three costs, and the evidence for each reading it was keeping came back empty. (1) A one-case name ending in `SA` still read a suffix (S6), so all 255 SÁ / SA pairs of tse-donors (#843) still parted. (2) The title reading took the slot in front of a name and alone after a family comma, which `ma` does not (`Smith, Sa` → title 'Sa' where `Smith, Ma` → given 'Ma'; `Smith, John PhD Sa` → title 'Sa', suffix 'PhD', unreported). (3) Every name read with `Sa` as a title in the committed and eight local corpora is a person whose given name is Sa (`Sa Liu`, `Sa Mahammed Ilyas`) or the Arabic Sa ad (`Sa ad Hariri`), or whose Portuguese surname Sá is written without its accent (`Sa Pinto`), none a rank; and every SA or S.A. suffix is a company (`Galderma S.A.`, `Novartis Biociências SA`), a South African qualifier (`FCP(SA)`), or initials typed without stops (`MANN, S.A.`, `BROWN, MARYAM SA`) — no person writes it as a credential, the 167 NPPES providers aside, whom `name_usage.py` counts and nobody has read by item. Readings now: `Maria de Sa`, `MARIA DE SA`, `maria de sa` → family 'de Sa'/'DE SA'/'de sa' (was family 'de', suffix 'Sa'/'SA'/'sa' since 1.4.0); `Joao Sa`, `ANA SA`, `ANA SILVA SA` → family 'Sa'/'SA'; `SA Smith` → given 'SA', family 'Smith'; `Smith, SA` → given 'SA' (was suffix 'SA', #296's audit row); `Mrs Sa Li` → title 'Mrs', given 'Sa' (the second title word is gone). Accepted: a credential `SA` after a mixed-case name is a name word, so `John Smith SA` reads middle 'Smith', family 'SA' (rules/S9 admits an unlisted all-caps word as a credential by shape only behind a comma, and this one has no vocabulary behind it); and so are the comma shapes, the same reading an unlisted two-letter caps word gets (`John Smith, XY`): `John Smith, SA` → given 'SA', family 'John Smith' (the name merged into the family, where 1.4.0 through 2.5.0 read given 'John', family 'Smith', suffix 'SA'), `John Smith, SA, PhD` → family 'John Smith', suffix 'PhD', `Smith, John SA` → middle 'SA', and `, SA` → given 'SA' (1.4.0 read title 'SA'; 2.5.0 suffix 'SA'). The corpora hold 15 such after-comma names in ctgov-officials and 10 in fec-individuals (`Spector, SA` went suffix → given 'SA'), and NPPES counts 167 US healthcare providers who write it, none read by item: if they turn out to be a credential, that is the evidence to restore it on. `Smith, John PhD Sa` keeps suffix 'PhD Sa' and reports SUFFIX_OR_NAME; `John Smith S.A.` and `Smith, S.A.` go through the unlisted-dotted rule and report SUFFIX_OR_NAME; a parenthesised `(SA)` after a credential, ClinicalTrials.gov's `FCP (SA)`, is a nickname 'SA' where it was inside the suffix (25 names in ctgov-officials). Local corpora, `snapshot.py --local` against the base 8c965d74 (2026-10-25): tse-donors moves 1,613 of 848,351 names, about 700 gaining the `SA` as part of the family name (`ADAO DE SA FERREIRA` → family 'DE SA FERREIRA', was family 'DE', suffix 'SA FERREIRA'; `WESLEY VITOR SA PEREIRA` → family 'PEREIRA'), and `local_corpora.py pairs tse-donors --agreement` reads 0 accent-only pairs with two surnames where the base read 255 SÁ / SA; ctgov-officials 54 of 352,104, 25 of them the `(SA)` nickname; psc-individuals 40 of 1,034,888 (three `Mrs Sa Li`-shaped names lose a title); ukec-individuals 0. Putting a reading back is cheap — add it and delete the matching `excluded` entry, which the build checks — but the form matters: a `credential` reading WITH `may_be_name` returns the costs the path above ended on (255 parting pairs, `Smith, Sa` reading a title once `military` is back), and one without it returns `Maria de Sa` reading suffix. A caller who needs `SA` as a credential adds it to their own lexicon. diff --git a/docs/design/decisions/comma-suffix-arc.md b/docs/design/decisions/comma-suffix-arc.md index 568e2582..b91b6ecc 100644 --- a/docs/design/decisions/comma-suffix-arc.md +++ b/docs/design/decisions/comma-suffix-arc.md @@ -12,13 +12,13 @@ its decline-by-measurement evidence in the issue body; "z domu" folded into it 2 Not standing: plan amendment 1's repair, carried out by #296's first commit — 2026-08-01. -Elsewhere: the audit table's dispositions as shipped, `md` kept a dual (until #853) and `ms` and `sa` kept out of the ambiguous set — decisions/C8's 2026-08-23 #296 and 2026-08-23 #428 Standing lines, the second also holding the repair's condition, no name word; the trailing slot read by vocabulary, a period-marked title word there a title (`John Smith Prof.` → title 'Prof.') — decisions/H5's 2026-09-08 #316 entry; the dotted shape class — decisions/S9's 2026-09-14 (Derek) entry headed #516's DOTTED HALF IS A SWITCH. +Elsewhere: the audit table's dispositions as shipped, `md` kept a dual (until #853) and `ms` and `sa` kept out of the ambiguous set (`sa` until 2026-10-25, #857, when it left the vocabulary) — decisions/C8's 2026-08-23 #296 and 2026-08-23 #428 Standing lines, the second also holding the repair's condition, no name word; the trailing slot read by vocabulary, a period-marked title word there a title (`John Smith Prof.` → title 'Prof.') — decisions/H5's 2026-09-08 #316 entry; the dotted shape class — decisions/S9's 2026-09-14 (Derek) entry headed #516's DOTTED HALF IS A SWITCH. Open: none in this file. ## Log -- 2026-07-26 #291/#296/#316 (its audit table's `ms` and `sa` joining the ambiguous set superseded by decisions/C8's 2026-08-23 (review of PR #428, four reviewers) entry; its trailing-abbreviation decline superseded for the dotted spelling by decisions/S9's 2026-09-14 (Derek) entry headed #516's DOTTED HALF IS A SWITCH) — THE COMMA-SUFFIX ARC, 2026-07-26 → 2026-08-01: the paragraph, the plan amendment and the Declined entries below. +- 2026-07-26 #291/#296/#316 (its audit table's `ms` and `sa` joining the ambiguous set superseded by decisions/C8's 2026-08-23 (review of PR #428, four reviewers) entry (`sa` left the vocabulary instead on 2026-10-25, #857, decisions/C8); its trailing-abbreviation decline superseded for the dotted spelling by decisions/S9's 2026-09-14 (Derek) entry headed #516's DOTTED HALF IS A SWITCH) — THE COMMA-SUFFIX ARC, 2026-07-26 → 2026-08-01: the paragraph, the plan amendment and the Declined entries below. The approved 19-word TITLES∩suffix audit table (2026-07-30/08-01, intent for the unshipped bundle): jr, junior, phd, md, do and se DROP from TITLES (se: Structural Engineer is the PE/SE post-nominal pair, no prenominal convention — v1 residue); dr drops from SUFFIX_WORDS and sra from the suffix sets (v1 residue); sr, lt, cpl, cpt, cpo, csm, sgm, ra and vc KEEP dual membership (position decides); ms and sa keep dual membership AND join the ambiguous set (the periods gate). Derek's framing thesis for the whole table: v1's sets encoded where the v1 PARSER needed words to be, not where words can occur. Amendment A6: once SUFFIX_PHRASES ships, the glued peel's token-level post-nominal test cannot step over a phrase segment ("김민준씨, LEED AP" will not peel) — #291 owns deciding that, with a case row either way. diff --git a/docs/design/decisions/suffix-acronym-collisions.md b/docs/design/decisions/suffix-acronym-collisions.md index 52647d00..4bd32548 100644 --- a/docs/design/decisions/suffix-acronym-collisions.md +++ b/docs/design/decisions/suffix-acronym-collisions.md @@ -14,14 +14,14 @@ Closes #342 (a wordlist question) and #454 (a rules.md question) together, becau - Under the default order, a marked member that reads as a name is taken into the family by a particle in front of it, with the words between (`John de Smith Chi` → family 'de Smith Chi', where `John de Smith CHI` keeps suffix 'CHI'), an accepted cost — 2026-10-21 #791; 2026-10-21. - 33 more credential readings, listed by `uv run python tools/vocab/query.py --issue 796` and `se` and `om` among them, are marked ambiguous rather than removed: 30 because the surname reading is substantial (US Census 2010 bearers ≥200, or Wikidata family bearers ≥20), a threshold that is a judgement and not a rule the data sets — 2026-10-21. - Where both counts are small, the surname count is weighed against the NPPES count of US healthcare providers writing the word, and the word is marked where the surname reading is the larger (`capa`, `ams`); a provider count is evidence that a credential is live, and a zero says nothing about a credential outside healthcare, so `ei`, whose credential is not a healthcare one, rests on the default under uncertainty instead — 2026-10-21. -- Held out of the marking: `am`, `sa` and `pla`, whose marking regressed CLDR names; `edd`, which would read `EdD` as the family whether marked or removed; `ra`, also a title; `mba`, whose credential reading dominates; and `cap` and `aas`, whose provider counts are as large as their surname counts — 2026-10-21. +- Held out of the marking: `am`, `sa` (left every vocabulary set on 2026-10-25, #857, decisions/C8) and `pla`, whose marking regressed CLDR names; `edd`, which would read `EdD` as the family whether marked or removed; `ra`, also a title; `mba`, whose credential reading dominates; and `cap` and `aas`, whose provider counts are as large as their surname counts — 2026-10-21. - 36 credentials that read as the given name after a comma because nothing listed them (`John Smith, OTR/L` → given 'OTR/L') are added, `pa` the one marked ambiguous (Census 739 family, SSA 1,343 given); `pi`, Principal Investigator and a surname, is refused — 2026-10-21 #797. - `ing`, the engineering degree of Czech, Slovak, Austrian and Romanian names, is a credential marked ambiguous (Census 1,814 family; a given name in Sweden), so `Viera Sevcikova, Ing` reads suffix 'Ing' where it reversed and `John Ing` keeps the family name; the period-marked `Ing.` after a full name and a comma moves with it from title to suffix, unlike `Mgr.` and `MUDr.`, and the capitals `ING.` and the dotted initials `I.N.G.` read as `MA.` and `M.A.` do, a name spelled in the initials lost where they stand last (`John I.N.G.`), both accepted costs — 2026-10-23 #848. - `mc` and `vd` stay unambiguous suffix acronyms (#454 is by design): neither is a borne name, and a bare trailing one reads as a post-nominal, leaving no family name (`Donald mc`, `Smith vd`); `Mc Donald` reads family 'Mc Donald' because `mc` is a never-given particle — 2026-09-07 #342. - `esq` is a suffix word and not an acronym, a contraction rather than an initialism, so no word is both a suffix acronym and a suffix word — 2026-09-08 #316. - The Wikipedia footnote marks `cva[22]`, `gaee[14]` and `np[18]` are out of the suffix acronyms, and a list imported from a wiki is checked for reference marks before it lands — 2026-10-18. -Elsewhere: the case lean that decides the capitals forms above is decisions/S6's 2026-09-14 #289 and 2026-09-14 (Derek) entries; the chunked dotted spelling passing the period gate (`Wang M.Eng.` → suffix) and the credential run that ends in a member keeping its suffix comma (`John Smith, PhD MEng` → suffix 'PhD MEng'), with the run a member leads keeping the cost (`John Smith MEng PhD` → family 'MEng'), are decisions/S7's 2026-09-27 (Derek), #544 entry headed A CREDENTIAL RUN KEEPS ITS AMBIGUOUS MEMBERS, which this file's 2026-09-27 (#544) entry indexes; the shape-plus-position heuristic this file parked shipped as the two switches of decisions/S9's 2026-09-14 (Derek) entries, the all-caps one reading the comma position by default since its 2026-10-01 (Derek), #564 entry; the roman-numeral accident's narrow retirement is decisions/S3's 2026-09-15 (Derek) entry, which this file's 2026-09-15 twin indexes; and `sa` stays a title and suffix dual that position decides by #296's audit (decisions/comma-suffix-arc's audit table, whose `sa` row moving it into the ambiguous set decisions/C8's 2026-08-23 (review of PR #428, four reviewers) entry declined), held out of the marking again by the 2026-10-21 entry. +Elsewhere: the case lean that decides the capitals forms above is decisions/S6's 2026-09-14 #289 and 2026-09-14 (Derek) entries; the chunked dotted spelling passing the period gate (`Wang M.Eng.` → suffix) and the credential run that ends in a member keeping its suffix comma (`John Smith, PhD MEng` → suffix 'PhD MEng'), with the run a member leads keeping the cost (`John Smith MEng PhD` → family 'MEng'), are decisions/S7's 2026-09-27 (Derek), #544 entry headed A CREDENTIAL RUN KEEPS ITS AMBIGUOUS MEMBERS, which this file's 2026-09-27 (#544) entry indexes; the shape-plus-position heuristic this file parked shipped as the two switches of decisions/S9's 2026-09-14 (Derek) entries, the all-caps one reading the comma position by default since its 2026-10-01 (Derek), #564 entry; the roman-numeral accident's narrow retirement is decisions/S3's 2026-09-15 (Derek) entry, which this file's 2026-09-15 twin indexes; and `sa` was a title and suffix dual that position decided by #296's audit (decisions/comma-suffix-arc's audit table, whose `sa` row moving it into the ambiguous set decisions/C8's 2026-08-23 (review of PR #428, four reviewers) entry declined), held out of the marking again by the 2026-10-21 entry, and has been in neither set since decisions/C8's 2026-10-25 #857 entry. Not standing: the #454 bullet's note that the `fix(suffix-routing)` ledger rule stays, the ledgers having been retired for the snapshot (decisions/differential-ledger, #664), and the repair of rules/P6's Accepted clause recorded in the "What #454 got right" bullet, carried out in the same change — 2026-09-07 #342. @@ -45,7 +45,7 @@ Open: #752 (phase 2 derives the ambiguous marking from clashing readings, which - **What #454 got right, and it is a rules.md defect that is now fixed.** rules/P6's Accepted clause said that under the default order a comma-less name keeps its positional reading, with "Jong Anke de" → family "de" as its only example. Measured, that holds for de and for do and fails for exactly the two words that are in BOTH the particle vocabulary and the UNAMBIGUOUS suffix vocabulary: "Donald mc" and "Smith vd" lose the family entirely to `suffix`. The clause is repaired to say so, and the two places that already conceded it in prose — the `'Donald mc'` bullet of #differential-ledger's fields-only arc and the corresponding comment in expected_since_1.4.0.toml — are amended from "recorded here rather than fixed" to point at the repair. No example line moves: the clause's own example is de-shaped and stays true, which is why no test caught the defect and why regenerating corpus_rules.jsonl is a no-op here. Recompute the membership with `L = Parser().lexicon; sorted(L.particles & (L.suffix_acronyms - L.suffix_acronyms_ambiguous))`, which gives ['mc', 'vd'] on 2026-09-07, and `sorted(L.particles & L.suffix_acronyms_ambiguous)`, which gives ['do']. - **Parking lot: the shape-plus-position credential heuristic.** (Superseded by the 2026-09-15 (#289/#516) entry below that closes it.) #490's idea without the "collides with vocabulary" part, and what would recover "John Smith RAI" without any wordlist at all. Two shapes: an ALL-CAPS acronym standing in the suffix position of a MIXED-CASE name ("John Smith XYZ", "John Smith, XYZ"), where the case contrast is the signal; and a DOTTED acronym in that position regardless of case ("John Smith X.Y.Z.", "john smith x.y.z."), where the periods are. Measured 2026-09-07, none of the four is read as a suffix today — the bare form gives family XYZ, the comma form given XYZ with family "John Smith", and the second two family X.Y.Z. — while the roman-numeral accident above reads three dotted forms as suffixes on a fork that is about generations, not credentials. That contrast is the evidence the dotted shape is unhandled: the parser already produces the wanted answer for X.Y.I. and R.A.V. by coincidence and the unwanted one for R.A.X. A follow-up issue carries it; nothing in this entry depends on it. - **esq left SUFFIX_ACRONYMS (2026-09-08, the #316/#489 bundle).** (Its `John Smith E.S.Q.` measurement superseded by decisions/S9's dotted shape, by which that spelling reads suffix again.) The same criterion asked of a word the frequency argument cannot reach: does the entry describe the WORD or the machinery. Esquire is a contraction, not an initialism, so the initialism set was the wrong home for it whatever its frequency — this is the criterion's third kind of answer, and it is the one rai and cha did not need. The entry arrived with the same 2019-12-11 bulk import (af5bdab, #93) that brought rai and cha and was never reviewed either; its only unique coverage was the multi-dot spelling `E.S.Q.`, the SUFFIX_WORDS membership carrying every single-token spelling. Measured over the corpus glob: one name moves, `John Smith E.S.Q.`, from suffix to family, at every baseline. What the removal buys is the invariant this entry's neighbours could not have: `SUFFIX_ACRONYMS ∩ SUFFIX_WORDS == ∅` is asserted at import (a vocabulary build check since 2026-10-18: one suffix kind per word, each an acronym or a word; decisions/vocabulary-model), esq having been the only member of that intersection and the reason the assert could not exist. Recompute the intersection with `L = Parser().lexicon; sorted(L.suffix_acronyms & L.suffix_words)`, which gives `[]` on 2026-09-08. The standing keep-out is the Excluded (SUFFIX_ACRONYMS — esq) block in decisions/vocabulary-exclusions. -- **Not decided here:** (Its se and om half superseded by the 2026-10-21 entry below, which marks both ambiguous.) sa stays where #296's audit put it (title and suffix dual, position decides); se and om stay unambiguous, there being no surname evidence worth standing behind and OM being the Order of Merit; the S2 comma-form boundary ("Smith, Ed" → given Ed) is not reopened. #342's own comment lists sa, se and om beside ba and cha as model recall rather than corpus-attested, and per the #360 lesson they need a human before they move. +- **Not decided here:** (Its se and om half superseded by the 2026-10-21 entry below, which marks both ambiguous; its sa half by decisions/C8's 2026-10-25 #857 entry, which removes sa from the vocabulary instead.) sa stays where #296's audit put it (title and suffix dual, position decides); se and om stay unambiguous, there being no surname evidence worth standing behind and OM being the Order of Merit; the S2 comma-form boundary ("Smith, Ed" → given Ed) is not reopened. #342's own comment lists sa, se and om beside ba and cha as model recall rather than corpus-attested, and per the #360 lesson they need a human before they move. - **Measurement (2026-09-07).** Five corpus names move, every one from the radar-tier corpus_issues.jsonl and with one cause (Aishwarya Rai, Lala Lajpat Rai, John Smith RAI, John Smith, RAI, Ahmad Jayadi, CHA), measured over the tools/differential/corpus*.jsonl glob against the shipped lexicon with the two entries restored (`Lexicon.default().add(suffix_acronyms={"rai","cha"})`); the ba step moves nothing, and Aishwarya Rai is 1.4.0's own reading (PR #515). - **2026-09-15 (#289/#516) (its `John Smith, XYZ` reading superseded by decisions/S9's 2026-10-01 #564 entry, which reads it suffix by default) — the parking lot is CLOSED, and this entry's "measured 2026-09-07, none of the four is read as a suffix today" sentence is superseded rather than rewritten.** The shape-plus-position heuristic this entry parked is what shipped: the DOTTED shape as `Policy.unlisted_dotted_suffixes`, default on, and the ALL-CAPS shape as `Policy.unlisted_caps_suffixes`, default off. Of the four names the bullet measured, `John Smith X.Y.Z.` and `john smith x.y.z.` now read suffix at the default, and `John Smith XYZ` / `John Smith, XYZ` read suffix only with the opt-in switch. See decisions/S9 for the whole decision and for why the two halves have different defaults. - **2026-09-15 (#289/#516) — the roman-numeral accident is RETIRED, and here is what narrowly means.** This entry's bullet recorded the accident as evidence rather than as behavior anyone relied on. It now fires no longer where every chunk the vocabulary matches is a single ASCII character — measured, `{i, v, 2}` — so `John Smith R.A.I.` and `John Smith X.Y.I.` reach the shape class instead and read by position, which gives the same fields for the first and 1.4.0's family reading for `Jack X.Y.I.`. `John Smith C.H.A.`, the bullet's other half, is unaffected: nothing claimed it before or after. The WIDE retirement was measured and rejected; decisions/S3 records the three names that pay for it. @@ -56,7 +56,7 @@ Open: #752 (phase 2 derives the ambiguous marking from clashing readings, which - **2026-10-18 — cva[22], gaee[14] and np[18] left SUFFIX_ACRONYMS: Wikipedia footnote marks, not credentials.** All three arrived with the `af5bdab` bulk import (2019-12-11), the bracketed number being the source page's reference mark copied along with the acronym; the 3.0 vocabulary spec's migration spike found them. They matched only a token written with the mark itself, so `John Smith NP[18]` read suffix `NP[18]` and now reads family `NP[18]`, and nothing else moves: `np` and `cva` have clean entries of their own, and `gaee` never had one (`John Smith GAEE` reads family `GAEE` before and after). Whether GAEE should be added as a clean entry is a separate vetting question, not taken here. A bulk import from a wiki page carries its reference marks: check a new list for brackets before it lands. Recompute that no vocabulary entry holds one: `uv run python -c "from nameparser import Lexicon; L = Lexicon.default(); print(sorted(w for f in ('titles', 'suffix_acronyms', 'suffix_words', 'particles', 'conjunctions', 'bound_given_names', 'maiden_markers') for w in getattr(L, f) if set(w) & set('[]')))"`. -- **2026-10-21 — 33 credential readings marked ambiguous (C-i's default), not removed.** The 33 readings are listed by `uv run python tools/vocab/query.py --issue 796`; each carries its Census, Wikidata and SSA counts with an `as_of` date, and an NPPES note. The candidate screen found 58 unflagged credentials with census or Wikidata surname evidence. The batch keeps the words whose surname reading is substantial (US Census 2010 bearers ≥200, or Wikidata family bearers ≥20). That threshold stood in for the credential half of C-i while no provider count existed. Since #798 the NPPES count (US healthcare providers writing the word, August 2026 file) measures that half, with one direction only: a count is evidence a credential is live, while a zero says nothing about a non-healthcare credential. Revised rule, dated 2026-10-21: when both counts are small, compare them and mark where the surname reading is the larger. Words that the rule re-adds: `capa` (surname 175 census / 7 Wikidata family, NPPES 11) and `ams` (135 / 2, NPPES 5). `ei` (127 / 3, NPPES 12; also a Korean given name in nam_dict) rests on C-i's default under uncertainty alone, because EI (usually Engineer Intern) is not a healthcare credential, so NPPES says nothing about it. Pinned by the `robert_capa_is_a_surname` and `all_caps_capa_is_still_a_credential` case rows. The census lookup folds a spelling to its letters (`name_usage.py`), so `pa-c` was counted as the surname `PAC`; it is dropped, since no `Pa-c` surname is measured. The threshold is a judgement, not a rule the data sets, and the held-out words below are the ones it cannot place. Dropped for weak surname evidence (14): `abd`, `ceh`, `fala`, `ches`, `cas`, `cia`, `res`, `obe`, `facha`, `ame`, `usa`, `aia`, `iso`, `rid`. Several of these are recognizable designations, which the threshold does not itself test. Dropped after the NPPES check (US healthcare providers, a live-credential count): `cap` (759 providers against 865 census bearers) and `aas` (528 against 238), where the credential reading is at least as large as the surname reading. Held out: `am`, `sa`, `pla` (CLDR regressions: Gaelic `am post`, Burmese `Khun Sa`, Spanish `Hernández Pla`); `edd` (`John Smith EdD` reads `EdD` as family under either option); `ra` (also a title: `John Smith Ra.` reads title `Ra.` with no flag); `mba` (its credential reading is the dominant one). `pe` is kept on surname evidence (582 census, 305 Wikidata family) although it is also a designation. Removal, measured on the 53-word set, is the worse answer on real designations: `john smith mba` reads `mba` as family, and `John Smith, MBA` reads `MBA` as a given name, where the flag keeps the suffix and reports the fork. Measured on the tree (30 words): the corpus names that move are re-recorded in `snapshot.jsonl`, with the agreement check unchanged (`Maheen Farooqi, ACA`, `Pèrs, Ciro di`), with 0 agreement regression lines across the CLDR, GND, Wikidata, native and conventions corpora. Every batch member moves bracketed, double-quoted and single-quoted credentials to nickname (rules/S1 is silent on that coupling; decisions/ma-do records it for ma and do). A single trailing period (`John Smith Caro.`) reads family under the flag too. The written case decides: UPPER moves nothing, per S6 before S4. +- **2026-10-21 — 33 credential readings marked ambiguous (C-i's default), not removed.** The 33 readings are listed by `uv run python tools/vocab/query.py --issue 796`; each carries its Census, Wikidata and SSA counts with an `as_of` date, and an NPPES note. The candidate screen found 58 unflagged credentials with census or Wikidata surname evidence. The batch keeps the words whose surname reading is substantial (US Census 2010 bearers ≥200, or Wikidata family bearers ≥20). That threshold stood in for the credential half of C-i while no provider count existed. Since #798 the NPPES count (US healthcare providers writing the word, August 2026 file) measures that half, with one direction only: a count is evidence a credential is live, while a zero says nothing about a non-healthcare credential. Revised rule, dated 2026-10-21: when both counts are small, compare them and mark where the surname reading is the larger. Words that the rule re-adds: `capa` (surname 175 census / 7 Wikidata family, NPPES 11) and `ams` (135 / 2, NPPES 5). `ei` (127 / 3, NPPES 12; also a Korean given name in nam_dict) rests on C-i's default under uncertainty alone, because EI (usually Engineer Intern) is not a healthcare credential, so NPPES says nothing about it. Pinned by the `robert_capa_is_a_surname` and `all_caps_capa_is_still_a_credential` case rows. The census lookup folds a spelling to its letters (`name_usage.py`), so `pa-c` was counted as the surname `PAC`; it is dropped, since no `Pa-c` surname is measured. The threshold is a judgement, not a rule the data sets, and the held-out words below are the ones it cannot place. Dropped for weak surname evidence (14): `abd`, `ceh`, `fala`, `ches`, `cas`, `cia`, `res`, `obe`, `facha`, `ame`, `usa`, `aia`, `iso`, `rid`. Several of these are recognizable designations, which the threshold does not itself test. Dropped after the NPPES check (US healthcare providers, a live-credential count): `cap` (759 providers against 865 census bearers) and `aas` (528 against 238), where the credential reading is at least as large as the surname reading. Held out (`sa` removed from the vocabulary after all on 2026-10-25, #857, decisions/C8; its `Khun Sa` regression no longer holds, the name reading family 'Sa' now): `am`, `sa`, `pla` (CLDR regressions: Gaelic `am post`, Burmese `Khun Sa`, Spanish `Hernández Pla`); `edd` (`John Smith EdD` reads `EdD` as family under either option); `ra` (also a title: `John Smith Ra.` reads title `Ra.` with no flag); `mba` (its credential reading is the dominant one). `pe` is kept on surname evidence (582 census, 305 Wikidata family) although it is also a designation. Removal, measured on the 53-word set, is the worse answer on real designations: `john smith mba` reads `mba` as family, and `John Smith, MBA` reads `MBA` as a given name, where the flag keeps the suffix and reports the fork. Measured on the tree (30 words): the corpus names that move are re-recorded in `snapshot.jsonl`, with the agreement check unchanged (`Maheen Farooqi, ACA`, `Pèrs, Ciro di`), with 0 agreement regression lines across the CLDR, GND, Wikidata, native and conventions corpora. Every batch member moves bracketed, double-quoted and single-quoted credentials to nickname (rules/S1 is silent on that coupling; decisions/ma-do records it for ma and do). A single trailing period (`John Smith Caro.`) reads family under the flag too. The written case decides: UPPER moves nothing, per S6 before S4. - **Order dependence (measured 2026-10-21, #796).** Recipe as in the #791 entry below (corrected 2026-10-21: this said "above", the entry standing after this one), over the 33 words: 9 templates × 2 name pairs × 3 casings (Title, UPPER, lower) = 1,782 cells per order, marked tree (`Lexicon.default()`) against the comparator `Lexicon.default().remove(suffix_acronyms_ambiguous=…33 words…)`. Moves: `GIVEN_FIRST` 1,122 (Title 594, UPPER 66, lower 462); `FAMILY_FIRST` 1,056 (Title 594, lower 462, UPPER 0); `FAMILY_FIRST_GIVEN_LAST` 1,056 (the same split as `FAMILY_FIRST`). The family-first orders move nothing in UPPER, the case signal (rules/S6) deciding there. Credential inside a multi-word family name (the `John de Smith Chi` class, which these templates do not contain), probed separately with `John de Smith W`, `John de W` and `John van der Smith W` in Title and lower casing, 198 cells: `GIVEN_FIRST` 99 moves, every one reading the credential inside the family (`John de Smith Caro` reads family `de Smith Caro`, with no suffix); `FAMILY_FIRST` and `FAMILY_FIRST_GIVEN_LAST` 99 moves with no credential inside a multi-word family. This is the surname reading the marking is for, and it is the accepted cost of a particle before the word under `GIVEN_FIRST`; it is not checked against 1.4.0 here. - **2026-10-21 (#797) — 36 credentials added where the comma path read them as the given name; the vocabulary half only.** `John Smith, MBChB` gave given 'MBChB' and family 'John Smith' because no list held the word and no shape reads it: C6's caps shape needs an all-capitals word, C5 reads an unlisted two-capital word as paired initials on purpose, and a hyphen or slash puts a word outside the caps shape. Whether an unlisted hyphen- or slash-joined run of capitals should count by shape stays open in #797 beside S9; this entry is the words. **Candidates** came from two sources, and only tokens that reverse `John Smith, X` today were kept. NPPES (`name_usage.py --rank 600 --unlisted`, August 2026 file): at least 2,000 providers, about rank 150, below which the list is fragments of split fields (`BC`, `PC`, `AA`). ClinicalTrials.gov study officials (`local_corpora.py tally ctgov-officials`, data 2026-10-21): at least 100 officials writing the token after a comma as typed. The cutoffs are a judgement; a mixed-case credential with no name evidence costs nothing to add below them, and 17 sit there at 30 to 73 officials (`MCh`, `FRCPath`, `MRCPsych`, `BPharm`, `MEd` and `AuD` among them). **Added, plain** (no Census 2010, SSA or Wikidata bearer above each source's floor, or a credential count far above the name count): `mbchb`, `mbbch`, `dmsc`, `dmsci`, `scd`, `dphil`, `mmed`, `mmsc`, `mphil`, `mhsc`, `scm`, `mdphd`, `dm`, `mb`, `bs`, `ld` (SSA given 45 against 11,703 providers), and 19 joined spellings (`ccc-slp`, `ccc/slp`, `otr/l`, `fnp-c`, `fnp-bc`, `np-c`, `pmhnp-bc`, `cota/l`, `lcsw-c`, `cf-slp`, `agacnp-bc`, `aprn-cnp`, `ccc-a`, `lcsw-r`, `mt-bc`, `cpnp-pc`, `lisw-s`, `whnp-bc`, `rpa-c`). A joined spelling is matched as one token like the listed `pa-c` (`John Smith, OTR/L` and `John Smith OTR/L` both read suffix, checked with a lexicon holding it), so listing them needs no shape rule. **Marked ambiguous**: `pa`, whose surname reading (Census 739, SSA 1,343 given, Wikidata 36 given) is over #796's threshold of 200 and whose NPPES count is 54,128: `John Smith, PA` keeps the credential and reports the fork, `John Pa` reads family 'Pa'. **Refused**: `pi`, Principal Investigator (102 officials after a comma, 8 NPPES providers), a role and not a credential, and a surname (Census 1,006, Wikidata 634 family), with an `excluded` entry carrying the reason. **Held out, not decided here**: `bc` (board certified) and `pc` (professional corporation), which are not credentials; `aa` (1,917 providers, under the cutoff); `pu-ph`, a French hospital-university post; the split spellings (`M. D.`, `Pharm. D.`, `MB ChB`, `MD; PhD`, `MD-PhD`, `MD/PhD`), a Phase 2 shape question; and the non-English titles after a comma (`Pr`, `Profesor`), issue 804. **Case**: the twelve mixed-case ones (`mbchb`, `mbbch`, `dphil`, `scd`, `dmsc`, `dmsci`, `mmed`, `mphil`, `mhsc`, `scm`, `mmsc`, `mdphd`) carry a `cased` mask, as `pharmd` does, so `capitalized()` writes `MBChB` and not `MBCHB`; the joined spellings carry none, since their case repair (`fnp-c` → `Fnp-C`, as the listed `pa-c` already gave `Pa-C`) is a rule question outside this entry (corrected 2026-10-22, #762: wrong — `pa-c` gave `Pa-C` before this change and `PA-C` after it, this entry listing `pa`); pinned by `test_a_mixed_case_credential_listed_for_797_repairs_to_its_mask`. **Accepted cost**: a person whose initials are written `DM`, `MB`, `BS` or `LD` as the whole given name after a comma (`Smith, DM`) now reads the letters as a credential with no given name, as `MS` and `MA` already do. **Measured** (2026-10-21, tree against master fd5ef886; snapshot.jsonl re-recorded, five snapshot entries move, four of them held only by corpus_cldr.jsonl): `snapshot.py --local ctgov-officials`, recorded from the base checkout with `--record --tree` first, moves 5,899 of 352,104 readings; of those, 3,607 move a batch word from given, middle or family into the suffix (`mbchb` 1,122, `mb` 590, `bs` 332, `scd` 306, `dmsc` 237, `dm` 231, `mbbch` 201, `dphil` 184, `mmed` 158, `mdphd` 125, `mphil` 96, the rest under 50), and 1,589 move only the reported ambiguities. The CLDR agreement check moves 8 field disagreements across 3 renderings (2 now agree in full, 1 improved but still disagrees on given_names). The figures are a dated snapshot of one fetch, and the remaining moved readings carry a listed word in a changed part of speech too (a title-to-suffix move for `mdphd` among them) or a punctuated spelling the token count does not match. **Recompute**: `uv run python tools/vocab/query.py --issue 797` lists the entries (the 36 additions and the `pi` exclusion on 2026-10-21); record the local baseline from a master checkout, check the branch, and count the moved readings whose old given, middle or family held a listed word as a token and whose new suffix does. Pinned by the `comma_mixed_case_credential_is_a_suffix`, `comma_slash_joined_credential_is_a_suffix`, `one_word_head_two_capital_credential`, `empty_head_joined_credential`, `comma_pa_keeps_its_suffix_and_reports_the_fork` and `john_pa_is_a_family_name` case rows. diff --git a/docs/design/decisions/vocabulary-exclusions.md b/docs/design/decisions/vocabulary-exclusions.md index 11bc1e7e..261861f3 100644 --- a/docs/design/decisions/vocabulary-exclusions.md +++ b/docs/design/decisions/vocabulary-exclusions.md @@ -22,7 +22,7 @@ Excluded (the never-given / ambiguous particle line, a particle reading's `may_b - Encoding rationale (#293, predating #360's membership questions): the data layer stores the NEVER-GIVEN set and derives the ambiguous one, because that is safe-by-default for new particles — a one-place addition — and the v1 shim translates by one-directional complement. And the constants are FROZEN specifically to kill the cached-Lexicon.default()-vs-fresh- Constants desync that runtime module-constant mutation caused. The data-layer half is superseded since 2026-10-18 (decisions/vocabulary-model): every particle reading states `may_be_given`, true or false, and neither side is a default; the freeze half stands. - Load-bearing dependency: TITLES ∩ ambiguous — {do, freiherr, st} until #296's audit, {freiherr, st} since — is what keeps the particle-or-given ambiguity emitter reachable at all; moving them all would leave the shipped vocabulary unable to reach it, which is why test_the_chained_emitter_is_reachable_by_construction (tests/v2/test_parser.py) pins reachability by construction rather than by shipped vocabulary — an empty intersection fails no test and does not mean the emitter is unreachable. -Open (contested vocabulary memberships — the rule is right, the word's set is questioned; the issue is canonical): none. The block held three entries and all three were answered inside a fortnight — #346, #343 and #344 together on 2026-09-06 (see #indic-honorifics) and #342 on 2026-09-07 (see #suffix-acronym-collisions). The heading stays with nothing under it on purpose, the same reason an empty ledger roster section is a statement and a missing one is nobody having looked. sa, se and om are named as the next candidates by #342's own comment and carry no issue, so they are not entries here. +Open (contested vocabulary memberships — the rule is right, the word's set is questioned; the issue is canonical): none. The block held three entries and all three were answered inside a fortnight — #346, #343 and #344 together on 2026-09-06 (see #indic-honorifics) and #342 on 2026-09-07 (see #suffix-acronym-collisions). The heading stays with nothing under it on purpose, the same reason an empty ledger roster section is a statement and a missing one is nobody having looked. sa, se and om (all three since decided, sa on 2026-10-25, #857) are named as the next candidates by #342's own comment and carry no issue, so they are not entries here. Excluded (SUFFIX_ACRONYMS — esq, removed 2026-09-08, #316/#489 bundle): diff --git a/docs/release_log.rst b/docs/release_log.rst index e096e78b..b4e4924b 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -12,6 +12,7 @@ Release Log **Behavior Changes** - **Fix credentials that are also surnames reading as credentials.** ``Luo Si`` and ``Robert Capa`` give a family name, where 2.0.0 through 2.5.0 gave suffix ``Si`` or ``Capa`` with no family name. A Title-case word after a full name takes the family name, as ``Ma`` does; the all-caps form stays a credential (``John Smith CAPA`` keeps suffix ``CAPA``). :issue:`791`, :issue:`796` + - **Fix Sa reading as a credential or a title.** ``Maria de Sa`` and ``MARIA DE SA`` give family ``de Sa`` and ``DE SA``, and ``Joao Sa`` gives family ``Sa``, where 1.4.0 through 2.5.0 gave suffix ``Sa`` (``Joao Sa``: 2.0.0 through 2.5.0); ``Mrs Sa Li`` gives given ``Sa``, where it read ``Sa`` as a second title. ``SA`` is no longer a credential: ``John Smith SA`` gives family ``SA`` and ``John Smith, SA`` gives given ``SA``, family ``John Smith``, where 1.4.0 through 2.5.0 gave suffix ``SA``; a caller who needs it adds it to their own lexicon. :issue:`857` - **Fix more credentials after a comma reading as the given name.** ``John Smith, MBChB`` and ``John Smith, OTR/L`` give suffix ``MBChB`` and ``OTR/L``, where 2.0.0 through 2.5.0 gave given ``MBChB`` and family ``John Smith``; 37 degrees and certifications are listed, and ``PA`` and ``Ing`` are marked ambiguous (``John Pa`` and ``John Ing`` keep the family name). ``Ing.`` after a full name and a comma gives suffix ``Ing.``, where 2.5.0 gave title ``Ing.``. An unlisted hyphen- or slash-joined credential still reverses. :issue:`797`, :issue:`848` - **Fix case repair writing a joined credential like a hyphenated name.** ``john smith, kchs/dchs`` repairs to ``John Smith KCHS/DCHS``, where 2.0.0 through 2.5.0 gave ``Kchs/Dchs``; a listed acronym joined by a hyphen or slash is written in capitals as one word, and a casing the caller records for it wins. Hyphenated names such as ``Smith-Jones`` keep their per-part casing. :issue:`808` - **Add Cllr as a title.** ``Cllr John Smith`` gives title ``Cllr``, first ``John``, last ``Smith``, where 1.4.0 through 2.5.0 gave first ``Cllr``, middle ``John``. A bare ``Cllr`` after the name is still a name word. :issue:`847` diff --git a/nameparser/config/_compiled.py b/nameparser/config/_compiled.py index 52c3fd0e..beb85ead 100644 --- a/nameparser/config/_compiled.py +++ b/nameparser/config/_compiled.py @@ -646,7 +646,6 @@ 'rouva', 'royal', 'rt', - 'sa', 'sailor', 'saint', 'sainte', @@ -2068,7 +2067,6 @@ 'rtrp', 'rvm', 'rvt', - 'sa', 'same', 'sasm', 'sccp', diff --git a/nameparser/vocab/en.toml b/nameparser/vocab/en.toml index 055f25b4..f6d3f8d9 100644 --- a/nameparser/vocab/en.toml +++ b/nameparser/vocab/en.toml @@ -759,8 +759,9 @@ military = { same_as = "radm" } [[word]] text = "sa" -military = { expands = "Seaman Apprentice" } -credential = { note = "NOT may_be_name, for the reason recorded on ms (decisions/C8)" } +family = { us_census_2010 = 1600, wikidata = 182, as_of = 2026-10-24, note = "The Census writes no diacritics, so its 1,600 is Sa and Sá together (37.88% Asian/PI, so much of it is Korean, Chinese or Vietnamese Sa); Wikidata is a lower bound: 110 as Sá (Portugal, Brazil) plus 72 as Sa" } +given = { us_ssa = 73, wikidata = 2, as_of = 2026-10-24, note = "US SSA births summed over both sexes; Wikidata is a lower bound; nam_dict band 5 (banded in 3 countries: Korea, Vietnam, China)" } +excluded = { military = { reason = "Seaman Apprentice, with no bearer in any corpus: every name read with SA as its title in the committed and local corpora is a person whose given name is Sa (Sa Liu, Mohamad Sa Haichal) or the Arabic Sa ad (Sa ad Hariri), or whose Portuguese surname Sá is written without its accent (Ricardo Sa Pinto); as a title it also took the slot after a family comma and ahead of a name", issue = 857, decided = 2026-10-25 }, credential = { reason = "no person in the corpora writes it: the corpora's SA and S.A. suffixes are companies (Galderma S.A.), the South African qualifier in FCP(SA), and the initials S.A. typed without stops (MANN, S.A.); the Portuguese surname Sá written without its accent (Maria de Sa) is the reading to protect, and as a credential it needed the may_be_name flag, which left a capitals name ending in SA a suffix (MARIA DE SA). NPPES lists 167 US healthcare providers who write it, unread by item", issue = 857, decided = 2026-10-25 } } [[word]] text = "scpo" diff --git a/tests/v2/cases.py b/tests/v2/cases.py index dcd1a268..9ac293ba 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -10575,13 +10575,91 @@ def _check_cjk_shape_purity(self) -> None: Case("audit_ms_before_a_name_after_comma_is_the_title", "Smith, Ms. Jane", {"title": "Ms.", "given": "Jane", "family": "Smith"}), - Case("audit_sa_after_comma_is_the_postnominal", "Smith, SA", - {"family": "Smith", "suffix": "SA"}, - classification="fix(#296)", - notes="the same dual: Special Agent leading, the business " - "form trailing"), - Case("audit_perioded_sa_is_the_postnominal", "Smith, S.A.", - {"family": "Smith", "suffix": "S.A."}), + # #857: sa is in NO vocabulary set -- not a credential, not a title + # (its military and credential readings are both excluded on the + # word). It is the Portuguese surname Sa written without its accent, + # so it reads as any unlisted word does, whatever the case. These + # rows pin that decision against the listed twin `ma`, which stays a + # credential in the same shapes: an unlisted all-caps word is a + # credential by SHAPE only behind a comma (S9), and the period + # spellings go through the unlisted-dotted-suffix rule (S3). + Case("sa_after_a_particle_is_the_surname", "Maria de Sa", + {"given": "Maria", "family": "de Sa"}, + classification="fix(#857)"), + Case("bare_sa_is_a_surname", "Joao Sa", + {"given": "Joao", "family": "Sa"}), + Case("capitals_sa_after_a_particle_is_the_surname", "MARIA DE SA", + {"given": "MARIA", "family": "DE SA"}, + classification="fix(#857)", + notes="the 255 SA / SA pairs of tse-donors (#843) read alike"), + Case("capitals_ma_after_a_particle_is_the_credential", "MARIA DE MA", + {"given": "MARIA", "family": "DE", "suffix": "MA"}, + ambiguities=("suffix-or-name",), + notes="the listed twin of the row above"), + Case("lowercase_sa_after_a_particle_is_the_surname", "maria de sa", + {"given": "maria", "family": "de sa"}, + classification="fix(#857)"), + Case("capitals_sa_with_words_to_spare_is_the_surname", "ANA SILVA SA", + {"given": "ANA", "middle": "SILVA", "family": "SA"}, + classification="fix(#857)"), + Case("capitals_sa_with_no_word_to_spare_is_the_surname", "ANA SA", + {"given": "ANA", "family": "SA"}), + Case("sa_after_a_middle_name_is_the_surname", "Maria Silva Sa", + {"given": "Maria", "middle": "Silva", "family": "Sa"}, + classification="fix(#857)"), + Case("sa_before_a_family_comma_is_the_surname", "Sa, Maria", + {"given": "Maria", "family": "Sa"}), + Case("sa_in_front_is_a_name_word", "SA Smith", + {"given": "SA", "family": "Smith"}, + classification="fix(#857)", + notes="not a title: Seaman Apprentice has no bearer in any " + "corpus (decisions/C8, #857)"), + Case("mrs_then_sa_is_one_title", "Mrs Sa Li", + {"title": "Mrs", "given": "Sa", "family": "Li"}, + classification="fix(#857)", + notes="`sa` was a second title word and took the given name"), + Case("sa_after_a_family_comma_is_the_given_name", "Smith, SA", + {"given": "SA", "family": "Smith"}, + classification="fix(#857)", + notes="2.5.0 read suffix 'SA' (#296's audit row); no person " + "writes it as a credential in any corpus (#857)"), + Case("sa_after_a_family_comma_and_a_name_is_a_middle_name", + "Smith, SA John", + {"given": "SA", "middle": "John", "family": "Smith"}, + classification="fix(#857)"), + Case("mixed_case_sa_after_a_full_name_is_the_family_name", + "John Smith SA", + {"given": "John", "middle": "Smith", "family": "SA"}, + classification="fix(#857)", + notes="accepted cost (#857): no shape reads an unlisted " + "all-caps tail as a credential at this slot (S9), so " + "with no vocabulary behind it SA is the family name"), + Case("sa_after_a_full_name_and_a_comma_merges_the_name", + "John Smith, SA", + {"given": "SA", "family": "John Smith"}, + classification="fix(#857)", + notes="accepted cost (#857): the reading an unlisted two-letter " + "caps word gets after a full name; 1.4.0 through 2.5.0 " + "read suffix 'SA'. The listed twin `John Smith, MA` " + "keeps the suffix"), + Case("ma_after_a_full_name_and_a_comma_is_the_credential", + "John Smith, MA", + {"given": "John", "family": "Smith", "suffix": "MA"}, + ambiguities=("suffix-or-name",)), + Case("sa_after_a_comma_with_no_head_is_the_given_name", ", SA", + {"given": "SA"}, + classification="fix(#857)", + notes="1.4.0 read title 'SA'; 2.5.0 suffix 'SA'"), + Case("sa_ending_the_given_part_is_a_middle_name", "Smith, John SA", + {"given": "John", "middle": "SA", "family": "Smith"}, + classification="fix(#857)"), + Case("sa_after_a_credential_joins_it", "Smith, John PhD Sa", + {"given": "John", "family": "Smith", "suffix": "PhD Sa"}, + ambiguities=("suffix-or-name",)), + Case("perioded_sa_after_a_family_comma_is_unlisted", + "Smith, S.A.", + {"given": "S.A.", "family": "Smith"}, + ambiguities=("suffix-or-name",)), # MOVED by #289, not deleted: 'DO' is written in capitals inside a # mixed-case name, so it now leans CREDENTIAL and the post-comma # slot takes it with one word to spare rather than falling to the diff --git a/tests/v2/test_cases.py b/tests/v2/test_cases.py index 1bdfa654..14d584ae 100644 --- a/tests/v2/test_cases.py +++ b/tests/v2/test_cases.py @@ -230,10 +230,14 @@ def test_the_field_sweep_sees_the_claims_it_checks() -> None: # 1399 -> 1400 the same day: #754's 'Beethoven Ludwig Maria van' # row, which carries its own policy, one particle-or-given claim; # 1400 -> 1403 the same day: #835's particle row 'Doe, Jane van - # Mrs', one particle-or-given claim under each order. - assert claims == 1403, ( - f"the field sweep checks {claims} claims, recorded as 1403 on " - f"2026-10-24") + # Mrs', one particle-or-given claim under each order; 1403 -> 1406 + # the same day: #857's sa rows, after the word left every set (the + # one reporting row, 'Smith, S.A.', claims under the orders it + # reaches); 1406 -> 1415 the same day: the review's comma rows for + # sa and the listed `ma` twins, three of them reporting. + assert claims == 1415, ( + f"the field sweep checks {claims} claims, recorded as 1415 on " + f"2026-10-25") def test_every_case_name_is_unique() -> None: diff --git a/tests/v2/test_local_corpora.py b/tests/v2/test_local_corpora.py index 8533a7d3..51ad3383 100644 --- a/tests/v2/test_local_corpora.py +++ b/tests/v2/test_local_corpora.py @@ -3671,19 +3671,19 @@ def test_the_pairs_view_counts_the_pairs_whose_spellings_share_a_surname() -> No def test_the_pairs_view_lists_the_accents_only_pairs_read_with_two_surnames() -> None: - # SA is a credential and SÁ is not, so the readings part; JÚNIOR and + # RA is a credential and RÁ is not, so the readings part; JÚNIOR and # JUNIOR are both suffixes (#843), and the accent on JOSÉ moves # nothing. Fewer than ten: not listed. - found = local.pair_counts(_paired(*[("ANA SÁ", "ANA SA")] * 10, + found = local.pair_counts(_paired(*[("ANA SILVA RÁ", "ANA SILVA RA")] * 10, *[("JOSE SILVA JÚNIOR", "JOSÉ SILVA JUNIOR")] * 10, *[("JOSE SILVA", "JOSÉ SILVA")] * 10, *[("JOSE SILVA MÁ", "JOSÉ SILVA MA")] * 9), agreement=True) assert found.agreement == {"accents only": (20, 39)} - assert found.accent_splits == {"SÁ / SA": 10, "MÁ / MA": 9} + assert found.accent_splits == {"RÁ / RA": 10, "MÁ / MA": 9} report = local.format_pairs(found, top=15) assert ("accents-only pairs read with two surnames, by their last words: " - "SÁ / SA 10") in report + "RÁ / RA 10") in report assert "MÁ / MA" not in report diff --git a/tools/differential/README.md b/tools/differential/README.md index 77d25c6e..ac47c66c 100644 --- a/tools/differential/README.md +++ b/tools/differential/README.md @@ -280,7 +280,7 @@ A live source has no fixed snapshot to pin, so the cached copy is the pin: `meta uv run python tools/differential/local_corpora.py pairs tse-donors --agreement ``` -On the fetch of 2026-10-22, 205,925 of `tse-donors`' 854,531 pairs differ. The commonest shapes: words left out (22.0%: `DE` left out of 13,412 pairs, `DA` 5,906, `DOS` 3,033, then second surnames, `SILVA` 2,869), accents only (21.1%), words shortened (15.8%), words misspelled (14.9%) and words left out and shortened (11.8%). Read the agreement per shape, since a shape can change the surname legitimately: a pair that leaves out the last surname ends the family in another word, which is likely most of why only 44% of words left out agree (not measured apart from the other causes). Accents only cannot change the surname, so its 1,304 of 43,552 pairs that disagree are the parser reading one name two ways: by their last words, `JÚNIOR` against `JUNIOR` in 866 (#843) and `SÁ` against `SA` in 255, where `MARIA DE SA` reads `SA` as a credential and `MARIA DE SÁ` as the surname. The run took 39s. +On the fetch of 2026-10-22, 205,925 of `tse-donors`' 854,531 pairs differ. The commonest shapes: words left out (22.0%: `DE` left out of 13,412 pairs, `DA` 5,906, `DOS` 3,033, then second surnames, `SILVA` 2,869), accents only (21.1%; 255 of them SÁ / SA pairs read with two surnames until #857), words shortened (15.8%), words misspelled (14.9%) and words left out and shortened (11.8%). Read the agreement per shape, since a shape can change the surname legitimately: a pair that leaves out the last surname ends the family in another word, which is likely most of why only 44% of words left out agree (not measured apart from the other causes). Accents only cannot change the surname, so its 1,304 of 43,552 pairs that disagree are the parser reading one name two ways: by their last words, `JÚNIOR` against `JUNIOR` in 866 (#843) and `SÁ` against `SA` in 255, where `MARIA DE SA` reads `SA` as a credential and `MARIA DE SÁ` as the surname. The run took 39s. `jrc-names` reads a gzip downloaded once into the cache, and refuses one whose sha256 is not `JRC_SHA256`: the JRC says the resource is updated daily, but the file served on 2026-10-24 was last modified on 2024-03-15, so a new file is measured and the pin moved by hand rather than read under constants measured on another. A download that is not the pinned file replaces nothing: it is kept beside the cached one as `entities.gzip.new`, to be measured, and adopted without another download once the pin moves to it, since the JRC serves only its current file and the pinned one could not be fetched again. For the same reason `fetch --refresh` downloads only where the cached file is not the pinned one; otherwise it reads the cached file again. `meta.json` records the pin, `JRC_READ_VERSION` and the Unicode version `script()` read, and a corpus fetched under another of any is refused on load, so a change to how a spelling is read or its script labelled is not counted under the old records. It keeps the person rows (`P`, 1,338,463 of 1,445,013 lines) and reads past the organisations (`O`). It drops every row the JRC marks with a language (`ru`, `ar`, `sl`, `et`, `pl` and others, 20,819 of the person rows), counted by language in `meta.json`'s detail, on the JRC's marking alone. Where measured, a marked row is a morphological variant of a spelling the JRC leaves unmarked (`u`): an inflected form, or an Arabic spelling with و, ل or ال glued on. Every marked row's entity held an unmarked spelling, and 18,594 of the 20,819 had as many words as one and differed from it only in endings or in glued fronts of three letters or fewer (the rule is stated exactly at `JRC_UNMARKED` in `local_jrc.py`); the other 2,225, 1,680 of them Arabic, fit neither as stated and were not measured further. The unmarked spellings still hold inflected forms the JRC did not mark (by the same rule, 22,732 of the 66,673 distinct unmarked Cyrillic spellings are another Cyrillic spelling of the same person), which are kept. A name's `+` and whitespace are read as single spaces, and an id is kept as written. A line that is not four well-formed columns (the file's first line, `#`, is one) is counted, and past `JRC_MAX_BAD` refuses the file, as do a gzip cut short, a file ending inside a line, person rows under `JRC_MIN_PERSONS` of the lines (a type renamed), more marked rows than `JRC_MAX_MARKED` (a file that marks every spelling), more persons left with only marked spellings than `JRC_MAX_LOST` (none were), and more repeated spellings than `JRC_MAX_REPEATED` (58 were). 99 spellings hold a replacement character (U+FFFD), damage in the source; they are kept and counted. The licence bars redistributing the resource or anything derived from it, so its snapshot, its views' output and every file derived from it stay in the cache, and what is cited from it is counts and shapes written with made-up names. A snapshot of it (`snapshot.py --local jrc-names`) recorded 1,315,051 names in 30s on 12 CPUs (2026-10-24, 128 MB). diff --git a/tools/differential/gnd_agreement.json b/tools/differential/gnd_agreement.json index 483d777e..2ddbec55 100644 --- a/tools/differential/gnd_agreement.json +++ b/tools/differential/gnd_agreement.json @@ -1846,14 +1846,14 @@ "text": "Ísleifur Gizursson" }, "136135064/direct-et-mm/givenFirst": { + "family": { + "expected": "", + "got": "Sa" + }, "given_names": { "expected": "Khun Sa", "got": "Khun" }, - "suffix": { - "expected": "", - "got": "Sa" - }, "text": "Khun Sa" }, "13628471X/direct-my-id/givenFirst": { diff --git a/tools/differential/snapshot.jsonl b/tools/differential/snapshot.jsonl index 57c6238d..f8b34342 100644 --- a/tools/differential/snapshot.jsonl +++ b/tools/differential/snapshot.jsonl @@ -2000,7 +2000,7 @@ {"name": "Khaled el-Hassani ben el-Hachemi", "given": "Khaled", "middle": "el-Hassani ben", "family": "el-Hachemi", "initials": "K. e. b. e.", "capitalized_force": "Khaled El-Hassani Ben El-Hachemi"} {"name": "Kho Liang Ie", "given": "Kho", "middle": "Liang", "family": "Ie", "initials": "K. L. I."} {"name": "Khun Malee", "given": "Khun", "family": "Malee", "initials": "K. M."} -{"name": "Khun Sa", "given": "Khun", "suffix": "Sa", "ambiguities": ["GIVEN_OR_FAMILY"], "initials": "K.", "capitalized_force": "Khun SA"} +{"name": "Khun Sa", "given": "Khun", "family": "Sa", "initials": "K. S."} {"name": "Ki, Tomonori", "given": "Tomonori", "family": "Ki", "initials": "T. K."} {"name": "Kim Min Ho", "order": "FAMILY_FIRST", "given": "Min", "middle": "Ho", "family": "Kim", "initials": "M. H. K."} {"name": "Kim, Jr.", "family": "Kim", "suffix": "Jr.", "initials": "K."}