From 15d5426f25d7864ac53fad39caecd6154e04e5bd Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sun, 25 Oct 2026 03:08:01 -0700 Subject: [PATCH 1/7] Read a credential joined by a hyphen or a slash as one where every chunk is listed (#752) Phase 2c. rules/S3 (credentials written in joined chunks, renamed from "credentials written with interior periods") extends its chunk reading to '-' and '/', with a stricter test than the periods' any: every chunk counts as a suffix as written, so a bare member of the ambiguous class ('MA/PhD') and a hyphenated name ('Smith-Jones') are left as they were. The test lives in suffix_as_written, so classify's tag, the comma's decision and every predicate that tests a word's text read a joined word alike. 'John Smith, MD/PhD' reads suffix 'MD/PhD' where 1.4.0 through 2.6.0 read given 'MD/PhD', family 'John Smith'. Measured against 2af8da8e: ctgov-officials 1,333 entries, fec-individuals 1, tse-donors, psc-individuals, the committed corpora and the agreement records none (decisions/S3). Co-Authored-By: Claude Opus 5.5 --- docs/customize.rst | 15 ++++++ docs/design/decisions/README.md | 2 +- ...3-credentials-written-in-joined-chunks.md} | 4 +- docs/design/decisions/vocabulary-model.md | 1 + docs/design/rules/README.md | 2 +- ...3-credentials-written-in-joined-chunks.md} | 26 ++++++++-- docs/design/vocabulary-model-rfc.md | 4 +- docs/release_log.rst | 4 ++ nameparser/_pipeline/_vocab.py | 35 +++++++++++-- tests/v2/cases.py | 50 +++++++++++++++++++ tests/v2/pipeline/test_script_segment.py | 14 +++--- tests/v2/pipeline/test_vocab.py | 33 ++++++++++-- tests/v2/rules_budget.json | 2 +- tests/v2/test_regex_sync.py | 3 ++ tools/differential/corpus_rules.jsonl | 11 ++++ tools/differential/snapshot.jsonl | 11 ++++ 16 files changed, 194 insertions(+), 23 deletions(-) rename docs/design/decisions/{S3-credentials-written-with-interior-periods.md => S3-credentials-written-in-joined-chunks.md} (71%) rename docs/design/rules/{S3-credentials-written-with-interior-periods.md => S3-credentials-written-in-joined-chunks.md} (73%) diff --git a/docs/customize.rst b/docs/customize.rst index 31d3346b..695ac223 100644 --- a/docs/customize.rst +++ b/docs/customize.rst @@ -1138,6 +1138,21 @@ Through 2.3 the undeclared reading was given ``RN``, family ``Jane Smith`` and suffix ``CRNA``, so the delimiter was the only way to get the name right. +A hyphen or a slash written between two credentials with no space needs +no declaring when every part is a listed credential that is not also a +name, because the word then reads as one credential (since 2.7). A part +that may be a name, such as ``MA``, keeps the word a name word, since a +hyphen also joins surnames: + +.. doctest:: + + >>> joined = parse("Jane Smith, RN/CRNA") + >>> joined.given, joined.family, joined.suffix + ('Jane', 'Smith', 'RN/CRNA') + >>> joined = parse("Jane Smith, MA/CRNA") + >>> joined.given, joined.family + ('MA/CRNA', 'Jane Smith') + .. _unlisted-credentials: Credentials the vocabulary doesn't list diff --git a/docs/design/decisions/README.md b/docs/design/decisions/README.md index b962220d..f73a0000 100644 --- a/docs/design/decisions/README.md +++ b/docs/design/decisions/README.md @@ -47,7 +47,7 @@ Entry conventions: - [S1 — a bracketed credential is not a nickname](S1-bracketed-credential-is-not-a-nickname.md) - [S2 — a suffix word ends a name and never opens the string](S2-suffix-word-ends-a-name-and-never-opens-the-string.md) -- [S3 — credentials written with interior periods](S3-credentials-written-with-interior-periods.md) +- [S3 — credentials written in joined chunks](S3-credentials-written-in-joined-chunks.md) - [S4 — the ambiguous acronym needs words to spare](S4-ambiguous-acronym-needs-words-to-spare.md) - [S5 — the given part's trailing slot after a family comma](S5-given-parts-trailing-slot-after-a-family-comma.md) - [S6 — the case signal at the suffix slot](S6-case-signal-at-the-suffix-slot.md) diff --git a/docs/design/decisions/S3-credentials-written-with-interior-periods.md b/docs/design/decisions/S3-credentials-written-in-joined-chunks.md similarity index 71% rename from docs/design/decisions/S3-credentials-written-with-interior-periods.md rename to docs/design/decisions/S3-credentials-written-in-joined-chunks.md index 1769c519..10026214 100644 --- a/docs/design/decisions/S3-credentials-written-with-interior-periods.md +++ b/docs/design/decisions/S3-credentials-written-in-joined-chunks.md @@ -1,4 +1,4 @@ -# S3 — credentials written with interior periods +# S3 — credentials written in joined chunks ## Standing @@ -6,6 +6,7 @@ - A digit-dotted token (`1.4.2`) loses the chunk claim and is no member of the shape class either, so it reads as a name word, silently (`John Smith 1.4.2` → family '1.4.2') — 2026-09-18. - An ambiguous acronym written with its periods, one after each letter or one after each of two or more letter chunks, counts as unambiguous, and a single trailing period does not (`Jack M.A.` → suffix, `Jack Ma.` → family); the clause stood in rules/S2 until it moved here — 2026-10-19. - Paired initials of a script with capitals (two single letters each followed by a period, the second optional) are no word of the ambiguous class at the trailing slots and read as the name word they are written as, reported, as the switch off reads them: `Ram Bahadur B.K.` → middle 'Bahadur', family 'B.K.'; `Bahadur, Ram B.K.` → middle 'B.K.'; `Ram Bahadur B.K., PhD` → family 'B.K.', suffix 'PhD'. At the first slot after a comma they stay C5's (`John Smith, X.Y. P.Q.` → suffix), behind an unambiguous suffix word they join the run (`Ram Bahadur PhD B.K.` → suffix 'PhD B.K.') while behind an ambiguous or by-shape credential with no unambiguous word in front of it they end the name and take it in (`Jane Doe MA R.T.` → middle 'Doe MA', family 'R.T.'), and three dotted letters, longer chunks and a caseless script's pair keep the position reading (`John Smith X.Y.Z.`, `John Smith B.Tech.`, `สมชาย ใจดี พ.บ.` → suffix) — 2026-10-24 #864. +- A word joined by a hyphen or a slash reads as a suffix where every chunk counts as a suffix as written, listed and not a bare member of the ambiguous class, a chunk's own periods counting as in a word of its own: `John Smith, MD/PhD` → suffix 'MD/PhD', `John Smith MD-PhD` → suffix 'MD-PhD', `John Smith, M.D./Ph.D.` → suffix; one chunk that does not count leaves the word as it was (`John Smith MD-Jones` → family, `John Smith, MA/PhD` → given 'MA/PhD', family 'John Smith'), and a joiner standing alone or glued to one side makes no joined word — 2026-10-25 #752. Open: none in this file. @@ -16,3 +17,4 @@ Open: none in this file. - 2026-10-17 #682 — SPLIT FROM decisions/S2. rules/S3 was already a rule of its own, but its history pointed at decisions/S2, so the entries above were written there; when #682 split S2 they moved here unchanged but for pointers the move broke. - 2026-10-24 #864 — PAIRED INITIALS ARE NO CREDENTIAL AT THE TRAILING SLOTS. SUPERSEDES, in part, decisions/S9's 2026-09-30 #563 entry: its choice (iii) kept #516's reading at the trailing slots and left the no-comma question open, and this answers it for paired initials. Since 2.4.0 the position reading took a surname written as two dotted initials wherever the name had a word to spare: Nepali surnames are written that way (B.K., K.C., B.C., D.C.), and `Ram Bahadur B.K.` read family 'Bahadur', suffix 'B.K.' where 2.0.0 through 2.3.0 read family 'B.K.' (wheels, measured 2026-10-24). Two dotted single letters is how initials are written, and C5 already reads the shape as a name at the first slot after a comma. THE BOUNDARY is the pair, not every run of single letters: measured with the wider variant (any number of single-letter chunks), 11 committed corpus names move, every one a decided credential example of this rule or S9 (`John Smith X.Y.Z.`, `Q.W.E.R.T.`, `J.u.n.i.o.r.`, `R.A.I.`, `Jane Doe nee Smith X.Y.Z.`) or CLDR's Russian `к.м.н.`, for 3 more Nepal names and 3 more readings in the local ClinicalTrials.gov and FEC corpora together. THE SCRIPT has capitals: CLDR's Thai renderings write degrees as dotted letters (`พ.บ.`, the MD, and `ท.บ.`), a caseless script having no capitals to write initials in, and the unscoped pair moved both into the name, one into the family and one into the middle. THE SEAM: classify writes the shape tag without the class tag, the state the switch off leaves every dotted word in, so every trailing slot (the end of the name, the word ending the given part after a family comma, a maiden clause's end, the part before a suffix comma) reads the pair as a name word and reports the fork through the switch-off branch that already existed; `_comma.decide` restores the class tag in the part it reads, where C5 decides. Narrowing the shape verdict itself, in `period_joined_vocab`, was measured first and declined: the comma's credential run reads the class too, so `John Smith, X.Y. P.Q.` lost its suffix (given 'X.Y.', middle 'P.Q.') and `John Smith, PhD X.Y.` gained a report; holding back only classify's tag, without the restore, cost the same. MOVED IN THE COMMITTED CORPORA: two names, each to 1.4.0's reading, `García Márquez, Ed G.J.` (middle 'G.J.', was suffix) and `De La Cruz, M.J. K.L.` (middle 'K.L.', was suffix); CLDR agreement unmoved. LOCAL CORPORA (counts only), each recorded first from origin/master 638685b4 with `--record --tree`: nepal-candidates, 1,095 of its 237,408 entries move (the romanized names of 1,221 of its 126,981 name pairs), every one to the dotted pair (or a GC, below) as the family name, and 31 of the corpus's ASCII names ending in a two-letter dotted word still do not read it as the family, 20 of them `D.C.`, 8 `G.M.` and 2 `M.C.`, all listed credentials, and one after a comma, C5's; fec-individuals, 293 move, 289 of them a pair ending the given part after a family comma moving from suffix to middle (in one, two pairs, both now middle) and 4 a lone `GC` after a comma moving to the given name; ctgov-officials, 9 of 352,104, 7 a dotted pair ending a name, mostly before a suffix comma, moving from suffix to family, none of them a listed credential, and 2 a `GC`; psc-individuals (local_agreement.py), 4 records progress and 1 regresses, a filed surname of a word and a dotted pair now read with the pair alone as the family; ukec-individuals, none. THE ACCEPTED COST is an unlisted two-letter credential written with its periods after a full name, which reads as the family name, reported: King's Counsel is the case (`John Smith K.C.`), unlisted, and written KC by 4 of 15,214 UK donors (`tools/name_usage.py kc`, the ukec-individuals row); 1.4.0 read the same. A SECOND ACCEPTED COST, found by the design-docs review: only an unambiguous suffix word in front makes the pair the credential run's (`Ram Bahadur PhD B.K.`, `Jane Doe Ph.D. R.T.` → suffix); behind a credential that is itself ambiguous, listed (`MA`, `MEng`) or read by position (`X.Y.Z.`, `XYZ` under EVERYWHERE), the pair ends the name and the credential is taken into it, `Jane Doe MA R.T.` reading middle 'Doe MA', family 'R.T.' where 2.4.0 through 2.5.0 read suffix 'MA R.T.' and 1.4.0 through 2.3.0 the same as now; 1 local entry takes this shape, in nepal-candidates (counted as paired initials of a cased script standing right after a word tagged `vocab:suffix-ambiguous` that is not also a suffix, the review's count; none in the other seven local corpora). An unambiguous word in front of the ambiguous one starts the run, which then takes both (`Jane Doe PhD MA R.T.` → suffix 'PhD MA R.T.', rules/S8). Recorded as a cost rather than changed (Derek, 2026-10-24). `D.C.`, `G.M.` and `M.C.` are listed and so untouched: `dc` stays on 99,172 NPPES providers against 21 Nepal name pairs, and `gm` and `mc` were not weighed here. `gc` left the suffix acronyms in the same change (decisions/suffix-acronym-collisions' 2026-10-24 #864 entry). Pinned by the #864 block of `tests/v2/cases.py` and the `B.K` and `พ.บ.` rows of `test_ambiguous_class_candidate_agrees_with_the_tag`. - 2026-10-19 — CLAUSE MOVED IN: an ambiguous acronym written with its periods, one after each letter or one after each of two or more letter chunks, counts unambiguously, and a single trailing period does not. It stood in rules/S2 until now and is a shape, the periods, which is this rule's signal, so it moved here, one rule per piece of evidence (docs/design/AGENTS.md). It was decided in #296's review (decisions/C8's PR #428 review entry) and widened to letter chunks by #544 (decisions/suffix-acronym-collisions' #544 entry), and no entry under decisions/S2 recorded it. Where earlier entries and code call it S2's period gate, it is this rule's now. +- 2026-10-25 #752 — JOINED CHUNKS: A WORD JOINED BY A HYPHEN OR A SLASH IS A CREDENTIAL WHERE EVERY CHUNK IS ONE (phase 2c, item 5 of decisions/vocabulary-model's 2026-10-23 #752 entry, agreed by Derek on 2026-10-23). The chunk reading this rule gives interior periods is extended to the two other joiners a credential is written with, by a stricter test: the periods' ANY becomes EVERY, and a chunk must count as a suffix as written, which a bare member of the ambiguous class (`MA`, `Ed`) does not, because a hyphen also joins names (`Smith-Jones`, `Mary-Jane`) and a slash sits between a name and a credential in typed data. The test is asked where a word's suffix reading as written is asked, so classify's tag, the comma's decision and every predicate testing a word's text read the joined word alike. A chunk is read as a word of its own, so a chunk written with its periods counts (`M.D./Ph.D.`, and `M.A./Ph.D.`, the ambiguous member written so). Not read: a joiner standing alone or glued to one side (`MD / PhD` already read as a run of suffix words; `MD/ PhD` keeps `MD/` as a name word, 10 ClinicalTrials.gov entries, and `MD /PhD` 1), the en dash and other dash characters (no entry joins two listed credentials with one), and `&` (63 entries join two listed credentials with it, left for its own change). MEASURED (2026-10-25, against 2af8da8e; every local corpus recorded from that base with `--tree`): `tools/differential/snapshot.py` moves nothing but the rule's own new examples; `cldr_agreement.py` matches every record; `local_agreement.py psc-individuals` matches its record; tse-donors, 0 of 848,351 entries; fec-individuals, 1 of 1,893,964, a doubled joined credential leaving the middle name for the suffix; ctgov-officials, 1,333 of 352,104, the population being the 1,592 words in it joined by a hyphen or a slash whose every chunk counts (the rest already read as suffixes, in a part past a suffix comma). Of the 1,333, 1,319 are a joined credential after a comma that had been read as the given name, the name reversed around it, now the suffix; 2 a joined credential ending a name without a comma, now the suffix where it was the family; and 12 move a report alone, 11 dropping a comma-structure report (a third part now read wholly as credentials) and 1 a suffix-or-name report. Of the 1,321 whose fields move, 3 also take a neighbouring word into the suffix with a new report, each as the unjoined twin reads: a word after the credential (`John Smith, MD/PhD Candidate` reads as `John Smith, PhD Candidate` does) and, once, a capitals credential ending the part before the comma (rules/C12). The joined words are 82 distinct credentials, every one a degree or certification combination, the commonest `MD-PhD` (494), `MD/PhD` (473), `PhD/MD` (65) and `MS-PT` (53); 68 entries hold a chunk written with its periods, which the 2026-10-23 measurement's letters-only test did not read (its 1,254). COST: a word holding a joiner pays three profiler frames at each site that asks its suffix reading as written, stopping at its first chunk that does not count, and a word holding none pays nothing, the joiners being tested in C first (`John Smith` 150 frames before and after, `Mary Smith-Jones` 156 → 159, `Smith-Jones, Mary-Jane` 169 → 175, py3.11). Pinned by the #752 rows of tests/v2/cases.py and `test_suffix_as_written_reads_a_joined_credential` (tests/v2/pipeline/test_vocab.py), whose refused rows pass on the pre-change tree and whose admitted rows fail on it. diff --git a/docs/design/decisions/vocabulary-model.md b/docs/design/decisions/vocabulary-model.md index 88f344bb..926cd555 100644 --- a/docs/design/decisions/vocabulary-model.md +++ b/docs/design/decisions/vocabulary-model.md @@ -188,3 +188,4 @@ Open: #752 (phase 2, derived ambiguity; its sub-phases 2a to 2m, each its own PR - **`is_cased_paired_initials(text)`**, rules/S3's paired initials in a script with capitals: classify asked the script half inline (#872, decisions/S3). This is the lift the 2026-10-24 #872 entry above names, for classify and the split merge to share. One overlap is left to the split-credential PR to state: item 6 refuses a split credential whose first chunk is a single letter, so for a split `A. B.` that length test answers before this predicate is asked. rules/C5's comma reading keeps the bare `is_paired_initials`, since after a comma a caseless pair reads as a name too (`John Smith, พ.บ.` → given 'พ.บ.'), and moving C5 onto this predicate would change that reading. - **Measured neutral** against the branch, first on 632685c8 and again on 2dd0414b once phase 2a had merged: a grid of 20,364 texts (22 heads covering no comma, a whole name, one word and nothing before a comma, a nickname-only head, a maiden clause, a family comma with a title-particle head and a quoted clause before the pair; 33 spellings including `MA`, `M.D.`, `X.Y.Z.`, `XYZ`, `Ph. D.` and its case variants, `V`, `VI`, `III`, `XI`, `M.J.`, `K.C.` and `พ.บ.`; each text also in lower and upper case), under all three orders, read identically (fields, ambiguity kinds and details, token roles and tags, `initials()`, `capitalized()`), and the same harness reported 45,871, 18, 3,974 and 2,986 differing readings with `split_pair`, C12's call to the numeral predicate, the cased half and the peel's call to it each knocked out. The old and new predicates gave the same answer on 19,769 direct calls. The committed snapshot, every CLDR, GND, Wikidata and conventions agreement record, psc-individuals' agreement record and the ctgov-officials, fec-individuals and tse-donors local snapshots (each recorded from the base with `--tree`) all match, on both bases. psc-individuals' record was itself stale against 632685c8 (3 regressions, 5 progress, 5 changed before this change), so it was re-recorded from each base first. - **Frames**, counted with `sys.setprofile` on both trees: `_CALL_BASELINE`'s reference name and `Smith, John` cost what they did (305 and 158). A word matching `SPLIT_PAIR_OPENS` (`Ph`, `ph.`) with a word after it pays one frame at each site that tests it, whether or not a `D.` follows (`John Smith Ph. D.` 233 → 234, `Smith, John Ph. D.` 239 → 241; the review counted `ph ph ph ph ph ph` 282 → 286), so the cost grows only with the number of such words. C12's evidence keeps its tag test ahead of the call, so it costs what it did (`John Smith Jr., AD` 372). That is a rare shape that no guard counts. +- **2026-10-25 #752 — phase 2c: a credential joined by a hyphen or a slash reads as one where every chunk counts as a suffix as written**, carrying out the listed half of item 5 of the 2026-10-23 #752 entry. `John Smith, MD/PhD` reads suffix 'MD/PhD' where it read given 'MD/PhD', family 'John Smith'; a bare member of the ambiguous class is no chunk the test counts (`John Smith, MA/PhD` keeps its reading). The record, with what was measured and what was left unread, is decisions/S3's entry of this date (rules/S3). diff --git a/docs/design/rules/README.md b/docs/design/rules/README.md index 8475dd32..6b72b715 100644 --- a/docs/design/rules/README.md +++ b/docs/design/rules/README.md @@ -140,7 +140,7 @@ Terms the rules use without defining them each time, each with the rule that def - [S1 — a bracketed credential is not a nickname](S1-bracketed-credential-is-not-a-nickname.md) - [S2 — a suffix word ends a name and never opens the string](S2-suffix-word-ends-a-name-and-never-opens-the-string.md) -- [S3 — credentials written with interior periods](S3-credentials-written-with-interior-periods.md) +- [S3 — credentials written in joined chunks](S3-credentials-written-in-joined-chunks.md) - [S4 — the ambiguous acronym needs words to spare](S4-ambiguous-acronym-needs-words-to-spare.md) - [S5 — the given part's trailing slot after a family comma](S5-given-parts-trailing-slot-after-a-family-comma.md) - [S6 — the case signal at the suffix slot](S6-case-signal-at-the-suffix-slot.md) diff --git a/docs/design/rules/S3-credentials-written-with-interior-periods.md b/docs/design/rules/S3-credentials-written-in-joined-chunks.md similarity index 73% rename from docs/design/rules/S3-credentials-written-with-interior-periods.md rename to docs/design/rules/S3-credentials-written-in-joined-chunks.md index cc6fcd86..f8a37eb6 100644 --- a/docs/design/rules/S3-credentials-written-with-interior-periods.md +++ b/docs/design/rules/S3-credentials-written-in-joined-chunks.md @@ -1,6 +1,6 @@ -# S3 — credentials written with interior periods +# S3 — credentials written in joined chunks -**When:** a word written with interior periods stands in a name, as in "John Smith Msc.Ed.". +**When:** a word written in chunks, joined by interior periods or by a hyphen or a slash, stands in a name, as in "John Smith Msc.Ed.". **Then:** a word with interior periods reads as a suffix when any of its period-separated chunks is suffix vocabulary, and a word of two or more period-separated chunks that no vocabulary claims is read by POSITION instead, as a bare ambiguous acronym is (S4). @@ -22,8 +22,21 @@ "john smith x.y.z." → suffix="x.y.z." "John Smith 1.4" → family="1.4" · boundary ``` +- **A word joined by a hyphen or a slash** is read by a stricter test than the periods' any: a word whose chunks are joined by a hyphen or a slash reads as a suffix when every chunk counts as a suffix as written, which is to say listed and not a bare member of the ambiguous class (S4), a chunk's own periods counting as they do in a word of its own. One chunk that does not count leaves the word as it would be without this bullet, and a joiner standing alone or glued to one side joins nothing. + ``` + "John Smith, MD/PhD" → suffix="MD/PhD" + "John Smith MD-PhD" → suffix="MD-PhD" + "Smith, John MD/PhD" → suffix="MD/PhD" + ", MD/PhD" → suffix="MD/PhD" + "John Smith, M.D./Ph.D." → suffix="M.D./Ph.D." + "john smith, ms-pt" → suffix="ms-pt" + "John Smith-Jones, MD-PhD" → family="Smith-Jones" + "John Smith-Jones" → family="Smith-Jones" · boundary + "Smith, Mary-Jane" → given="Mary-Jane" · boundary + "John Smith MD-Jones" → family="MD-Jones" · boundary + ``` -**Because:** credentials are often written run together with periods, and the chunks between the periods are what carry the vocabulary. A word of several chunks that no vocabulary knows is still written the way a credential is written. An acronym that is also an ordinary name is unmistakably a credential when its periods are written. +**Because:** credentials are often written run together, with periods between their parts or a hyphen or a slash between two of them, and the chunks are what carry the vocabulary. A word of several chunks that no vocabulary knows is still written the way a credential is written. An acronym that is also an ordinary name is unmistakably a credential when its periods are written. ``` "John Smith Msc.Ed." → suffix="Msc.Ed." @@ -64,6 +77,11 @@ **Accepted:** +- A joined credential with a chunk that is a bare member of the ambiguous class keeps the reading it had before joined chunks were read, where a name in front of a comma is reversed: the hyphen joins names too, and a name word joined to a listed credential is not one the stricter test may take. + ``` + "John Smith, MA/PhD" → given="MA/PhD" + "John Smith, MA/PhD" → family="John Smith" + ``` - An unlisted credential of two letters written with its periods behind a full name reads as the family name, and the reading is reported: King's Counsel, which no vocabulary lists, is the case. ``` "John Smith K.C." → family="K.C." @@ -76,5 +94,5 @@ - The initialless-script clause carries no example line of its own. Every input that exercises it composes a script that writes no abbreviations with a period that only a Latin convention writes — `John Smith 田.中.` is the shape — and a composed form no writing system produces is tolerated input rather than contract, which an example of a normative rule would promise. So the witnesses are the tolerated row tests/v2/cases.py's an_initialless_script_glued_into_periods_is_not_this_shape and the CJK assertions in tests/v2/pipeline/test_vocab.py's test_period_joined_vocab_retires_the_single_character_chunk. W3 states the same precedence for the same reason. --- -**Signal:** shape: a word with interior periods, read by its chunks or as the acronym its letters spell, and for paired initials whether its script writes capitals +**Signal:** shape: a word in chunks joined by interior periods, a hyphen or a slash, read by its chunks or as the acronym its letters spell, and for paired initials whether its script writes capitals **history:** decisions/S3 · **implemented:** nameparser/_pipeline/_classify.py, nameparser/_pipeline/_comma.py, nameparser/_pipeline/_pieces.py, nameparser/_pipeline/_vocab.py diff --git a/docs/design/vocabulary-model-rfc.md b/docs/design/vocabulary-model-rfc.md index f510365f..94344291 100644 --- a/docs/design/vocabulary-model-rfc.md +++ b/docs/design/vocabulary-model-rfc.md @@ -64,7 +64,7 @@ Every clash the parser decides is between readings of one token, and phase 2 put - **Vocabulary readings**, the TOML kinds: titles, suffixes, particles, conjunctions, bound given names, birth markers. - **Declared name readings**, `given` and `family`, written by hand on a word some file claims or refuses (decision 11), with dated counts as provenance that never compute a reading (decision 12). Nobody maintains surname lists. -- **Shape readings**, which today are rules standing around the vocabulary: the initial ([rules/T4](rules/T4-bare-letter-as-an-initial.md), a bare letter as an initial; [rules/T5](rules/T5-written-shape-of-an-initial.md), the written shape of an initial), the leading abbreviation title ([rules/H2](rules/H2-leading-abbreviation-title.md)), the dotted acronym ([rules/S3](rules/S3-credentials-written-with-interior-periods.md)), the capitals acronym ([rules/S9](rules/S9-unlisted-words-that-join-the-ambiguous-class-by-shape.md), unlisted words that join the ambiguous class by shape), the roman numeral past the listed ones ([rules/S10](rules/S10-trailing-roman-numeral.md)), paired initials ([rules/C5](rules/C5-paired-initials-after-the-comma.md), paired initials after the comma), and the joined and split spellings of a credential that phase 2 adds. +- **Shape readings**, which today are rules standing around the vocabulary: the initial ([rules/T4](rules/T4-bare-letter-as-an-initial.md), a bare letter as an initial; [rules/T5](rules/T5-written-shape-of-an-initial.md), the written shape of an initial), the leading abbreviation title ([rules/H2](rules/H2-leading-abbreviation-title.md)), the dotted acronym ([rules/S3](rules/S3-credentials-written-in-joined-chunks.md)), the capitals acronym ([rules/S9](rules/S9-unlisted-words-that-join-the-ambiguous-class-by-shape.md), unlisted words that join the ambiguous class by shape), the roman numeral past the listed ones ([rules/S10](rules/S10-trailing-roman-numeral.md)), paired initials ([rules/C5](rules/C5-paired-initials-after-the-comma.md), paired initials after the comma), and the joined and split spellings of a credential that phase 2 adds. Shape readings join the pool because of decision 20, which lets a pack's preferred reading outrank a shape-derived one, and which phase 2 applies in the default as well (an abbreviated given name over the leading-abbreviation title, below): a reading can outrank a shape-derived one only if the shape's answer is a reading the gate weighs, not a verdict already written into a tag. Settling a clash by rewriting a tag also erases the word's other readings: retagging a capitals `DO` as an unambiguous credential let its particle reading take it into the family (`Doe, John DO`), which is why the case cue for credentials (#785) waits for the pool. @@ -112,7 +112,7 @@ A reading whose writing offers no cue is one the gate could only guess, so it is **`fils` and the agnomens show what kind-specific cues buy.** `fils` is gated by its case and the count, so `Alexandre Dumas fils` is to read suffix `fils` while `Marie Fils` keeps her family name, reported. The Brazilian agnomens are to be gated by the count alone: case would say nothing, and the plain generation reading breaks real surnames (`Ana Neto`) and given names. -**A credential's spelling is a form cue.** A credential joined by `-` or `/` is to read as one when every chunk is listed and none is ambiguous, since a hyphen also joins names (`Smith-Jones`); an unlisted word of capitals joined that way is to join the ambiguous class after a comma, gated by the name's case contrast. Today `John Smith, MD/PhD` reads given `MD/PhD`. A credential split across a space (`Pharm. D.`, generalizing the `Ph. D.` merge) is to merge only where its first chunk has two or more letters, the merged piece a member of the ambiguous class, decided by the comma count and words to spare: an opening single letter is a middle initial (`SMITH, JOHN A. MS.` holds an initial and a salutation, not a credential `AMS`). #872 made the same judgment for the joined spelling (`A.B.`), in scripts with capitals and outside the slot right after a comma. The merged piece is also to start the credential's run (#827), once the numeral predicate answers the numeral before a credential; since [rules/S8](rules/S8-credentials-trailing-run.md) refuses a member of the ambiguous class as a run start, how the two combine is the split-credential PR's to state (today's `Ph. D.` merge is unambiguous, and S8 already covers it). +**A credential's spelling is a form cue.** A credential joined by `-` or `/` is to read as one when every chunk is listed and none is ambiguous, since a hyphen also joins names (`Smith-Jones`); an unlisted word of capitals joined that way is to join the ambiguous class after a comma, gated by the name's case contrast. `John Smith, MD/PhD` read given `MD/PhD` until phase 2c (2026-10-25) read the listed half: a joined word whose every chunk counts as a suffix as written ([rules/S3](rules/S3-credentials-written-in-joined-chunks.md)). A credential split across a space (`Pharm. D.`, generalizing the `Ph. D.` merge) is to merge only where its first chunk has two or more letters, the merged piece a member of the ambiguous class, decided by the comma count and words to spare: an opening single letter is a middle initial (`SMITH, JOHN A. MS.` holds an initial and a salutation, not a credential `AMS`). #872 made the same judgment for the joined spelling (`A.B.`), in scripts with capitals and outside the slot right after a comma. The merged piece is also to start the credential's run (#827), once the numeral predicate answers the numeral before a credential; since [rules/S8](rules/S8-credentials-trailing-run.md) refuses a member of the ambiguous class as a run start, how the two combine is the split-credential PR's to state (today's `Ph. D.` merge is unambiguous, and S8 already covers it). ### The front of the name diff --git a/docs/release_log.rst b/docs/release_log.rst index 2aad3e20..1b2ccd94 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -4,6 +4,10 @@ Release Log nameparser 2.7 is under development. + **Behavior Changes** + + - **Read a credential joined by a hyphen or a slash as a suffix.** ``John Smith, MD/PhD`` gives first ``John``, last ``Smith``, suffix ``MD/PhD``, where 1.4.0 through 2.6.0 gave first ``MD/PhD``, last ``John Smith``; ``MD-PhD`` and ``PhD/MD`` read the same. Every part has to be a listed credential that is not also a name, so ``Smith-Jones`` is untouched and ``John Smith, MA/PhD`` still reverses. :issue:`752` + * 2.6.0 - October 25, 2026 nameparser 2.6 ships the default vocabulary as per-word data files, adds an opt-in reading for names with two family names, and fixes words read as a credential or a title that are part of the name. Highlights: diff --git a/nameparser/_pipeline/_vocab.py b/nameparser/_pipeline/_vocab.py index 78a61e75..b943299b 100644 --- a/nameparser/_pipeline/_vocab.py +++ b/nameparser/_pipeline/_vocab.py @@ -560,9 +560,38 @@ def suffix_as_written(n: str, text: str, lexicon: Lexicon) -> bool: a = n.replace(".", "") if a in lexicon.suffix_acronyms_ambiguous and _dotted(text): return True - return (a in lexicon.suffix_acronyms - and a not in lexicon.suffix_acronyms_ambiguous) \ - or n in lexicon.suffix_words + if ((a in lexicon.suffix_acronyms + and a not in lexicon.suffix_acronyms_ambiguous) + or n in lexicon.suffix_words): + return True + # the joiners asked in C first, so a word holding neither pays no + # frame + return ("-" in n or "/" in n) and _joined_credential(text, lexicon) + + +#: What joins the chunks of a joined credential ('MD/PhD', 'MD-PhD'). +_JOINERS = re.compile(r"[-/]") + + +# rules/S3: "a word whose chunks are joined by a hyphen or a slash +# reads as a suffix when every chunk counts as a suffix as written" -- +# the stricter test, every chunk rather than any, because a hyphen also +# joins names (decisions/S3 (#752)) +def _joined_credential(text: str, lexicon: Lexicon) -> bool: + """Whether TEXT is two or more chunks joined by '-' or '/', each + non-empty and each counting as a suffix as written: listed, and + not a bare member of the ambiguous class ('MD/PhD', 'M.A.-Ph.D.', + but not 'MA/PhD', 'Smith-Jones' or 'MD/'). A chunk holds no joiner, + so the recursion is one level deep. A loop rather than `all()` over + a generator, which would cost a frame per chunk: a hyphenated name + ('Smith-Jones') stops at its first chunk.""" + chunks = _JOINERS.split(text) + if len(chunks) < 2: + return False + for c in chunks: + if not (c and suffix_as_written(_normalize(c), c, lexicon)): + return False + return True def _is_suffix_strict_n(n: str, text: str, lexicon: Lexicon, diff --git a/tests/v2/cases.py b/tests/v2/cases.py index 8ee5889b..0cd59908 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -3491,6 +3491,56 @@ def _check_cjk_shape_purity(self) -> None: notes="the boundary on the vocabulary side: Doctor of " "Chiropractic is listed, so the shape question is never " "asked (99,172 NPPES providers write it, decisions/S3)"), + # #752 (phase 2c): a word joined by a hyphen or a slash is a + # credential where every chunk counts as a suffix as written + # (rules/S3). Each moved row's reading is 1.4.0's through 2.6.0's + # otherwise (wheels, 2026-10-25), stated per row. + Case("a_joined_credential_after_a_comma_keeps_the_name_in_order", + "John Smith, MD/PhD", + {"given": "John", "family": "Smith", "suffix": "MD/PhD"}, + classification="fix(#752)", + notes="1.4.0 through 2.6.0 read given 'MD/PhD', family 'John " + "Smith', the name reversed around the credential"), + Case("a_hyphen_joined_credential_ending_a_name", + "John Smith MD-PhD", + {"given": "John", "family": "Smith", "suffix": "MD-PhD"}, + classification="fix(#752)", + notes="1.4.0 through 2.6.0 read middle 'Smith', family " + "'MD-PhD'"), + Case("a_joined_credential_ending_the_given_part", + "Smith, John MD/PhD", + {"given": "John", "family": "Smith", "suffix": "MD/PhD"}, + classification="fix(#752)", + notes="1.4.0 through 2.6.0 read middle 'MD/PhD'"), + Case("a_joined_credential_with_nothing_before_the_comma", + ", MD/PhD", + {"suffix": "MD/PhD"}, + classification="fix(#752)", + notes="the empty-head comma shape; 1.4.0 through 2.6.0 read " + "given 'MD/PhD'"), + Case("a_joined_credential_of_dotted_chunks", + "John Smith, M.D./Ph.D.", + {"given": "John", "family": "Smith", "suffix": "M.D./Ph.D."}, + classification="fix(#752)", + notes="a chunk counts as it does as a word of its own, periods " + "and all; 1.4.0 through 2.6.0 read the name reversed"), + Case("a_joined_credential_behind_a_hyphenated_surname", + "John Smith-Jones, MD-PhD", + {"given": "John", "family": "Smith-Jones", "suffix": "MD-PhD"}, + classification="fix(#752)", + notes="the two hyphens side by side: the surname's chunks are " + "not listed, the credential's are; 1.4.0 through 2.6.0 " + "read given 'MD-PhD', family 'John Smith-Jones'"), + Case("a_joined_word_with_one_unlisted_chunk_is_a_name_word", + "John Smith MD-Jones", + {"given": "John", "middle": "Smith", "family": "MD-Jones"}, + notes="the boundary: every chunk, not any, because a hyphen " + "also joins names"), + Case("a_joined_credential_with_a_bare_ambiguous_chunk_keeps_its_reading", + "John Smith, MA/PhD", + {"given": "MA/PhD", "family": "John Smith"}, + notes="rules/S3's Accepted cost: a bare member of the ambiguous " + "class is no chunk the stricter test counts"), Case("paired_initials_in_a_caseless_script_keep_the_position_reading", "สมชาย ใจดี พ.บ.", {"given": "สมชาย", "family": "ใจดี", "suffix": "พ.บ."}, diff --git a/tests/v2/pipeline/test_script_segment.py b/tests/v2/pipeline/test_script_segment.py index 3241df38..b7834410 100644 --- a/tests/v2/pipeline/test_script_segment.py +++ b/tests/v2/pipeline/test_script_segment.py @@ -908,18 +908,20 @@ def test_a_configured_suffix_delimiter_reaches_the_peel() -> None: # question it is borrowing. Two routes into that answer, neither # reachable from the other's input, so both are pinned: a # delimiter WITHOUT whitespace leaves one token no core equals - # ("PhD/MD"), admitted by splits_into_suffixes, while a + # ("PhD+MD"), admitted by splits_into_suffixes, while a # whitespace-padded one surfaces as a standalone token ("-"), # admitted by plain core membership. Each peels only with its # delimiter configured; the bare-policy line under each is the # baseline that says so. lex = _LEX_TAILS.add(suffix_words={"v"}, suffix_acronyms={"md", "phd"}) - slash = Policy(extra_suffix_delimiters=frozenset({"/"})) - assert _texts(_run("田中さん, PhD/MD", policy=slash, - lexicon=lex)) == ["田中", "さん", "PhD/MD"] - assert _texts(_run("田中さん, PhD/MD", policy=_HANGUL, - lexicon=lex)) == ["田中さん", "PhD/MD"] + # not '/': a word joined by a slash or a hyphen whose every chunk + # is listed reads as a suffix with no delimiter configured (#752) + plus = Policy(extra_suffix_delimiters=frozenset({"+"})) + assert _texts(_run("田中さん, PhD+MD", policy=plus, + lexicon=lex)) == ["田中", "さん", "PhD+MD"] + assert _texts(_run("田中さん, PhD+MD", policy=_HANGUL, + lexicon=lex)) == ["田中さん", "PhD+MD"] dash = Policy(extra_suffix_delimiters=frozenset({" - "})) assert _texts(_run("田中さん, Jr. - V.", policy=dash, lexicon=lex)) == ["田中", "さん", "Jr.", "-", "V."] diff --git a/tests/v2/pipeline/test_vocab.py b/tests/v2/pipeline/test_vocab.py index 8678e1f6..3014db8a 100644 --- a/tests/v2/pipeline/test_vocab.py +++ b/tests/v2/pipeline/test_vocab.py @@ -15,7 +15,7 @@ is_single_letter_numeral, split_pair, is_suffix_lenient, is_suffix_strict, is_title_shaped, is_wholly_suffix, maiden_marker_run, name_word_count, period_joined_vocab, written_as_a_name, - resolve_script_set, single_script, + resolve_script_set, single_script, suffix_as_written, ) from nameparser._policy import (Policy, Script, _NO_INITIALS, _SCRIPT_RANGES) @@ -270,9 +270,10 @@ def test_is_wholly_suffix() -> None: # diagnosed a long way from its cause. assert is_wholly_suffix(["Lt.Jr."], lex, pol) # period-joined assert not is_wholly_suffix(["Lt.Smith"], lex, pol) # no suffix chunk - delim = Policy(extra_suffix_delimiters=frozenset({"/"})) - assert is_wholly_suffix(["PhD/MD"], lex, delim) # delimiter split - assert not is_wholly_suffix(["PhD/MD"], lex, pol) + delim = Policy(extra_suffix_delimiters=frozenset({"+"})) + assert is_wholly_suffix(["PhD+MD"], lex, delim) # delimiter split + assert not is_wholly_suffix(["PhD+MD"], lex, pol) + assert is_wholly_suffix(["PhD/MD"], lex, pol) # joined (S3) def test_is_wholly_suffix_empty_run_is_false() -> None: @@ -395,6 +396,30 @@ def test_period_joined_vocab_retires_the_single_character_chunk() -> None: assert period_joined_vocab("たな.か.", lex) is None +@pytest.mark.parametrize("text, expected", [ + ("MD/PhD", True), + ("MD-PhD", True), + ("MD/PhD/MPH", True), + ("md-phd", True), + ("M.D./Ph.D.", True), # a chunk counts as written, periods and all + ("M.A./Ph.D.", True), # an ambiguous acronym written with its periods + ("MA/PhD", False), # a bare member of the ambiguous class + ("PhD/Ed", False), + ("Smith-Jones", False), # a hyphen also joins names + ("MD-Jones", False), # every chunk, not any + ("MD/", False), # an empty chunk is no chunk + ("/MD", False), + ("MD//PhD", False), + ("MD+PhD", False), # '+' is no joiner +]) +def test_suffix_as_written_reads_a_joined_credential( + text: str, expected: bool) -> None: + # a word joined by a hyphen or a slash is a suffix only + # where EVERY chunk counts as a suffix as written (#752) + lex = Lexicon.default() + assert suffix_as_written(_normalize(text), text, lex) is expected + + def test_ambiguous_class_candidate_admits_a_by_shape_member() -> None: # #516: the comma form's own candidate test reaches a by-shape # member too, once Policy admits it -- case-free either way, the diff --git a/tests/v2/rules_budget.json b/tests/v2/rules_budget.json index 204e8262..81b13010 100644 --- a/tests/v2/rules_budget.json +++ b/tests/v2/rules_budget.json @@ -54,7 +54,7 @@ "R5": {"words": 250, "then_words": 154, "because_sentences": 2, "except": 0, "precedence": 0, "accepted": 0, "accepted_words": 0}, "S1": {"words": 274, "then_words": 26, "because_sentences": 1, "except": 2, "precedence": 2, "accepted": 0, "accepted_words": 0}, "S2": {"words": 641, "then_words": 103, "because_sentences": 1, "except": 0, "precedence": 7, "accepted": 3, "accepted_words": 214}, - "S3": {"words": 593, "then_words": 265, "because_sentences": 3, "except": 3, "precedence": 3, "accepted": 3, "accepted_words": 179}, + "S3": {"words": 714, "then_words": 366, "because_sentences": 3, "except": 3, "precedence": 3, "accepted": 4, "accepted_words": 236}, "S4": {"words": 298, "then_words": 156, "because_sentences": 2, "except": 0, "precedence": 3, "accepted": 0, "accepted_words": 0}, "S5": {"words": 632, "then_words": 317, "because_sentences": 1, "except": 2, "precedence": 1, "accepted": 0, "accepted_words": 0}, "S6": {"words": 337, "then_words": 121, "because_sentences": 2, "except": 0, "precedence": 2, "accepted": 0, "accepted_words": 0}, diff --git a/tests/v2/test_regex_sync.py b/tests/v2/test_regex_sync.py index f7e2e74d..cfb9443a 100644 --- a/tests/v2/test_regex_sync.py +++ b/tests/v2/test_regex_sync.py @@ -138,6 +138,9 @@ def test_dotted_initial_is_the_period_alternative_of_initial() -> None: # #544: the chunked sibling of _DOTTED ('M.Eng.'), S3's period gate # for a member with a lower-case tail; no config key to mirror ("_vocab", "_CHUNKED"): None, + # the joiners of a joined credential ('MD/PhD', rules/S3, #752); no + # config key to mirror + ("_vocab", "_JOINERS"): None, # #563: paired initials ('M.J.'), C5's exception to C4's name-word # count after a comma; no config key to mirror ("_vocab", "_PAIRED_INITIALS"): None, diff --git a/tools/differential/corpus_rules.jsonl b/tools/differential/corpus_rules.jsonl index d0f9a937..97dd21c8 100644 --- a/tools/differential/corpus_rules.jsonl +++ b/tools/differential/corpus_rules.jsonl @@ -7,6 +7,7 @@ ", Ana García López" ", Jane" ", Jane Mrs" +", MD/PhD" ".," "1. née Smith" "A.B. Vajpayee" @@ -318,6 +319,8 @@ "John Smith K.C." "John Smith M.A." "John Smith MD PhD" +"John Smith MD-Jones" +"John Smith MD-PhD" "John Smith Ma" "John Smith Mc V" "John Smith Msc.Ed." @@ -361,8 +364,10 @@ "John Smith, Jr., abdul Dr. Smith" "John Smith, Jr., van der Berg Prof." "John Smith, LEED AP" +"John Smith, M.D./Ph.D." "John Smith, M.D.; Ph.D." "John Smith, MA" +"John Smith, MA/PhD" "John Smith, MA; PhD" "John Smith, MD ; PhD" "John Smith, MD, Bart" @@ -371,6 +376,7 @@ "John Smith, MD, R.A.I." "John Smith, MD, XYZ" "John Smith, MD,, Jr." +"John Smith, MD/PhD" "John Smith, MD; PhD" "John Smith, MD;PhD" "John Smith, Ma" @@ -394,6 +400,8 @@ "John Smith, XYZ" "John Smith, otr/l" "John Smith, vd Ma" +"John Smith-Jones" +"John Smith-Jones, MD-PhD" "John V" "John V, Dr." "John W Ingram V" @@ -573,6 +581,7 @@ "Smith, Jan van Prof." "Smith, Jane Mrs" "Smith, John" +"Smith, John MD/PhD" "Smith, John PhD I." "Smith, John PhD Jones" "Smith, John PhD de Jr." @@ -597,6 +606,7 @@ "Smith, Ma" "Smith, Ma. John" "Smith, Major. John" +"Smith, Mary-Jane" "Smith, Ms Jane" "Smith, Ms Ma" "Smith, Ms." @@ -713,6 +723,7 @@ "john smith vi" "john smith x.y.z." "john smith xi" +"john smith, ms-pt" "john smith, otr/l" "john smith, v" "john smith, vi" diff --git a/tools/differential/snapshot.jsonl b/tools/differential/snapshot.jsonl index afe7ab01..9591e4cc 100644 --- a/tools/differential/snapshot.jsonl +++ b/tools/differential/snapshot.jsonl @@ -13,6 +13,7 @@ {"name": ", Jane", "given": "Jane", "initials": "J."} {"name": ", Jane Mrs", "title": "Mrs", "family": "Jane", "initials": "J."} {"name": ", John", "given": "John", "initials": "J."} +{"name": ", MD/PhD", "suffix": "MD/PhD"} {"name": ".,"} {"name": "1 & 2, 3 4 5, Mr.", "title": "Mr.", "given": "3", "middle": "4 5", "family": "1 & 2", "initials": "3. 4. 5. 1. 2."} {"name": "1. née Smith", "given": "1.", "maiden": "Smith", "initials": "1."} @@ -1624,6 +1625,8 @@ {"name": "John Smith MA", "given": "John", "family": "Smith", "suffix": "MA", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. S."} {"name": "John Smith MC", "given": "John", "family": "Smith", "suffix": "MC", "initials": "J. S."} {"name": "John Smith MD PhD", "given": "John", "family": "Smith", "suffix": "MD PhD", "initials": "J. S."} +{"name": "John Smith MD-Jones", "given": "John", "middle": "Smith", "family": "MD-Jones", "initials": "J. S. M.", "capitalized_force": "John Smith Md-Jones"} +{"name": "John Smith MD-PhD", "given": "John", "family": "Smith", "suffix": "MD-PhD", "initials": "J. S."} {"name": "John Smith MEng PhD", "given": "John", "middle": "Smith", "family": "MEng", "suffix": "PhD", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. S. M.", "capitalized_force": "John Smith Meng PhD"} {"name": "John Smith Ma", "given": "John", "middle": "Smith", "family": "Ma", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. S. M."} {"name": "John Smith Mc V", "given": "John", "family": "Smith", "suffix": "Mc V", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. S.", "capitalized_force": "John Smith MC V"} @@ -1682,8 +1685,10 @@ {"name": "John Smith, Jr., abdul Dr. Smith", "title": "Dr.", "given": "John", "family": "Smith", "suffix": "Jr., abdul Smith", "ambiguities": ["COMMA_STRUCTURE"], "initials": "J. S.", "capitalized_force": "Dr. John Smith Jr., Abdul Smith"} {"name": "John Smith, Jr., van der Berg Prof.", "title": "Prof.", "given": "John", "family": "Smith", "suffix": "Jr., van der Berg", "ambiguities": ["COMMA_STRUCTURE"], "initials": "J. S.", "capitalized_force": "Prof. John Smith Jr., Van Der Berg"} {"name": "John Smith, LEED AP", "given": "John", "family": "Smith", "suffix": "LEED AP", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. S."} +{"name": "John Smith, M.D./Ph.D.", "given": "John", "family": "Smith", "suffix": "M.D./Ph.D.", "initials": "J. S."} {"name": "John Smith, M.D.; Ph.D.", "given": "John", "family": "Smith", "suffix": "M.D.; Ph.D.", "initials": "J. S."} {"name": "John Smith, MA", "given": "John", "family": "Smith", "suffix": "MA", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. S."} +{"name": "John Smith, MA/PhD", "given": "MA/PhD", "family": "John Smith", "initials": "M. J. S.", "capitalized_force": "Ma/PhD John Smith"} {"name": "John Smith, MA; PhD", "given": "John", "family": "Smith", "suffix": "MA; PhD", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. S."} {"name": "John Smith, MD ; PhD", "given": "John", "family": "Smith", "suffix": "MD, PhD", "initials": "J. S."} {"name": "John Smith, MD MEng", "given": "John", "family": "Smith", "suffix": "MD MEng", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. S.", "capitalized_force": "John Smith MD MENG"} @@ -1693,6 +1698,7 @@ {"name": "John Smith, MD, R.A.I.", "given": "John", "family": "Smith", "suffix": "MD, R.A.I.", "initials": "J. S."} {"name": "John Smith, MD, XYZ", "given": "John", "family": "Smith", "suffix": "MD, XYZ", "ambiguities": ["COMMA_STRUCTURE"], "initials": "J. S.", "capitalized_force": "John Smith MD, Xyz"} {"name": "John Smith, MD,, Jr.", "given": "John", "family": "Smith", "suffix": "MD, Jr.", "initials": "J. S."} +{"name": "John Smith, MD/PhD", "given": "John", "family": "Smith", "suffix": "MD/PhD", "initials": "J. S."} {"name": "John Smith, MD; PhD", "given": "John", "family": "Smith", "suffix": "MD; PhD", "initials": "J. S."} {"name": "John Smith, MD;PhD", "given": "John", "family": "Smith", "suffix": "MD;PhD", "initials": "J. S."} {"name": "John Smith, MEng PhD", "given": "John", "family": "Smith", "suffix": "MEng PhD", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. S.", "capitalized_force": "John Smith MENG PhD"} @@ -1725,6 +1731,8 @@ {"name": "John Smith, XYZ", "given": "John", "family": "Smith", "suffix": "XYZ", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. S."} {"name": "John Smith, otr/l", "given": "John", "family": "Smith", "suffix": "otr/l", "initials": "J. S.", "capitalized_force": "John Smith OTR/L"} {"name": "John Smith, vd Ma", "given": "John", "family": "Smith", "suffix": "vd Ma", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. S.", "capitalized_force": "John Smith VD MA"} +{"name": "John Smith-Jones", "given": "John", "family": "Smith-Jones", "initials": "J. S."} +{"name": "John Smith-Jones, MD-PhD", "given": "John", "family": "Smith-Jones", "suffix": "MD-PhD", "initials": "J. S."} {"name": "John V", "given": "John", "family": "V", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. V."} {"name": "John V, Dr.", "title": "Dr.", "given": "John", "family": "V", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. V."} {"name": "John V. Smith", "given": "John", "middle": "V.", "family": "Smith", "initials": "J. V. S."} @@ -2993,6 +3001,7 @@ {"name": "Smith, John E, III, Jr", "given": "John", "middle": "E", "family": "Smith", "suffix": "III, Jr", "initials": "J. E. S."} {"name": "Smith, John I", "given": "John", "family": "Smith", "suffix": "I", "initials": "J. S."} {"name": "Smith, John Jr.", "given": "John", "family": "Smith", "suffix": "Jr.", "initials": "J. S."} +{"name": "Smith, John MD/PhD", "given": "John", "family": "Smith", "suffix": "MD/PhD", "initials": "J. S."} {"name": "Smith, John MEng", "given": "John", "middle": "MEng", "family": "Smith", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. M. S.", "capitalized_force": "John Meng Smith"} {"name": "Smith, John PhD I.", "given": "John", "family": "Smith", "suffix": "PhD I.", "initials": "J. S."} {"name": "Smith, John PhD Jones", "given": "John", "family": "Smith", "suffix": "PhD Jones", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. S."} @@ -3025,6 +3034,7 @@ {"name": "Smith, Ma", "given": "Ma", "family": "Smith", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "M. S."} {"name": "Smith, Ma. John", "title": "Ma.", "given": "John", "family": "Smith", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. S."} {"name": "Smith, Major. John", "title": "Major.", "given": "John", "family": "Smith", "initials": "J. S."} +{"name": "Smith, Mary-Jane", "given": "Mary-Jane", "family": "Smith", "initials": "M. S."} {"name": "Smith, Ms Jane", "title": "Ms", "given": "Jane", "family": "Smith", "initials": "J. S."} {"name": "Smith, Ms Ma", "title": "Ms", "given": "Ma", "family": "Smith", "initials": "M. S."} {"name": "Smith, Ms RA Ma", "title": "Ms RA", "given": "Ma", "family": "Smith", "initials": "M. S.", "capitalized_force": "Ms Ra Ma Smith"} @@ -3594,6 +3604,7 @@ {"name": "john smith xi", "given": "john", "middle": "smith", "family": "xi", "initials": "j. s. x.", "capitalized": "John Smith Xi"} {"name": "john smith, ma", "given": "john", "family": "smith", "suffix": "ma", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "j. s.", "capitalized": "John Smith MA"} {"name": "john smith, md ma", "given": "john", "family": "smith", "suffix": "md ma", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "j. s.", "capitalized": "John Smith MD MA"} +{"name": "john smith, ms-pt", "given": "john", "family": "smith", "suffix": "ms-pt", "initials": "j. s.", "capitalized": "John Smith MS-PT"} {"name": "john smith, otr/l", "given": "john", "family": "smith", "suffix": "otr/l", "initials": "j. s.", "capitalized": "John Smith OTR/L"} {"name": "john smith, phd meng", "given": "john", "family": "smith", "suffix": "phd meng", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "j. s.", "capitalized": "John Smith PhD MENG"} {"name": "john smith, v", "given": "john", "family": "smith", "suffix": "v", "initials": "j. s.", "capitalized": "John Smith V"} From 5bb7ce6504818c686dacf61e74d20f6e3fb251b7 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sun, 25 Oct 2026 03:27:40 -0700 Subject: [PATCH 2/7] Read an unlisted word of joined capitals after a comma as a credential (#752) Phase 2d, #797's half A. An unlisted word of capitals whose letter chunks are joined by '-' or '/', one of two or more letters, is the caps shape (rules/S9): 'John Smith, PU-PH' reads suffix 'PU-PH', reported, where 1.4.0 through 2.6.0 read given 'PU-PH', family 'John Smith'. It joins the class where and as the unjoined word does, so C4's count and C6's case contrast gate it after a comma by default ('JOHN SMITH, PU-PH' and 'Smith, PU-PH' keep the given name), and EVERYWHERE takes it at every suffix position. The three call sites' C-level pre-filters (the comma's candidates, C12's case-repair marks, classify under EVERYWHERE) asked isalpha() ahead of the predicate and widen with it; without that nothing moves (mechanisms.md's field notes). Measured against the 2c commit: ctgov-officials 529 entries; fec, tse, psc-individuals, the committed corpora and the agreement records none (decisions/S9). A capitals word between MD and the comma is taken into the suffix as the comma-less name already reads it (rules/S8's run), recorded there rather than changed. Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 2 +- docs/customize.rst | 14 +++++--- ...-that-join-the-ambiguous-class-by-shape.md | 2 ++ docs/design/decisions/vocabulary-model.md | 1 + docs/design/mechanisms.md | 1 + ...caps-word-needs-the-names-case-contrast.md | 3 ++ ...S3-credentials-written-in-joined-chunks.md | 6 +++- ...-that-join-the-ambiguous-class-by-shape.md | 12 +++++-- docs/release_log.rst | 2 +- nameparser/_pipeline/_classify.py | 4 ++- nameparser/_pipeline/_comma.py | 12 +++++-- nameparser/_pipeline/_vocab.py | 32 ++++++++++++++--- tests/v2/cases.py | 33 +++++++++++++++++ tests/v2/pipeline/test_vocab.py | 35 +++++++++++++++++++ tests/v2/rules_budget.json | 4 +-- tests/v2/test_cases.py | 8 +++-- tests/v2/test_name_usage_tool.py | 12 ++++--- tests/v2/test_regex_sync.py | 4 +++ tools/differential/corpus_rules.jsonl | 9 +++++ tools/differential/snapshot.jsonl | 9 +++++ 20 files changed, 178 insertions(+), 27 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 350fa3ae..e7f3829b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -186,7 +186,7 @@ The library has two layers: `nameparser/config/` (data) and `nameparser/parser.p **Sweep the forms a change can reach: comma shapes, then `name_order`.** No comma, a FULL name before the comma, a ONE-WORD name before the comma and NOTHING before the comma are four paths, and the last two are the misses — #429, #430 and #432 are all the one-word path disagreeing with the full-name path on inputs the full-name path reads correctly, and #613's PR review found `, PhD` and `, Jr.` (an empty surname field, as CSV data writes it) read as given and title by a decision that returned early on an empty head, after two review rounds whose every grid had a name before the comma. Put an empty head and a nickname-only head (`(Bob), PhD`) in every comma grid. Count the one-word path the way C3 counts, which is neither tokens nor pieces: a particle run and the one word it attaches to are one word (#575), a connective join and a bound given-name pair count as their words (#613, so `Ortega y Gasset, Dr.` is a whole name), and a suffix beside the surname counts as no name word at all. #603's review found the suffix half (`Vega y Lopez Jr., PhD Jones` lost its family where `Vega y Lopez, PhD Jones` kept it); #613 then reversed the connective half, and both now read given 'Vega', middle 'y', family 'Lopez' — so test a particle surname, a connective surname and a surname with a suffix, each before a comma. For orders the sweep already exists (`tests/v2/test_cases.py` runs every row under all three) but carries one assertion, R2's family partition, so it checks nothing a new change moves; coverage there has been vacuous before (PR #394's review found the suite passed with `name_order` discarded from grouping). Parse your change's names in each comma shape and each order, and read the ones you did not predict. And read a generated grid's moves by SHAPE, not by corpus membership: #614's first design passed the suite, the five gates and a fingerprint of every corpus name, because its grid moves were sorted into 'corpus name moved' and 'garbage', and the review found the moves that mattered among the 'garbage' -- a bare salutation (`Mr. and Mrs.` lost its connective to the family name) and a name with one trailing word (`Freiherr von Berg X.Y.Z.`). Pull out the head-plus-one-word slice of the grid and a salutation with nothing after it, and read every move there. Two more were the second review's (#614): no grid put TWO words that are both a title and a particle at the head (`Freiherr St van Berg MA`, where an index into the pieces was applied to a merged view of them -- `TITLES ∩ particles_ambiguous` is `{freiherr, st}`, so put both in every head list, each in its bare and its period-marked spelling -- #637's grid had `St` and `Freiherr` alone and missed that `St.` and `Freiherr.` are declined by the bound join and then attached by P6), and the grids compared ambiguity KINDS, so a report telling the caller the wrong field (`Freiherr von Berg Ma`, 'read as a given name' of a word in the family) passed every pin. #701's review added one: put a title/suffix dual (`Ms`, `Sr.`, `Cpl`) SECOND in a title run, behind an ordinary title. A stop that asks only "is this word suffix vocabulary" passes every row that opens with the dual and fails `Rev. Sr. Smith, Mary`; it was found by comparing `T Smith, John` against `T John Smith` over every title word, and a comma-versus-comma-less comparison is the cheap check for any rule that moves a word between the two slots. Compare each report's `detail` against the field its tokens land in -- since #626 `test_a_report_names_the_field_its_word_lands_in` (tests/v2/test_cases.py) does it over every non-locale case row under all three orders (a row carrying its own policy only as declared), for the phrasings its `_FIELD_CLAIMS` lists -- so a row with the shape is the test only where the report's wording is listed: a new emitter's wording goes in that list, and segment's report on a part past the second comma names no field (#629). #626's own first grid then missed an emitter for want of one comma shape (`, Ma Dr.`: an empty head, an ambiguous word, a trailing title, which H1 moves to the family): a trailing title belongs in every empty-head comma grid too. And #620's: an exclusion list in code being replaced is a set of rules, not noise -- #614's merged copy excluded every word the trailing read weighs from its units, the first #620 prototype dropped the list as a symptom of the copy, and its review found the list had been keeping positional counting's rule that a member in front counts as a word (`Freiherr von Berg MA X.Y.Z.` lost its suffix). Before deleting one, mutate each entry out and read what moves. The converse holds for replacing a hand copy WITH the shared predicate: it is a behavior change wherever the two disagreed, and #735's /simplify shipped one as "no reading moves" because the suite and the snapshot were its only check (`St Sir de Mesnil` family-first, a token test replaced by P1's piece test); diff the two predicates' answers over a grid before calling the swap a refactor. -**Then sweep the spellings: a class reached by vocabulary is also reached by SHAPE.** A test that uses only the listed spelling of a word class walks only the vocabulary path, and the shape path is a separate branch that can be missing while every test passes. Roman numerals are the sharpest case: `ii`, `iii` and `iv` are suffix vocabulary; `i` and `v` are vocabulary too but initial-shaped, so `is_suffix_piece` vetoes them and only the numeral fork reads them; and `vi` through `x` are in no list at all, read by the fork's shape test alone (checked 2026-10-04 against `Lexicon.default()`) -- which stops at `x`: `_vocab._ROMAN` matches I to X, so `John Smith XI` reads `XI` as a name word (re-checked 2026-10-07, #614's /simplify, which found a docstring claiming "any length"). And the fork counts a ONE-letter numeral besides, since #718 (rules/S10): it is the suffix only where the name has a word to spare, so `John V` and `John Smith V` take different branches of one fork, a one-case name being counted like capitals whatever the letter -- since #721 a one-case one-letter numeral carries the `initial` tag its capital does, and since #723 every one-case bare letter does (rules/T4), and a bare letter in front of the numeral is initial-shaped as its capital is, so `john v` reads as `JOHN V` does over `test_a_one_case_name_reads_as_its_capitals`'s grid (comma shapes, orders, a maiden clause, a particle chain, a bare letter in front, a trailing title, a non-ASCII letter in front: 315 of its then 1,872 pairs disagreed at 2bfa9533, before #721, and none after). Its first version claimed "every shape" with no bare letter in front and no trailing title in the grid, and #721's reviews found both broken; a grid's claim is the grid -- put a head-plus-numeral row beside every words-to-spare row. So a change that handles `V` and `III` can still miss `VI` — #610 found exactly that, the maiden take's candidate loop admitting a shape-only numeral in one position where the peel reads it in two (`Jane Doe nee Smith VI Prof.` kept maiden `Smith VI` while `V` and `III` worked). #656's sweep found the same class from the other side: C12 takes the word ending the name before a comma as EVIDENCE, and its first draft asked the suffix-word tag, so `John Smith VI, AD` missed the rule `III` reached, and then counted the whole part, so `Smith VI, AD` fired with `VI` as its own name word and `Smith V, AD` with a `V` that S10 reads as the family. A rule that weighs a numeral as a generation must ask the predicate that READS it as one (the trailing fork, `_vocab.reads_as_generation`, and its words-to-spare count), not the vocabulary that lists some of its spellings. The same split runs through the credentials (`MA` listed, `X.Y.Z.` by the dotted shape, `XYZ` by the caps shape, and `Ph. D.` merged by group into one flagged piece the peel's walk never holds -- #603 found #602's run-start predicate refusing it too, so `Smith, John Ph. D. Jones` kept middle 'Jones' where `Smith, John PhD Jones` read suffix 'PhD Jones') and the titles (`Prof.` period-marked and read at the end of a name, `Prof` bare and a name word there, and `Mr. and Mrs.` a run P3 joins, carrying a piece tag a one-word vocabulary test never sees -- PR #735's review found C12's hand copy of C13's title test (C8's until its 2026-10-19 split) reading `John Smith Jr., Mr. and Mrs. AD` as all suffix; ask C13's predicates in C13's order, `not is_suffix_piece` then `is_title_piece`, not the token tags, since `is_title_piece` alone reads the duals `Ms` and `Sr.` as titles). When a change touches a word class, test one spelling from each path. The same holds for a separator's SPACING: a comma (or any punctuation) can stand alone, glue to the word before it or glue to the word after, and each is a different token stream. #653's first fix swept `(née, Jones)` and missed `(née , Jones)`, where the kept comma became a word after the marker and M3 read the clause as maiden ', Jones'; its review found it. Put all three spacings in any grid around a separator. And a separator glued to a word hides the word from every vocabulary test, not only the one being changed: the semicolon delimiter's first draft read a glued word's stem in C1's run test alone and put 365 of ClinicalTrials.gov's reversed names right, missing the ones C13, C4 and C6 read off a word's TAGS (`MBBS;`, `MA;`); tagging the stem put 412 right, and reading it again in the comma decision's shape tests, which read a word's text rather than its tags (`MBBS;`), 431 (decisions/C10, 2026-10-24). A rule over a glued spelling reads the stem everywhere the word is read, its tags and every test of its text, and is swept against the space twin -- with a declared word delimiter too, which a substring test finds inside other words ('Stand' under ' and ', the review of the same change). A full stop INSIDE a word has four spellings of its own: initials run together (`J.R.R.`), joined words each closed by a stop (`Lt.Gov.`, `Msc.Ed.`), a joined word or name closed by none (`St.John`, `J.Smith`), and an abbreviation's own ending, closed or not (Italian `Sig.ra`, Latvian `Dr.philol.`). Shape tells the last from none of the others, so a rule over the shape decides which it costs; #838 found the ending only through the snapshot's CLDR row `Dut.ra Prof.ra`, and its first narrowing traded `St.John` for `Sig.ra` while the snapshot listed the `Dr.philol.` move among moves read as intended, until the design-docs review named it. Initials run together are also how some SURNAMES are written (Nepali `B.K.`, `K.C.`), and a caseless script writes its abbreviations, degrees included, as dotted letters (Thai `พ.บ.`): #864 found S3's position reading taking the first from 2.4.0 on, visible only once #866's Nepal candidates gave the corpora a population that writes them, and its first fix moved CLDR's Thai degrees into the name. A rule over dotted letters reads both before it ships. +**Then sweep the spellings: a class reached by vocabulary is also reached by SHAPE.** A test that uses only the listed spelling of a word class walks only the vocabulary path, and the shape path is a separate branch that can be missing while every test passes. Roman numerals are the sharpest case: `ii`, `iii` and `iv` are suffix vocabulary; `i` and `v` are vocabulary too but initial-shaped, so `is_suffix_piece` vetoes them and only the numeral fork reads them; and `vi` through `x` are in no list at all, read by the fork's shape test alone (checked 2026-10-04 against `Lexicon.default()`) -- which stops at `x`: `_vocab._ROMAN` matches I to X, so `John Smith XI` reads `XI` as a name word (re-checked 2026-10-07, #614's /simplify, which found a docstring claiming "any length"). And the fork counts a ONE-letter numeral besides, since #718 (rules/S10): it is the suffix only where the name has a word to spare, so `John V` and `John Smith V` take different branches of one fork, a one-case name being counted like capitals whatever the letter -- since #721 a one-case one-letter numeral carries the `initial` tag its capital does, and since #723 every one-case bare letter does (rules/T4), and a bare letter in front of the numeral is initial-shaped as its capital is, so `john v` reads as `JOHN V` does over `test_a_one_case_name_reads_as_its_capitals`'s grid (comma shapes, orders, a maiden clause, a particle chain, a bare letter in front, a trailing title, a non-ASCII letter in front: 315 of its then 1,872 pairs disagreed at 2bfa9533, before #721, and none after). Its first version claimed "every shape" with no bare letter in front and no trailing title in the grid, and #721's reviews found both broken; a grid's claim is the grid -- put a head-plus-numeral row beside every words-to-spare row. So a change that handles `V` and `III` can still miss `VI` — #610 found exactly that, the maiden take's candidate loop admitting a shape-only numeral in one position where the peel reads it in two (`Jane Doe nee Smith VI Prof.` kept maiden `Smith VI` while `V` and `III` worked). #656's sweep found the same class from the other side: C12 takes the word ending the name before a comma as EVIDENCE, and its first draft asked the suffix-word tag, so `John Smith VI, AD` missed the rule `III` reached, and then counted the whole part, so `Smith VI, AD` fired with `VI` as its own name word and `Smith V, AD` with a `V` that S10 reads as the family. A rule that weighs a numeral as a generation must ask the predicate that READS it as one (the trailing fork, `_vocab.reads_as_generation`, and its words-to-spare count), not the vocabulary that lists some of its spellings. The same split runs through the credentials (`MA` listed, `X.Y.Z.` by the dotted shape, `XYZ` by the caps shape, `MD/PhD` by its listed chunks and `PU-PH` by the caps shape's joined spelling, #752, each with its joiner glued, spaced and glued to one side, which are three token streams, and `Ph. D.` merged by group into one flagged piece the peel's walk never holds -- #603 found #602's run-start predicate refusing it too, so `Smith, John Ph. D. Jones` kept middle 'Jones' where `Smith, John PhD Jones` read suffix 'PhD Jones') and the titles (`Prof.` period-marked and read at the end of a name, `Prof` bare and a name word there, and `Mr. and Mrs.` a run P3 joins, carrying a piece tag a one-word vocabulary test never sees -- PR #735's review found C12's hand copy of C13's title test (C8's until its 2026-10-19 split) reading `John Smith Jr., Mr. and Mrs. AD` as all suffix; ask C13's predicates in C13's order, `not is_suffix_piece` then `is_title_piece`, not the token tags, since `is_title_piece` alone reads the duals `Ms` and `Sr.` as titles). When a change touches a word class, test one spelling from each path. The same holds for a separator's SPACING: a comma (or any punctuation) can stand alone, glue to the word before it or glue to the word after, and each is a different token stream. #653's first fix swept `(née, Jones)` and missed `(née , Jones)`, where the kept comma became a word after the marker and M3 read the clause as maiden ', Jones'; its review found it. Put all three spacings in any grid around a separator. And a separator glued to a word hides the word from every vocabulary test, not only the one being changed: the semicolon delimiter's first draft read a glued word's stem in C1's run test alone and put 365 of ClinicalTrials.gov's reversed names right, missing the ones C13, C4 and C6 read off a word's TAGS (`MBBS;`, `MA;`); tagging the stem put 412 right, and reading it again in the comma decision's shape tests, which read a word's text rather than its tags (`MBBS;`), 431 (decisions/C10, 2026-10-24). A rule over a glued spelling reads the stem everywhere the word is read, its tags and every test of its text, and is swept against the space twin -- with a declared word delimiter too, which a substring test finds inside other words ('Stand' under ' and ', the review of the same change). A full stop INSIDE a word has four spellings of its own: initials run together (`J.R.R.`), joined words each closed by a stop (`Lt.Gov.`, `Msc.Ed.`), a joined word or name closed by none (`St.John`, `J.Smith`), and an abbreviation's own ending, closed or not (Italian `Sig.ra`, Latvian `Dr.philol.`). Shape tells the last from none of the others, so a rule over the shape decides which it costs; #838 found the ending only through the snapshot's CLDR row `Dut.ra Prof.ra`, and its first narrowing traded `St.John` for `Sig.ra` while the snapshot listed the `Dr.philol.` move among moves read as intended, until the design-docs review named it. Initials run together are also how some SURNAMES are written (Nepali `B.K.`, `K.C.`), and a caseless script writes its abbreviations, degrees included, as dotted letters (Thai `พ.บ.`): #864 found S3's position reading taking the first from 2.4.0 on, visible only once #866's Nepal candidates gave the corpora a population that writes them, and its first fix moved CLDR's Thai degrees into the name. A rule over dotted letters reads both before it ships. **And the encoding: a letter class written `[^\W\d_]` or tested with `isalpha()` is blind to a combining mark.** Python's `\w` and `isalpha()` do not match Mn, so a letter written as base plus accent (NFD) ends a word early; the vocabulary path is immune because `_normalize` NFC-folds, the shape path is not. #652: `Dr. inž. Janez Novak` read `inž.` as the given name when the ž was decomposed, and `Janez Novak, ŽŠČ` reversed, in two families that share the blind spot — the letter-shape regexes (`period_abbreviation`, `_DOTTED`, `_CHUNKED`, `_PAIRED_INITIALS`), now spelled with `_vocab._LETTER` (a letter plus the general combining blocks; `test_regex_sync` compares the two `_MARKS` strings and pins `period_abbreviation` against config's copy), and the caps predicates, which NFC-compose a non-ASCII word before `isalpha()` (`is_caps_word`, behind C-level `isalpha()`/`is_normalized` tests, so already-composed text pays no frame — measured 2026-10-12, `Smith, José` was 193 frames before and after; the first version called it for every non-ASCII word and cost one frame each, and no guard saw it because the reference name is ASCII). A width test must count letters, not code points (`composed_len`): admitting a decomposed word to the caps set put `John Smith, MÁ` on the pair exception's wrong side, and the review's grid found it where the rows (all three letters long) did not — put a two-letter word in every caps grid. The initial shape (`_INITIAL`, `\w\.`, v1-verbatim and public) is not widened but asked of the COMPOSED word, in `is_initial_shaped` and in `_render`'s two copies; and the single-letter connective tests count composed letters, since a decomposed `й` is two code points (#665, where `É. y É. Smith` and `John й Smith` read differently decomposed). Compose the WHOLE predicate, not one half: `is_initial` is a shape ANDed with the #320 script veto, and composing the shape alone made a decomposed `씨.` an initial, since decomposed hangul is conjoining jamo outside every script range (#665's first version; `in_initialless_script` composes too now). Not covered, deliberately: the marks only some scripts write (Indic vowel signs, Arabic and Hebrew points), declined in decisions/nfd-shapes because widening the class would move readings nobody had measured; rules/H2's Accepted clause records the one consequence it names, that a period-marked abbreviation in Bengali or Devanagari never reads as a title by shape. The guard is `test_a_decomposed_name_renders_as_its_composed_twin` over the CLDR corpus plus `test_a_decomposed_abbreviation_reads_as_its_composed_twin` (rows × three orders × EVERYWHERE); the corpus guard reaches a shape only if some corpus name does, which is how the caps shape was missed until review, so when adding a shape rule add a row there. Four of #652's rows and two of #665's pass with every fix reverted (measured) and are parity pins, and the test compares ambiguity kinds and `initials()` too, since #665's `John й Smith` differed in its report; `_DOTTED` has no observable input and is pinned by a direct call. diff --git a/docs/customize.rst b/docs/customize.rst index 695ac223..270e3e68 100644 --- a/docs/customize.rst +++ b/docs/customize.rst @@ -1142,16 +1142,17 @@ A hyphen or a slash written between two credentials with no space needs no declaring when every part is a listed credential that is not also a name, because the word then reads as one credential (since 2.7). A part that may be a name, such as ``MA``, keeps the word a name word, since a -hyphen also joins surnames: +hyphen also joins surnames, unless the word is written in capitals, +which :ref:`all-caps-credentials` reads on its own terms: .. doctest:: >>> joined = parse("Jane Smith, RN/CRNA") >>> joined.given, joined.family, joined.suffix ('Jane', 'Smith', 'RN/CRNA') - >>> joined = parse("Jane Smith, MA/CRNA") + >>> joined = parse("Jane Smith, MA/PhD") >>> joined.given, joined.family - ('MA/CRNA', 'Jane Smith') + ('MA/PhD', 'Jane Smith') .. _unlisted-credentials: @@ -1268,7 +1269,9 @@ All-caps words ^^^^^^^^^^^^^^ ``unlisted_caps_suffixes`` reads an unlisted word of two or more -capital letters, with no period in it. Its value is a +capital letters, with no period in it, and since 2.7 one of capitals +joined by a hyphen or a slash (``PU-PH``, ``MS-OMPT``) with a part of +two or more letters. Its value is a :class:`~nameparser.CapsSuffixes`, because where capitals mean a credential depends on a second convention: many records write the SURNAME in capitals (``"Jean DUPONT"``, ``"DUPONT, Jean"``). @@ -1297,6 +1300,9 @@ all-caps surname convention never writes capitals there: 'XYZ' >>> parse("JOHN SMITH, XYZ").given 'XYZ' + >>> cred = parse("John Smith, PU-PH") + >>> cred.family, cred.suffix + ('Smith', 'PU-PH') ``EVERYWHERE`` """""""""""""" diff --git a/docs/design/decisions/S9-unlisted-words-that-join-the-ambiguous-class-by-shape.md b/docs/design/decisions/S9-unlisted-words-that-join-the-ambiguous-class-by-shape.md index 0bf14899..f094b97e 100644 --- a/docs/design/decisions/S9-unlisted-words-that-join-the-ambiguous-class-by-shape.md +++ b/docs/design/decisions/S9-unlisted-words-that-join-the-ambiguous-class-by-shape.md @@ -13,6 +13,7 @@ Examples name each field as the default order reads it; under the family-first o - The comma's case contrast is carried by a word of the name before the comma that holds a capital, ends in a lowercase cased letter and is claimed by no wordlist as a non-name, so an all-caps record keeps its given name (`MÜLLER WEIß, HANS`, `LLOYD WEBBER, ANDREW PhD`) and a name written in lowercase reads the listing form (`john smith, XYZ` → given 'XYZ'); a lone two-letter word is never admitted at the comma (`García Márquez, MJ` → given 'MJ') — 2026-10-01 #564. - Accepted: a given name written in capitals behind a two-word surname reads as the credential (`García Márquez, GABRIEL` → suffix 'GABRIEL', reported), OFF being the escape; and case repair follows the reading, a caps word in the part a suffix comma opened being tagged under AFTER_COMMA, one in a third or later part only under EVERYWHERE — 2026-10-01 #564. - Under EVERYWHERE the cost is the three-word name (`Jean Pierre DUPONT` → family 'Pierre', suffix 'DUPONT'), while a two-word name keeps its family and reports (`Jean DUPONT`); unlisted is checked by wordlist membership rather than tags (`John Smith NEE` keeps family 'NEE'), and a roman numeral is the numeral fork's before the shape is asked (`John Smith VI` → suffix 'VI') — 2026-09-14 (Derek), corrected 2026-09-18. +- The caps shape has a joined spelling: an unlisted word of capitals whose letter chunks are joined by a hyphen or a slash, one of them of two or more letters, joins the class wherever the caps shape is admitted, gated as it is (`John Smith, PU-PH` → suffix 'PU-PH', reported; `JOHN SMITH, PU-PH` and `Smith, PU-PH` → given; `John Smith, A-B` and `John Smith, Pu-Ph` → given), and under EVERYWHERE at every suffix position as the unjoined word is (`Jean Pierre DUPONT-MORETTI` → suffix) — 2026-10-25 #752. - Neither switch reaches the v1 shim, which translates neither: `HumanName` follows both defaults and a v1 caller cannot turn them off (`HumanName("John Smith X.Y.Z.").suffix` and `HumanName("John Smith, XYZ").suffix` are 'X.Y.Z.' and 'XYZ'), the caps half since the AFTER_COMMA default — 2026-09-14 (Derek), corrected 2026-09-18; 2026-10-01 #564. - A rule's statement names no `Policy` field; the example annotations carry the field names — 2026-09-18, RULES.MD STATEMENTS. - An example's text is a default-policy corpus name whatever its annotation, so an annotated-only example is checked for a default-policy diff like any other — 2026-09-18, corrected. @@ -50,3 +51,4 @@ Open: none in this file. COST, `sys.setprofile` call counts against `git archive origin/master`: ordinary and listed-credential comma names unchanged, `John Smith, XYZ` 251 → 266, and an all-caps record with a capitalized given name and a credential (`LLOYD WEBBER, ANDREW PhD`) a constant +26 (the second draft's per-character contrast test had made `'LLOYD'*64 + ' WEBBER, ANDREW PhD'` +350), the contrast linear in the words before the comma and constant in a word's letters, all paid only once a caps word is in hand (PR #577). The comma test is on by default, so three C-level checks go before any call: two or more words before the comma (the first draft lacked it, and `Smith, JOHN`, a common record format, paid +25 for a flip it can never make — the code review), the first word in capitals, and the lone-two-letter length. Every caller of the caps predicate asks first, in C, what it would decline anyway — alphabetic capitals, not a listed suffix word — so a listed credential never pays for the call (the review of the first fix found `John Smith, CPA` +4 and `John Smith, Ph. D.` +5 before these). - 2026-10-17 #682 — SPLIT FROM decisions/S2 with the rule. The entries above were written under decisions/S2 while this rule was part of rules/S2, and moved here unchanged but for pointers the move broke; where one says rules/S2 it means the rule as it then stood, which held this one. - 2026-10-24 #864 — PAIRED INITIALS LEAVE THE CLASS AT THE TRAILING SLOTS. The 2026-09-30 #563 entry's choice (iii) left those slots with #516's reading; for paired initials of a script with capitals that question is answered under decisions/S3, the rule whose position reading took them, and this bullet is S9's pointer to it. +- 2026-10-25 #752 — THE CAPS SHAPE'S JOINED SPELLING (phase 2d, the unlisted half of item 5 of decisions/vocabulary-model's 2026-10-23 #752 entry, agreed by Derek on 2026-10-23; #797's half A). An unlisted word of capitals in letter chunks joined by `-` or `/`, one chunk of two or more letters, is the caps shape: `John Smith, PU-PH` reads suffix 'PU-PH' with the class's report where 1.4.0 through 2.6.0 read given 'PU-PH', family 'John Smith'. It joins the class exactly where and as the unjoined word does, so the C4 count and C6's case contrast gate it after a comma by default (`JOHN SMITH, PU-PH`, `Smith, PU-PH` → given) and EVERYWHERE takes it at every suffix position, its cost the unjoined word's (`Jean Pierre DUPONT-MORETTI` reads as `Jean Pierre DUPONT` does); keeping it to the comma under EVERYWHERE too was not built, since it would make one shape two by position. "Unlisted" is the whole word in no wordlist, as for the unjoined word; a word whose every chunk counts as a suffix as written is rules/S3's joined credential, read unambiguously before this shape is asked (`MD-PHD`), so the two overlap in no reading. Single-letter chunks alone are how initials are joined (`A-B`) and stay out, and a chunk holding a digit or a period is no letter chunk. REACHABILITY: the comma's candidate list, C12's case-repair marks and classify under EVERYWHERE each ask `isalpha()` and `isupper()` in C before calling the predicate, and the joined spelling fails `isalpha()`, so each pre-filter widened with it to a capitals word holding a joiner; the 2026-10-23 measurement's first run moved nothing for the same reason. MEASURED (2026-10-25, against the phase 2c commit 15d5426f, every local corpus recorded from it with `--tree`): `tools/differential/snapshot.py` moves nothing but the rules' own new examples; `cldr_agreement.py` matches every record; `local_agreement.py psc-individuals` matches its record; tse-donors and fec-individuals, none (FEC is written in one case, so the contrast never admits a word there); ctgov-officials, 529 of 352,104 entries. Of the 529, 519 are a joined capitals word after a comma, now the suffix with a suffix-or-name report (5 of them in a third part, where the comma-structure report stays): 516 had read it as the given name with the name reversed around it, and 3 had read a listed credential opening the part as the given name and the joined word as the middle (`John Smith, MSN ACNP-BC`), the whole part now the suffix. 10 move only in `capitalized(force=True)`, a joined capitals word already in a suffix run now keeping its capitals (rules/R4). The words are 104 distinct credentials and job titles, `PU-PH` (258) and `MCU-PH` (30), the French hospital-university ranks, and `MS-OMPT` (32) the commonest; none is a name. The 2026-10-23 figure, 759, was measured without phase 2c, which now reads about 230 of them as listed credentials. One of the 516 also takes a word before the comma into the suffix, as the unjoined twin reads: a capitals word between a listed credential and the comma (`Jean Claude MD DUPONT, PU-PH` → family 'Claude', suffix 'MD DUPONT, PU-PH', reported), where the part before the comma reads as the name without it (rules/C1) and `Jean Claude MD DUPONT` alone already gives suffix 'MD DUPONT' by rules/S8's run; ACCEPTED here, since it is S8's reading of a capitals surname behind a credential, the same with `XYZ` or `PhD` after the comma. COST: a capitals word holding a joiner pays the caps shape's path, as an unjoined capitals word already does (`SMITH-JONES, MARY-JANE` 175 → 189 profiler frames, py3.11), and a word with a lowercase letter or no joiner pays nothing new (`Mary Smith-Jones` 159, `John Smith, XYZ` 264, before and after). Pinned by the #752 rows of tests/v2/cases.py and `test_caps_shape_candidate_reads_joined_capitals` (tests/v2/pipeline/test_vocab.py). diff --git a/docs/design/decisions/vocabulary-model.md b/docs/design/decisions/vocabulary-model.md index 926cd555..72d117d5 100644 --- a/docs/design/decisions/vocabulary-model.md +++ b/docs/design/decisions/vocabulary-model.md @@ -189,3 +189,4 @@ Open: #752 (phase 2, derived ambiguity; its sub-phases 2a to 2m, each its own PR - **Measured neutral** against the branch, first on 632685c8 and again on 2dd0414b once phase 2a had merged: a grid of 20,364 texts (22 heads covering no comma, a whole name, one word and nothing before a comma, a nickname-only head, a maiden clause, a family comma with a title-particle head and a quoted clause before the pair; 33 spellings including `MA`, `M.D.`, `X.Y.Z.`, `XYZ`, `Ph. D.` and its case variants, `V`, `VI`, `III`, `XI`, `M.J.`, `K.C.` and `พ.บ.`; each text also in lower and upper case), under all three orders, read identically (fields, ambiguity kinds and details, token roles and tags, `initials()`, `capitalized()`), and the same harness reported 45,871, 18, 3,974 and 2,986 differing readings with `split_pair`, C12's call to the numeral predicate, the cased half and the peel's call to it each knocked out. The old and new predicates gave the same answer on 19,769 direct calls. The committed snapshot, every CLDR, GND, Wikidata and conventions agreement record, psc-individuals' agreement record and the ctgov-officials, fec-individuals and tse-donors local snapshots (each recorded from the base with `--tree`) all match, on both bases. psc-individuals' record was itself stale against 632685c8 (3 regressions, 5 progress, 5 changed before this change), so it was re-recorded from each base first. - **Frames**, counted with `sys.setprofile` on both trees: `_CALL_BASELINE`'s reference name and `Smith, John` cost what they did (305 and 158). A word matching `SPLIT_PAIR_OPENS` (`Ph`, `ph.`) with a word after it pays one frame at each site that tests it, whether or not a `D.` follows (`John Smith Ph. D.` 233 → 234, `Smith, John Ph. D.` 239 → 241; the review counted `ph ph ph ph ph ph` 282 → 286), so the cost grows only with the number of such words. C12's evidence keeps its tag test ahead of the call, so it costs what it did (`John Smith Jr., AD` 372). That is a rare shape that no guard counts. - **2026-10-25 #752 — phase 2c: a credential joined by a hyphen or a slash reads as one where every chunk counts as a suffix as written**, carrying out the listed half of item 5 of the 2026-10-23 #752 entry. `John Smith, MD/PhD` reads suffix 'MD/PhD' where it read given 'MD/PhD', family 'John Smith'; a bare member of the ambiguous class is no chunk the test counts (`John Smith, MA/PhD` keeps its reading). The record, with what was measured and what was left unread, is decisions/S3's entry of this date (rules/S3). +- **2026-10-25 #752 — phase 2d: an unlisted word of capitals joined by a hyphen or a slash is the caps shape**, carrying out the unlisted half of item 5 of the 2026-10-23 #752 entry (#797's half A). `John Smith, PU-PH` reads suffix 'PU-PH', reported, where it read given 'PU-PH', family 'John Smith'; C6's case contrast gates it after a comma as it gates the unjoined word. The record is decisions/S9's entry of this date (rules/S9, rules/C6). diff --git a/docs/design/mechanisms.md b/docs/design/mechanisms.md index 41d9d0e0..b2e1878a 100644 --- a/docs/design/mechanisms.md +++ b/docs/design/mechanisms.md @@ -432,6 +432,7 @@ A second axis cuts across the jobs: whether the names are real or invented, labe - Test a function INSIDE a stage on the inputs its caller hands it, not on the stage's finished output. #610's first sweep test built pieces by running `group` whole and handed them to the maiden take's candidate function, but the take runs before any join, so the test read `de VI Prof.` as one piece where production hands it three: a guard checking a structure production never passes in, green for the wrong reason. Review moved the oracle to the take itself by wrapping it (tests/v2/pipeline/test_pieces.py), and recorded the old loop's misses there as the negative control (432), which is what shows the moved harness can still see a difference. - Enumerate the rules that BUILD a structure; do not recall them. #395's unit walk was written three times in one PR — P2's chain missing, then the conjunction and bound-given branches absorbing one token where the particle branch absorbed a unit, then the suffix stop — and each miss came from listing the joining rules from memory instead of reading them out of the rules. Both later misses reproduced the very defect the first fix had just removed, mirrored. +- Widening a predicate widens nothing its callers filter out first. A predicate on a hot path often sits behind a C-level pre-check at each call site, a necessary half of it inlined so a word that cannot pass pays no frame (`isalpha()` and `isupper()` ahead of `caps_shape_candidate`), and a widened predicate behind an unwidened pre-check is unreachable: #752's joined capitals (`PU-PH`) failed `isalpha()` at all three call sites, and the 2026-10-23 measurement of the change first moved nothing. Before measuring a widened predicate, grep its call sites for the pre-check and widen each with it, and hold one input the change must move. - Prefer monkeypatching a module-level helper to re-exec'ing a module: a caller looks a module global up at CALL time, so replacing `_units` or `_name_positions` reaches it and none of the traps below apply. Re-exec is only needed for logic INLINE in a function, and "the thing I want to mutate is not a module-level helper" is a hint that it could be one. - Re-exec'ing a module redefines the classes that module DEFINES, not the ones it imports — the exec re-runs its own import statements, which resolve through sys.modules. So `is` comparisons break selectively: re-exec'ing `_group.py` yields a second `BoundJoin` (defined there) while `Structure` (imported) stays identical, and `bound_join is not BoundJoin.DISABLED` is then always true, joining a family comma's own segment and reporting 3,150 phantom movers. Rebind what the module defines (`mod.__dict__["BoundJoin"] = real.BoundJoin`) and assert the identity rather than assuming it either way. - `_pipeline.STAGES` holds function OBJECTS captured at import, so re-exec'ing a stage module leaves the pipeline running the old function and every result is a false negative. Rewire `STAGES` and assert the new function is in it. diff --git a/docs/design/rules/C6-unlisted-all-caps-word-needs-the-names-case-contrast.md b/docs/design/rules/C6-unlisted-all-caps-word-needs-the-names-case-contrast.md index e653ef00..5d9f2401 100644 --- a/docs/design/rules/C6-unlisted-all-caps-word-needs-the-names-case-contrast.md +++ b/docs/design/rules/C6-unlisted-all-caps-word-needs-the-names-case-contrast.md @@ -18,6 +18,9 @@ "John Smith, LEED AP" → suffix="LEED AP" "Smith, XYZ" → given="XYZ" · boundary "John Smith, PhD XYZ" → suffix="PhD XYZ" +"John Smith, PU-PH" → suffix="PU-PH" +"JOHN SMITH, PU-PH" → given="PU-PH" · boundary +"Smith, PU-PH" → given="PU-PH" · boundary ``` --- diff --git a/docs/design/rules/S3-credentials-written-in-joined-chunks.md b/docs/design/rules/S3-credentials-written-in-joined-chunks.md index f8a37eb6..8bc9e391 100644 --- a/docs/design/rules/S3-credentials-written-in-joined-chunks.md +++ b/docs/design/rules/S3-credentials-written-in-joined-chunks.md @@ -71,13 +71,17 @@ **Precedence:** +- **A joined word in capitals with a chunk the stricter test refuses** ("John Smith, MA/PHD"): S9's caps shape reads it, as it reads any unlisted word written in capitals, so after a comma where the name carries C6's case contrast it joins the ambiguous class, where the same word with a lowercase letter stays the name word this rule leaves it. + ``` + "John Smith, MA/PHD" → suffix="MA/PHD" + ``` - **An ambiguous credential between an unambiguous one and paired initials** ("Jane Doe PhD MA R.T."): the unambiguous word starts the run and the run takes both, the pair included (S8). - **Paired initials at the first slot after a comma** ("García Márquez, G.J."): they read as the given name unless something in the part speaks for them (C5). - **A flip to the credential run at the first slot after a comma** ("John Smith, X.Y.Z."): where no listed member of the class takes part it is not reported, unless paired initials speak only for each other (C11). **Accepted:** -- A joined credential with a chunk that is a bare member of the ambiguous class keeps the reading it had before joined chunks were read, where a name in front of a comma is reversed: the hyphen joins names too, and a name word joined to a listed credential is not one the stricter test may take. +- A joined credential with a chunk that is a bare member of the ambiguous class, written with a lowercase letter, keeps the reading it had before joined chunks were read, where a name in front of a comma is reversed: the hyphen joins names too, and a name word joined to a listed credential is not one the stricter test may take. ``` "John Smith, MA/PhD" → given="MA/PhD" "John Smith, MA/PhD" → family="John Smith" diff --git a/docs/design/rules/S9-unlisted-words-that-join-the-ambiguous-class-by-shape.md b/docs/design/rules/S9-unlisted-words-that-join-the-ambiguous-class-by-shape.md index 26b4c11e..84136b28 100644 --- a/docs/design/rules/S9-unlisted-words-that-join-the-ambiguous-class-by-shape.md +++ b/docs/design/rules/S9-unlisted-words-that-join-the-ambiguous-class-by-shape.md @@ -4,7 +4,15 @@ **Then:** an unlisted word joins the ambiguous class (S4) by SHAPE, in the positions each shape is admitted: the dotted shape by default, the caps shape by default in one position and elsewhere where the caller asks for it. -- **The two shapes.** Two or more period-separated chunks is one such shape, admitted by default (S3); an unlisted all-caps alphabetic word of two or more letters, standing in a suffix position of a mixed-case name and belonging to no wordlist, is the other. +- **The two shapes.** Two or more period-separated chunks is one such shape, admitted by default (S3); an unlisted all-caps alphabetic word of two or more letters, standing in a suffix position of a mixed-case name and belonging to no wordlist, is the other, and an unlisted word of capitals whose letter chunks are joined by a hyphen or a slash, one of them of two or more letters, is that shape too, admitted where it is. + ``` + "John Smith, PU-PH" → suffix="PU-PH" + "John Smith, MS-OMPT" → suffix="MS-OMPT" + "John Smith, PU/PH, Jr." → suffix="PU/PH, Jr." + "John Smith, A-B" → given="A-B" · boundary + "John Smith, Pu-Ph" → given="Pu-Ph" · boundary + "Jean Pierre DUPONT-MORETTI" unlisted_caps_suffixes-everywhere → suffix="DUPONT-MORETTI" + ``` - **Where the caps shape is admitted by default.** By default that second shape is admitted in one position only, the part after a comma with two or more name words before it (C4, C6): French and Korean records write the SURNAME in capitals, and that convention puts the capitals at the end of a name or before a comma, never after a comma behind a full name. ``` "John Smith XYZ" → family="XYZ" · boundary @@ -27,4 +35,4 @@ --- **Signal:** shape: an unlisted word written in capitals or in dotted chunks, standing in a suffix position -**history:** decisions/S9 · **implemented:** nameparser/_pipeline/_assign.py, nameparser/_pipeline/_classify.py +**history:** decisions/S9 · **implemented:** nameparser/_pipeline/_assign.py, nameparser/_pipeline/_classify.py, nameparser/_pipeline/_vocab.py diff --git a/docs/release_log.rst b/docs/release_log.rst index 1b2ccd94..d3f16614 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -6,7 +6,7 @@ Release Log **Behavior Changes** - - **Read a credential joined by a hyphen or a slash as a suffix.** ``John Smith, MD/PhD`` gives first ``John``, last ``Smith``, suffix ``MD/PhD``, where 1.4.0 through 2.6.0 gave first ``MD/PhD``, last ``John Smith``; ``MD-PhD`` and ``PhD/MD`` read the same. Every part has to be a listed credential that is not also a name, so ``Smith-Jones`` is untouched and ``John Smith, MA/PhD`` still reverses. :issue:`752` + - **Read a credential joined by a hyphen or a slash as a suffix.** ``John Smith, MD/PhD`` gives first ``John``, last ``Smith``, suffix ``MD/PhD``, where 1.4.0 through 2.6.0 gave first ``MD/PhD``, last ``John Smith``; ``MD-PhD`` and ``PhD/MD`` read the same. An unlisted one written in capitals after a comma reads as a credential too where the name before the comma is in mixed case: ``John Smith, PU-PH`` gives suffix ``PU-PH``, where 1.4.0 through 2.6.0 gave first ``PU-PH``. Otherwise every part has to be a listed credential that is not also a name, so ``Smith-Jones`` is untouched and ``John Smith, MA/PhD`` still reverses. :issue:`752`, :issue:`797` * 2.6.0 - October 25, 2026 diff --git a/nameparser/_pipeline/_classify.py b/nameparser/_pipeline/_classify.py index de1d1f13..f86b98dc 100644 --- a/nameparser/_pipeline/_classify.py +++ b/nameparser/_pipeline/_classify.py @@ -281,7 +281,9 @@ def _tags_for(token: WorkToken, n: str, state: ParseState, # C: a listed member ('John Smith, MA'), and a word that # is not alphabetic capitals ('John Smith, Ph. D.') and n not in lex.suffix_acronyms_ambiguous - and (token.text.isalpha() and token.text.isupper() + and (token.text.isupper() + and (token.text.isalpha() or "-" in token.text + or "/" in token.text) or not unicodedata.is_normalized("NFC", token.text)) and caps_shape_candidate(token.text, lex, state.policy, one_case)): diff --git a/nameparser/_pipeline/_comma.py b/nameparser/_pipeline/_comma.py index 681a9eaa..43f12640 100644 --- a/nameparser/_pipeline/_comma.py +++ b/nameparser/_pipeline/_comma.py @@ -353,8 +353,12 @@ def decide(state: ParseState) -> ParseState: maybe = [p[0] for p in p1 if len(p) == 1 and (AMBIGUOUS_ACRONYM_TAG in (tags := tokens[p[0]].tags) or ("vocab:suffix" not in tags - and (texts[p[0]].isalpha() - and texts[p[0]].isupper() + and (texts[p[0]].isupper() + and (texts[p[0]].isalpha() + # the joined spelling ('PU-PH'), the + # predicate's other half (rules/S9) + or "-" in texts[p[0]] + or "/" in texts[p[0]]) or not unicodedata.is_normalized( "NFC", texts[p[0]]) and is_caps_word(texts[p[0]]))))] @@ -518,7 +522,9 @@ def decide(state: ParseState) -> ParseState: # caps shape writes them, so case repair keeps 'AD' # (rules/R4); the C-level filter `maybe` uses first if ("vocab:suffix" not in tokens[i].tags - and (text.isalpha() and text.isupper() + and (text.isupper() + and (text.isalpha() or "-" in text + or "/" in text) or not unicodedata.is_normalized( "NFC", text) and is_caps_word(text))): caps.add(i) diff --git a/nameparser/_pipeline/_vocab.py b/nameparser/_pipeline/_vocab.py index b943299b..2725c963 100644 --- a/nameparser/_pipeline/_vocab.py +++ b/nameparser/_pipeline/_vocab.py @@ -845,9 +845,11 @@ class at all). def caps_shape_candidate(text: str, lexicon: Lexicon, policy: Policy, one_case: bool | None) -> bool: """Whether TEXT is an UNLISTED all-caps credential candidate: two - or more alphabetic characters, no period (a period fails - `isalpha()` outright, which is what excludes the dotted spelling - here for free -- no separate '.' test needed), written in a name + or more alphabetic characters, or letter chunks joined by '-' or + '/' (`_joined_capitals`, #752), no period (a period fails + `isalpha()` and the joined pattern outright, which is what excludes + the dotted spelling here for free -- no separate '.' test needed), + written in a name `one_case` says is mixed (`one_case is False`; `None`, "not established", declines exactly as if the name were one case). @@ -886,11 +888,33 @@ def caps_shape_candidate(text: str, lexicon: Lexicon, policy: Policy, return False if not text.isascii(): text = unicodedata.normalize("NFC", text) - if not (len(text) >= 2 and text.isalpha() and text.isupper()): + if not (len(text) >= 2 and text.isalpha() and text.isupper() + or _joined_capitals(text)): return False return not in_any_wordlist(_normalize(text), lexicon) +#: A word of letter chunks joined by '-' or '/' ('PU-PH', 'MD/MBA'). +_JOINED_LETTERS = re.compile( + r'(?:' + _LETTER + r')+(?:[-/](?:' + _LETTER + r')+)+') + + +# rules/S9: "an unlisted word of capitals whose letter chunks are joined +# by a hyphen or a slash, one of them of two or more letters, is that +# shape too" -- the caps shape's joined spelling (decisions/S9 (#752)) +def _joined_capitals(text: str) -> bool: + """Whether TEXT is capitals in letter chunks joined by '-' or '/', + one chunk of two or more letters ('PU-PH', 'MS-OMPT', not 'A-B', + 'MD/' or 'PU-PH2'). Its callers test `isupper()` and a joiner in C + first, so a word holding neither pays no frame.""" + if not (text.isupper() and _JOINED_LETTERS.fullmatch(text)): + return False + for chunk in _JOINERS.split(text): + if composed_len(chunk) >= 2: + return True + return False + + def is_caps_word(text: str) -> bool: """Whether TEXT is alphabetic capitals once NFC-composed: `isalpha()` is False for a combining mark, so the decomposed spelling of diff --git a/tests/v2/cases.py b/tests/v2/cases.py index 0cd59908..481b052e 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -3541,6 +3541,39 @@ def _check_cjk_shape_purity(self) -> None: {"given": "MA/PhD", "family": "John Smith"}, notes="rules/S3's Accepted cost: a bare member of the ambiguous " "class is no chunk the stricter test counts"), + # #752 (phase 2d): an unlisted word of capitals joined by a hyphen + # or a slash is the caps shape (rules/S9), gated by the name's case + # contrast after a comma (rules/C6) as the unjoined word is. 1.4.0 + # through 2.6.0 read each moved row's word as the given name, the + # name reversed around it (wheels, 2026-10-25). + Case("joined_capitals_after_a_comma_join_the_class", + "John Smith, PU-PH", + {"given": "John", "family": "Smith", "suffix": "PU-PH"}, + ambiguities=("suffix-or-name",), + classification="fix(#752)", + notes="the French hospital-university rank; 1.4.0 through " + "2.6.0 read given 'PU-PH', family 'John Smith'"), + Case("slash_joined_capitals_before_a_third_part", + "John Smith, PU/PH, Jr.", + {"given": "John", "family": "Smith", "suffix": "PU/PH, Jr."}, + ambiguities=("suffix-or-name",), + classification="fix(#752)", + notes="1.4.0 through 2.6.0 read given 'PU/PH', suffix 'Jr.'"), + Case("joined_capitals_in_a_one_case_name_stay_the_given_name", + "JOHN SMITH, PU-PH", + {"given": "PU-PH", "family": "JOHN SMITH"}, + notes="the boundary: no case contrast, so the capitals say " + "nothing (rules/C6)"), + Case("joined_capitals_behind_one_name_word_stay_the_given_name", + "Smith, PU-PH", + {"given": "PU-PH", "family": "Smith"}, + notes="the boundary: one name word before the comma leaves " + "the part the given name (rules/C4)"), + Case("joined_single_letters_are_no_caps_shape", + "John Smith, A-B", + {"given": "A-B", "family": "John Smith"}, + notes="the boundary: a chunk of two or more letters is needed, " + "single letters joined being how initials are written"), Case("paired_initials_in_a_caseless_script_keep_the_position_reading", "สมชาย ใจดี พ.บ.", {"given": "สมชาย", "family": "ใจดี", "suffix": "พ.บ."}, diff --git a/tests/v2/pipeline/test_vocab.py b/tests/v2/pipeline/test_vocab.py index 3014db8a..999ce0ed 100644 --- a/tests/v2/pipeline/test_vocab.py +++ b/tests/v2/pipeline/test_vocab.py @@ -420,6 +420,41 @@ def test_suffix_as_written_reads_a_joined_credential( assert suffix_as_written(_normalize(text), text, lex) is expected +@pytest.mark.parametrize("text, expected", [ + ("PU-PH", True), + ("PU/PH", True), + ("MS-OMPT", True), + ("MD/PU-PH", True), # either joiner, more than two chunks + ("A-BC", True), # one chunk of two letters is enough + ("A-B", False), # single letters alone are how initials join + ("Pu-Ph", False), # capitals, not Title case + ("PU-PH2", False), # a digit is no letter chunk + ("PU-", False), # an empty chunk is no chunk + ("PU--PH", False), + ("P.U.-PH", False), # a period is the dotted shape's, not this + ("XYZ", True), # the unjoined word, as before +]) +def test_caps_shape_candidate_reads_joined_capitals( + text: str, expected: bool) -> None: + # the caps shape's joined spelling (#752): an unlisted word of + # capitals in letter chunks joined by a hyphen or a slash + lex = Lexicon.default() + pol = Policy() + assert caps_shape_candidate(text, lex, pol, one_case=False) is expected + # gated as the unjoined word is: a name in one case says nothing + assert caps_shape_candidate(text, lex, pol, one_case=True) is False + + +def test_caps_shape_candidate_refuses_a_listed_joined_word() -> None: + # unlisted means the whole word in no wordlist, as for the unjoined + # word + lex = Lexicon.default().add(surnames={"dupont-moretti"}) + assert not caps_shape_candidate("DUPONT-MORETTI", lex, Policy(), + one_case=False) + assert caps_shape_candidate("DUPONT-MORETTI", Lexicon.default(), + Policy(), one_case=False) + + def test_ambiguous_class_candidate_admits_a_by_shape_member() -> None: # #516: the comma form's own candidate test reaches a by-shape # member too, once Policy admits it -- case-free either way, the diff --git a/tests/v2/rules_budget.json b/tests/v2/rules_budget.json index 81b13010..f9cf280a 100644 --- a/tests/v2/rules_budget.json +++ b/tests/v2/rules_budget.json @@ -54,13 +54,13 @@ "R5": {"words": 250, "then_words": 154, "because_sentences": 2, "except": 0, "precedence": 0, "accepted": 0, "accepted_words": 0}, "S1": {"words": 274, "then_words": 26, "because_sentences": 1, "except": 2, "precedence": 2, "accepted": 0, "accepted_words": 0}, "S2": {"words": 641, "then_words": 103, "because_sentences": 1, "except": 0, "precedence": 7, "accepted": 3, "accepted_words": 214}, - "S3": {"words": 714, "then_words": 366, "because_sentences": 3, "except": 3, "precedence": 3, "accepted": 4, "accepted_words": 236}, + "S3": {"words": 775, "then_words": 366, "because_sentences": 3, "except": 3, "precedence": 4, "accepted": 4, "accepted_words": 241}, "S4": {"words": 298, "then_words": 156, "because_sentences": 2, "except": 0, "precedence": 3, "accepted": 0, "accepted_words": 0}, "S5": {"words": 632, "then_words": 317, "because_sentences": 1, "except": 2, "precedence": 1, "accepted": 0, "accepted_words": 0}, "S6": {"words": 337, "then_words": 121, "because_sentences": 2, "except": 0, "precedence": 2, "accepted": 0, "accepted_words": 0}, "S7": {"words": 570, "then_words": 388, "because_sentences": 1, "except": 1, "precedence": 2, "accepted": 1, "accepted_words": 147}, "S8": {"words": 354, "then_words": 240, "because_sentences": 1, "except": 1, "precedence": 1, "accepted": 0, "accepted_words": 0}, - "S9": {"words": 430, "then_words": 229, "because_sentences": 2, "except": 0, "precedence": 3, "accepted": 0, "accepted_words": 0}, + "S9": {"words": 463, "then_words": 262, "because_sentences": 2, "except": 0, "precedence": 3, "accepted": 0, "accepted_words": 0}, "S10": {"words": 401, "then_words": 140, "because_sentences": 3, "except": 4, "precedence": 2, "accepted": 2, "accepted_words": 73}, "T1": {"words": 93, "then_words": 18, "because_sentences": 1, "except": 1, "precedence": 0, "accepted": 3, "accepted_words": 83}, "T2": {"words": 43, "then_words": 14, "because_sentences": 1, "except": 0, "precedence": 0, "accepted": 0, "accepted_words": 0}, diff --git a/tests/v2/test_cases.py b/tests/v2/test_cases.py index 14d584ae..d37716ce 100644 --- a/tests/v2/test_cases.py +++ b/tests/v2/test_cases.py @@ -234,9 +234,11 @@ def test_the_field_sweep_sees_the_claims_it_checks() -> None: # the same day: #857's sa rows, after the word left every set (the # one reporting row, 'Smith, S.A.', claims under the orders it # reaches); 1406 -> 1415 the same day: the review's comma rows for - # sa and the listed `ma` twins, three of them reporting. - assert claims == 1415, ( - f"the field sweep checks {claims} claims, recorded as 1415 on " + # sa and the listed `ma` twins, three of them reporting; 1415 -> + # 1421 the same day: #752's two joined-capitals rows that report, + # one suffix-or-name claim under each order. + assert claims == 1421, ( + f"the field sweep checks {claims} claims, recorded as 1421 on " f"2026-10-25") diff --git a/tests/v2/test_name_usage_tool.py b/tests/v2/test_name_usage_tool.py index 4d46f936..372a808d 100644 --- a/tests/v2/test_name_usage_tool.py +++ b/tests/v2/test_name_usage_tool.py @@ -1985,14 +1985,16 @@ def test_main_exits_2_on_an_unusable_tally_without_losing_the_other_sources( def test_main_rank_lists_credentials_with_vocabulary_and_readings( monkeypatch: pytest.MonkeyPatch, tmp_path: Path, capsys: pytest.CaptureFixture[str]) -> None: - _cache_tally(monkeypatch, tmp_path, _tally(RA=50, **{"QRS-C": 40}, ESQ=3)) + # Q-R-S still reverses after a comma: joined single letters are no + # caps shape (rules/S9), so the given-name readout stays exercised + _cache_tally(monkeypatch, tmp_path, _tally(RA=50, **{"Q-R-S": 40}, ESQ=3)) assert tool.main(["--no-wikidata", "--json", "--rank", "5"]) == 0 rows = {r["credential"]: r for r in json.loads(capsys.readouterr().out)["rows"]} assert [r["rank"] for r in rows.values()] == [1, 2, 3] assert rows["RA"]["vocabulary"] == "acronym, title" assert (rows["RA"]["after_comma"], rows["RA"]["bare"]) == ("suffix", "suffix") - assert rows["QRS-C"]["vocabulary"] == "-" - assert (rows["QRS-C"]["after_comma"], rows["QRS-C"]["bare"]) == ("given", "family") + assert rows["Q-R-S"]["vocabulary"] == "-" + assert (rows["Q-R-S"]["after_comma"], rows["Q-R-S"]["bare"]) == ("given", "family") assert rows["ESQ"]["vocabulary"] == "suffix word" assert rows["RA"]["census"] == "not downloaded" @@ -2152,12 +2154,12 @@ def test_main_rank_prints_a_table( capsys: pytest.CaptureFixture[str]) -> None: # Every other rank test reads --json, so a bad format spec in the # table would have crashed the default output unseen. - _cache_tally(monkeypatch, tmp_path, _tally(MD=5000, **{"QRS-C": 40})) + _cache_tally(monkeypatch, tmp_path, _tally(MD=5000, **{"Q-R-S": 40})) assert tool.main(["--no-wikidata", "--rank", "2"]) == 0 lines = capsys.readouterr().out.splitlines() assert lines[0].startswith("5,040 of 5,040 individual providers") assert lines[1].split()[:3] == ["rank", "credential", "providers"] - assert lines[3].split()[:4] == ["2", "QRS-C", "40", "0.79%"] + assert lines[3].split()[:4] == ["2", "Q-R-S", "40", "0.79%"] assert lines[3].split()[4:7] == ["-", "given", "family"] diff --git a/tests/v2/test_regex_sync.py b/tests/v2/test_regex_sync.py index cfb9443a..413a4a97 100644 --- a/tests/v2/test_regex_sync.py +++ b/tests/v2/test_regex_sync.py @@ -141,6 +141,10 @@ def test_dotted_initial_is_the_period_alternative_of_initial() -> None: # the joiners of a joined credential ('MD/PhD', rules/S3, #752); no # config key to mirror ("_vocab", "_JOINERS"): None, + # the caps shape's joined spelling ('PU-PH', rules/S9, #752), built + # from _LETTER, whose _MARKS the test above holds to config's; no + # config key to mirror + ("_vocab", "_JOINED_LETTERS"): None, # #563: paired initials ('M.J.'), C5's exception to C4's name-word # count after a comma; no config key to mirror ("_vocab", "_PAIRED_INITIALS"): None, diff --git a/tools/differential/corpus_rules.jsonl b/tools/differential/corpus_rules.jsonl index 97dd21c8..3b70be9d 100644 --- a/tools/differential/corpus_rules.jsonl +++ b/tools/differential/corpus_rules.jsonl @@ -208,6 +208,7 @@ "JOHN SMITH NEE X X" "JOHN SMITH PH.D." "JOHN SMITH, FNP-BC" +"JOHN SMITH, PU-PH" "JOHN SMITH, VD MA" "JOHN V" "JOSE ORTEGA-Y-GASSET" @@ -280,6 +281,7 @@ "Jean 'JD' Smith" "Jean DUPONT" "Jean Pierre DUPONT" +"Jean Pierre DUPONT-MORETTI" "Jiří Dienstbier ml." "John . Smith" "John Doctor Smith" @@ -351,6 +353,7 @@ "John Smith XYZ" "John Smith Xyz." "John Smith vi" +"John Smith, A-B" "John Smith, A.B." "John Smith, A.B. Ph.D." "John Smith, AD" @@ -367,6 +370,7 @@ "John Smith, M.D./Ph.D." "John Smith, M.D.; Ph.D." "John Smith, MA" +"John Smith, MA/PHD" "John Smith, MA/PhD" "John Smith, MA; PhD" "John Smith, MD ; PhD" @@ -379,10 +383,13 @@ "John Smith, MD/PhD" "John Smith, MD; PhD" "John Smith, MD;PhD" +"John Smith, MS-OMPT" "John Smith, Ma" "John Smith, Mr." "John Smith, Mr. Jr." "John Smith, Mr. and Mrs." +"John Smith, PU-PH" +"John Smith, PU/PH, Jr." "John Smith, Ph. D. Jones" "John Smith, Ph. D.; MD" "John Smith, PhD" @@ -392,6 +399,7 @@ "John Smith, PhD X.Y." "John Smith, PhD XYZ" "John Smith, PhD vd DO" +"John Smith, Pu-Ph" "John Smith, RA Jones" "John Smith, V." "John Smith, V; MD" @@ -613,6 +621,7 @@ "Smith, Ms. Jane" "Smith, PSM I" "Smith, PSM I." +"Smith, PU-PH" "Smith, Ph. D. Jr." "Smith, PhD" "Smith, PhD MEng" diff --git a/tools/differential/snapshot.jsonl b/tools/differential/snapshot.jsonl index 9591e4cc..8eae356a 100644 --- a/tools/differential/snapshot.jsonl +++ b/tools/differential/snapshot.jsonl @@ -1340,6 +1340,7 @@ {"name": "JOHN SMITH PH.D.", "given": "JOHN", "family": "SMITH", "suffix": "PH.D.", "initials": "J. S.", "capitalized": "John Smith Ph.D."} {"name": "JOHN SMITH, FNP-BC", "given": "JOHN", "family": "SMITH", "suffix": "FNP-BC", "initials": "J. S.", "capitalized": "John Smith FNP-BC"} {"name": "JOHN SMITH, MA", "given": "JOHN", "family": "SMITH", "suffix": "MA", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. S.", "capitalized": "John Smith MA"} +{"name": "JOHN SMITH, PU-PH", "given": "PU-PH", "family": "JOHN SMITH", "initials": "P. J. S.", "capitalized": "Pu-Ph John Smith"} {"name": "JOHN SMITH, VD MA", "given": "JOHN", "family": "SMITH", "suffix": "VD MA", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. S.", "capitalized": "John Smith VD MA"} {"name": "JOHN V", "given": "JOHN", "family": "V", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. V.", "capitalized": "John V"} {"name": "JOSE E MARIA SANTOS", "given": "JOSE", "middle": "E MARIA", "family": "SANTOS", "ambiguities": ["CONJUNCTION_OR_INITIAL"], "initials": "J. E. M. S.", "capitalized": "Jose E Maria Santos"} @@ -1481,6 +1482,7 @@ {"name": "Jean Joseph Antoine Marie de Witte", "given": "Jean", "middle": "Joseph Antoine Marie", "family": "de Witte", "initials": "J. J. A. M. W."} {"name": "Jean Louis Charles d'Orléans-Longueville", "given": "Jean", "middle": "Louis Charles", "family": "d'Orléans-Longueville", "initials": "J. L. C. d.", "capitalized_force": "Jean Louis Charles D'Orléans-Longueville"} {"name": "Jean Pierre DUPONT", "given": "Jean", "middle": "Pierre", "family": "DUPONT", "initials": "J. P. D.", "capitalized_force": "Jean Pierre Dupont"} +{"name": "Jean Pierre DUPONT-MORETTI", "given": "Jean", "middle": "Pierre", "family": "DUPONT-MORETTI", "initials": "J. P. D.", "capitalized_force": "Jean Pierre Dupont-Moretti"} {"name": "Jean St Clair", "given": "Jean", "family": "St Clair", "initials": "J. C.", "capitalized_force": "Jean st Clair"} {"name": "Jean de Beaumanoir de Lavardin", "given": "Jean", "middle": "de Beaumanoir", "family": "de Lavardin", "initials": "J. B. L."} {"name": "Jean de Mesnil", "given": "Jean", "family": "de Mesnil", "initials": "J. M."} @@ -1667,6 +1669,7 @@ {"name": "John Smith vi", "given": "John", "family": "Smith", "suffix": "vi", "initials": "J. S.", "capitalized_force": "John Smith VI"} {"name": "John Smith 田.中.", "given": "John", "middle": "Smith", "family": "田.中.", "initials": "J. S. 田."} {"name": "John Smith, 1.4", "given": "1.4", "family": "John Smith", "initials": "1. J. S."} +{"name": "John Smith, A-B", "given": "A-B", "family": "John Smith", "initials": "A. J. S."} {"name": "John Smith, A.B.", "given": "A.B.", "family": "John Smith", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "A. J. S."} {"name": "John Smith, A.B. Ph.D.", "given": "A.B.", "family": "John Smith", "suffix": "Ph.D.", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "A. J. S."} {"name": "John Smith, AD", "given": "AD", "family": "John Smith", "initials": "A. J. S.", "capitalized_force": "Ad John Smith"} @@ -1688,6 +1691,7 @@ {"name": "John Smith, M.D./Ph.D.", "given": "John", "family": "Smith", "suffix": "M.D./Ph.D.", "initials": "J. S."} {"name": "John Smith, M.D.; Ph.D.", "given": "John", "family": "Smith", "suffix": "M.D.; Ph.D.", "initials": "J. S."} {"name": "John Smith, MA", "given": "John", "family": "Smith", "suffix": "MA", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. S."} +{"name": "John Smith, MA/PHD", "given": "John", "family": "Smith", "suffix": "MA/PHD", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. S.", "capitalized_force": "John Smith MA/PhD"} {"name": "John Smith, MA/PhD", "given": "MA/PhD", "family": "John Smith", "initials": "M. J. S.", "capitalized_force": "Ma/PhD John Smith"} {"name": "John Smith, MA; PhD", "given": "John", "family": "Smith", "suffix": "MA; PhD", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. S."} {"name": "John Smith, MD ; PhD", "given": "John", "family": "Smith", "suffix": "MD, PhD", "initials": "J. S."} @@ -1702,11 +1706,14 @@ {"name": "John Smith, MD; PhD", "given": "John", "family": "Smith", "suffix": "MD; PhD", "initials": "J. S."} {"name": "John Smith, MD;PhD", "given": "John", "family": "Smith", "suffix": "MD;PhD", "initials": "J. S."} {"name": "John Smith, MEng PhD", "given": "John", "family": "Smith", "suffix": "MEng PhD", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. S.", "capitalized_force": "John Smith MENG PhD"} +{"name": "John Smith, MS-OMPT", "given": "John", "family": "Smith", "suffix": "MS-OMPT", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. S."} {"name": "John Smith, Ma", "given": "John", "family": "Smith", "suffix": "Ma", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. S.", "capitalized_force": "John Smith MA"} {"name": "John Smith, Mr.", "title": "Mr.", "given": "John", "family": "Smith", "initials": "J. S."} {"name": "John Smith, Mr. Jr.", "title": "Mr.", "given": "John", "family": "Smith", "suffix": "Jr.", "initials": "J. S."} {"name": "John Smith, Mr. and Mrs.", "title": "Mr. and Mrs.", "given": "John", "family": "Smith", "initials": "J. S."} {"name": "John Smith, Ms Ma", "given": "John", "family": "Smith", "suffix": "Ms Ma", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. S.", "capitalized_force": "John Smith MS MA"} +{"name": "John Smith, PU-PH", "given": "John", "family": "Smith", "suffix": "PU-PH", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. S."} +{"name": "John Smith, PU/PH, Jr.", "given": "John", "family": "Smith", "suffix": "PU/PH, Jr.", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. S."} {"name": "John Smith, Ph. D.", "given": "John", "family": "Smith", "suffix": "Ph. D.", "initials": "J. S."} {"name": "John Smith, Ph. D. Jones", "given": "John", "family": "Smith", "suffix": "Ph. D. Jones", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. S."} {"name": "John Smith, Ph. D.; MD", "title": "Ph.", "given": "D.;", "family": "John Smith", "suffix": "MD", "initials": "D. J. S."} @@ -1719,6 +1726,7 @@ {"name": "John Smith, PhD X.Y.", "given": "John", "family": "Smith", "suffix": "PhD X.Y.", "initials": "J. S."} {"name": "John Smith, PhD XYZ", "given": "John", "family": "Smith", "suffix": "PhD XYZ", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. S."} {"name": "John Smith, PhD vd DO", "given": "John", "family": "Smith", "suffix": "PhD vd DO", "initials": "J. S.", "capitalized_force": "John Smith PhD VD DO"} +{"name": "John Smith, Pu-Ph", "given": "Pu-Ph", "family": "John Smith", "initials": "P. J. S."} {"name": "John Smith, RA Jones", "title": "RA", "given": "Jones", "family": "John Smith", "initials": "J. J. S.", "capitalized_force": "Ra Jones John Smith"} {"name": "John Smith, RAI", "given": "John", "family": "Smith", "suffix": "RAI", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. S."} {"name": "John Smith, V Jr.", "given": "John", "family": "Smith", "suffix": "V Jr.", "initials": "J. S."} @@ -3042,6 +3050,7 @@ {"name": "Smith, Ms. Jane", "title": "Ms.", "given": "Jane", "family": "Smith", "initials": "J. S."} {"name": "Smith, PSM I", "family": "Smith", "suffix": "PSM I", "initials": "S."} {"name": "Smith, PSM I.", "family": "Smith", "suffix": "PSM I.", "initials": "S."} +{"name": "Smith, PU-PH", "given": "PU-PH", "family": "Smith", "initials": "P. S.", "capitalized_force": "Pu-Ph Smith"} {"name": "Smith, Ph. D.", "family": "Smith", "suffix": "Ph. D.", "initials": "S."} {"name": "Smith, Ph. D. III", "family": "Smith", "suffix": "Ph. D. III", "initials": "S."} {"name": "Smith, Ph. D. Jr.", "family": "Smith", "suffix": "Ph. D. Jr.", "initials": "S."} From eee2242c85a08216d006de4974c2741c8d5cdab6 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sun, 25 Oct 2026 03:40:10 -0700 Subject: [PATCH 3/7] Keep the single-character exception for joined credentials, record the Korean cost (#752) Review of phases 2c and 2d. A joined word whose every chunk is a single ASCII character ('Jack I-V') read as a suffix with no family; rules/S3's exception binds it now, as it binds 'Jack X.Y.I.'. S3's Then states the joined reading, and its bullet names what joins nothing (other dashes, 'MD/Ph. D.'). Accepted: a Korean given name hyphenated between two listed credentials ('Kim, Ae-Ra') reads as the credential, as the spaced spelling already did on the base (decisions/S3). The release bullet names C4's two name words. ctgov-officials still moves 1,862 entries over both phases, fec 1. Co-Authored-By: Claude Opus 5.5 --- .../S3-credentials-written-in-joined-chunks.md | 3 ++- .../rules/S3-credentials-written-in-joined-chunks.md | 12 ++++++++++-- docs/release_log.rst | 2 +- nameparser/_pipeline/_vocab.py | 12 +++++++++--- tests/v2/pipeline/test_vocab.py | 2 ++ tests/v2/rules_budget.json | 2 +- tools/differential/corpus_rules.jsonl | 5 +++++ tools/differential/snapshot.jsonl | 5 +++++ 8 files changed, 35 insertions(+), 8 deletions(-) diff --git a/docs/design/decisions/S3-credentials-written-in-joined-chunks.md b/docs/design/decisions/S3-credentials-written-in-joined-chunks.md index 10026214..0ee31877 100644 --- a/docs/design/decisions/S3-credentials-written-in-joined-chunks.md +++ b/docs/design/decisions/S3-credentials-written-in-joined-chunks.md @@ -6,7 +6,7 @@ - A digit-dotted token (`1.4.2`) loses the chunk claim and is no member of the shape class either, so it reads as a name word, silently (`John Smith 1.4.2` → family '1.4.2') — 2026-09-18. - An ambiguous acronym written with its periods, one after each letter or one after each of two or more letter chunks, counts as unambiguous, and a single trailing period does not (`Jack M.A.` → suffix, `Jack Ma.` → family); the clause stood in rules/S2 until it moved here — 2026-10-19. - Paired initials of a script with capitals (two single letters each followed by a period, the second optional) are no word of the ambiguous class at the trailing slots and read as the name word they are written as, reported, as the switch off reads them: `Ram Bahadur B.K.` → middle 'Bahadur', family 'B.K.'; `Bahadur, Ram B.K.` → middle 'B.K.'; `Ram Bahadur B.K., PhD` → family 'B.K.', suffix 'PhD'. At the first slot after a comma they stay C5's (`John Smith, X.Y. P.Q.` → suffix), behind an unambiguous suffix word they join the run (`Ram Bahadur PhD B.K.` → suffix 'PhD B.K.') while behind an ambiguous or by-shape credential with no unambiguous word in front of it they end the name and take it in (`Jane Doe MA R.T.` → middle 'Doe MA', family 'R.T.'), and three dotted letters, longer chunks and a caseless script's pair keep the position reading (`John Smith X.Y.Z.`, `John Smith B.Tech.`, `สมชาย ใจดี พ.บ.` → suffix) — 2026-10-24 #864. -- A word joined by a hyphen or a slash reads as a suffix where every chunk counts as a suffix as written, listed and not a bare member of the ambiguous class, a chunk's own periods counting as in a word of its own: `John Smith, MD/PhD` → suffix 'MD/PhD', `John Smith MD-PhD` → suffix 'MD-PhD', `John Smith, M.D./Ph.D.` → suffix; one chunk that does not count leaves the word as it was (`John Smith MD-Jones` → family, `John Smith, MA/PhD` → given 'MA/PhD', family 'John Smith'), and a joiner standing alone or glued to one side makes no joined word — 2026-10-25 #752. +- A word joined by a hyphen or a slash reads as a suffix where every chunk counts as a suffix as written, listed and not a bare member of the ambiguous class, a chunk's own periods counting as in a word of its own: `John Smith, MD/PhD` → suffix 'MD/PhD', `John Smith MD-PhD` → suffix 'MD-PhD', `John Smith, M.D./Ph.D.` → suffix; one chunk that does not count leaves the word as it was (`John Smith MD-Jones` → family, `John Smith, MA/PhD` → given 'MA/PhD', family 'John Smith'), a word of single-ASCII-character chunks is none (`Jack I-V` → family 'I-V'), and a joiner standing alone or glued to one side, a dash other than the hyphen-minus and a joined split credential make no joined word — 2026-10-25 #752. Open: none in this file. @@ -18,3 +18,4 @@ Open: none in this file. - 2026-10-24 #864 — PAIRED INITIALS ARE NO CREDENTIAL AT THE TRAILING SLOTS. SUPERSEDES, in part, decisions/S9's 2026-09-30 #563 entry: its choice (iii) kept #516's reading at the trailing slots and left the no-comma question open, and this answers it for paired initials. Since 2.4.0 the position reading took a surname written as two dotted initials wherever the name had a word to spare: Nepali surnames are written that way (B.K., K.C., B.C., D.C.), and `Ram Bahadur B.K.` read family 'Bahadur', suffix 'B.K.' where 2.0.0 through 2.3.0 read family 'B.K.' (wheels, measured 2026-10-24). Two dotted single letters is how initials are written, and C5 already reads the shape as a name at the first slot after a comma. THE BOUNDARY is the pair, not every run of single letters: measured with the wider variant (any number of single-letter chunks), 11 committed corpus names move, every one a decided credential example of this rule or S9 (`John Smith X.Y.Z.`, `Q.W.E.R.T.`, `J.u.n.i.o.r.`, `R.A.I.`, `Jane Doe nee Smith X.Y.Z.`) or CLDR's Russian `к.м.н.`, for 3 more Nepal names and 3 more readings in the local ClinicalTrials.gov and FEC corpora together. THE SCRIPT has capitals: CLDR's Thai renderings write degrees as dotted letters (`พ.บ.`, the MD, and `ท.บ.`), a caseless script having no capitals to write initials in, and the unscoped pair moved both into the name, one into the family and one into the middle. THE SEAM: classify writes the shape tag without the class tag, the state the switch off leaves every dotted word in, so every trailing slot (the end of the name, the word ending the given part after a family comma, a maiden clause's end, the part before a suffix comma) reads the pair as a name word and reports the fork through the switch-off branch that already existed; `_comma.decide` restores the class tag in the part it reads, where C5 decides. Narrowing the shape verdict itself, in `period_joined_vocab`, was measured first and declined: the comma's credential run reads the class too, so `John Smith, X.Y. P.Q.` lost its suffix (given 'X.Y.', middle 'P.Q.') and `John Smith, PhD X.Y.` gained a report; holding back only classify's tag, without the restore, cost the same. MOVED IN THE COMMITTED CORPORA: two names, each to 1.4.0's reading, `García Márquez, Ed G.J.` (middle 'G.J.', was suffix) and `De La Cruz, M.J. K.L.` (middle 'K.L.', was suffix); CLDR agreement unmoved. LOCAL CORPORA (counts only), each recorded first from origin/master 638685b4 with `--record --tree`: nepal-candidates, 1,095 of its 237,408 entries move (the romanized names of 1,221 of its 126,981 name pairs), every one to the dotted pair (or a GC, below) as the family name, and 31 of the corpus's ASCII names ending in a two-letter dotted word still do not read it as the family, 20 of them `D.C.`, 8 `G.M.` and 2 `M.C.`, all listed credentials, and one after a comma, C5's; fec-individuals, 293 move, 289 of them a pair ending the given part after a family comma moving from suffix to middle (in one, two pairs, both now middle) and 4 a lone `GC` after a comma moving to the given name; ctgov-officials, 9 of 352,104, 7 a dotted pair ending a name, mostly before a suffix comma, moving from suffix to family, none of them a listed credential, and 2 a `GC`; psc-individuals (local_agreement.py), 4 records progress and 1 regresses, a filed surname of a word and a dotted pair now read with the pair alone as the family; ukec-individuals, none. THE ACCEPTED COST is an unlisted two-letter credential written with its periods after a full name, which reads as the family name, reported: King's Counsel is the case (`John Smith K.C.`), unlisted, and written KC by 4 of 15,214 UK donors (`tools/name_usage.py kc`, the ukec-individuals row); 1.4.0 read the same. A SECOND ACCEPTED COST, found by the design-docs review: only an unambiguous suffix word in front makes the pair the credential run's (`Ram Bahadur PhD B.K.`, `Jane Doe Ph.D. R.T.` → suffix); behind a credential that is itself ambiguous, listed (`MA`, `MEng`) or read by position (`X.Y.Z.`, `XYZ` under EVERYWHERE), the pair ends the name and the credential is taken into it, `Jane Doe MA R.T.` reading middle 'Doe MA', family 'R.T.' where 2.4.0 through 2.5.0 read suffix 'MA R.T.' and 1.4.0 through 2.3.0 the same as now; 1 local entry takes this shape, in nepal-candidates (counted as paired initials of a cased script standing right after a word tagged `vocab:suffix-ambiguous` that is not also a suffix, the review's count; none in the other seven local corpora). An unambiguous word in front of the ambiguous one starts the run, which then takes both (`Jane Doe PhD MA R.T.` → suffix 'PhD MA R.T.', rules/S8). Recorded as a cost rather than changed (Derek, 2026-10-24). `D.C.`, `G.M.` and `M.C.` are listed and so untouched: `dc` stays on 99,172 NPPES providers against 21 Nepal name pairs, and `gm` and `mc` were not weighed here. `gc` left the suffix acronyms in the same change (decisions/suffix-acronym-collisions' 2026-10-24 #864 entry). Pinned by the #864 block of `tests/v2/cases.py` and the `B.K` and `พ.บ.` rows of `test_ambiguous_class_candidate_agrees_with_the_tag`. - 2026-10-19 — CLAUSE MOVED IN: an ambiguous acronym written with its periods, one after each letter or one after each of two or more letter chunks, counts unambiguously, and a single trailing period does not. It stood in rules/S2 until now and is a shape, the periods, which is this rule's signal, so it moved here, one rule per piece of evidence (docs/design/AGENTS.md). It was decided in #296's review (decisions/C8's PR #428 review entry) and widened to letter chunks by #544 (decisions/suffix-acronym-collisions' #544 entry), and no entry under decisions/S2 recorded it. Where earlier entries and code call it S2's period gate, it is this rule's now. - 2026-10-25 #752 — JOINED CHUNKS: A WORD JOINED BY A HYPHEN OR A SLASH IS A CREDENTIAL WHERE EVERY CHUNK IS ONE (phase 2c, item 5 of decisions/vocabulary-model's 2026-10-23 #752 entry, agreed by Derek on 2026-10-23). The chunk reading this rule gives interior periods is extended to the two other joiners a credential is written with, by a stricter test: the periods' ANY becomes EVERY, and a chunk must count as a suffix as written, which a bare member of the ambiguous class (`MA`, `Ed`) does not, because a hyphen also joins names (`Smith-Jones`, `Mary-Jane`) and a slash sits between a name and a credential in typed data. The test is asked where a word's suffix reading as written is asked, so classify's tag, the comma's decision and every predicate testing a word's text read the joined word alike. A chunk is read as a word of its own, so a chunk written with its periods counts (`M.D./Ph.D.`, and `M.A./Ph.D.`, the ambiguous member written so). Not read: a joiner standing alone or glued to one side (`MD / PhD` already read as a run of suffix words; `MD/ PhD` keeps `MD/` as a name word, 10 ClinicalTrials.gov entries, and `MD /PhD` 1), the en dash and other dash characters (no entry joins two listed credentials with one), and `&` (63 entries join two listed credentials with it, left for its own change). MEASURED (2026-10-25, against 2af8da8e; every local corpus recorded from that base with `--tree`): `tools/differential/snapshot.py` moves nothing but the rule's own new examples; `cldr_agreement.py` matches every record; `local_agreement.py psc-individuals` matches its record; tse-donors, 0 of 848,351 entries; fec-individuals, 1 of 1,893,964, a doubled joined credential leaving the middle name for the suffix; ctgov-officials, 1,333 of 352,104, the population being the 1,592 words in it joined by a hyphen or a slash whose every chunk counts (the rest already read as suffixes, in a part past a suffix comma). Of the 1,333, 1,319 are a joined credential after a comma that had been read as the given name, the name reversed around it, now the suffix; 2 a joined credential ending a name without a comma, now the suffix where it was the family; and 12 move a report alone, 11 dropping a comma-structure report (a third part now read wholly as credentials) and 1 a suffix-or-name report. Of the 1,321 whose fields move, 3 also take a neighbouring word into the suffix with a new report, each as the unjoined twin reads: a word after the credential (`John Smith, MD/PhD Candidate` reads as `John Smith, PhD Candidate` does) and, once, a capitals credential ending the part before the comma (rules/C12). The joined words are 82 distinct credentials, every one a degree or certification combination, the commonest `MD-PhD` (494), `MD/PhD` (473), `PhD/MD` (65) and `MS-PT` (53); 68 entries hold a chunk written with its periods, which the 2026-10-23 measurement's letters-only test did not read (its 1,254). COST: a word holding a joiner pays three profiler frames at each site that asks its suffix reading as written, stopping at its first chunk that does not count, and a word holding none pays nothing, the joiners being tested in C first (`John Smith` 150 frames before and after, `Mary Smith-Jones` 156 → 159, `Smith-Jones, Mary-Jane` 169 → 175, py3.11). Pinned by the #752 rows of tests/v2/cases.py and `test_suffix_as_written_reads_a_joined_credential` (tests/v2/pipeline/test_vocab.py), whose refused rows pass on the pre-change tree and whose admitted rows fail on it. +- 2026-10-25 #752, review of phase 2c — THREE BOUNDARIES THE FIRST VERSION MISSED, found by the PR's code and design-docs reviews. (i) The single-character exception binds a joined word too: `i` and `v` are suffix words, so `Jack I-V` read suffix 'I-V' with no family where 1.4.0 through 2.6.0 read family 'I-V' and the period twin `Jack I.V.` still does; a joined word now needs a chunk that is not a single ASCII character (`John Smith, MS-V` keeps suffix 'MS-V'), and the Except bullet carries the joined row. No local corpus entry moved back: the measured 1,333 hold no such word. (ii) ACCEPTED: a Korean given name romanized with a hyphen between two syllables the vocabulary lists as unambiguous credentials, `ae` and `ra` (`Kim, Ae-Ra`, `Shin Ae-ra`), reads as the credential, unreported after a comma. The spaced spelling already did on the base (`Kim, Ae Ra` → suffix 'Ae Ra', `Shin Ae Ra` → suffix 'Ae Ra'), so this is the vocabulary collision decisions/vocabulary-collisions weighs, which the hyphen had been hiding, not a new reading of the joiner; giving `ae` and `ra` a `given` reading, which would make both ambiguous by derivation (decisions/vocabulary-model's 2026-10-25 #752 entry), moves the spaced spelling too and is left for its own change. The reviewer's scan of every romanized Korean syllable found `ae`, `am`, `ra` and `i` counting as suffixes; `i` alone is no joined credential since (i), and no committed or local corpus entry holds a name-cased hyphenated pair of them. (iii) A split credential joined to another (`MD/Ph. D.`) and the dash characters other than the hyphen-minus join nothing, now stated in the rule. diff --git a/docs/design/rules/S3-credentials-written-in-joined-chunks.md b/docs/design/rules/S3-credentials-written-in-joined-chunks.md index 8bc9e391..03321a7a 100644 --- a/docs/design/rules/S3-credentials-written-in-joined-chunks.md +++ b/docs/design/rules/S3-credentials-written-in-joined-chunks.md @@ -2,7 +2,7 @@ **When:** a word written in chunks, joined by interior periods or by a hyphen or a slash, stands in a name, as in "John Smith Msc.Ed.". -**Then:** a word with interior periods reads as a suffix when any of its period-separated chunks is suffix vocabulary, and a word of two or more period-separated chunks that no vocabulary claims is read by POSITION instead, as a bare ambiguous acronym is (S4). +**Then:** a word with interior periods reads as a suffix when any of its period-separated chunks is suffix vocabulary, a word joined by a hyphen or a slash when every one of its chunks is, and a word of two or more period-separated chunks that no vocabulary claims is read by POSITION instead, as a bare ambiguous acronym is (S4). - **An ambiguous acronym written with its periods**, one after each letter or one after each of two or more letter chunks, counts unambiguously; a single trailing period is the abbreviation shape any word can wear and does not. Written so, it is not the bare acronym S4 counts words to spare for: the periods are how a credential is written and a name is not. ``` @@ -22,7 +22,7 @@ "john smith x.y.z." → suffix="x.y.z." "John Smith 1.4" → family="1.4" · boundary ``` -- **A word joined by a hyphen or a slash** is read by a stricter test than the periods' any: a word whose chunks are joined by a hyphen or a slash reads as a suffix when every chunk counts as a suffix as written, which is to say listed and not a bare member of the ambiguous class (S4), a chunk's own periods counting as they do in a word of its own. One chunk that does not count leaves the word as it would be without this bullet, and a joiner standing alone or glued to one side joins nothing. +- **A word joined by a hyphen or a slash** is read by a stricter test than the periods' any: a word whose chunks are joined by a hyphen or a slash reads as a suffix when every chunk counts as a suffix as written, which is to say listed and not a bare member of the ambiguous class (S4), a chunk's own periods counting as they do in a word of its own. One chunk that does not count leaves the word as it would be without this bullet, and a joiner standing alone or glued to one side joins nothing, as do the dash characters other than the hyphen-minus and a split credential's halves (`MD/Ph. D.`). ``` "John Smith, MD/PhD" → suffix="MD/PhD" "John Smith MD-PhD" → suffix="MD-PhD" @@ -53,6 +53,8 @@ - **Every chunk the vocabulary matches is a single ASCII character**, the roman numerals and the lone digit the vocabulary lists, which are about generations rather than credentials: those chunks do not make the word a suffix. ``` "Jack X.Y.I." → family="X.Y.I." · boundary + "Jack I-V" → family="I-V" · boundary + "John Smith, MS-V" → suffix="MS-V" ``` - **Paired initials are the exception**, as C5 defines them, in a script that writes capitals: where S4 would count words to spare for them they are the name word they are written as, as the switch off reads every such word, and the fork is still reported. Two dotted single letters is how initials are written in such a script, and how some surnames are (Nepali B.K., K.C.), while a script without capitals writes its abbreviations that way, degrees among them; behind an unambiguous suffix word the pair joins the credential run that word starts (S8). ``` @@ -86,6 +88,12 @@ "John Smith, MA/PhD" → given="MA/PhD" "John Smith, MA/PhD" → family="John Smith" ``` +- A Korean given name romanized with a hyphen between two syllables the vocabulary lists as unambiguous credentials reads as the credential, unreported after a comma, as its spaced spelling already did: `ae` and `ra` are such credentials, and the hyphen no longer keeps the name a name word. + ``` + "Kim, Ae-Ra" → suffix="Ae-Ra" + "Kim, Ae Ra" → suffix="Ae Ra" + "Shin Ae-ra" → suffix="Ae-ra" + ``` - An unlisted credential of two letters written with its periods behind a full name reads as the family name, and the reading is reported: King's Counsel, which no vocabulary lists, is the case. ``` "John Smith K.C." → family="K.C." diff --git a/docs/release_log.rst b/docs/release_log.rst index d3f16614..e594301b 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -6,7 +6,7 @@ Release Log **Behavior Changes** - - **Read a credential joined by a hyphen or a slash as a suffix.** ``John Smith, MD/PhD`` gives first ``John``, last ``Smith``, suffix ``MD/PhD``, where 1.4.0 through 2.6.0 gave first ``MD/PhD``, last ``John Smith``; ``MD-PhD`` and ``PhD/MD`` read the same. An unlisted one written in capitals after a comma reads as a credential too where the name before the comma is in mixed case: ``John Smith, PU-PH`` gives suffix ``PU-PH``, where 1.4.0 through 2.6.0 gave first ``PU-PH``. Otherwise every part has to be a listed credential that is not also a name, so ``Smith-Jones`` is untouched and ``John Smith, MA/PhD`` still reverses. :issue:`752`, :issue:`797` + - **Read a credential joined by a hyphen or a slash as a suffix.** ``John Smith, MD/PhD`` gives first ``John``, last ``Smith``, suffix ``MD/PhD``, where 1.4.0 through 2.6.0 gave first ``MD/PhD``, last ``John Smith``; ``MD-PhD`` and ``PhD/MD`` read the same. An unlisted one written in capitals reads as a credential too after a comma behind two or more name words written in mixed case: ``John Smith, PU-PH`` gives suffix ``PU-PH``, where 1.4.0 through 2.6.0 gave first ``PU-PH``. Otherwise every part has to be a listed credential that is not also a name, so ``Smith-Jones`` is untouched and ``John Smith, MA/PhD`` still reverses; a hyphenated Korean given name whose syllables are both listed credentials reads as one, as its spaced spelling already did (``Kim, Ae-Ra`` gives suffix ``Ae-Ra``). :issue:`752`, :issue:`797` * 2.6.0 - October 25, 2026 diff --git a/nameparser/_pipeline/_vocab.py b/nameparser/_pipeline/_vocab.py index 2725c963..c5aaeace 100644 --- a/nameparser/_pipeline/_vocab.py +++ b/nameparser/_pipeline/_vocab.py @@ -580,18 +580,24 @@ def suffix_as_written(n: str, text: str, lexicon: Lexicon) -> bool: def _joined_credential(text: str, lexicon: Lexicon) -> bool: """Whether TEXT is two or more chunks joined by '-' or '/', each non-empty and each counting as a suffix as written: listed, and - not a bare member of the ambiguous class ('MD/PhD', 'M.A.-Ph.D.', - but not 'MA/PhD', 'Smith-Jones' or 'MD/'). A chunk holds no joiner, + not a bare member of the ambiguous class, and not every one a + single ASCII character ('MD/PhD', 'M.A.-Ph.D.', but not 'MA/PhD', + 'Smith-Jones', 'I-V' or 'MD/'). A chunk holds no joiner, so the recursion is one level deep. A loop rather than `all()` over a generator, which would cost a frame per chunk: a hyphenated name ('Smith-Jones') stops at its first chunk.""" chunks = _JOINERS.split(text) if len(chunks) < 2: return False + whole = False # a chunk that is not a single ASCII character for c in chunks: if not (c and suffix_as_written(_normalize(c), c, lexicon)): return False - return True + whole = whole or len(c) > 1 or not c.isascii() + # rules/S3: "Every chunk the vocabulary matches is a single ASCII + # character" -- the numerals and the digit, which are about + # generations, make no joined word a suffix either ('Jack I-V') + return whole def _is_suffix_strict_n(n: str, text: str, lexicon: Lexicon, diff --git a/tests/v2/pipeline/test_vocab.py b/tests/v2/pipeline/test_vocab.py index 999ce0ed..ae70c50f 100644 --- a/tests/v2/pipeline/test_vocab.py +++ b/tests/v2/pipeline/test_vocab.py @@ -411,6 +411,8 @@ def test_period_joined_vocab_retires_the_single_character_chunk() -> None: ("/MD", False), ("MD//PhD", False), ("MD+PhD", False), # '+' is no joiner + ("I-V", False), # every chunk a single ASCII character + ("MS-V", True), # one chunk is not ]) def test_suffix_as_written_reads_a_joined_credential( text: str, expected: bool) -> None: diff --git a/tests/v2/rules_budget.json b/tests/v2/rules_budget.json index f9cf280a..f0939243 100644 --- a/tests/v2/rules_budget.json +++ b/tests/v2/rules_budget.json @@ -54,7 +54,7 @@ "R5": {"words": 250, "then_words": 154, "because_sentences": 2, "except": 0, "precedence": 0, "accepted": 0, "accepted_words": 0}, "S1": {"words": 274, "then_words": 26, "because_sentences": 1, "except": 2, "precedence": 2, "accepted": 0, "accepted_words": 0}, "S2": {"words": 641, "then_words": 103, "because_sentences": 1, "except": 0, "precedence": 7, "accepted": 3, "accepted_words": 214}, - "S3": {"words": 775, "then_words": 366, "because_sentences": 3, "except": 3, "precedence": 4, "accepted": 4, "accepted_words": 241}, + "S3": {"words": 807, "then_words": 398, "because_sentences": 3, "except": 3, "precedence": 4, "accepted": 5, "accepted_words": 289}, "S4": {"words": 298, "then_words": 156, "because_sentences": 2, "except": 0, "precedence": 3, "accepted": 0, "accepted_words": 0}, "S5": {"words": 632, "then_words": 317, "because_sentences": 1, "except": 2, "precedence": 1, "accepted": 0, "accepted_words": 0}, "S6": {"words": 337, "then_words": 121, "because_sentences": 2, "except": 0, "precedence": 2, "accepted": 0, "accepted_words": 0}, diff --git a/tools/differential/corpus_rules.jsonl b/tools/differential/corpus_rules.jsonl index 3b70be9d..c4c65ad6 100644 --- a/tools/differential/corpus_rules.jsonl +++ b/tools/differential/corpus_rules.jsonl @@ -215,6 +215,7 @@ "JOSÉ GARCÍA" "JUAN GARCIA Jr." "JUAN GARCIA Y LOPEZ" +"Jack I-V" "Jack MA" "Jack Ma" "Jack Ma." @@ -384,6 +385,7 @@ "John Smith, MD; PhD" "John Smith, MD;PhD" "John Smith, MS-OMPT" +"John Smith, MS-V" "John Smith, Ma" "John Smith, Mr." "John Smith, Mr. Jr." @@ -467,6 +469,8 @@ "Juan y Garcia née Jones" "Juan Ó Pérez" "Juan Ó. Pérez" +"Kim, Ae Ra" +"Kim, Ae-Ra" "Kim김민준씨, MA" "King Smith, John" "LT.GOV. JOHN DOE" @@ -552,6 +556,7 @@ "Sheik abdul Jr Smith" "Sheik abdul salam" "Sheik abdul salam Jr" +"Shin Ae-ra" "Shirley Maclaine" "Sidorov Ivan Petrovich Jr." "Sir John" diff --git a/tools/differential/snapshot.jsonl b/tools/differential/snapshot.jsonl index 8eae356a..b2ea2cbe 100644 --- a/tools/differential/snapshot.jsonl +++ b/tools/differential/snapshot.jsonl @@ -1357,6 +1357,7 @@ {"name": "Jacinto López Moreno", "given": "Jacinto", "middle": "López", "family": "Moreno", "initials": "J. L. M."} {"name": "Jack", "given": "Jack", "ambiguities": ["GIVEN_OR_FAMILY"], "initials": "J."} {"name": "Jack", "order": "FAMILY_FIRST", "family": "Jack", "ambiguities": ["GIVEN_OR_FAMILY"], "initials": "J."} +{"name": "Jack I-V", "given": "Jack", "family": "I-V", "initials": "J. I."} {"name": "Jack M.A.", "given": "Jack", "suffix": "M.A.", "ambiguities": ["GIVEN_OR_FAMILY"], "initials": "J."} {"name": "Jack MA", "given": "Jack", "suffix": "MA", "ambiguities": ["GIVEN_OR_FAMILY", "SUFFIX_OR_NAME"], "initials": "J."} {"name": "Jack MA.", "given": "Jack", "suffix": "MA.", "ambiguities": ["GIVEN_OR_FAMILY", "SUFFIX_OR_NAME"], "initials": "J."} @@ -1707,6 +1708,7 @@ {"name": "John Smith, MD;PhD", "given": "John", "family": "Smith", "suffix": "MD;PhD", "initials": "J. S."} {"name": "John Smith, MEng PhD", "given": "John", "family": "Smith", "suffix": "MEng PhD", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. S.", "capitalized_force": "John Smith MENG PhD"} {"name": "John Smith, MS-OMPT", "given": "John", "family": "Smith", "suffix": "MS-OMPT", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. S."} +{"name": "John Smith, MS-V", "given": "John", "family": "Smith", "suffix": "MS-V", "initials": "J. S."} {"name": "John Smith, Ma", "given": "John", "family": "Smith", "suffix": "Ma", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. S.", "capitalized_force": "John Smith MA"} {"name": "John Smith, Mr.", "title": "Mr.", "given": "John", "family": "Smith", "initials": "J. S."} {"name": "John Smith, Mr. Jr.", "title": "Mr.", "given": "John", "family": "Smith", "suffix": "Jr.", "initials": "J. S."} @@ -2020,6 +2022,8 @@ {"name": "Khun Sa", "given": "Khun", "family": "Sa", "initials": "K. S."} {"name": "Ki, Tomonori", "given": "Tomonori", "family": "Ki", "initials": "T. K."} {"name": "Kim Min Ho", "order": "FAMILY_FIRST", "given": "Min", "middle": "Ho", "family": "Kim", "initials": "M. H. K."} +{"name": "Kim, Ae Ra", "family": "Kim", "suffix": "Ae Ra", "initials": "K.", "capitalized_force": "Kim AE RA"} +{"name": "Kim, Ae-Ra", "family": "Kim", "suffix": "Ae-Ra", "initials": "K.", "capitalized_force": "Kim AE-RA"} {"name": "Kim, Jr.", "family": "Kim", "suffix": "Jr.", "initials": "K."} {"name": "Kim김민준씨, MA", "family": "Kim김민준", "suffix": "씨, MA", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "K."} {"name": "King Curtis", "title": "King", "given": "Curtis", "initials": "C."} @@ -2898,6 +2902,7 @@ {"name": "Sheik abdul salam", "title": "Sheik", "given": "abdul salam", "initials": "a. s.", "capitalized_force": "Sheik Abdul Salam"} {"name": "Sheik abdul salam Jr", "title": "Sheik", "given": "abdul salam", "suffix": "Jr", "initials": "a. s.", "capitalized_force": "Sheik Abdul Salam Jr"} {"name": "Shemuʾel ben Ḥofni", "given": "Shemuʾel", "middle": "ben", "family": "Ḥofni", "initials": "S. b. Ḥ.", "capitalized_force": "Shemuʾel Ben Ḥofni"} +{"name": "Shin Ae-ra", "given": "Shin", "suffix": "Ae-ra", "ambiguities": ["GIVEN_OR_FAMILY"], "initials": "S.", "capitalized_force": "Shin AE-RA"} {"name": "Shirley Maclaine", "given": "Shirley", "family": "Maclaine", "initials": "S. M.", "capitalized_force": "Shirley MacLaine"} {"name": "Shree Lalit Kumaar Bharati Jr, MP", "given": "Shree", "middle": "Lalit Kumaar", "family": "Bharati", "suffix": "Jr, MP", "initials": "S. L. K. B."} {"name": "Shtober Zaziliya Hemish", "order": "FAMILY_FIRST", "given": "Zaziliya", "middle": "Hemish", "family": "Shtober", "initials": "Z. H. S."} From 4a3b19c511995585e8351239c1225473f2d123ba Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sun, 25 Oct 2026 03:46:17 -0700 Subject: [PATCH 4/7] Set a chunk's periods aside in the single-character test (#752) The second review of the fix: 'Jack I.-V.' still read suffix with no family, the length test counting the chunk's period where a word split at its periods never does. AGENTS.md's spelling sweep records the miss. Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 2 +- .../decisions/S3-credentials-written-in-joined-chunks.md | 2 +- docs/design/rules/S3-credentials-written-in-joined-chunks.md | 4 +++- nameparser/_pipeline/_vocab.py | 5 ++++- tests/v2/pipeline/test_vocab.py | 2 ++ tests/v2/rules_budget.json | 2 +- tools/differential/corpus_rules.jsonl | 2 ++ tools/differential/snapshot.jsonl | 2 ++ 8 files changed, 16 insertions(+), 5 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index e7f3829b..011d13c6 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -186,7 +186,7 @@ The library has two layers: `nameparser/config/` (data) and `nameparser/parser.p **Sweep the forms a change can reach: comma shapes, then `name_order`.** No comma, a FULL name before the comma, a ONE-WORD name before the comma and NOTHING before the comma are four paths, and the last two are the misses — #429, #430 and #432 are all the one-word path disagreeing with the full-name path on inputs the full-name path reads correctly, and #613's PR review found `, PhD` and `, Jr.` (an empty surname field, as CSV data writes it) read as given and title by a decision that returned early on an empty head, after two review rounds whose every grid had a name before the comma. Put an empty head and a nickname-only head (`(Bob), PhD`) in every comma grid. Count the one-word path the way C3 counts, which is neither tokens nor pieces: a particle run and the one word it attaches to are one word (#575), a connective join and a bound given-name pair count as their words (#613, so `Ortega y Gasset, Dr.` is a whole name), and a suffix beside the surname counts as no name word at all. #603's review found the suffix half (`Vega y Lopez Jr., PhD Jones` lost its family where `Vega y Lopez, PhD Jones` kept it); #613 then reversed the connective half, and both now read given 'Vega', middle 'y', family 'Lopez' — so test a particle surname, a connective surname and a surname with a suffix, each before a comma. For orders the sweep already exists (`tests/v2/test_cases.py` runs every row under all three) but carries one assertion, R2's family partition, so it checks nothing a new change moves; coverage there has been vacuous before (PR #394's review found the suite passed with `name_order` discarded from grouping). Parse your change's names in each comma shape and each order, and read the ones you did not predict. And read a generated grid's moves by SHAPE, not by corpus membership: #614's first design passed the suite, the five gates and a fingerprint of every corpus name, because its grid moves were sorted into 'corpus name moved' and 'garbage', and the review found the moves that mattered among the 'garbage' -- a bare salutation (`Mr. and Mrs.` lost its connective to the family name) and a name with one trailing word (`Freiherr von Berg X.Y.Z.`). Pull out the head-plus-one-word slice of the grid and a salutation with nothing after it, and read every move there. Two more were the second review's (#614): no grid put TWO words that are both a title and a particle at the head (`Freiherr St van Berg MA`, where an index into the pieces was applied to a merged view of them -- `TITLES ∩ particles_ambiguous` is `{freiherr, st}`, so put both in every head list, each in its bare and its period-marked spelling -- #637's grid had `St` and `Freiherr` alone and missed that `St.` and `Freiherr.` are declined by the bound join and then attached by P6), and the grids compared ambiguity KINDS, so a report telling the caller the wrong field (`Freiherr von Berg Ma`, 'read as a given name' of a word in the family) passed every pin. #701's review added one: put a title/suffix dual (`Ms`, `Sr.`, `Cpl`) SECOND in a title run, behind an ordinary title. A stop that asks only "is this word suffix vocabulary" passes every row that opens with the dual and fails `Rev. Sr. Smith, Mary`; it was found by comparing `T Smith, John` against `T John Smith` over every title word, and a comma-versus-comma-less comparison is the cheap check for any rule that moves a word between the two slots. Compare each report's `detail` against the field its tokens land in -- since #626 `test_a_report_names_the_field_its_word_lands_in` (tests/v2/test_cases.py) does it over every non-locale case row under all three orders (a row carrying its own policy only as declared), for the phrasings its `_FIELD_CLAIMS` lists -- so a row with the shape is the test only where the report's wording is listed: a new emitter's wording goes in that list, and segment's report on a part past the second comma names no field (#629). #626's own first grid then missed an emitter for want of one comma shape (`, Ma Dr.`: an empty head, an ambiguous word, a trailing title, which H1 moves to the family): a trailing title belongs in every empty-head comma grid too. And #620's: an exclusion list in code being replaced is a set of rules, not noise -- #614's merged copy excluded every word the trailing read weighs from its units, the first #620 prototype dropped the list as a symptom of the copy, and its review found the list had been keeping positional counting's rule that a member in front counts as a word (`Freiherr von Berg MA X.Y.Z.` lost its suffix). Before deleting one, mutate each entry out and read what moves. The converse holds for replacing a hand copy WITH the shared predicate: it is a behavior change wherever the two disagreed, and #735's /simplify shipped one as "no reading moves" because the suite and the snapshot were its only check (`St Sir de Mesnil` family-first, a token test replaced by P1's piece test); diff the two predicates' answers over a grid before calling the swap a refactor. -**Then sweep the spellings: a class reached by vocabulary is also reached by SHAPE.** A test that uses only the listed spelling of a word class walks only the vocabulary path, and the shape path is a separate branch that can be missing while every test passes. Roman numerals are the sharpest case: `ii`, `iii` and `iv` are suffix vocabulary; `i` and `v` are vocabulary too but initial-shaped, so `is_suffix_piece` vetoes them and only the numeral fork reads them; and `vi` through `x` are in no list at all, read by the fork's shape test alone (checked 2026-10-04 against `Lexicon.default()`) -- which stops at `x`: `_vocab._ROMAN` matches I to X, so `John Smith XI` reads `XI` as a name word (re-checked 2026-10-07, #614's /simplify, which found a docstring claiming "any length"). And the fork counts a ONE-letter numeral besides, since #718 (rules/S10): it is the suffix only where the name has a word to spare, so `John V` and `John Smith V` take different branches of one fork, a one-case name being counted like capitals whatever the letter -- since #721 a one-case one-letter numeral carries the `initial` tag its capital does, and since #723 every one-case bare letter does (rules/T4), and a bare letter in front of the numeral is initial-shaped as its capital is, so `john v` reads as `JOHN V` does over `test_a_one_case_name_reads_as_its_capitals`'s grid (comma shapes, orders, a maiden clause, a particle chain, a bare letter in front, a trailing title, a non-ASCII letter in front: 315 of its then 1,872 pairs disagreed at 2bfa9533, before #721, and none after). Its first version claimed "every shape" with no bare letter in front and no trailing title in the grid, and #721's reviews found both broken; a grid's claim is the grid -- put a head-plus-numeral row beside every words-to-spare row. So a change that handles `V` and `III` can still miss `VI` — #610 found exactly that, the maiden take's candidate loop admitting a shape-only numeral in one position where the peel reads it in two (`Jane Doe nee Smith VI Prof.` kept maiden `Smith VI` while `V` and `III` worked). #656's sweep found the same class from the other side: C12 takes the word ending the name before a comma as EVIDENCE, and its first draft asked the suffix-word tag, so `John Smith VI, AD` missed the rule `III` reached, and then counted the whole part, so `Smith VI, AD` fired with `VI` as its own name word and `Smith V, AD` with a `V` that S10 reads as the family. A rule that weighs a numeral as a generation must ask the predicate that READS it as one (the trailing fork, `_vocab.reads_as_generation`, and its words-to-spare count), not the vocabulary that lists some of its spellings. The same split runs through the credentials (`MA` listed, `X.Y.Z.` by the dotted shape, `XYZ` by the caps shape, `MD/PhD` by its listed chunks and `PU-PH` by the caps shape's joined spelling, #752, each with its joiner glued, spaced and glued to one side, which are three token streams, and `Ph. D.` merged by group into one flagged piece the peel's walk never holds -- #603 found #602's run-start predicate refusing it too, so `Smith, John Ph. D. Jones` kept middle 'Jones' where `Smith, John PhD Jones` read suffix 'PhD Jones') and the titles (`Prof.` period-marked and read at the end of a name, `Prof` bare and a name word there, and `Mr. and Mrs.` a run P3 joins, carrying a piece tag a one-word vocabulary test never sees -- PR #735's review found C12's hand copy of C13's title test (C8's until its 2026-10-19 split) reading `John Smith Jr., Mr. and Mrs. AD` as all suffix; ask C13's predicates in C13's order, `not is_suffix_piece` then `is_title_piece`, not the token tags, since `is_title_piece` alone reads the duals `Ms` and `Sr.` as titles). When a change touches a word class, test one spelling from each path. The same holds for a separator's SPACING: a comma (or any punctuation) can stand alone, glue to the word before it or glue to the word after, and each is a different token stream. #653's first fix swept `(née, Jones)` and missed `(née , Jones)`, where the kept comma became a word after the marker and M3 read the clause as maiden ', Jones'; its review found it. Put all three spacings in any grid around a separator. And a separator glued to a word hides the word from every vocabulary test, not only the one being changed: the semicolon delimiter's first draft read a glued word's stem in C1's run test alone and put 365 of ClinicalTrials.gov's reversed names right, missing the ones C13, C4 and C6 read off a word's TAGS (`MBBS;`, `MA;`); tagging the stem put 412 right, and reading it again in the comma decision's shape tests, which read a word's text rather than its tags (`MBBS;`), 431 (decisions/C10, 2026-10-24). A rule over a glued spelling reads the stem everywhere the word is read, its tags and every test of its text, and is swept against the space twin -- with a declared word delimiter too, which a substring test finds inside other words ('Stand' under ' and ', the review of the same change). A full stop INSIDE a word has four spellings of its own: initials run together (`J.R.R.`), joined words each closed by a stop (`Lt.Gov.`, `Msc.Ed.`), a joined word or name closed by none (`St.John`, `J.Smith`), and an abbreviation's own ending, closed or not (Italian `Sig.ra`, Latvian `Dr.philol.`). Shape tells the last from none of the others, so a rule over the shape decides which it costs; #838 found the ending only through the snapshot's CLDR row `Dut.ra Prof.ra`, and its first narrowing traded `St.John` for `Sig.ra` while the snapshot listed the `Dr.philol.` move among moves read as intended, until the design-docs review named it. Initials run together are also how some SURNAMES are written (Nepali `B.K.`, `K.C.`), and a caseless script writes its abbreviations, degrees included, as dotted letters (Thai `พ.บ.`): #864 found S3's position reading taking the first from 2.4.0 on, visible only once #866's Nepal candidates gave the corpora a population that writes them, and its first fix moved CLDR's Thai degrees into the name. A rule over dotted letters reads both before it ships. +**Then sweep the spellings: a class reached by vocabulary is also reached by SHAPE.** A test that uses only the listed spelling of a word class walks only the vocabulary path, and the shape path is a separate branch that can be missing while every test passes. Roman numerals are the sharpest case: `ii`, `iii` and `iv` are suffix vocabulary; `i` and `v` are vocabulary too but initial-shaped, so `is_suffix_piece` vetoes them and only the numeral fork reads them; and `vi` through `x` are in no list at all, read by the fork's shape test alone (checked 2026-10-04 against `Lexicon.default()`) -- which stops at `x`: `_vocab._ROMAN` matches I to X, so `John Smith XI` reads `XI` as a name word (re-checked 2026-10-07, #614's /simplify, which found a docstring claiming "any length"). And the fork counts a ONE-letter numeral besides, since #718 (rules/S10): it is the suffix only where the name has a word to spare, so `John V` and `John Smith V` take different branches of one fork, a one-case name being counted like capitals whatever the letter -- since #721 a one-case one-letter numeral carries the `initial` tag its capital does, and since #723 every one-case bare letter does (rules/T4), and a bare letter in front of the numeral is initial-shaped as its capital is, so `john v` reads as `JOHN V` does over `test_a_one_case_name_reads_as_its_capitals`'s grid (comma shapes, orders, a maiden clause, a particle chain, a bare letter in front, a trailing title, a non-ASCII letter in front: 315 of its then 1,872 pairs disagreed at 2bfa9533, before #721, and none after). Its first version claimed "every shape" with no bare letter in front and no trailing title in the grid, and #721's reviews found both broken; a grid's claim is the grid -- put a head-plus-numeral row beside every words-to-spare row. So a change that handles `V` and `III` can still miss `VI` — #610 found exactly that, the maiden take's candidate loop admitting a shape-only numeral in one position where the peel reads it in two (`Jane Doe nee Smith VI Prof.` kept maiden `Smith VI` while `V` and `III` worked). #656's sweep found the same class from the other side: C12 takes the word ending the name before a comma as EVIDENCE, and its first draft asked the suffix-word tag, so `John Smith VI, AD` missed the rule `III` reached, and then counted the whole part, so `Smith VI, AD` fired with `VI` as its own name word and `Smith V, AD` with a `V` that S10 reads as the family. A rule that weighs a numeral as a generation must ask the predicate that READS it as one (the trailing fork, `_vocab.reads_as_generation`, and its words-to-spare count), not the vocabulary that lists some of its spellings. The same split runs through the credentials (`MA` listed, `X.Y.Z.` by the dotted shape, `XYZ` by the caps shape, `MD/PhD` by its listed chunks and `PU-PH` by the caps shape's joined spelling, #752, each with its joiner glued, spaced and glued to one side, which are three token streams -- and a new chunk reading inherits the old one's exceptions, which #752's first draft did not (`Jack I-V` lost its family to the listed numerals, and the fix's length test counted a chunk's period, so `Jack I.-V.` did too, found by the second review), and `Ph. D.` merged by group into one flagged piece the peel's walk never holds -- #603 found #602's run-start predicate refusing it too, so `Smith, John Ph. D. Jones` kept middle 'Jones' where `Smith, John PhD Jones` read suffix 'PhD Jones') and the titles (`Prof.` period-marked and read at the end of a name, `Prof` bare and a name word there, and `Mr. and Mrs.` a run P3 joins, carrying a piece tag a one-word vocabulary test never sees -- PR #735's review found C12's hand copy of C13's title test (C8's until its 2026-10-19 split) reading `John Smith Jr., Mr. and Mrs. AD` as all suffix; ask C13's predicates in C13's order, `not is_suffix_piece` then `is_title_piece`, not the token tags, since `is_title_piece` alone reads the duals `Ms` and `Sr.` as titles). When a change touches a word class, test one spelling from each path. The same holds for a separator's SPACING: a comma (or any punctuation) can stand alone, glue to the word before it or glue to the word after, and each is a different token stream. #653's first fix swept `(née, Jones)` and missed `(née , Jones)`, where the kept comma became a word after the marker and M3 read the clause as maiden ', Jones'; its review found it. Put all three spacings in any grid around a separator. And a separator glued to a word hides the word from every vocabulary test, not only the one being changed: the semicolon delimiter's first draft read a glued word's stem in C1's run test alone and put 365 of ClinicalTrials.gov's reversed names right, missing the ones C13, C4 and C6 read off a word's TAGS (`MBBS;`, `MA;`); tagging the stem put 412 right, and reading it again in the comma decision's shape tests, which read a word's text rather than its tags (`MBBS;`), 431 (decisions/C10, 2026-10-24). A rule over a glued spelling reads the stem everywhere the word is read, its tags and every test of its text, and is swept against the space twin -- with a declared word delimiter too, which a substring test finds inside other words ('Stand' under ' and ', the review of the same change). A full stop INSIDE a word has four spellings of its own: initials run together (`J.R.R.`), joined words each closed by a stop (`Lt.Gov.`, `Msc.Ed.`), a joined word or name closed by none (`St.John`, `J.Smith`), and an abbreviation's own ending, closed or not (Italian `Sig.ra`, Latvian `Dr.philol.`). Shape tells the last from none of the others, so a rule over the shape decides which it costs; #838 found the ending only through the snapshot's CLDR row `Dut.ra Prof.ra`, and its first narrowing traded `St.John` for `Sig.ra` while the snapshot listed the `Dr.philol.` move among moves read as intended, until the design-docs review named it. Initials run together are also how some SURNAMES are written (Nepali `B.K.`, `K.C.`), and a caseless script writes its abbreviations, degrees included, as dotted letters (Thai `พ.บ.`): #864 found S3's position reading taking the first from 2.4.0 on, visible only once #866's Nepal candidates gave the corpora a population that writes them, and its first fix moved CLDR's Thai degrees into the name. A rule over dotted letters reads both before it ships. **And the encoding: a letter class written `[^\W\d_]` or tested with `isalpha()` is blind to a combining mark.** Python's `\w` and `isalpha()` do not match Mn, so a letter written as base plus accent (NFD) ends a word early; the vocabulary path is immune because `_normalize` NFC-folds, the shape path is not. #652: `Dr. inž. Janez Novak` read `inž.` as the given name when the ž was decomposed, and `Janez Novak, ŽŠČ` reversed, in two families that share the blind spot — the letter-shape regexes (`period_abbreviation`, `_DOTTED`, `_CHUNKED`, `_PAIRED_INITIALS`), now spelled with `_vocab._LETTER` (a letter plus the general combining blocks; `test_regex_sync` compares the two `_MARKS` strings and pins `period_abbreviation` against config's copy), and the caps predicates, which NFC-compose a non-ASCII word before `isalpha()` (`is_caps_word`, behind C-level `isalpha()`/`is_normalized` tests, so already-composed text pays no frame — measured 2026-10-12, `Smith, José` was 193 frames before and after; the first version called it for every non-ASCII word and cost one frame each, and no guard saw it because the reference name is ASCII). A width test must count letters, not code points (`composed_len`): admitting a decomposed word to the caps set put `John Smith, MÁ` on the pair exception's wrong side, and the review's grid found it where the rows (all three letters long) did not — put a two-letter word in every caps grid. The initial shape (`_INITIAL`, `\w\.`, v1-verbatim and public) is not widened but asked of the COMPOSED word, in `is_initial_shaped` and in `_render`'s two copies; and the single-letter connective tests count composed letters, since a decomposed `й` is two code points (#665, where `É. y É. Smith` and `John й Smith` read differently decomposed). Compose the WHOLE predicate, not one half: `is_initial` is a shape ANDed with the #320 script veto, and composing the shape alone made a decomposed `씨.` an initial, since decomposed hangul is conjoining jamo outside every script range (#665's first version; `in_initialless_script` composes too now). Not covered, deliberately: the marks only some scripts write (Indic vowel signs, Arabic and Hebrew points), declined in decisions/nfd-shapes because widening the class would move readings nobody had measured; rules/H2's Accepted clause records the one consequence it names, that a period-marked abbreviation in Bengali or Devanagari never reads as a title by shape. The guard is `test_a_decomposed_name_renders_as_its_composed_twin` over the CLDR corpus plus `test_a_decomposed_abbreviation_reads_as_its_composed_twin` (rows × three orders × EVERYWHERE); the corpus guard reaches a shape only if some corpus name does, which is how the caps shape was missed until review, so when adding a shape rule add a row there. Four of #652's rows and two of #665's pass with every fix reverted (measured) and are parity pins, and the test compares ambiguity kinds and `initials()` too, since #665's `John й Smith` differed in its report; `_DOTTED` has no observable input and is pinned by a direct call. diff --git a/docs/design/decisions/S3-credentials-written-in-joined-chunks.md b/docs/design/decisions/S3-credentials-written-in-joined-chunks.md index 0ee31877..c625693a 100644 --- a/docs/design/decisions/S3-credentials-written-in-joined-chunks.md +++ b/docs/design/decisions/S3-credentials-written-in-joined-chunks.md @@ -18,4 +18,4 @@ Open: none in this file. - 2026-10-24 #864 — PAIRED INITIALS ARE NO CREDENTIAL AT THE TRAILING SLOTS. SUPERSEDES, in part, decisions/S9's 2026-09-30 #563 entry: its choice (iii) kept #516's reading at the trailing slots and left the no-comma question open, and this answers it for paired initials. Since 2.4.0 the position reading took a surname written as two dotted initials wherever the name had a word to spare: Nepali surnames are written that way (B.K., K.C., B.C., D.C.), and `Ram Bahadur B.K.` read family 'Bahadur', suffix 'B.K.' where 2.0.0 through 2.3.0 read family 'B.K.' (wheels, measured 2026-10-24). Two dotted single letters is how initials are written, and C5 already reads the shape as a name at the first slot after a comma. THE BOUNDARY is the pair, not every run of single letters: measured with the wider variant (any number of single-letter chunks), 11 committed corpus names move, every one a decided credential example of this rule or S9 (`John Smith X.Y.Z.`, `Q.W.E.R.T.`, `J.u.n.i.o.r.`, `R.A.I.`, `Jane Doe nee Smith X.Y.Z.`) or CLDR's Russian `к.м.н.`, for 3 more Nepal names and 3 more readings in the local ClinicalTrials.gov and FEC corpora together. THE SCRIPT has capitals: CLDR's Thai renderings write degrees as dotted letters (`พ.บ.`, the MD, and `ท.บ.`), a caseless script having no capitals to write initials in, and the unscoped pair moved both into the name, one into the family and one into the middle. THE SEAM: classify writes the shape tag without the class tag, the state the switch off leaves every dotted word in, so every trailing slot (the end of the name, the word ending the given part after a family comma, a maiden clause's end, the part before a suffix comma) reads the pair as a name word and reports the fork through the switch-off branch that already existed; `_comma.decide` restores the class tag in the part it reads, where C5 decides. Narrowing the shape verdict itself, in `period_joined_vocab`, was measured first and declined: the comma's credential run reads the class too, so `John Smith, X.Y. P.Q.` lost its suffix (given 'X.Y.', middle 'P.Q.') and `John Smith, PhD X.Y.` gained a report; holding back only classify's tag, without the restore, cost the same. MOVED IN THE COMMITTED CORPORA: two names, each to 1.4.0's reading, `García Márquez, Ed G.J.` (middle 'G.J.', was suffix) and `De La Cruz, M.J. K.L.` (middle 'K.L.', was suffix); CLDR agreement unmoved. LOCAL CORPORA (counts only), each recorded first from origin/master 638685b4 with `--record --tree`: nepal-candidates, 1,095 of its 237,408 entries move (the romanized names of 1,221 of its 126,981 name pairs), every one to the dotted pair (or a GC, below) as the family name, and 31 of the corpus's ASCII names ending in a two-letter dotted word still do not read it as the family, 20 of them `D.C.`, 8 `G.M.` and 2 `M.C.`, all listed credentials, and one after a comma, C5's; fec-individuals, 293 move, 289 of them a pair ending the given part after a family comma moving from suffix to middle (in one, two pairs, both now middle) and 4 a lone `GC` after a comma moving to the given name; ctgov-officials, 9 of 352,104, 7 a dotted pair ending a name, mostly before a suffix comma, moving from suffix to family, none of them a listed credential, and 2 a `GC`; psc-individuals (local_agreement.py), 4 records progress and 1 regresses, a filed surname of a word and a dotted pair now read with the pair alone as the family; ukec-individuals, none. THE ACCEPTED COST is an unlisted two-letter credential written with its periods after a full name, which reads as the family name, reported: King's Counsel is the case (`John Smith K.C.`), unlisted, and written KC by 4 of 15,214 UK donors (`tools/name_usage.py kc`, the ukec-individuals row); 1.4.0 read the same. A SECOND ACCEPTED COST, found by the design-docs review: only an unambiguous suffix word in front makes the pair the credential run's (`Ram Bahadur PhD B.K.`, `Jane Doe Ph.D. R.T.` → suffix); behind a credential that is itself ambiguous, listed (`MA`, `MEng`) or read by position (`X.Y.Z.`, `XYZ` under EVERYWHERE), the pair ends the name and the credential is taken into it, `Jane Doe MA R.T.` reading middle 'Doe MA', family 'R.T.' where 2.4.0 through 2.5.0 read suffix 'MA R.T.' and 1.4.0 through 2.3.0 the same as now; 1 local entry takes this shape, in nepal-candidates (counted as paired initials of a cased script standing right after a word tagged `vocab:suffix-ambiguous` that is not also a suffix, the review's count; none in the other seven local corpora). An unambiguous word in front of the ambiguous one starts the run, which then takes both (`Jane Doe PhD MA R.T.` → suffix 'PhD MA R.T.', rules/S8). Recorded as a cost rather than changed (Derek, 2026-10-24). `D.C.`, `G.M.` and `M.C.` are listed and so untouched: `dc` stays on 99,172 NPPES providers against 21 Nepal name pairs, and `gm` and `mc` were not weighed here. `gc` left the suffix acronyms in the same change (decisions/suffix-acronym-collisions' 2026-10-24 #864 entry). Pinned by the #864 block of `tests/v2/cases.py` and the `B.K` and `พ.บ.` rows of `test_ambiguous_class_candidate_agrees_with_the_tag`. - 2026-10-19 — CLAUSE MOVED IN: an ambiguous acronym written with its periods, one after each letter or one after each of two or more letter chunks, counts unambiguously, and a single trailing period does not. It stood in rules/S2 until now and is a shape, the periods, which is this rule's signal, so it moved here, one rule per piece of evidence (docs/design/AGENTS.md). It was decided in #296's review (decisions/C8's PR #428 review entry) and widened to letter chunks by #544 (decisions/suffix-acronym-collisions' #544 entry), and no entry under decisions/S2 recorded it. Where earlier entries and code call it S2's period gate, it is this rule's now. - 2026-10-25 #752 — JOINED CHUNKS: A WORD JOINED BY A HYPHEN OR A SLASH IS A CREDENTIAL WHERE EVERY CHUNK IS ONE (phase 2c, item 5 of decisions/vocabulary-model's 2026-10-23 #752 entry, agreed by Derek on 2026-10-23). The chunk reading this rule gives interior periods is extended to the two other joiners a credential is written with, by a stricter test: the periods' ANY becomes EVERY, and a chunk must count as a suffix as written, which a bare member of the ambiguous class (`MA`, `Ed`) does not, because a hyphen also joins names (`Smith-Jones`, `Mary-Jane`) and a slash sits between a name and a credential in typed data. The test is asked where a word's suffix reading as written is asked, so classify's tag, the comma's decision and every predicate testing a word's text read the joined word alike. A chunk is read as a word of its own, so a chunk written with its periods counts (`M.D./Ph.D.`, and `M.A./Ph.D.`, the ambiguous member written so). Not read: a joiner standing alone or glued to one side (`MD / PhD` already read as a run of suffix words; `MD/ PhD` keeps `MD/` as a name word, 10 ClinicalTrials.gov entries, and `MD /PhD` 1), the en dash and other dash characters (no entry joins two listed credentials with one), and `&` (63 entries join two listed credentials with it, left for its own change). MEASURED (2026-10-25, against 2af8da8e; every local corpus recorded from that base with `--tree`): `tools/differential/snapshot.py` moves nothing but the rule's own new examples; `cldr_agreement.py` matches every record; `local_agreement.py psc-individuals` matches its record; tse-donors, 0 of 848,351 entries; fec-individuals, 1 of 1,893,964, a doubled joined credential leaving the middle name for the suffix; ctgov-officials, 1,333 of 352,104, the population being the 1,592 words in it joined by a hyphen or a slash whose every chunk counts (the rest already read as suffixes, in a part past a suffix comma). Of the 1,333, 1,319 are a joined credential after a comma that had been read as the given name, the name reversed around it, now the suffix; 2 a joined credential ending a name without a comma, now the suffix where it was the family; and 12 move a report alone, 11 dropping a comma-structure report (a third part now read wholly as credentials) and 1 a suffix-or-name report. Of the 1,321 whose fields move, 3 also take a neighbouring word into the suffix with a new report, each as the unjoined twin reads: a word after the credential (`John Smith, MD/PhD Candidate` reads as `John Smith, PhD Candidate` does) and, once, a capitals credential ending the part before the comma (rules/C12). The joined words are 82 distinct credentials, every one a degree or certification combination, the commonest `MD-PhD` (494), `MD/PhD` (473), `PhD/MD` (65) and `MS-PT` (53); 68 entries hold a chunk written with its periods, which the 2026-10-23 measurement's letters-only test did not read (its 1,254). COST: a word holding a joiner pays three profiler frames at each site that asks its suffix reading as written, stopping at its first chunk that does not count, and a word holding none pays nothing, the joiners being tested in C first (`John Smith` 150 frames before and after, `Mary Smith-Jones` 156 → 159, `Smith-Jones, Mary-Jane` 169 → 175, py3.11). Pinned by the #752 rows of tests/v2/cases.py and `test_suffix_as_written_reads_a_joined_credential` (tests/v2/pipeline/test_vocab.py), whose refused rows pass on the pre-change tree and whose admitted rows fail on it. -- 2026-10-25 #752, review of phase 2c — THREE BOUNDARIES THE FIRST VERSION MISSED, found by the PR's code and design-docs reviews. (i) The single-character exception binds a joined word too: `i` and `v` are suffix words, so `Jack I-V` read suffix 'I-V' with no family where 1.4.0 through 2.6.0 read family 'I-V' and the period twin `Jack I.V.` still does; a joined word now needs a chunk that is not a single ASCII character (`John Smith, MS-V` keeps suffix 'MS-V'), and the Except bullet carries the joined row. No local corpus entry moved back: the measured 1,333 hold no such word. (ii) ACCEPTED: a Korean given name romanized with a hyphen between two syllables the vocabulary lists as unambiguous credentials, `ae` and `ra` (`Kim, Ae-Ra`, `Shin Ae-ra`), reads as the credential, unreported after a comma. The spaced spelling already did on the base (`Kim, Ae Ra` → suffix 'Ae Ra', `Shin Ae Ra` → suffix 'Ae Ra'), so this is the vocabulary collision decisions/vocabulary-collisions weighs, which the hyphen had been hiding, not a new reading of the joiner; giving `ae` and `ra` a `given` reading, which would make both ambiguous by derivation (decisions/vocabulary-model's 2026-10-25 #752 entry), moves the spaced spelling too and is left for its own change. The reviewer's scan of every romanized Korean syllable found `ae`, `am`, `ra` and `i` counting as suffixes; `i` alone is no joined credential since (i), and no committed or local corpus entry holds a name-cased hyphenated pair of them. (iii) A split credential joined to another (`MD/Ph. D.`) and the dash characters other than the hyphen-minus join nothing, now stated in the rule. +- 2026-10-25 #752, review of phase 2c — THREE BOUNDARIES THE FIRST VERSION MISSED, found by the PR's code and design-docs reviews. (i) The single-character exception binds a joined word too: `i` and `v` are suffix words, so `Jack I-V` read suffix 'I-V' with no family where 1.4.0 through 2.6.0 read family 'I-V' and the period twin `Jack I.V.` still does; a joined word now needs a chunk that is not a single ASCII character once its own periods are set aside, as a period-split word's chunks are (`Jack I.-V.` keeps family 'I.-V.', the second review's find against the first fix, which counted the period; `John Smith, MS-V` keeps suffix 'MS-V'), and the Except bullet carries the joined rows. No local corpus entry moved back: the measured 1,333 hold no such word. (ii) ACCEPTED: a Korean given name romanized with a hyphen between two syllables the vocabulary lists as unambiguous credentials, `ae` and `ra` (`Kim, Ae-Ra`, `Shin Ae-ra`), reads as the credential, unreported after a comma and leaving no family without one (`Shin Ae-ra` → given 'Shin', suffix 'Ae-ra'). The spaced spelling already did on the base (`Kim, Ae Ra` → suffix 'Ae Ra', `Shin Ae Ra` → suffix 'Ae Ra'), so this is the vocabulary collision decisions/vocabulary-collisions weighs, which the hyphen had been hiding, not a new reading of the joiner; giving `ae` and `ra` a `given` reading, which would make both ambiguous by derivation (decisions/vocabulary-model's 2026-10-25 #752 entry), moves the spaced spelling too and is left for its own change. The reviewer's scan of every romanized Korean syllable found `ae`, `am`, `ra` and `i` counting as suffixes; `i` alone is no joined credential since (i), and no committed or local corpus entry holds a name-cased hyphenated pair of them. (iii) A split credential joined to another (`MD/Ph. D.`) and the dash characters other than the hyphen-minus join nothing, now stated in the rule. diff --git a/docs/design/rules/S3-credentials-written-in-joined-chunks.md b/docs/design/rules/S3-credentials-written-in-joined-chunks.md index 03321a7a..d5d47f43 100644 --- a/docs/design/rules/S3-credentials-written-in-joined-chunks.md +++ b/docs/design/rules/S3-credentials-written-in-joined-chunks.md @@ -54,6 +54,7 @@ ``` "Jack X.Y.I." → family="X.Y.I." · boundary "Jack I-V" → family="I-V" · boundary + "Jack I.-V." → family="I.-V." · boundary "John Smith, MS-V" → suffix="MS-V" ``` - **Paired initials are the exception**, as C5 defines them, in a script that writes capitals: where S4 would count words to spare for them they are the name word they are written as, as the switch off reads every such word, and the fork is still reported. Two dotted single letters is how initials are written in such a script, and how some surnames are (Nepali B.K., K.C.), while a script without capitals writes its abbreviations that way, degrees among them; behind an unambiguous suffix word the pair joins the credential run that word starts (S8). @@ -88,11 +89,12 @@ "John Smith, MA/PhD" → given="MA/PhD" "John Smith, MA/PhD" → family="John Smith" ``` -- A Korean given name romanized with a hyphen between two syllables the vocabulary lists as unambiguous credentials reads as the credential, unreported after a comma, as its spaced spelling already did: `ae` and `ra` are such credentials, and the hyphen no longer keeps the name a name word. +- A Korean given name romanized with a hyphen between two syllables the vocabulary lists as unambiguous credentials reads as the credential, unreported after a comma and leaving no family without one, as its spaced spelling already did: `ae` and `ra` are such credentials, and the hyphen no longer keeps the name a name word. ``` "Kim, Ae-Ra" → suffix="Ae-Ra" "Kim, Ae Ra" → suffix="Ae Ra" "Shin Ae-ra" → suffix="Ae-ra" + "Shin Ae Ra" → suffix="Ae Ra" ``` - An unlisted credential of two letters written with its periods behind a full name reads as the family name, and the reading is reported: King's Counsel, which no vocabulary lists, is the case. ``` diff --git a/nameparser/_pipeline/_vocab.py b/nameparser/_pipeline/_vocab.py index c5aaeace..84296937 100644 --- a/nameparser/_pipeline/_vocab.py +++ b/nameparser/_pipeline/_vocab.py @@ -593,7 +593,10 @@ def _joined_credential(text: str, lexicon: Lexicon) -> bool: for c in chunks: if not (c and suffix_as_written(_normalize(c), c, lexicon)): return False - whole = whole or len(c) > 1 or not c.isascii() + # its periods are not the chunk's ('I.' is 'i'), as in a word + # split at its periods + bare = c.replace(".", "") + whole = whole or len(bare) > 1 or not bare.isascii() # rules/S3: "Every chunk the vocabulary matches is a single ASCII # character" -- the numerals and the digit, which are about # generations, make no joined word a suffix either ('Jack I-V') diff --git a/tests/v2/pipeline/test_vocab.py b/tests/v2/pipeline/test_vocab.py index ae70c50f..41937644 100644 --- a/tests/v2/pipeline/test_vocab.py +++ b/tests/v2/pipeline/test_vocab.py @@ -412,6 +412,8 @@ def test_period_joined_vocab_retires_the_single_character_chunk() -> None: ("MD//PhD", False), ("MD+PhD", False), # '+' is no joiner ("I-V", False), # every chunk a single ASCII character + ("I.-V.", False), # its periods are not the chunk's + ("I-V.", False), ("MS-V", True), # one chunk is not ]) def test_suffix_as_written_reads_a_joined_credential( diff --git a/tests/v2/rules_budget.json b/tests/v2/rules_budget.json index f0939243..639ee676 100644 --- a/tests/v2/rules_budget.json +++ b/tests/v2/rules_budget.json @@ -54,7 +54,7 @@ "R5": {"words": 250, "then_words": 154, "because_sentences": 2, "except": 0, "precedence": 0, "accepted": 0, "accepted_words": 0}, "S1": {"words": 274, "then_words": 26, "because_sentences": 1, "except": 2, "precedence": 2, "accepted": 0, "accepted_words": 0}, "S2": {"words": 641, "then_words": 103, "because_sentences": 1, "except": 0, "precedence": 7, "accepted": 3, "accepted_words": 214}, - "S3": {"words": 807, "then_words": 398, "because_sentences": 3, "except": 3, "precedence": 4, "accepted": 5, "accepted_words": 289}, + "S3": {"words": 807, "then_words": 398, "because_sentences": 3, "except": 3, "precedence": 4, "accepted": 5, "accepted_words": 295}, "S4": {"words": 298, "then_words": 156, "because_sentences": 2, "except": 0, "precedence": 3, "accepted": 0, "accepted_words": 0}, "S5": {"words": 632, "then_words": 317, "because_sentences": 1, "except": 2, "precedence": 1, "accepted": 0, "accepted_words": 0}, "S6": {"words": 337, "then_words": 121, "because_sentences": 2, "except": 0, "precedence": 2, "accepted": 0, "accepted_words": 0}, diff --git a/tools/differential/corpus_rules.jsonl b/tools/differential/corpus_rules.jsonl index c4c65ad6..b22b7f12 100644 --- a/tools/differential/corpus_rules.jsonl +++ b/tools/differential/corpus_rules.jsonl @@ -216,6 +216,7 @@ "JUAN GARCIA Jr." "JUAN GARCIA Y LOPEZ" "Jack I-V" +"Jack I.-V." "Jack MA" "Jack Ma" "Jack Ma." @@ -556,6 +557,7 @@ "Sheik abdul Jr Smith" "Sheik abdul salam" "Sheik abdul salam Jr" +"Shin Ae Ra" "Shin Ae-ra" "Shirley Maclaine" "Sidorov Ivan Petrovich Jr." diff --git a/tools/differential/snapshot.jsonl b/tools/differential/snapshot.jsonl index b2ea2cbe..bf9c5d55 100644 --- a/tools/differential/snapshot.jsonl +++ b/tools/differential/snapshot.jsonl @@ -1358,6 +1358,7 @@ {"name": "Jack", "given": "Jack", "ambiguities": ["GIVEN_OR_FAMILY"], "initials": "J."} {"name": "Jack", "order": "FAMILY_FIRST", "family": "Jack", "ambiguities": ["GIVEN_OR_FAMILY"], "initials": "J."} {"name": "Jack I-V", "given": "Jack", "family": "I-V", "initials": "J. I."} +{"name": "Jack I.-V.", "given": "Jack", "family": "I.-V.", "initials": "J. I."} {"name": "Jack M.A.", "given": "Jack", "suffix": "M.A.", "ambiguities": ["GIVEN_OR_FAMILY"], "initials": "J."} {"name": "Jack MA", "given": "Jack", "suffix": "MA", "ambiguities": ["GIVEN_OR_FAMILY", "SUFFIX_OR_NAME"], "initials": "J."} {"name": "Jack MA.", "given": "Jack", "suffix": "MA.", "ambiguities": ["GIVEN_OR_FAMILY", "SUFFIX_OR_NAME"], "initials": "J."} @@ -2902,6 +2903,7 @@ {"name": "Sheik abdul salam", "title": "Sheik", "given": "abdul salam", "initials": "a. s.", "capitalized_force": "Sheik Abdul Salam"} {"name": "Sheik abdul salam Jr", "title": "Sheik", "given": "abdul salam", "suffix": "Jr", "initials": "a. s.", "capitalized_force": "Sheik Abdul Salam Jr"} {"name": "Shemuʾel ben Ḥofni", "given": "Shemuʾel", "middle": "ben", "family": "Ḥofni", "initials": "S. b. Ḥ.", "capitalized_force": "Shemuʾel Ben Ḥofni"} +{"name": "Shin Ae Ra", "given": "Shin", "suffix": "Ae Ra", "ambiguities": ["GIVEN_OR_FAMILY"], "initials": "S.", "capitalized_force": "Shin AE RA"} {"name": "Shin Ae-ra", "given": "Shin", "suffix": "Ae-ra", "ambiguities": ["GIVEN_OR_FAMILY"], "initials": "S.", "capitalized_force": "Shin AE-RA"} {"name": "Shirley Maclaine", "given": "Shirley", "family": "Maclaine", "initials": "S. M.", "capitalized_force": "Shirley MacLaine"} {"name": "Shree Lalit Kumaar Bharati Jr, MP", "given": "Shree", "middle": "Lalit Kumaar", "family": "Bharati", "suffix": "Jr, MP", "initials": "S. L. K. B."} From 8ae97c1a552df44660516b0ede947f4aa0e259dd Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sun, 25 Oct 2026 10:57:02 -0700 Subject: [PATCH 5/7] Drop an unreachable chunk-count test from the joined reading (#752) Codecov's patch check found it uncovered: suffix_as_written calls _joined_credential only for a word holding a joiner, so the split always gives two chunks. Co-Authored-By: Claude Opus 5.5 --- nameparser/_pipeline/_vocab.py | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/nameparser/_pipeline/_vocab.py b/nameparser/_pipeline/_vocab.py index 84296937..c952b073 100644 --- a/nameparser/_pipeline/_vocab.py +++ b/nameparser/_pipeline/_vocab.py @@ -586,9 +586,8 @@ def _joined_credential(text: str, lexicon: Lexicon) -> bool: so the recursion is one level deep. A loop rather than `all()` over a generator, which would cost a frame per chunk: a hyphenated name ('Smith-Jones') stops at its first chunk.""" + # the caller found a joiner, so there are two chunks at least chunks = _JOINERS.split(text) - if len(chunks) < 2: - return False whole = False # a chunk that is not a single ASCII character for c in chunks: if not (c and suffix_as_written(_normalize(c), c, lexicon)): From 6b7e8c9d3aa90e19aaa107d68c0fcd06ba2b253b Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sun, 25 Oct 2026 11:11:13 -0700 Subject: [PATCH 6/7] Read a hyphenated word written as a name as a name word (#752) Derek's decision on the Korean given names the code review found. A word joined by a hyphen whose first chunk is in Title case and whose other chunks are in Title case or lower case is a name word, whatever its chunks are listed as: 'Kim, Ae-Ra' and 'Shin Ae-ra' keep their 1.4.0 readings. A chunk with an inner capital ('PhD-Md'), capitals ('MD-PhD'), a lower-case opening ('md-phd') and a slash keep the joined reading. Accepted in its place: 'KIM, AE-RA' reads suffix. No local corpus holds a Title-case joined credential, so phase 2c's 1,333 ctgov moves all stand (decisions/S3). Co-Authored-By: Claude Opus 5.5 --- ...S3-credentials-written-in-joined-chunks.md | 3 ++- ...S3-credentials-written-in-joined-chunks.md | 19 +++++++++++----- docs/release_log.rst | 2 +- nameparser/_pipeline/_vocab.py | 22 ++++++++++++++----- tests/v2/cases.py | 12 ++++++++++ tests/v2/pipeline/test_vocab.py | 10 +++++++++ tests/v2/rules_budget.json | 2 +- tools/differential/corpus_rules.jsonl | 8 +++++-- tools/differential/snapshot.jsonl | 12 ++++++---- 9 files changed, 70 insertions(+), 20 deletions(-) diff --git a/docs/design/decisions/S3-credentials-written-in-joined-chunks.md b/docs/design/decisions/S3-credentials-written-in-joined-chunks.md index c625693a..cc912e7e 100644 --- a/docs/design/decisions/S3-credentials-written-in-joined-chunks.md +++ b/docs/design/decisions/S3-credentials-written-in-joined-chunks.md @@ -6,7 +6,7 @@ - A digit-dotted token (`1.4.2`) loses the chunk claim and is no member of the shape class either, so it reads as a name word, silently (`John Smith 1.4.2` → family '1.4.2') — 2026-09-18. - An ambiguous acronym written with its periods, one after each letter or one after each of two or more letter chunks, counts as unambiguous, and a single trailing period does not (`Jack M.A.` → suffix, `Jack Ma.` → family); the clause stood in rules/S2 until it moved here — 2026-10-19. - Paired initials of a script with capitals (two single letters each followed by a period, the second optional) are no word of the ambiguous class at the trailing slots and read as the name word they are written as, reported, as the switch off reads them: `Ram Bahadur B.K.` → middle 'Bahadur', family 'B.K.'; `Bahadur, Ram B.K.` → middle 'B.K.'; `Ram Bahadur B.K., PhD` → family 'B.K.', suffix 'PhD'. At the first slot after a comma they stay C5's (`John Smith, X.Y. P.Q.` → suffix), behind an unambiguous suffix word they join the run (`Ram Bahadur PhD B.K.` → suffix 'PhD B.K.') while behind an ambiguous or by-shape credential with no unambiguous word in front of it they end the name and take it in (`Jane Doe MA R.T.` → middle 'Doe MA', family 'R.T.'), and three dotted letters, longer chunks and a caseless script's pair keep the position reading (`John Smith X.Y.Z.`, `John Smith B.Tech.`, `สมชาย ใจดี พ.บ.` → suffix) — 2026-10-24 #864. -- A word joined by a hyphen or a slash reads as a suffix where every chunk counts as a suffix as written, listed and not a bare member of the ambiguous class, a chunk's own periods counting as in a word of its own: `John Smith, MD/PhD` → suffix 'MD/PhD', `John Smith MD-PhD` → suffix 'MD-PhD', `John Smith, M.D./Ph.D.` → suffix; one chunk that does not count leaves the word as it was (`John Smith MD-Jones` → family, `John Smith, MA/PhD` → given 'MA/PhD', family 'John Smith'), a word of single-ASCII-character chunks is none (`Jack I-V` → family 'I-V'), and a joiner standing alone or glued to one side, a dash other than the hyphen-minus and a joined split credential make no joined word — 2026-10-25 #752. +- A word joined by a hyphen or a slash reads as a suffix where every chunk counts as a suffix as written, listed and not a bare member of the ambiguous class, a chunk's own periods counting as in a word of its own: `John Smith, MD/PhD` → suffix 'MD/PhD', `John Smith MD-PhD` → suffix 'MD-PhD', `John Smith, M.D./Ph.D.` → suffix; one chunk that does not count leaves the word as it was (`John Smith MD-Jones` → family, `John Smith, MA/PhD` → given 'MA/PhD', family 'John Smith'), a word of single-ASCII-character chunks is none (`Jack I-V` → family 'I-V'), a hyphen-joined word written as a hyphenated name is (first chunk in Title case, the rest in Title case or lower case) is a name word (`Kim, Ae-Ra`, `Shin Ae-ra` keep the given and family), while capitals keep the joined reading (`KIM, AE-RA` → suffix), and a joiner standing alone or glued to one side, a dash other than the hyphen-minus and a joined split credential make no joined word — 2026-10-25 #752. Open: none in this file. @@ -19,3 +19,4 @@ Open: none in this file. - 2026-10-19 — CLAUSE MOVED IN: an ambiguous acronym written with its periods, one after each letter or one after each of two or more letter chunks, counts unambiguously, and a single trailing period does not. It stood in rules/S2 until now and is a shape, the periods, which is this rule's signal, so it moved here, one rule per piece of evidence (docs/design/AGENTS.md). It was decided in #296's review (decisions/C8's PR #428 review entry) and widened to letter chunks by #544 (decisions/suffix-acronym-collisions' #544 entry), and no entry under decisions/S2 recorded it. Where earlier entries and code call it S2's period gate, it is this rule's now. - 2026-10-25 #752 — JOINED CHUNKS: A WORD JOINED BY A HYPHEN OR A SLASH IS A CREDENTIAL WHERE EVERY CHUNK IS ONE (phase 2c, item 5 of decisions/vocabulary-model's 2026-10-23 #752 entry, agreed by Derek on 2026-10-23). The chunk reading this rule gives interior periods is extended to the two other joiners a credential is written with, by a stricter test: the periods' ANY becomes EVERY, and a chunk must count as a suffix as written, which a bare member of the ambiguous class (`MA`, `Ed`) does not, because a hyphen also joins names (`Smith-Jones`, `Mary-Jane`) and a slash sits between a name and a credential in typed data. The test is asked where a word's suffix reading as written is asked, so classify's tag, the comma's decision and every predicate testing a word's text read the joined word alike. A chunk is read as a word of its own, so a chunk written with its periods counts (`M.D./Ph.D.`, and `M.A./Ph.D.`, the ambiguous member written so). Not read: a joiner standing alone or glued to one side (`MD / PhD` already read as a run of suffix words; `MD/ PhD` keeps `MD/` as a name word, 10 ClinicalTrials.gov entries, and `MD /PhD` 1), the en dash and other dash characters (no entry joins two listed credentials with one), and `&` (63 entries join two listed credentials with it, left for its own change). MEASURED (2026-10-25, against 2af8da8e; every local corpus recorded from that base with `--tree`): `tools/differential/snapshot.py` moves nothing but the rule's own new examples; `cldr_agreement.py` matches every record; `local_agreement.py psc-individuals` matches its record; tse-donors, 0 of 848,351 entries; fec-individuals, 1 of 1,893,964, a doubled joined credential leaving the middle name for the suffix; ctgov-officials, 1,333 of 352,104, the population being the 1,592 words in it joined by a hyphen or a slash whose every chunk counts (the rest already read as suffixes, in a part past a suffix comma). Of the 1,333, 1,319 are a joined credential after a comma that had been read as the given name, the name reversed around it, now the suffix; 2 a joined credential ending a name without a comma, now the suffix where it was the family; and 12 move a report alone, 11 dropping a comma-structure report (a third part now read wholly as credentials) and 1 a suffix-or-name report. Of the 1,321 whose fields move, 3 also take a neighbouring word into the suffix with a new report, each as the unjoined twin reads: a word after the credential (`John Smith, MD/PhD Candidate` reads as `John Smith, PhD Candidate` does) and, once, a capitals credential ending the part before the comma (rules/C12). The joined words are 82 distinct credentials, every one a degree or certification combination, the commonest `MD-PhD` (494), `MD/PhD` (473), `PhD/MD` (65) and `MS-PT` (53); 68 entries hold a chunk written with its periods, which the 2026-10-23 measurement's letters-only test did not read (its 1,254). COST: a word holding a joiner pays three profiler frames at each site that asks its suffix reading as written, stopping at its first chunk that does not count, and a word holding none pays nothing, the joiners being tested in C first (`John Smith` 150 frames before and after, `Mary Smith-Jones` 156 → 159, `Smith-Jones, Mary-Jane` 169 → 175, py3.11). Pinned by the #752 rows of tests/v2/cases.py and `test_suffix_as_written_reads_a_joined_credential` (tests/v2/pipeline/test_vocab.py), whose refused rows pass on the pre-change tree and whose admitted rows fail on it. - 2026-10-25 #752, review of phase 2c — THREE BOUNDARIES THE FIRST VERSION MISSED, found by the PR's code and design-docs reviews. (i) The single-character exception binds a joined word too: `i` and `v` are suffix words, so `Jack I-V` read suffix 'I-V' with no family where 1.4.0 through 2.6.0 read family 'I-V' and the period twin `Jack I.V.` still does; a joined word now needs a chunk that is not a single ASCII character once its own periods are set aside, as a period-split word's chunks are (`Jack I.-V.` keeps family 'I.-V.', the second review's find against the first fix, which counted the period; `John Smith, MS-V` keeps suffix 'MS-V'), and the Except bullet carries the joined rows. No local corpus entry moved back: the measured 1,333 hold no such word. (ii) ACCEPTED: a Korean given name romanized with a hyphen between two syllables the vocabulary lists as unambiguous credentials, `ae` and `ra` (`Kim, Ae-Ra`, `Shin Ae-ra`), reads as the credential, unreported after a comma and leaving no family without one (`Shin Ae-ra` → given 'Shin', suffix 'Ae-ra'). The spaced spelling already did on the base (`Kim, Ae Ra` → suffix 'Ae Ra', `Shin Ae Ra` → suffix 'Ae Ra'), so this is the vocabulary collision decisions/vocabulary-collisions weighs, which the hyphen had been hiding, not a new reading of the joiner; giving `ae` and `ra` a `given` reading, which would make both ambiguous by derivation (decisions/vocabulary-model's 2026-10-25 #752 entry), moves the spaced spelling too and is left for its own change. The reviewer's scan of every romanized Korean syllable found `ae`, `am`, `ra` and `i` counting as suffixes; `i` alone is no joined credential since (i), and no committed or local corpus entry holds a name-cased hyphenated pair of them. (iii) A split credential joined to another (`MD/Ph. D.`) and the dash characters other than the hyphen-minus join nothing, now stated in the rule. +- 2026-10-25 #752 (Derek) — A HYPHENATED NAME'S WRITING IS A FORM CUE. SUPERSEDES the review entry's (ii), which accepted `Kim, Ae-Ra` reading suffix 'Ae-Ra'. Derek: "refuse a hyphen join whose every chunk is Title case and then mark AE-RA as the accepted". A word joined by a hyphen whose first chunk is in Title case (one capital, then lower case, its periods set aside) and whose every other chunk is in Title case or lower case is a name word whatever its chunks are listed as, the design's cue that Title case leans to the name (the 2026-10-23 #752 entry's form-cue table, decisions/vocabulary-model). The lower-case tail is this change's widening of Derek's wording, because the Revised Romanization writes a given name's second syllable in lower case (`Shin Ae-ra`), the spelling the code review's case was in; `Ae-Ra` and `Ae-ra` both keep the name. Left to the every-chunk test: a chunk with a capital past its first letter (`PhD-Md` → suffix), a chunk in capitals (`MD-PhD`), a word opening in lower case (`md-phd`, the lower-case writing of a credential) and a slash, which joins no name; no local corpus holds a slash-joined credential in Title case, so the slash's exemption is decided by what the cue is for rather than by a count. ACCEPTED in its place: the capitals (`KIM, AE-RA`, `Kim, AE-RA`) and the lower-case writing (`kim, ae-ra`) still read as the credential, a name in capitals having no Title case for the writing to show; the spaced `Kim, Ae Ra` losing its given name is older than phase 2c and stays the `ae`/`ra` vocabulary question. MEASURED (2026-10-25): no word in any local corpus joins two listed credentials with a hyphen or a slash in Title case throughout (ctgov-officials, fec-individuals, tse-donors, psc-individuals, ukec-individuals, jrc-names, nepal-candidates; FEC's 31 joined credentials are all capitals), so the cue gives back none of phase 2c's 1,333 ctgov moves; ctgov-officials still moves 1,862 entries over both phases and fec-individuals 1; the committed corpora move only the rules' and cases' new rows. Pinned by `Kim, Ae-Ra` and `Shin Ae-ra` in tests/v2/cases.py, 1.4.0's readings (wheel), and the cue's rows in `test_suffix_as_written_reads_a_joined_credential`. diff --git a/docs/design/rules/S3-credentials-written-in-joined-chunks.md b/docs/design/rules/S3-credentials-written-in-joined-chunks.md index d5d47f43..a8d63a29 100644 --- a/docs/design/rules/S3-credentials-written-in-joined-chunks.md +++ b/docs/design/rules/S3-credentials-written-in-joined-chunks.md @@ -30,6 +30,8 @@ ", MD/PhD" → suffix="MD/PhD" "John Smith, M.D./Ph.D." → suffix="M.D./Ph.D." "john smith, ms-pt" → suffix="ms-pt" + "John Smith, PhD-Md" → suffix="PhD-Md" + "John Smith, md-phd" → suffix="md-phd" "John Smith-Jones, MD-PhD" → family="Smith-Jones" "John Smith-Jones" → family="Smith-Jones" · boundary "Smith, Mary-Jane" → given="Mary-Jane" · boundary @@ -50,6 +52,12 @@ **Except:** +- **A word joined by a hyphen and written the way a hyphenated name is, its first chunk in Title case and every other in Title case or lower case, is a name word**, whatever its chunks are listed as: that is how a hyphenated given name is written (`Ae-Ra`, `Ae-ra`, the Korean syllables `ae` and `ra` both being listed credentials), and a credential's chunks are written otherwise. A chunk with a capital past its first letter (`PhD`), a chunk in capitals, a word opening in lower case and a slash, which joins no name, leave the word to the joined reading's every-chunk test. + ``` + "Kim, Ae-Ra" → given="Ae-Ra" + "Shin Ae-ra" → family="Ae-ra" + "John Smith, Md-Rn" → given="Md-Rn" + ``` - **Every chunk the vocabulary matches is a single ASCII character**, the roman numerals and the lone digit the vocabulary lists, which are about generations rather than credentials: those chunks do not make the word a suffix. ``` "Jack X.Y.I." → family="X.Y.I." · boundary @@ -89,12 +97,11 @@ "John Smith, MA/PhD" → given="MA/PhD" "John Smith, MA/PhD" → family="John Smith" ``` -- A Korean given name romanized with a hyphen between two syllables the vocabulary lists as unambiguous credentials reads as the credential, unreported after a comma and leaving no family without one, as its spaced spelling already did: `ae` and `ra` are such credentials, and the hyphen no longer keeps the name a name word. +- A Korean given name romanized with a hyphen between two syllables the vocabulary lists as unambiguous credentials, written in capitals or opening in lower case, reads as the credential, unreported after a comma: a name in capitals has no Title case for the writing to show, and `ae` and `ra` are such credentials. ``` - "Kim, Ae-Ra" → suffix="Ae-Ra" - "Kim, Ae Ra" → suffix="Ae Ra" - "Shin Ae-ra" → suffix="Ae-ra" - "Shin Ae Ra" → suffix="Ae Ra" + "KIM, AE-RA" → suffix="AE-RA" + "Kim, AE-RA" → suffix="AE-RA" + "kim, ae-ra" → suffix="ae-ra" ``` - An unlisted credential of two letters written with its periods behind a full name reads as the family name, and the reading is reported: King's Counsel, which no vocabulary lists, is the case. ``` @@ -108,5 +115,5 @@ - The initialless-script clause carries no example line of its own. Every input that exercises it composes a script that writes no abbreviations with a period that only a Latin convention writes — `John Smith 田.中.` is the shape — and a composed form no writing system produces is tolerated input rather than contract, which an example of a normative rule would promise. So the witnesses are the tolerated row tests/v2/cases.py's an_initialless_script_glued_into_periods_is_not_this_shape and the CJK assertions in tests/v2/pipeline/test_vocab.py's test_period_joined_vocab_retires_the_single_character_chunk. W3 states the same precedence for the same reason. --- -**Signal:** shape: a word in chunks joined by interior periods, a hyphen or a slash, read by its chunks or as the acronym its letters spell, and for paired initials whether its script writes capitals +**Signal:** shape: a word in chunks joined by interior periods, a hyphen or a slash, read by its chunks or as the acronym its letters spell, for paired initials whether its script writes capitals, and for a hyphen-joined word whether it is written as a hyphenated name is **history:** decisions/S3 · **implemented:** nameparser/_pipeline/_classify.py, nameparser/_pipeline/_comma.py, nameparser/_pipeline/_pieces.py, nameparser/_pipeline/_vocab.py diff --git a/docs/release_log.rst b/docs/release_log.rst index e594301b..8e01e48b 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -6,7 +6,7 @@ Release Log **Behavior Changes** - - **Read a credential joined by a hyphen or a slash as a suffix.** ``John Smith, MD/PhD`` gives first ``John``, last ``Smith``, suffix ``MD/PhD``, where 1.4.0 through 2.6.0 gave first ``MD/PhD``, last ``John Smith``; ``MD-PhD`` and ``PhD/MD`` read the same. An unlisted one written in capitals reads as a credential too after a comma behind two or more name words written in mixed case: ``John Smith, PU-PH`` gives suffix ``PU-PH``, where 1.4.0 through 2.6.0 gave first ``PU-PH``. Otherwise every part has to be a listed credential that is not also a name, so ``Smith-Jones`` is untouched and ``John Smith, MA/PhD`` still reverses; a hyphenated Korean given name whose syllables are both listed credentials reads as one, as its spaced spelling already did (``Kim, Ae-Ra`` gives suffix ``Ae-Ra``). :issue:`752`, :issue:`797` + - **Read a credential joined by a hyphen or a slash as a suffix.** ``John Smith, MD/PhD`` gives first ``John``, last ``Smith``, suffix ``MD/PhD``, where 1.4.0 through 2.6.0 gave first ``MD/PhD``, last ``John Smith``; ``MD-PhD`` and ``PhD/MD`` read the same. An unlisted one written in capitals reads as a credential too after a comma behind two or more name words written in mixed case: ``John Smith, PU-PH`` gives suffix ``PU-PH``, where 1.4.0 through 2.6.0 gave first ``PU-PH``. Otherwise every part has to be a listed credential that is not also a name, so ``Smith-Jones`` is untouched and ``John Smith, MA/PhD`` still reverses; a hyphenated name written in Title case keeps its reading even where both parts are listed credentials (``Kim, Ae-Ra``), though its capitals spelling does not (``KIM, AE-RA`` gives suffix ``AE-RA``). :issue:`752`, :issue:`797` * 2.6.0 - October 25, 2026 diff --git a/nameparser/_pipeline/_vocab.py b/nameparser/_pipeline/_vocab.py index c952b073..7c0a4dec 100644 --- a/nameparser/_pipeline/_vocab.py +++ b/nameparser/_pipeline/_vocab.py @@ -581,25 +581,37 @@ def _joined_credential(text: str, lexicon: Lexicon) -> bool: """Whether TEXT is two or more chunks joined by '-' or '/', each non-empty and each counting as a suffix as written: listed, and not a bare member of the ambiguous class, and not every one a - single ASCII character ('MD/PhD', 'M.A.-Ph.D.', but not 'MA/PhD', - 'Smith-Jones', 'I-V' or 'MD/'). A chunk holds no joiner, + single ASCII character, and -- joined by a hyphen -- not + written as a hyphenated name is ('MD/PhD', 'M.A.-Ph.D.', 'PhD-Md', + 'md-phd', but not 'MA/PhD', 'Smith-Jones', 'I-V', 'Ae-Ra', 'Ae-ra' + or 'MD/'). A chunk holds no joiner, so the recursion is one level deep. A loop rather than `all()` over a generator, which would cost a frame per chunk: a hyphenated name ('Smith-Jones') stops at its first chunk.""" # the caller found a joiner, so there are two chunks at least chunks = _JOINERS.split(text) whole = False # a chunk that is not a single ASCII character - for c in chunks: + named = "-" in text # every chunk so far in Title case + for k, c in enumerate(chunks): if not (c and suffix_as_written(_normalize(c), c, lexicon)): return False # its periods are not the chunk's ('I.' is 'i'), as in a word # split at its periods bare = c.replace(".", "") whole = whole or len(bare) > 1 or not bare.isascii() + # the first chunk in Title case, a later one in Title case or + # lower case ('Ae-Ra', 'Ae-ra') + named = named and len(bare) > 1 and ( + bare[0].isupper() and bare[1:].islower() + or k > 0 and bare.islower()) # rules/S3: "Every chunk the vocabulary matches is a single ASCII # character" -- the numerals and the digit, which are about - # generations, make no joined word a suffix either ('Jack I-V') - return whole + # generations, make no joined word a suffix either ('Jack I-V'); + # rules/S3: "a word joined by a hyphen and written the way a + # hyphenated name is, its first chunk in Title case and every other + # in Title case or lower case, is a name word" -- where a + # credential's chunks are not (decisions/S3 (#752)) + return whole and not named def _is_suffix_strict_n(n: str, text: str, lexicon: Lexicon, diff --git a/tests/v2/cases.py b/tests/v2/cases.py index 481b052e..219023cb 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -3531,6 +3531,18 @@ def _check_cjk_shape_purity(self) -> None: notes="the two hyphens side by side: the surname's chunks are " "not listed, the credential's are; 1.4.0 through 2.6.0 " "read given 'MD-PhD', family 'John Smith-Jones'"), + Case("a_hyphenated_given_name_in_title_case_stays_the_given_name", + "Kim, Ae-Ra", + {"given": "Ae-Ra", "family": "Kim"}, + notes="pins rules/S3's name-writing cue: 'ae' and 'ra' are both " + "listed credentials, so without the cue the joined " + "reading takes the given name (as the spaced 'Kim, Ae " + "Ra' already loses it to the two credentials)"), + Case("a_hyphenated_given_name_with_a_lower_case_tail_stays_the_name", + "Shin Ae-ra", + {"given": "Shin", "family": "Ae-ra"}, + notes="the standard romanization writes the second syllable in " + "lower case, which the cue reads as a name too"), Case("a_joined_word_with_one_unlisted_chunk_is_a_name_word", "John Smith MD-Jones", {"given": "John", "middle": "Smith", "family": "MD-Jones"}, diff --git a/tests/v2/pipeline/test_vocab.py b/tests/v2/pipeline/test_vocab.py index 41937644..a8f4df84 100644 --- a/tests/v2/pipeline/test_vocab.py +++ b/tests/v2/pipeline/test_vocab.py @@ -415,6 +415,16 @@ def test_period_joined_vocab_retires_the_single_character_chunk() -> None: ("I.-V.", False), # its periods are not the chunk's ("I-V.", False), ("MS-V", True), # one chunk is not + ("Ae-Ra", False), # every chunk Title case: a hyphenated name + ("Md-Rn", False), + ("Ae-ra", False), # the standard romanization's lower-case tail + ("ae-Ra", True), # a name opens in Title case + ("Md-phd", False), + ("PhD-Md", True), # one chunk is not Title case + ("Ph.D.-Md", True), + ("AE-RA", True), # capitals: the cue cannot read a name in them + ("ae-ra", True), + ("Ae/Ra", True), # a slash joins no name, so it takes no cue ]) def test_suffix_as_written_reads_a_joined_credential( text: str, expected: bool) -> None: diff --git a/tests/v2/rules_budget.json b/tests/v2/rules_budget.json index 639ee676..962aa24e 100644 --- a/tests/v2/rules_budget.json +++ b/tests/v2/rules_budget.json @@ -54,7 +54,7 @@ "R5": {"words": 250, "then_words": 154, "because_sentences": 2, "except": 0, "precedence": 0, "accepted": 0, "accepted_words": 0}, "S1": {"words": 274, "then_words": 26, "because_sentences": 1, "except": 2, "precedence": 2, "accepted": 0, "accepted_words": 0}, "S2": {"words": 641, "then_words": 103, "because_sentences": 1, "except": 0, "precedence": 7, "accepted": 3, "accepted_words": 214}, - "S3": {"words": 807, "then_words": 398, "because_sentences": 3, "except": 3, "precedence": 4, "accepted": 5, "accepted_words": 295}, + "S3": {"words": 910, "then_words": 398, "because_sentences": 3, "except": 4, "precedence": 4, "accepted": 5, "accepted_words": 294}, "S4": {"words": 298, "then_words": 156, "because_sentences": 2, "except": 0, "precedence": 3, "accepted": 0, "accepted_words": 0}, "S5": {"words": 632, "then_words": 317, "because_sentences": 1, "except": 2, "precedence": 1, "accepted": 0, "accepted_words": 0}, "S6": {"words": 337, "then_words": 121, "because_sentences": 2, "except": 0, "precedence": 2, "accepted": 0, "accepted_words": 0}, diff --git a/tools/differential/corpus_rules.jsonl b/tools/differential/corpus_rules.jsonl index b22b7f12..39a9ba1a 100644 --- a/tools/differential/corpus_rules.jsonl +++ b/tools/differential/corpus_rules.jsonl @@ -388,6 +388,7 @@ "John Smith, MS-OMPT" "John Smith, MS-V" "John Smith, Ma" +"John Smith, Md-Rn" "John Smith, Mr." "John Smith, Mr. Jr." "John Smith, Mr. and Mrs." @@ -402,6 +403,7 @@ "John Smith, PhD X.Y." "John Smith, PhD XYZ" "John Smith, PhD vd DO" +"John Smith, PhD-Md" "John Smith, Pu-Ph" "John Smith, RA Jones" "John Smith, V." @@ -409,6 +411,7 @@ "John Smith, X.Y. P.Q." "John Smith, X.Y.Z." "John Smith, XYZ" +"John Smith, md-phd" "John Smith, otr/l" "John Smith, vd Ma" "John Smith-Jones" @@ -470,7 +473,8 @@ "Juan y Garcia née Jones" "Juan Ó Pérez" "Juan Ó. Pérez" -"Kim, Ae Ra" +"KIM, AE-RA" +"Kim, AE-RA" "Kim, Ae-Ra" "Kim김민준씨, MA" "King Smith, John" @@ -557,7 +561,6 @@ "Sheik abdul Jr Smith" "Sheik abdul salam" "Sheik abdul salam Jr" -"Shin Ae Ra" "Shin Ae-ra" "Shirley Maclaine" "Sidorov Ivan Petrovich Jr." @@ -763,6 +766,7 @@ "juan mcdonald" "juan y-garcia" "juan y-garcia-lopez" +"kim, ae-ra" "lt.gov john doe" "m.ʿali smith" "maria da-silva" diff --git a/tools/differential/snapshot.jsonl b/tools/differential/snapshot.jsonl index bf9c5d55..91682885 100644 --- a/tools/differential/snapshot.jsonl +++ b/tools/differential/snapshot.jsonl @@ -1711,6 +1711,7 @@ {"name": "John Smith, MS-OMPT", "given": "John", "family": "Smith", "suffix": "MS-OMPT", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. S."} {"name": "John Smith, MS-V", "given": "John", "family": "Smith", "suffix": "MS-V", "initials": "J. S."} {"name": "John Smith, Ma", "given": "John", "family": "Smith", "suffix": "Ma", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. S.", "capitalized_force": "John Smith MA"} +{"name": "John Smith, Md-Rn", "given": "Md-Rn", "family": "John Smith", "initials": "M. J. S."} {"name": "John Smith, Mr.", "title": "Mr.", "given": "John", "family": "Smith", "initials": "J. S."} {"name": "John Smith, Mr. Jr.", "title": "Mr.", "given": "John", "family": "Smith", "suffix": "Jr.", "initials": "J. S."} {"name": "John Smith, Mr. and Mrs.", "title": "Mr. and Mrs.", "given": "John", "family": "Smith", "initials": "J. S."} @@ -1729,6 +1730,7 @@ {"name": "John Smith, PhD X.Y.", "given": "John", "family": "Smith", "suffix": "PhD X.Y.", "initials": "J. S."} {"name": "John Smith, PhD XYZ", "given": "John", "family": "Smith", "suffix": "PhD XYZ", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. S."} {"name": "John Smith, PhD vd DO", "given": "John", "family": "Smith", "suffix": "PhD vd DO", "initials": "J. S.", "capitalized_force": "John Smith PhD VD DO"} +{"name": "John Smith, PhD-Md", "given": "John", "family": "Smith", "suffix": "PhD-Md", "initials": "J. S.", "capitalized_force": "John Smith PhD-MD"} {"name": "John Smith, Pu-Ph", "given": "Pu-Ph", "family": "John Smith", "initials": "P. J. S."} {"name": "John Smith, RA Jones", "title": "RA", "given": "Jones", "family": "John Smith", "initials": "J. J. S.", "capitalized_force": "Ra Jones John Smith"} {"name": "John Smith, RAI", "given": "John", "family": "Smith", "suffix": "RAI", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. S."} @@ -1740,6 +1742,7 @@ {"name": "John Smith, X.Y.Z.", "given": "John", "family": "Smith", "suffix": "X.Y.Z.", "initials": "J. S."} {"name": "John Smith, X.Y.Z. MA", "given": "John", "family": "Smith", "suffix": "X.Y.Z. MA", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. S."} {"name": "John Smith, XYZ", "given": "John", "family": "Smith", "suffix": "XYZ", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. S."} +{"name": "John Smith, md-phd", "given": "John", "family": "Smith", "suffix": "md-phd", "initials": "J. S.", "capitalized_force": "John Smith MD-PhD"} {"name": "John Smith, otr/l", "given": "John", "family": "Smith", "suffix": "otr/l", "initials": "J. S.", "capitalized_force": "John Smith OTR/L"} {"name": "John Smith, vd Ma", "given": "John", "family": "Smith", "suffix": "vd Ma", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. S.", "capitalized_force": "John Smith VD MA"} {"name": "John Smith-Jones", "given": "John", "family": "Smith-Jones", "initials": "J. S."} @@ -1971,6 +1974,7 @@ {"name": "K. Sivakumar", "given": "K.", "family": "Sivakumar", "initials": "K. S."} {"name": "KALAM Alɛ Mustafa", "order": "FAMILY_FIRST", "given": "Alɛ", "middle": "Mustafa", "family": "KALAM", "initials": "A. M. K.", "capitalized_force": "Alɛ Mustafa Kalam"} {"name": "KANSULO Iiɖa", "order": "FAMILY_FIRST", "given": "Iiɖa", "family": "KANSULO", "initials": "I. K.", "capitalized_force": "Iiɖa Kansulo"} +{"name": "KIM, AE-RA", "family": "KIM", "suffix": "AE-RA", "initials": "K.", "capitalized": "Kim AE-RA"} {"name": "Kajal Patel Hiring Globally", "given": "Kajal", "middle": "Patel Hiring", "family": "Globally", "initials": "K. P. H. G."} {"name": "Kalniņa Aina", "order": "FAMILY_FIRST", "given": "Aina", "family": "Kalniņa", "initials": "A. K."} {"name": "Kamāl Yūsuf al-Ḥāǧǧ", "given": "Kamāl", "middle": "Yūsuf", "family": "al-Ḥāǧǧ", "initials": "K. Y. a."} @@ -2023,8 +2027,8 @@ {"name": "Khun Sa", "given": "Khun", "family": "Sa", "initials": "K. S."} {"name": "Ki, Tomonori", "given": "Tomonori", "family": "Ki", "initials": "T. K."} {"name": "Kim Min Ho", "order": "FAMILY_FIRST", "given": "Min", "middle": "Ho", "family": "Kim", "initials": "M. H. K."} -{"name": "Kim, Ae Ra", "family": "Kim", "suffix": "Ae Ra", "initials": "K.", "capitalized_force": "Kim AE RA"} -{"name": "Kim, Ae-Ra", "family": "Kim", "suffix": "Ae-Ra", "initials": "K.", "capitalized_force": "Kim AE-RA"} +{"name": "Kim, AE-RA", "family": "Kim", "suffix": "AE-RA", "initials": "K."} +{"name": "Kim, Ae-Ra", "given": "Ae-Ra", "family": "Kim", "initials": "A. K."} {"name": "Kim, Jr.", "family": "Kim", "suffix": "Jr.", "initials": "K."} {"name": "Kim김민준씨, MA", "family": "Kim김민준", "suffix": "씨, MA", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "K."} {"name": "King Curtis", "title": "King", "given": "Curtis", "initials": "C."} @@ -2903,8 +2907,7 @@ {"name": "Sheik abdul salam", "title": "Sheik", "given": "abdul salam", "initials": "a. s.", "capitalized_force": "Sheik Abdul Salam"} {"name": "Sheik abdul salam Jr", "title": "Sheik", "given": "abdul salam", "suffix": "Jr", "initials": "a. s.", "capitalized_force": "Sheik Abdul Salam Jr"} {"name": "Shemuʾel ben Ḥofni", "given": "Shemuʾel", "middle": "ben", "family": "Ḥofni", "initials": "S. b. Ḥ.", "capitalized_force": "Shemuʾel Ben Ḥofni"} -{"name": "Shin Ae Ra", "given": "Shin", "suffix": "Ae Ra", "ambiguities": ["GIVEN_OR_FAMILY"], "initials": "S.", "capitalized_force": "Shin AE RA"} -{"name": "Shin Ae-ra", "given": "Shin", "suffix": "Ae-ra", "ambiguities": ["GIVEN_OR_FAMILY"], "initials": "S.", "capitalized_force": "Shin AE-RA"} +{"name": "Shin Ae-ra", "given": "Shin", "family": "Ae-ra", "initials": "S. A.", "capitalized_force": "Shin Ae-Ra"} {"name": "Shirley Maclaine", "given": "Shirley", "family": "Maclaine", "initials": "S. M.", "capitalized_force": "Shirley MacLaine"} {"name": "Shree Lalit Kumaar Bharati Jr, MP", "given": "Shree", "middle": "Lalit Kumaar", "family": "Bharati", "suffix": "Jr, MP", "initials": "S. L. K. B."} {"name": "Shtober Zaziliya Hemish", "order": "FAMILY_FIRST", "given": "Zaziliya", "middle": "Hemish", "family": "Shtober", "initials": "Z. H. S."} @@ -3650,6 +3653,7 @@ {"name": "juan y garcia", "given": "juan", "middle": "y", "family": "garcia", "initials": "j. y. g.", "capitalized": "Juan y Garcia"} {"name": "juan y-garcia", "given": "juan", "family": "y-garcia", "initials": "j. y.", "capitalized": "Juan Y-Garcia"} {"name": "juan y-garcia-lopez", "given": "juan", "family": "y-garcia-lopez", "initials": "j. y.", "capitalized": "Juan Y-Garcia-Lopez"} +{"name": "kim, ae-ra", "family": "kim", "suffix": "ae-ra", "initials": "k.", "capitalized": "Kim AE-RA"} {"name": "larry james edward johnson v", "given": "larry", "middle": "james edward", "family": "johnson", "suffix": "v", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "l. j. e. j.", "capitalized": "Larry James Edward Johnson V"} {"name": "lt. gen. john a. kenneth doe iv", "title": "lt. gen.", "given": "john", "middle": "a. kenneth", "family": "doe", "suffix": "iv", "initials": "j. a. k. d.", "capitalized": "Lt. Gen. John A. Kenneth Doe IV"} {"name": "lt.gov john doe", "title": "lt.gov", "given": "john", "family": "doe", "initials": "j. d.", "capitalized": "Lt.Gov John Doe"} From 824c721de6ac18129b64d4339a82b808815cb8dd Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sun, 25 Oct 2026 11:20:49 -0700 Subject: [PATCH 7/7] Keep the name-writing cue to hyphen-only words, and state its edges (#752) Review of the cue: a word holding a slash took it ('Kim, Ae-Ra/Md'), which the rule says a slash does not; a one-letter lower-case tail ('Md-v') did not, which the rule says it does. S3 now says a word with an interior period is read by its periods first, and records a credential typed in Title case throughout ('Phd-Md') as the cue's cost; no local corpus writes one. Co-Authored-By: Claude Opus 5.5 --- .../decisions/S3-credentials-written-in-joined-chunks.md | 2 +- .../rules/S3-credentials-written-in-joined-chunks.md | 6 +++++- nameparser/_pipeline/_vocab.py | 8 +++++--- tests/v2/pipeline/test_vocab.py | 3 +++ tests/v2/rules_budget.json | 2 +- tools/differential/corpus_rules.jsonl | 1 + tools/differential/snapshot.jsonl | 1 + 7 files changed, 17 insertions(+), 6 deletions(-) diff --git a/docs/design/decisions/S3-credentials-written-in-joined-chunks.md b/docs/design/decisions/S3-credentials-written-in-joined-chunks.md index cc912e7e..43643fe1 100644 --- a/docs/design/decisions/S3-credentials-written-in-joined-chunks.md +++ b/docs/design/decisions/S3-credentials-written-in-joined-chunks.md @@ -19,4 +19,4 @@ Open: none in this file. - 2026-10-19 — CLAUSE MOVED IN: an ambiguous acronym written with its periods, one after each letter or one after each of two or more letter chunks, counts unambiguously, and a single trailing period does not. It stood in rules/S2 until now and is a shape, the periods, which is this rule's signal, so it moved here, one rule per piece of evidence (docs/design/AGENTS.md). It was decided in #296's review (decisions/C8's PR #428 review entry) and widened to letter chunks by #544 (decisions/suffix-acronym-collisions' #544 entry), and no entry under decisions/S2 recorded it. Where earlier entries and code call it S2's period gate, it is this rule's now. - 2026-10-25 #752 — JOINED CHUNKS: A WORD JOINED BY A HYPHEN OR A SLASH IS A CREDENTIAL WHERE EVERY CHUNK IS ONE (phase 2c, item 5 of decisions/vocabulary-model's 2026-10-23 #752 entry, agreed by Derek on 2026-10-23). The chunk reading this rule gives interior periods is extended to the two other joiners a credential is written with, by a stricter test: the periods' ANY becomes EVERY, and a chunk must count as a suffix as written, which a bare member of the ambiguous class (`MA`, `Ed`) does not, because a hyphen also joins names (`Smith-Jones`, `Mary-Jane`) and a slash sits between a name and a credential in typed data. The test is asked where a word's suffix reading as written is asked, so classify's tag, the comma's decision and every predicate testing a word's text read the joined word alike. A chunk is read as a word of its own, so a chunk written with its periods counts (`M.D./Ph.D.`, and `M.A./Ph.D.`, the ambiguous member written so). Not read: a joiner standing alone or glued to one side (`MD / PhD` already read as a run of suffix words; `MD/ PhD` keeps `MD/` as a name word, 10 ClinicalTrials.gov entries, and `MD /PhD` 1), the en dash and other dash characters (no entry joins two listed credentials with one), and `&` (63 entries join two listed credentials with it, left for its own change). MEASURED (2026-10-25, against 2af8da8e; every local corpus recorded from that base with `--tree`): `tools/differential/snapshot.py` moves nothing but the rule's own new examples; `cldr_agreement.py` matches every record; `local_agreement.py psc-individuals` matches its record; tse-donors, 0 of 848,351 entries; fec-individuals, 1 of 1,893,964, a doubled joined credential leaving the middle name for the suffix; ctgov-officials, 1,333 of 352,104, the population being the 1,592 words in it joined by a hyphen or a slash whose every chunk counts (the rest already read as suffixes, in a part past a suffix comma). Of the 1,333, 1,319 are a joined credential after a comma that had been read as the given name, the name reversed around it, now the suffix; 2 a joined credential ending a name without a comma, now the suffix where it was the family; and 12 move a report alone, 11 dropping a comma-structure report (a third part now read wholly as credentials) and 1 a suffix-or-name report. Of the 1,321 whose fields move, 3 also take a neighbouring word into the suffix with a new report, each as the unjoined twin reads: a word after the credential (`John Smith, MD/PhD Candidate` reads as `John Smith, PhD Candidate` does) and, once, a capitals credential ending the part before the comma (rules/C12). The joined words are 82 distinct credentials, every one a degree or certification combination, the commonest `MD-PhD` (494), `MD/PhD` (473), `PhD/MD` (65) and `MS-PT` (53); 68 entries hold a chunk written with its periods, which the 2026-10-23 measurement's letters-only test did not read (its 1,254). COST: a word holding a joiner pays three profiler frames at each site that asks its suffix reading as written, stopping at its first chunk that does not count, and a word holding none pays nothing, the joiners being tested in C first (`John Smith` 150 frames before and after, `Mary Smith-Jones` 156 → 159, `Smith-Jones, Mary-Jane` 169 → 175, py3.11). Pinned by the #752 rows of tests/v2/cases.py and `test_suffix_as_written_reads_a_joined_credential` (tests/v2/pipeline/test_vocab.py), whose refused rows pass on the pre-change tree and whose admitted rows fail on it. - 2026-10-25 #752, review of phase 2c — THREE BOUNDARIES THE FIRST VERSION MISSED, found by the PR's code and design-docs reviews. (i) The single-character exception binds a joined word too: `i` and `v` are suffix words, so `Jack I-V` read suffix 'I-V' with no family where 1.4.0 through 2.6.0 read family 'I-V' and the period twin `Jack I.V.` still does; a joined word now needs a chunk that is not a single ASCII character once its own periods are set aside, as a period-split word's chunks are (`Jack I.-V.` keeps family 'I.-V.', the second review's find against the first fix, which counted the period; `John Smith, MS-V` keeps suffix 'MS-V'), and the Except bullet carries the joined rows. No local corpus entry moved back: the measured 1,333 hold no such word. (ii) ACCEPTED: a Korean given name romanized with a hyphen between two syllables the vocabulary lists as unambiguous credentials, `ae` and `ra` (`Kim, Ae-Ra`, `Shin Ae-ra`), reads as the credential, unreported after a comma and leaving no family without one (`Shin Ae-ra` → given 'Shin', suffix 'Ae-ra'). The spaced spelling already did on the base (`Kim, Ae Ra` → suffix 'Ae Ra', `Shin Ae Ra` → suffix 'Ae Ra'), so this is the vocabulary collision decisions/vocabulary-collisions weighs, which the hyphen had been hiding, not a new reading of the joiner; giving `ae` and `ra` a `given` reading, which would make both ambiguous by derivation (decisions/vocabulary-model's 2026-10-25 #752 entry), moves the spaced spelling too and is left for its own change. The reviewer's scan of every romanized Korean syllable found `ae`, `am`, `ra` and `i` counting as suffixes; `i` alone is no joined credential since (i), and no committed or local corpus entry holds a name-cased hyphenated pair of them. (iii) A split credential joined to another (`MD/Ph. D.`) and the dash characters other than the hyphen-minus join nothing, now stated in the rule. -- 2026-10-25 #752 (Derek) — A HYPHENATED NAME'S WRITING IS A FORM CUE. SUPERSEDES the review entry's (ii), which accepted `Kim, Ae-Ra` reading suffix 'Ae-Ra'. Derek: "refuse a hyphen join whose every chunk is Title case and then mark AE-RA as the accepted". A word joined by a hyphen whose first chunk is in Title case (one capital, then lower case, its periods set aside) and whose every other chunk is in Title case or lower case is a name word whatever its chunks are listed as, the design's cue that Title case leans to the name (the 2026-10-23 #752 entry's form-cue table, decisions/vocabulary-model). The lower-case tail is this change's widening of Derek's wording, because the Revised Romanization writes a given name's second syllable in lower case (`Shin Ae-ra`), the spelling the code review's case was in; `Ae-Ra` and `Ae-ra` both keep the name. Left to the every-chunk test: a chunk with a capital past its first letter (`PhD-Md` → suffix), a chunk in capitals (`MD-PhD`), a word opening in lower case (`md-phd`, the lower-case writing of a credential) and a slash, which joins no name; no local corpus holds a slash-joined credential in Title case, so the slash's exemption is decided by what the cue is for rather than by a count. ACCEPTED in its place: the capitals (`KIM, AE-RA`, `Kim, AE-RA`) and the lower-case writing (`kim, ae-ra`) still read as the credential, a name in capitals having no Title case for the writing to show; the spaced `Kim, Ae Ra` losing its given name is older than phase 2c and stays the `ae`/`ra` vocabulary question. MEASURED (2026-10-25): no word in any local corpus joins two listed credentials with a hyphen or a slash in Title case throughout (ctgov-officials, fec-individuals, tse-donors, psc-individuals, ukec-individuals, jrc-names, nepal-candidates; FEC's 31 joined credentials are all capitals), so the cue gives back none of phase 2c's 1,333 ctgov moves; ctgov-officials still moves 1,862 entries over both phases and fec-individuals 1; the committed corpora move only the rules' and cases' new rows. Pinned by `Kim, Ae-Ra` and `Shin Ae-ra` in tests/v2/cases.py, 1.4.0's readings (wheel), and the cue's rows in `test_suffix_as_written_reads_a_joined_credential`. +- 2026-10-25 #752 (Derek) — A HYPHENATED NAME'S WRITING IS A FORM CUE. SUPERSEDES the review entry's (ii), which accepted `Kim, Ae-Ra` reading suffix 'Ae-Ra'. Derek: "refuse a hyphen join whose every chunk is Title case and then mark AE-RA as the accepted". A word joined by a hyphen whose first chunk is in Title case (one capital, then lower case, its periods set aside) and whose every other chunk is in Title case or lower case is a name word whatever its chunks are listed as, the design's cue that Title case leans to the name (the 2026-10-23 #752 entry's form-cue table, decisions/vocabulary-model). Its review (2026-10-25) narrowed and stated three edges: a word holding a slash anywhere takes no cue (`Kim, Ae-Ra/Md` keeps the joined reading); a one-letter later chunk in lower case is a name's tail like any other (`Md-v`); and a word holding an interior period is read by its periods first, so the cue never sees it (`Kim, Ae.-Ra.` → suffix by the chunk rule, an unwritten shape). The lower-case tail is this change's widening of Derek's wording, because the Revised Romanization writes a given name's second syllable in lower case (`Shin Ae-ra`), the spelling the code review's case was in; `Ae-Ra` and `Ae-ra` both keep the name. Left to the every-chunk test: a chunk with a capital past its first letter (`PhD-Md` → suffix), a chunk in capitals (`MD-PhD`), a word opening in lower case (`md-phd`, the lower-case writing of a credential) and a slash, which joins no name; no local corpus holds a slash-joined credential in Title case, so the slash's exemption is decided by what the cue is for rather than by a count. ACCEPTED in its place: the capitals (`KIM, AE-RA`, `Kim, AE-RA`) and the lower-case writing (`kim, ae-ra`) still read as the credential, a name in capitals having no Title case for the writing to show; the spaced `Kim, Ae Ra` losing its given name is older than phase 2c and stays the `ae`/`ra` vocabulary question. ACCEPTED too: a credential whose chunks are all typed in Title case reads as a name word, reversing the name after a comma (`John Smith, Phd-Md` → given 'Phd-Md'; likewise `Jr-Md`, `Md-Rn`), the writing being the cue; none of the 34 distinct hyphen-joined credential words in the local corpora is written that way (the review's scan, 2026-10-25). MEASURED (2026-10-25): no word in any local corpus joins two listed credentials with a hyphen or a slash in Title case throughout (ctgov-officials, fec-individuals, tse-donors, psc-individuals, ukec-individuals, jrc-names, nepal-candidates; FEC's 31 joined credentials are all capitals), so the cue gives back none of phase 2c's 1,333 ctgov moves; ctgov-officials still moves 1,862 entries over both phases and fec-individuals 1; the committed corpora move only the rules' and cases' new rows. Pinned by `Kim, Ae-Ra` and `Shin Ae-ra` in tests/v2/cases.py, 1.4.0's readings (wheel), and the cue's rows in `test_suffix_as_written_reads_a_joined_credential`. diff --git a/docs/design/rules/S3-credentials-written-in-joined-chunks.md b/docs/design/rules/S3-credentials-written-in-joined-chunks.md index a8d63a29..a2a6003d 100644 --- a/docs/design/rules/S3-credentials-written-in-joined-chunks.md +++ b/docs/design/rules/S3-credentials-written-in-joined-chunks.md @@ -52,7 +52,7 @@ **Except:** -- **A word joined by a hyphen and written the way a hyphenated name is, its first chunk in Title case and every other in Title case or lower case, is a name word**, whatever its chunks are listed as: that is how a hyphenated given name is written (`Ae-Ra`, `Ae-ra`, the Korean syllables `ae` and `ra` both being listed credentials), and a credential's chunks are written otherwise. A chunk with a capital past its first letter (`PhD`), a chunk in capitals, a word opening in lower case and a slash, which joins no name, leave the word to the joined reading's every-chunk test. +- **A word joined by a hyphen and written the way a hyphenated name is, its first chunk in Title case and every other in Title case or lower case, is a name word**, whatever its chunks are listed as: that is how a hyphenated given name is written (`Ae-Ra`, `Ae-ra`, the Korean syllables `ae` and `ra` both being listed credentials), and a credential's chunks are written otherwise. A chunk with a capital past its first letter (`PhD`), a chunk in capitals, a word opening in lower case and a slash anywhere in the word, a slash joining no name, leave the word to the joined reading's every-chunk test, and a word holding an interior period is read by its periods first. ``` "Kim, Ae-Ra" → given="Ae-Ra" "Shin Ae-ra" → family="Ae-ra" @@ -103,6 +103,10 @@ "Kim, AE-RA" → suffix="AE-RA" "kim, ae-ra" → suffix="ae-ra" ``` +- A credential whose chunks are all typed in Title case reads as the name word its writing shows, since the writing is the cue and it cannot tell the two apart; after a comma the name is reversed around it. + ``` + "John Smith, Phd-Md" → given="Phd-Md" + ``` - An unlisted credential of two letters written with its periods behind a full name reads as the family name, and the reading is reported: King's Counsel, which no vocabulary lists, is the case. ``` "John Smith K.C." → family="K.C." diff --git a/nameparser/_pipeline/_vocab.py b/nameparser/_pipeline/_vocab.py index 7c0a4dec..a692fd7f 100644 --- a/nameparser/_pipeline/_vocab.py +++ b/nameparser/_pipeline/_vocab.py @@ -591,7 +591,9 @@ def _joined_credential(text: str, lexicon: Lexicon) -> bool: # the caller found a joiner, so there are two chunks at least chunks = _JOINERS.split(text) whole = False # a chunk that is not a single ASCII character - named = "-" in text # every chunk so far in Title case + # every chunk so far written as a name's; a slash joins no name, so + # a word holding one takes no cue + named = "/" not in text for k, c in enumerate(chunks): if not (c and suffix_as_written(_normalize(c), c, lexicon)): return False @@ -601,8 +603,8 @@ def _joined_credential(text: str, lexicon: Lexicon) -> bool: whole = whole or len(bare) > 1 or not bare.isascii() # the first chunk in Title case, a later one in Title case or # lower case ('Ae-Ra', 'Ae-ra') - named = named and len(bare) > 1 and ( - bare[0].isupper() and bare[1:].islower() + named = named and ( + len(bare) > 1 and bare[0].isupper() and bare[1:].islower() or k > 0 and bare.islower()) # rules/S3: "Every chunk the vocabulary matches is a single ASCII # character" -- the numerals and the digit, which are about diff --git a/tests/v2/pipeline/test_vocab.py b/tests/v2/pipeline/test_vocab.py index a8f4df84..b8aa650c 100644 --- a/tests/v2/pipeline/test_vocab.py +++ b/tests/v2/pipeline/test_vocab.py @@ -425,6 +425,9 @@ def test_period_joined_vocab_retires_the_single_character_chunk() -> None: ("AE-RA", True), # capitals: the cue cannot read a name in them ("ae-ra", True), ("Ae/Ra", True), # a slash joins no name, so it takes no cue + ("Ae-Ra/Md", True), # nor does a word holding a slash anywhere + ("Md-v", False), # a one-letter tail in lower case is a name's + ("Phd-Md", False), # a credential typed in Title case: S3 Accepted ]) def test_suffix_as_written_reads_a_joined_credential( text: str, expected: bool) -> None: diff --git a/tests/v2/rules_budget.json b/tests/v2/rules_budget.json index 962aa24e..81a2242f 100644 --- a/tests/v2/rules_budget.json +++ b/tests/v2/rules_budget.json @@ -54,7 +54,7 @@ "R5": {"words": 250, "then_words": 154, "because_sentences": 2, "except": 0, "precedence": 0, "accepted": 0, "accepted_words": 0}, "S1": {"words": 274, "then_words": 26, "because_sentences": 1, "except": 2, "precedence": 2, "accepted": 0, "accepted_words": 0}, "S2": {"words": 641, "then_words": 103, "because_sentences": 1, "except": 0, "precedence": 7, "accepted": 3, "accepted_words": 214}, - "S3": {"words": 910, "then_words": 398, "because_sentences": 3, "except": 4, "precedence": 4, "accepted": 5, "accepted_words": 294}, + "S3": {"words": 928, "then_words": 398, "because_sentences": 3, "except": 4, "precedence": 4, "accepted": 6, "accepted_words": 334}, "S4": {"words": 298, "then_words": 156, "because_sentences": 2, "except": 0, "precedence": 3, "accepted": 0, "accepted_words": 0}, "S5": {"words": 632, "then_words": 317, "because_sentences": 1, "except": 2, "precedence": 1, "accepted": 0, "accepted_words": 0}, "S6": {"words": 337, "then_words": 121, "because_sentences": 2, "except": 0, "precedence": 2, "accepted": 0, "accepted_words": 0}, diff --git a/tools/differential/corpus_rules.jsonl b/tools/differential/corpus_rules.jsonl index 39a9ba1a..a16f1a7b 100644 --- a/tools/differential/corpus_rules.jsonl +++ b/tools/differential/corpus_rules.jsonl @@ -404,6 +404,7 @@ "John Smith, PhD XYZ" "John Smith, PhD vd DO" "John Smith, PhD-Md" +"John Smith, Phd-Md" "John Smith, Pu-Ph" "John Smith, RA Jones" "John Smith, V." diff --git a/tools/differential/snapshot.jsonl b/tools/differential/snapshot.jsonl index 91682885..715069d7 100644 --- a/tools/differential/snapshot.jsonl +++ b/tools/differential/snapshot.jsonl @@ -1731,6 +1731,7 @@ {"name": "John Smith, PhD XYZ", "given": "John", "family": "Smith", "suffix": "PhD XYZ", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. S."} {"name": "John Smith, PhD vd DO", "given": "John", "family": "Smith", "suffix": "PhD vd DO", "initials": "J. S.", "capitalized_force": "John Smith PhD VD DO"} {"name": "John Smith, PhD-Md", "given": "John", "family": "Smith", "suffix": "PhD-Md", "initials": "J. S.", "capitalized_force": "John Smith PhD-MD"} +{"name": "John Smith, Phd-Md", "given": "Phd-Md", "family": "John Smith", "initials": "P. J. S.", "capitalized_force": "PhD-Md John Smith"} {"name": "John Smith, Pu-Ph", "given": "Pu-Ph", "family": "John Smith", "initials": "P. J. S."} {"name": "John Smith, RA Jones", "title": "RA", "given": "Jones", "family": "John Smith", "initials": "J. J. S.", "capitalized_force": "Ra Jones John Smith"} {"name": "John Smith, RAI", "given": "John", "family": "Smith", "suffix": "RAI", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. S."}