From 5a6b7bbef36c27608bbc38f9d165819982aa1439 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sun, 18 Oct 2026 13:32:24 -0700 Subject: [PATCH 1/9] Read a one-case one-letter numeral as its capital, and no connective at a part's end (#721) 'john v' read suffix 'v' with no family while 'JOHN V' read family V: the capital carries the initial tag every trailing reader honors, and a lowercase 'v' was a plain suffix word. In a name written wholly in one case, classify now tags a one-letter roman numeral an initial as its capital is tagged (the name-level one-case fact, so a maiden clause's 'v' is one too). A lower/upper invariant over 1,197 one-case pairs (comma shapes, orders, clause, chain, credential) disagreed on 171 on master and on none now; it is a property test, control recorded. And, per Derek on the issue, a connective letter that ends its part is not reported conjunction-or-initial (rules/P3: it connects nothing); 'john i' reports suffix-or-name from the numeral fork instead, which retires #718's stand-down for a letter already reported as an initial. Docs: S10's Accepted clause for 'v' goes; P3 gains its exception and a boundary; S7's 'doe, jane v phd do' example reads middle 'v' as its capital twin does; decisions/S10 and decisions/P3 (the off switch no longer restores 2.3.0's one-case readings) record it. Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 2 +- docs/design/decisions/P3-connectives.md | 1 + .../decisions/S10-trailing-roman-numeral.md | 2 + docs/design/rules/P3-connectives.md | 5 +- .../rules/S10-trailing-roman-numeral.md | 21 ++------ ...S7-company-of-an-unambiguous-credential.md | 2 +- docs/release_log.rst | 2 +- nameparser/_pipeline/_assign.py | 26 ++------- nameparser/_pipeline/_classify.py | 34 ++++++++++-- tests/v2/cases.py | 20 +++---- tests/v2/pipeline/test_assemble.py | 7 +-- tests/v2/pipeline/test_pieces.py | 2 +- tests/v2/test_cases.py | 7 +-- tests/v2/test_facade.py | 18 ++++--- tests/v2/test_properties.py | 54 +++++++++++++++++++ tools/differential/corpus_rules.jsonl | 2 +- tools/differential/snapshot.jsonl | 14 ++--- 17 files changed, 143 insertions(+), 76 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 25ce3de2..68ac7ac3 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -173,7 +173,7 @@ The library has two layers: `nameparser/config/` (data) and `nameparser/parser.p **Sweep the forms a change can reach: comma shapes, then `name_order`.** No comma, a FULL name before the comma, a ONE-WORD name before the comma and NOTHING before the comma are four paths, and the last two are the misses — #429, #430 and #432 are all the one-word path disagreeing with the full-name path on inputs the full-name path reads correctly, and #613's PR review found `, PhD` and `, Jr.` (an empty surname field, as CSV data writes it) read as given and title by a decision that returned early on an empty head, after two review rounds whose every grid had a name before the comma. Put an empty head and a nickname-only head (`(Bob), PhD`) in every comma grid. Count the one-word path the way C1 counts, which is neither tokens nor pieces: a particle run and the one word it attaches to are one word (#575), a connective join and a bound given-name pair count as their words (#613, so `Ortega y Gasset, Dr.` is a whole name), and a suffix beside the surname counts as no name word at all. #603's review found the suffix half (`Vega y Lopez Jr., PhD Jones` lost its family where `Vega y Lopez, PhD Jones` kept it); #613 then reversed the connective half, and both now read given 'Vega', middle 'y', family 'Lopez' — so test a particle surname, a connective surname and a surname with a suffix, each before a comma. For orders the sweep already exists (`tests/v2/test_cases.py` runs every row under all three) but carries one assertion, R2's family partition, so it checks nothing a new change moves; coverage there has been vacuous before (PR #394's review found the suite passed with `name_order` discarded from grouping). Parse your change's names in each comma shape and each order, and read the ones you did not predict. And read a generated grid's moves by SHAPE, not by corpus membership: #614's first design passed the suite, the five gates and a fingerprint of every corpus name, because its grid moves were sorted into 'corpus name moved' and 'garbage', and the review found the moves that mattered among the 'garbage' -- a bare salutation (`Mr. and Mrs.` lost its connective to the family name) and a name with one trailing word (`Freiherr von Berg X.Y.Z.`). Pull out the head-plus-one-word slice of the grid and a salutation with nothing after it, and read every move there. Two more were the second review's (#614): no grid put TWO words that are both a title and a particle at the head (`Freiherr St van Berg MA`, where an index into the pieces was applied to a merged view of them -- `TITLES ∩ particles_ambiguous` is `{freiherr, st}`, so put both in every head list, each in its bare and its period-marked spelling -- #637's grid had `St` and `Freiherr` alone and missed that `St.` and `Freiherr.` are declined by the bound join and then attached by P6), and the grids compared ambiguity KINDS, so a report telling the caller the wrong field (`Freiherr von Berg Ma`, 'read as a given name' of a word in the family) passed every pin. #701's review added one: put a title/suffix dual (`Ms`, `Sr.`, `Cpl`) SECOND in a title run, behind an ordinary title. A stop that asks only "is this word suffix vocabulary" passes every row that opens with the dual and fails `Rev. Sr. Smith, Mary`; it was found by comparing `T Smith, John` against `T John Smith` over every title word, and a comma-versus-comma-less comparison is the cheap check for any rule that moves a word between the two slots. Compare each report's `detail` against the field its tokens land in -- since #626 `test_a_report_names_the_field_its_word_lands_in` (tests/v2/test_cases.py) does it over every non-locale case row under all three orders (a row carrying its own policy only as declared), for the phrasings its `_FIELD_CLAIMS` lists -- so a row with the shape is the test only where the report's wording is listed: a new emitter's wording goes in that list, and segment's report on a part past the second comma names no field (#629). #626's own first grid then missed an emitter for want of one comma shape (`, Ma Dr.`: an empty head, an ambiguous word, a trailing title, which H1 moves to the family): a trailing title belongs in every empty-head comma grid too. And #620's: an exclusion list in code being replaced is a set of rules, not noise -- #614's merged copy excluded every word the trailing read weighs from its units, the first #620 prototype dropped the list as a symptom of the copy, and its review found the list had been keeping positional counting's rule that a member in front counts as a word (`Freiherr von Berg MA X.Y.Z.` lost its suffix). Before deleting one, mutate each entry out and read what moves. -**Then sweep the spellings: a class reached by vocabulary is also reached by SHAPE.** A test that uses only the listed spelling of a word class walks only the vocabulary path, and the shape path is a separate branch that can be missing while every test passes. Roman numerals are the sharpest case: `ii`, `iii` and `iv` are suffix vocabulary; `i` and `v` are vocabulary too but initial-shaped, so `is_suffix_piece` vetoes them and only the numeral fork reads them; and `vi` through `x` are in no list at all, read by the fork's shape test alone (checked 2026-10-04 against `Lexicon.default()`) -- which stops at `x`: `_vocab._ROMAN` matches I to X, so `John Smith XI` reads `XI` as a name word (re-checked 2026-10-07, #614's /simplify, which found a docstring claiming "any length"). And the fork counts a ONE-letter numeral besides, since #718 (rules/S10): it is the suffix only where the name has a word to spare, so `John V` and `John Smith V` take different branches of one fork, while a one-case lowercase `v`, a listed suffix word, never reaches it and is not counted (`john v` keeps suffix 'v' where `john x` and `john i` read family -- `i` arrives as P3's connective, read as an initial) -- put a head-plus-numeral row beside every words-to-spare row. So a change that handles `V` and `III` can still miss `VI` — #610 found exactly that, the maiden take's candidate loop admitting a shape-only numeral in one position where the peel reads it in two (`Jane Doe nee Smith VI Prof.` kept maiden `Smith VI` while `V` and `III` worked). The same split runs through the credentials (`MA` listed, `X.Y.Z.` by the dotted shape, `XYZ` by the caps shape, and `Ph. D.` merged by group into one flagged piece the peel's walk never holds -- #603 found #602's run-start predicate refusing it too, so `Smith, John Ph. D. Jones` kept middle 'Jones' where `Smith, John PhD Jones` read suffix 'PhD Jones') and the titles (`Prof.` period-marked and read at the end of a name, `Prof` bare and a name word there). When a change touches a word class, test one spelling from each path. The same holds for a separator's SPACING: a comma (or any punctuation) can stand alone, glue to the word before it or glue to the word after, and each is a different token stream. #653's first fix swept `(née, Jones)` and missed `(née , Jones)`, where the kept comma became a word after the marker and M3 read the clause as maiden ', Jones'; its review found it. Put all three spacings in any grid around a separator. +**Then sweep the spellings: a class reached by vocabulary is also reached by SHAPE.** A test that uses only the listed spelling of a word class walks only the vocabulary path, and the shape path is a separate branch that can be missing while every test passes. Roman numerals are the sharpest case: `ii`, `iii` and `iv` are suffix vocabulary; `i` and `v` are vocabulary too but initial-shaped, so `is_suffix_piece` vetoes them and only the numeral fork reads them; and `vi` through `x` are in no list at all, read by the fork's shape test alone (checked 2026-10-04 against `Lexicon.default()`) -- which stops at `x`: `_vocab._ROMAN` matches I to X, so `John Smith XI` reads `XI` as a name word (re-checked 2026-10-07, #614's /simplify, which found a docstring claiming "any length"). And the fork counts a ONE-letter numeral besides, since #718 (rules/S10): it is the suffix only where the name has a word to spare, so `John V` and `John Smith V` take different branches of one fork, a one-case name being counted like capitals whatever the letter -- since #721 a one-case one-letter numeral carries the `initial` tag its capital does, so `john v` reads as `JOHN V` does in every shape (a lower/upper grid over comma shapes, orders, a maiden clause and a particle chain found 171 disagreeing pairs on master and none after) -- put a head-plus-numeral row beside every words-to-spare row. So a change that handles `V` and `III` can still miss `VI` — #610 found exactly that, the maiden take's candidate loop admitting a shape-only numeral in one position where the peel reads it in two (`Jane Doe nee Smith VI Prof.` kept maiden `Smith VI` while `V` and `III` worked). The same split runs through the credentials (`MA` listed, `X.Y.Z.` by the dotted shape, `XYZ` by the caps shape, and `Ph. D.` merged by group into one flagged piece the peel's walk never holds -- #603 found #602's run-start predicate refusing it too, so `Smith, John Ph. D. Jones` kept middle 'Jones' where `Smith, John PhD Jones` read suffix 'PhD Jones') and the titles (`Prof.` period-marked and read at the end of a name, `Prof` bare and a name word there). When a change touches a word class, test one spelling from each path. The same holds for a separator's SPACING: a comma (or any punctuation) can stand alone, glue to the word before it or glue to the word after, and each is a different token stream. #653's first fix swept `(née, Jones)` and missed `(née , Jones)`, where the kept comma became a word after the marker and M3 read the clause as maiden ', Jones'; its review found it. Put all three spacings in any grid around a separator. **And the encoding: a letter class written `[^\W\d_]` or tested with `isalpha()` is blind to a combining mark.** Python's `\w` and `isalpha()` do not match Mn, so a letter written as base plus accent (NFD) ends a word early; the vocabulary path is immune because `_normalize` NFC-folds, the shape path is not. #652: `Dr. inž. Janez Novak` read `inž.` as the given name when the ž was decomposed, and `Janez Novak, ŽŠČ` reversed, in two families that share the blind spot — the letter-shape regexes (`period_abbreviation`, `_DOTTED`, `_CHUNKED`, `_PAIRED_INITIALS`), now spelled with `_vocab._LETTER` (a letter plus the general combining blocks; `test_regex_sync` compares the two `_MARKS` strings and pins `period_abbreviation` against config's copy), and the caps predicates, which NFC-compose a non-ASCII word before `isalpha()` (`is_caps_word`, behind C-level `isalpha()`/`is_normalized` tests, so already-composed text pays no frame — measured 2026-10-12, `Smith, José` was 193 frames before and after; the first version called it for every non-ASCII word and cost one frame each, and no guard saw it because the reference name is ASCII). A width test must count letters, not code points (`composed_len`): admitting a decomposed word to the caps set put `John Smith, MÁ` on the pair exception's wrong side, and the review's grid found it where the rows (all three letters long) did not — put a two-letter word in every caps grid. The initial shape (`_INITIAL`, `\w\.`, v1-verbatim and public) is not widened but asked of the COMPOSED word, in `is_initial_shaped` and in `_render`'s two copies; and the single-letter connective tests count composed letters, since a decomposed `й` is two code points (#665, where `É. y É. Smith` and `John й Smith` read differently decomposed). Compose the WHOLE predicate, not one half: `is_initial` is a shape ANDed with the #320 script veto, and composing the shape alone made a decomposed `씨.` an initial, since decomposed hangul is conjoining jamo outside every script range (#665's first version; `in_initialless_script` composes too now). Not covered, deliberately: the marks only some scripts write (Indic vowel signs, Arabic and Hebrew points; rules/H2's Accepted clause). The guard is `test_a_decomposed_name_renders_as_its_composed_twin` over the CLDR corpus plus `test_a_decomposed_abbreviation_reads_as_its_composed_twin` (rows × three orders × EVERYWHERE); the corpus guard reaches a shape only if some corpus name does, which is how the caps shape was missed until review, so when adding a shape rule add a row there. Four of #652's rows and two of #665's pass with every fix reverted (measured) and are parity pins, and the test compares ambiguity kinds and `initials()` too, since #665's `John й Smith` differed in its report; `_DOTTED` has no observable input and is pinned by a direct call. diff --git a/docs/design/decisions/P3-connectives.md b/docs/design/decisions/P3-connectives.md index 6f030eff..feadb514 100644 --- a/docs/design/decisions/P3-connectives.md +++ b/docs/design/decisions/P3-connectives.md @@ -62,3 +62,4 @@ - 2026-10-07 #614 — CROSS-REFERENCE: P3's joins are made before S2's trailing run is read, so the run counts a joined part as one name word, as this rule says every count does (`Juan y Garcia X.Y.Z.` keeps family 'X.Y.Z.'), and a salutation keeps its connective (`Mr. and Mrs.` reads title 'Mr. and Mrs.'). #614's first design read before these joins too and broke both; #614's review reversed it (decisions/S2, this date). - Provenance: the single-letter-connective guard is v1's fix for Google Code issue 11 ("john e smith", 2013, commit 33676c9) — the "#11" citations that circulated pointed at a GitHub accident, not the real source. Recorded so the archaeology stays done. +- 2026-10-18, #721 — THE OFF SWITCH NO LONGER RESTORES THE ONE-CASE ROLE MOVERS. With `i` removed from the conjunctions, a one-case `i` is a one-letter roman numeral, which since #721 is the initial its capital is (rules/S10), so `rovira, i`, `john smith i jr` and `maier, amy i, jr.` read with the switch off exactly as with it on, and as their all-caps twins read either way — where 2.3.0 read them as suffixes, the disagreement with the capitals this paragraph's 2026-09-20 entry measured. The property the off-switch grid holds is unchanged, its exemption (2) — the readings disagree about the letter being an initial — covering it. And a connective letter ending its part is not reported (rules/P3, "unless nothing of its part stands to its right"): the fork had no other way to go. diff --git a/docs/design/decisions/S10-trailing-roman-numeral.md b/docs/design/decisions/S10-trailing-roman-numeral.md index c9a3d796..271fff8c 100644 --- a/docs/design/decisions/S10-trailing-roman-numeral.md +++ b/docs/design/decisions/S10-trailing-roman-numeral.md @@ -10,3 +10,5 @@ ONE STAND-DOWN, FOUND BY INV10 rather than designed. In a one-case name the connective `i` reaches the fork as a letter classify has already reported as `conjunction-or-initial`, read as an initial, and the new report put a second report on the same token: 138 rows of the connective grid (measured on this branch with the `reported_as_initial` check out of assign's numeral report, a dated snapshot) in tests/v2/test_properties.py (`AND ROVIRA I`, `HENRY I`) tripped INV10's kind pair. The two reports did not contradict each other — both read the letter as a name word — but they named one reading twice, so the numeral's report stands down where the initial reading is already reported, the shape of rules/S7's "a member its own capitals already made the credential reports nothing new". The guard was left untouched: its kind pair still means "two readings" everywhere else, and narrowing it to suit this change would have retaken its recorded control. `HENRY I` therefore reports `conjunction-or-initial` alone, and moved in test_assemble's contrast list from the withdrawn side to the side where the fork resolved to a name word. - 2026-10-18, #718 review — THE COUNT IS NOT ASKED IN THE GIVEN PART AFTER A FAMILY COMMA. The code review's grid (family-comma and empty-head shapes behind a title-particle, about 558,000 parses) found 2,880 parses where the first form moved a numeral from the suffix into the given or family name with no report: `Smith, Freiherr von Berg V` read given 'von Berg V', `, Freiherr von V` family 'von V', `Smith, St van V` given 'van V', where `Smith, von V` and `Smith, Freiherr John V` kept suffix V. The cause was not a choice: assign's walk over that part never counts the numeral (rules/S4: the given part's trailing slot "is read without it (S5)"), but the particle chain's bound there asks `tail_reading`, which the count had reached, so the chain took a word assign would have read as the suffix. The bound now reads with the count off (`trailing_run_start`, `count_numeral=False`), and all four read as they did on master (2.2.0 onward, since #424; 1.4.0 read the numeral into the name). The class is the one the AGENTS.md gotcha on group-side predictors of the given part names, met from the other side: the predictor did not change, the peel it asks did. The S4 analogue, `Smith, Freiherr von Berg Ma` → given 'von Berg Ma', is master's reading and is left alone here. - 2026-10-18, #718's PR review — TWO OF THE THREE REPORT SITES ARE ONE, AND THE CLAUSE ASKS THE CASE. The test-coverage review mutated each new emitter out in a copy of the tree. Removing the particle chain's numeral report moved no field and no kind — assign's pick loop reported the same `suffix-or-name`, worded from the final role ('read as a family name') — so the chain's branch was only choosing a sentence, and it is gone: a declined numeral stays one of the read's picks, and assign alone reports it, the decision having been the read's and not the chain's. The maiden clause's stand-down on a letter already reported as an initial could not fire at all (classify reads no connective inside a clause), so that call is gone too. And the clause's report on a numeral its take gave up did not ask the case: `Jane Doe nee Smith vi` and `x` reported where every other path had gone silent, the drift mechanisms.md#ONE-PREDICATE-PER-QUESTION names; the take now asks `numeral_settled_by_case` as assign does, `VI` still reporting beside them. Five rows the first draft labelled `fix(#718)` read field for field as 1.4.0 does and are `parity`, as are the two Freiherr rows added here; a restored 1.4.0 reading is parity by the table's own definition (rules/S4's `bare_surname_is_not_a_credential` is the precedent). +- 2026-10-18 (Derek), #721 — A ONE-CASE ONE-LETTER NUMERAL IS THE INITIAL ITS CAPITAL IS, AND A LETTER ENDING ITS PART IS NO CONNECTIVE. Two Accepted gaps of the entries above close. First, `john v` read suffix 'v' with no family while `JOHN V` and `john x` were counted: `V` carries the `initial` tag (a bare capital) and every reader of the trailing run honors it — the suffix veto, the particle chain, the credential run — while a lowercase `v` was a plain listed suffix word. A first fix routed a one-case `v` to the numeral fork at the walk's last position only, and a lower/upper invariant (every one-case name parses as its capitals, fields and report kinds, over 19 shapes × 7 numerals × 3 heads × 3 orders, 1,197 pairs) showed it too shallow: 171 pairs disagreed on master, 33 still did (`john v phd`, `john v, phd`, `freiherr von smith v`). The fix that holds tags the letter where its capital is tagged: in a name written in one case, a one-letter roman numeral is an `initial` (classify, reading the NAME-level one-case fact so a maiden clause's `v` is one too, as #516's caps branch reads it), and the grid then shows 0. Second, Derek on the issue: `john i` reporting `conjunction-or-initial` is wrong "because there's nothing for a conjunction to join to there". P3 already said a connective with nothing to its right connects nothing; classify's report now skips a letter that is the last own word of its part — before a comma, a marker's clause or the end. Such a letter is the numeral fork's, which reports `suffix-or-name` (`john i`, `HENRY I`, `carod i`), and the stand-down the entry above added for a letter already reported as an initial is gone: no declined numeral can carry that report any more, measured by disabling it (the same eight failures with and without, INV10 passing both ways). Two comma rows go silent rather than move — `rovira, i` and `maier, amy i, jr.` — since the comma path has never reported the numeral (decisions/C1, "Not done"), and `jones, john e`, a corpus name, loses its report the same way. + WHAT MOVES, the snapshot against master at 2bfa9533: six corpus names, `john v` (family 'v'), `john smith v` and `larry james edward johnson v` (gain the report), `doe, jane v phd do` (middle 'v', suffix 'phd do', as `DOE, JANE V PHD DO` and `Doe, Jane V PhD Do` read; rules/S7's example keeps its input, whose point — `do` anchored by the degree in front — is unchanged), `john i` (report kind), `jones, john e` (report dropped). And decisions/P3's off switch no longer restores 2.3.0's three one-case readings (`rovira, i` suffix 'i', and the two beside it): with `i` out of the conjunctions the letter is a one-letter numeral, so it is the initial `I` is there, switch on or off — those 2.3.0 readings were themselves a lower/upper disagreement. The switch still unjoins the link (`Josep Carod i Rovira`). diff --git a/docs/design/rules/P3-connectives.md b/docs/design/rules/P3-connectives.md index 505fcb69..43ead296 100644 --- a/docs/design/rules/P3-connectives.md +++ b/docs/design/rules/P3-connectives.md @@ -17,7 +17,9 @@ P3. Rationale: connective words ("y", "of the") bind name words into marks as reading both ways. A letter the vocabulary marks as reading both ways, read as an initial in a name written wholly in one case, is a call that could - have gone the other way, and is reported. + have gone the other way, and is reported — unless nothing of its + part stands to its right, where it connects nothing and there is + no other way for the call to go; S10 reads such a letter. The joined part is ONE name word wherever another rule counts them, so a rule taking "one name word" takes the whole join and never half of it — except C1's count of the words before a comma, @@ -74,6 +76,7 @@ P3. Rationale: connective words ("y", "of the") bind name words into "Josep i Rovira" → middle="i" "John Quincy Smith i" → suffix="i" "Carod i" → suffix="i" + "carod i" → ambiguities=("suffix-or-name",) · boundary "Josep Lluis Carod i III" → suffix="i III" "Carod y de Rovira i" → middle="y" "John й Smith" → middle="й" diff --git a/docs/design/rules/S10-trailing-roman-numeral.md b/docs/design/rules/S10-trailing-roman-numeral.md index a6ba308f..e1d93a43 100644 --- a/docs/design/rules/S10-trailing-roman-numeral.md +++ b/docs/design/rules/S10-trailing-roman-numeral.md @@ -15,9 +15,7 @@ S10. Rationale: a roman numeral ending a name is its generation, being no word to spare, and either reading is reported, as either reading of an ambiguous acronym is. A numeral read by its shape alone reports too; a numeral the vocabulary lists in two - or more letters is not in question and reports nothing, and a - letter already reported as read as an initial reports nothing - new. Written case + or more letters is not in question and reports nothing. Written case settles the fork in one way only: a numeral written in lower case in a name of mixed case is the generation and reports nothing, since nobody writes an initial that way. A name written @@ -43,21 +41,12 @@ S10. Rationale: a roman numeral ending a name is its generation, "John Smith V." → family="V." · boundary "John J. V" → family="V" · boundary "John Smith XI" → family="XI" · boundary - Accepted: a lower-case `v` in a name written wholly in lower - case is read as the suffix word the vocabulary lists, words to - spare or not, and reports nothing, where the same letter in - capitals is counted and reported; `i`, which is also a - connective, is read as an initial there (P3) and counted. - "john v" → family="" - "john smith v" → ambiguities=() + "john v" → family="v" + "john smith v" → ambiguities=("suffix-or-name",) "john i" → family="i" - "john i" → ambiguities=("conjunction-or-initial",) - Accepted: inside a maiden marker's clause the connective is not - read (M2), so a one-case `i` there is the suffix word the - vocabulary lists, as `v` is. - "john nee smith i" → suffix="i" + "john i" → ambiguities=("suffix-or-name",) Accepted: a numeral with another suffix word behind it is not the last word of the name and is read as any word there is, so a one-letter numeral before a credential is the family name. "John Smith V PhD" → family="V" - history: decisions/S10 · interacts: S2, S4, S7, C1, P2, P3, M2, H1, H3 · implemented: nameparser/_pipeline/_assign.py, nameparser/_pipeline/_group.py, nameparser/_pipeline/_pieces.py, nameparser/_pipeline/_vocab.py + history: decisions/S10 · interacts: S2, S4, S7, C1, P2, P3, M2, H1, H3 · implemented: nameparser/_pipeline/_assign.py, nameparser/_pipeline/_classify.py, nameparser/_pipeline/_group.py, nameparser/_pipeline/_pieces.py, nameparser/_pipeline/_vocab.py diff --git a/docs/design/rules/S7-company-of-an-unambiguous-credential.md b/docs/design/rules/S7-company-of-an-unambiguous-credential.md index 94ac75e9..ffad9fa8 100644 --- a/docs/design/rules/S7-company-of-an-unambiguous-credential.md +++ b/docs/design/rules/S7-company-of-an-unambiguous-credential.md @@ -44,7 +44,7 @@ S7. Rationale: post-nominals are written as a run, and an "Smith, PhD Ma" → ambiguities=("suffix-or-name",) "Smith, MD PhD Ma" → given="PhD" · boundary "Doe, Jane PhD MEng" → suffix="PhD MEng" - "doe, jane v phd do" → suffix="v phd do" + "doe, jane v phd do" → suffix="phd do" Accepted: a name-leaning member with nothing in front of it to speak for it ends the name, words to spare or not, and whatever stands in front of it is name text. Where a credential does diff --git a/docs/release_log.rst b/docs/release_log.rst index 96d99047..a02be1e6 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -14,7 +14,7 @@ Release Log - **Fix a surname particle such as d' or al- reading as a middle name when a listing puts it after the given name.** ``Ivoi, Paul d'`` gives last ``d' Ivoi``, where 1.4.0 through 2.4.2 gave middle ``d'``. The Arabic article reads the same in its plain and assimilated spellings (``al-``, ``az-``, ``aš-``). A particle written joined to the surname, as in ``Paul d'Ivoi``, is unchanged. :issue:`700` - **Fix case repair capitalizing the Dutch particle 't.** ``jacobus henricus van 't hoff`` repairs to ``Jacobus Henricus van 't Hoff``, where 1.4.0 through 2.4.2 gave ``van 'T Hoff``. :issue:`700` - **Add Gov as a title.** ``Gov Smith`` gives title ``Gov``, last ``Smith``, where 1.4.0 through 2.4.2 gave first ``Gov``, and ``Lt. Gov. Smith, John`` gives title ``Lt. Gov.`` with the change above. :issue:`701` - - **Fix a capital V, I or X after a lone first name being read as a generational suffix, leaving no last name.** ``John V`` gives first ``John``, last ``V``, and ``Malcolm X`` last ``X``, where 2.0.0 through 2.4.2 gave suffix ``V`` and ``X`` with no last name, restoring 1.4.0's reading. A title is no word to spare, so ``Dr. Smith V`` gives first ``Smith``, last ``V`` (1.4.0's reading too) where 2.0.0 through 2.4.2 gave last ``Smith``, suffix ``V``; a name with a word to spare keeps the suffix, and ``John Smith V`` still gives suffix ``V``. :issue:`718` + - **Fix a capital V, I or X after a lone first name being read as a generational suffix, leaving no last name.** ``John V`` gives first ``John``, last ``V``, and ``Malcolm X`` last ``X``, where 2.0.0 through 2.4.2 gave suffix ``V`` and ``X`` with no last name, restoring 1.4.0's reading. A title is no word to spare, so ``Dr. Smith V`` gives first ``Smith``, last ``V`` (1.4.0's reading too) where 2.0.0 through 2.4.2 gave last ``Smith``, suffix ``V``; a name with a word to spare keeps the suffix, and ``John Smith V`` still gives suffix ``V``. A name written all in lower case reads the same way (``john v`` gives last ``v``). :issue:`718`, :issue:`721` * 2.4.2 - October 13, 2026 diff --git a/nameparser/_pipeline/_assign.py b/nameparser/_pipeline/_assign.py index f4a885b6..9fdbcce0 100644 --- a/nameparser/_pipeline/_assign.py +++ b/nameparser/_pipeline/_assign.py @@ -306,21 +306,6 @@ def _placed(read: TailRead, rest: list[int], peel.absorbed) -# rules/S10: "a letter already reported as read as an initial reports -# nothing new" -- asked only by the numeral fork's declined pick -# (#718). The maiden clause's report on the numeral it keeps cannot -# meet the case: classify reads no connective inside a clause -def _reported_as_initial(ambiguities: Sequence[PendingAmbiguity], - index: int) -> bool: - """Whether classify's connective-or-initial fork already reported - the token at `index`, reading it as an initial. Declining the - numeral reads the letter as that same name word, so a second - report would name one reading twice. A rare shape, so the scan - costs ordinary names nothing.""" - return any(a.kind is AmbiguityKind.CONJUNCTION_OR_INITIAL - and index in a.indices for a in ambiguities) - - def _assign_main(seg_idx: int, state: ParseState, tokens: list[WorkToken], ambiguities: list[PendingAmbiguity], @@ -550,12 +535,11 @@ def _assign_main(seg_idx: int, state: ParseState, # declined pick (#718), which carries neither credential tag # (the acronym fork's picks carry one) and is never a suffix if token.tags.isdisjoint(_AMBIGUOUS_CREDENTIAL_TAGS): - if not _reported_as_initial(ambiguities, piece[0]): - ambiguities.append(PendingAmbiguity( - AmbiguityKind.SUFFIX_OR_NAME, - f"{token.text!r} is a roman numeral, but the name " - f"has no other word to spare for it; read as ", piece, - field_tail=" rather than a generational suffix")) + ambiguities.append(PendingAmbiguity( + AmbiguityKind.SUFFIX_OR_NAME, + f"{token.text!r} is a roman numeral, but the name has " + f"no other word to spare for it; read as ", piece, + field_tail=" rather than a generational suffix")) continue declined = ("a name part" if token.role is Role.SUFFIX else "a post-nominal") diff --git a/nameparser/_pipeline/_classify.py b/nameparser/_pipeline/_classify.py index 9017b519..fbecafcd 100644 --- a/nameparser/_pipeline/_classify.py +++ b/nameparser/_pipeline/_classify.py @@ -1,6 +1,7 @@ """Stage: classify. -Consumes: tokens, comma_offsets (with token roles, the two halves of +Consumes: tokens, segments (where a part ends, for the connective +fork's report, #721), comma_offsets (with token roles, the two halves of the structural-boundary test the marker pass applies -- see _vocab.tag_marker_runs), and one_case where an earlier stage recorded it -- segment writes it lazily where C2's flag on a part past the @@ -63,6 +64,7 @@ from nameparser._types import AmbiguityKind, Role from nameparser._pipeline._vocab import ( caps_shape_candidate, composed_len, is_initial, is_one_case, + is_single_letter_numeral, period_joined_vocab, suffix_as_written, tag_marker_runs, ) from nameparser._pipeline._pieces import own_words @@ -106,7 +108,8 @@ def _tags_for(token: WorkToken, n: str, state: ParseState, clause cut, never from the name's own writing (#516 review round, a second reviewer's finding). `single_letter_connective` below keeps `one_case_own`: that fork is genuinely about the OWN-WORDS - span, and reads a clause's word as no evidence on purpose.""" + span, and reads a clause's word as no evidence on purpose. The + one-case numeral initial reads `one_case` too (#721).""" lex = state.lexicon tags = set(token.tags) if marker_tag is not None: @@ -159,7 +162,17 @@ def _tags_for(token: WorkToken, n: str, state: ParseState, # the mixed-case rule, unchanged. v1's is_conjunction excludes # initials: 'e.' in 'john e. smith' is a middle initial, not # the Spanish conjunction 'e' - initial = is_initial(token.text) + # rules/S10: "A name written wholly in one case says nothing + # about any letter in it" -- so a one-letter roman numeral + # there is the initial its capital is, and every reader that + # vetoes 'V' vetoes 'v' too (#721). The NAME-level fact, as + # #516's caps branch reads it: a capital 'V' in a maiden clause + # is an initial, so a one-case 'v' there is one too. `len` + # first, C-level, so only one-character words of a one-case + # name pay the call + initial = is_initial(token.text) or ( + one_case and len(token.text) == 1 + and is_single_letter_numeral(token.text)) if n in lex.conjunctions and not initial: tags.add("conjunction") if initial: @@ -305,6 +318,17 @@ def classify(state: ParseState) -> ParseState: # rather than at the escape itself, which runs before tokenize and # so has no token index to point at. ambiguities = list(state.ambiguities) + # rules/P3: "A connective with nothing to its right is connecting + # nothing" -- so a letter ending its part (the last own word before + # a comma, a marker's clause or the end of the name) offers the + # fork no connective reading, and it is not reported here (#721); + # the numeral fork reads such a letter instead (rules/S10). Built + # only for a one-case name, the one the emitter below can reach. + ends = frozenset( + own_seg[-1] for seg in state.segments + if (own_seg := [j for j in seg + if j < clause_at and tokens[j].role is None]) + ) if one_case else frozenset() for i, token in enumerate(tokens): if (token.role is Role.NICKNAME and AMBIGUOUS_ACRONYM_TAG in token.tags): @@ -339,11 +363,13 @@ def classify(state: ParseState) -> ParseState: # clause's words were never eligible for the fork, so they must # never be eligible to report either. Emitted at the decision # site (mechanisms.md#AMBIGUITY-AT-THE-DECISION-SITE), per - # token -- 'e and e' reports twice. + # token -- 'e and e e' reports twice, the last 'e' connecting + # nothing (#721). if (one_case and "initial" in token.tags and folded[i] in state.lexicon.conjunctions_ambiguous and folded[i] in state.lexicon.conjunctions and i < clause_at and token.role is None + and i not in ends and (len(token.text) == 1 or not token.text.isascii() and composed_len(token.text) == 1)): diff --git a/tests/v2/cases.py b/tests/v2/cases.py index 7a7328ef..06fb90a6 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -1574,7 +1574,8 @@ def _check_cjk_shape_purity(self) -> None: "read middle 'ed'"), Case("an_anchored_particle_member_outranks_p6", "doe, jane v phd do", - {"given": "jane", "family": "doe", "suffix": "v phd do"}, + {"given": "jane", "middle": "v", "family": "doe", + "suffix": "phd do"}, classification="fix(#436/#437)", ambiguities=("suffix-or-name",), notes="'do' is a member AND a particle, and written in one " @@ -1583,7 +1584,9 @@ def _check_cjk_shape_purity(self) -> None: "in front outranks the attachment as the capitals do " "(Derek, 2026-09-27). The roles are 1.4.0's; 2.3.0 " "read family 'do doe', and 1.4.0 wrote the suffix " - "'v, phd, do'", + "'v, phd, do'. The 'v' is the one-case initial its " + "capital is since #721, as 'DOE, JANE V PHD DO' reads " + "middle 'V' (rules/S10)", shape=2), Case("by_design_trailing_mc_reads_as_a_credential", "Donald Mc", {"given": "Donald", "suffix": "Mc"}, @@ -4499,7 +4502,6 @@ def _check_cjk_shape_purity(self) -> None: "rovira, i", {"given": "i", "family": "rovira"}, classification="fix(#397)", - ambiguities=("conjunction-or-initial",), notes="written wholly in lower case, so the letter reads as " "an INITIAL and takes the given slot the comma leaves " "for it. 2.3.0 read family 'rovira' with suffix 'i'; " @@ -4532,7 +4534,6 @@ def _check_cjk_shape_purity(self) -> None: {"given": "amy", "middle": "i", "family": "maier", "suffix": "jr."}, classification="fix(#397)", - ambiguities=("conjunction-or-initial",), notes="the third comma shape, and the one whose MIXED-CASE " "twin is already in this table: 'Maier, Amy I, Jr.' " "reads middle 'I' and does not move, the writing " @@ -7513,13 +7514,14 @@ def _check_cjk_shape_purity(self) -> None: "declined numeral is still the read's pick, reported " "from the role it ends in. 2.2.0 to 2.4.2 read suffix " "'V'; 1.4.0 read last 'von Richthofen V'"), - Case("a_lone_numeral_the_chain_takes_already_an_initial", + Case("a_lone_one_case_connective_numeral_the_chain_takes_reports", "FREIHERR VON RICHTHOFEN I", {"title": "FREIHERR", "family": "VON RICHTHOFEN I"}, - ambiguities=("conjunction-or-initial", "particle-or-given"), - notes="the row above in one case with the connective 'I', " - "which classify has already reported as an initial: the " - "declined numeral reports nothing new (S10)"), + ambiguities=("particle-or-given", "suffix-or-name"), + notes="the row above in one case with the connective 'I'. " + "Ending the name it connects nothing, so classify does " + "not report it (P3, #721), and the declined numeral is " + "what reports, as for 'V'"), Case("a_lowercase_shape_numeral_in_a_mixed_case_name_is_settled", "John x", {"given": "John", "suffix": "x"}, ambiguities=("given-or-family",), classification="fix(#718)", diff --git a/tests/v2/pipeline/test_assemble.py b/tests/v2/pipeline/test_assemble.py index 425e67d9..1d2041fe 100644 --- a/tests/v2/pipeline/test_assemble.py +++ b/tests/v2/pipeline/test_assemble.py @@ -164,10 +164,11 @@ def test_a_connective_or_initial_report_is_withdrawn_where_the_role_is_a_suffix( # the contrast that keeps the withdrawal honest: where the parse # roles the same letter a NAME word the fork DID resolve to one # of its branches, and the report stands - # -- 'HENRY I' among them since #718: with no word to spare the - # letter is the family name (rules/S10), where it was the suffix + # -- each with a word to the letter's right: ending its part, as + # in 'HENRY I', the letter connects nothing and classify reports + # nothing (rules/P3, #721) for text in ("JOSEP CAROD I ROVIRA", "josep carod i rovira", - "JOHN I SMITH", "JOSEP CAROD I ROVIRA III", "HENRY I"): + "JOHN I SMITH", "JOSEP CAROD I ROVIRA III"): name = parse(text) assert "conjunction-or-initial" in [ a.kind.value for a in name.ambiguities], text diff --git a/tests/v2/pipeline/test_pieces.py b/tests/v2/pipeline/test_pieces.py index 08dcc9fa..79bb3198 100644 --- a/tests/v2/pipeline/test_pieces.py +++ b/tests/v2/pipeline/test_pieces.py @@ -517,7 +517,7 @@ def test_a_particle_in_suffix_vocabulary_anchors_nothing( # MEMBER is still anchored by a credential in front of it. n = parse(text) assert {k: v for k, v in n.as_dict().items() if v} == fields - assert parse("doe, jane v phd do").suffix == "v phd do" + assert parse("doe, jane v phd do").suffix == "phd do" def test_anchor_in_reach_is_false_only_where_the_pass_is() -> None: diff --git a/tests/v2/test_cases.py b/tests/v2/test_cases.py index 1cc3e0ab..e8ee387a 100644 --- a/tests/v2/test_cases.py +++ b/tests/v2/test_cases.py @@ -190,9 +190,10 @@ def test_the_field_sweep_sees_the_claims_it_checks() -> None: # #718's declined-numeral report, which names the field the letter # was read into, on those rows under each order; 1191 -> 1210 the # same day: the PR review's rows (the chain-taken, family-first, - # lowercase and maiden-clause numerals). - assert claims == 1210, ( - f"the field sweep checks {claims} claims, recorded as 1210 on " + # lowercase and maiden-clause numerals); 1210 -> 1213 the same day: + # #721's one-case lowercase numerals, counted and reported. + assert claims == 1213, ( + f"the field sweep checks {claims} claims, recorded as 1213 on " f"2026-10-18") diff --git a/tests/v2/test_facade.py b/tests/v2/test_facade.py index bfae80cd..80a2adfd 100644 --- a/tests/v2/test_facade.py +++ b/tests/v2/test_facade.py @@ -1059,8 +1059,12 @@ def test_the_v1_off_switch_restores_the_pre_397_fields() -> None: The three role movers of decisions/P3's one-case paragraph are the rows, because they are the ones whose FIELDS move: every - value on the right below is what the released 2.3.0 wheel gives, - measured 2026-09-20. + value on the right below was what the released 2.3.0 wheel gives, + measured 2026-09-20 -- until #721, which made a one-case + one-letter numeral the initial its capital is (rules/S10), so with + `i` out of the conjunctions the letter now reads as 'I' does in + each, switch on or off. What the switch still moves is the join, + below. A local `Constants`, never the shared `CONSTANTS`: a removal on the singleton would leak into every later test in the process. @@ -1070,17 +1074,17 @@ def test_the_v1_off_switch_restores_the_pre_397_fields() -> None: for text, on_fields, off_fields in ( ("rovira, i", {"first": "i", "last": "rovira", "suffix": ""}, - {"first": "", "last": "rovira", "suffix": "i"}), + {"first": "i", "last": "rovira", "suffix": ""}), ("john smith i jr", {"first": "john", "middle": "smith", "last": "i", "suffix": "jr"}, - {"first": "john", "middle": "", "last": "smith", - "suffix": "i jr"}), + {"first": "john", "middle": "smith", "last": "i", + "suffix": "jr"}), ("maier, amy i, jr.", {"first": "amy", "middle": "i", "last": "maier", "suffix": "jr."}, - {"first": "amy", "middle": "", "last": "maier", - "suffix": "i, jr."})): + {"first": "amy", "middle": "i", "last": "maier", + "suffix": "jr."})): on = HumanName(text) off = HumanName(text, constants) for field, value in on_fields.items(): diff --git a/tests/v2/test_properties.py b/tests/v2/test_properties.py index 251009b1..93f730c7 100644 --- a/tests/v2/test_properties.py +++ b/tests/v2/test_properties.py @@ -3332,3 +3332,57 @@ def test_every_one_letter_particle_veto_is_load_bearing( monkeypatch.setattr(importlib.import_module(module), "_spells_an_initial", lambda n, text: False) assert len(_p7_disagreements(lexicon)) == _P7_SITE_EFFECT[module] + + +# --- #721: a name written in one case reads as its capitals --------- +# S10 (the trailing roman numeral) says a name written wholly in one +# case says nothing about any letter in it -- so a wholly lower-case name parses exactly as the +# same name in capitals, fields (compared case-folded) and report +# kinds alike. The grid crosses the shapes a one-letter numeral meets +# -- every comma shape, a maiden clause, a title, a particle chain, a +# bound given name, an initial, a nickname and a credential before or +# after it -- with the numerals a listed and a shape-only spelling of +# each length, three heads (one Catalan, one holding an ambiguous +# acronym) and all three orders: +# 1,197 pairs, about 2,400 parses, 0.19s on py3.11 (2026-10-18). +_ONE_CASE_SHAPES = ( + "{a} {n}", "{a} {b} {n}", "{a} {n}, phd", "{a} {b} {n}, phd", + "{a} {n}, jr.", "{b}, {a} {n}", ", {a} {n}", "{a} nee {b} {n}", + "{a} {b} nee {c} {n}", "dr {a} {n}", "dr {a} {b} {n}", "sir {a} {n}", + "{a} van {b} {n}", "freiherr von {b} {n}", "abdul {a} {n}", + "{a} j. {n}", "({c}) {a} {n}", "{a} {b} jr {n}", "{a} {n} phd") +_ONE_CASE_NUMERALS = ("v", "i", "x", "vi", "iii", "ii", "iv") +_ONE_CASE_HEADS = (("john", "smith", "jones"), ("carod", "puig", "rovira"), + ("mary", "ma", "lee")) +#: the negative control: the pairs that disagree with classify's +#: one-case numeral initial taken out, measured 2026-10-18 -- the +#: same 171 master disagreed on before #721, so the tag is the whole +#: of the fix the grid sees. +_ONE_CASE_UNGUARDED = 171 + + +def _one_case_disagreements() -> list[str]: + def sig(name: ParsedName) -> tuple[object, ...]: + return ({k: v.lower() for k, v in name.as_dict().items() if v}, + tuple(sorted(a.kind.value for a in name.ambiguities))) + out = [] + for order in (GIVEN_FIRST, FAMILY_FIRST, FAMILY_FIRST_GIVEN_LAST): + parser = Parser(policy=Policy(name_order=order)) + for shape, numeral, (a, b, c) in itertools.product( + _ONE_CASE_SHAPES, _ONE_CASE_NUMERALS, _ONE_CASE_HEADS): + lower = shape.format(a=a, b=b, c=c, n=numeral) + if sig(parser.parse(lower)) != sig(parser.parse(lower.upper())): + out.append(f"[{order}] {lower!r}") + return out + + +def test_a_one_case_name_reads_as_its_capitals() -> None: + failures = _one_case_disagreements() + assert not failures, "\n".join(failures[:10]) + + +def test_the_one_case_grid_sees_the_numeral_initial( + monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr("nameparser._pipeline._classify." + "is_single_letter_numeral", lambda text: False) + assert len(_one_case_disagreements()) == _ONE_CASE_UNGUARDED diff --git a/tools/differential/corpus_rules.jsonl b/tools/differential/corpus_rules.jsonl index 904c4500..6fda5167 100644 --- a/tools/differential/corpus_rules.jsonl +++ b/tools/differential/corpus_rules.jsonl @@ -491,6 +491,7 @@ "abdul V Smith" "abdul salam ahmed salem" "anh van do" +"carod i" "de" "de Anke van y" "de Mesnil Jean" @@ -511,7 +512,6 @@ "john doe q.c." "john e smith" "john i" -"john nee smith i" "john smith EdD" "john smith MEng" "john smith bsc" diff --git a/tools/differential/snapshot.jsonl b/tools/differential/snapshot.jsonl index 8a5b5bab..1729e4d2 100644 --- a/tools/differential/snapshot.jsonl +++ b/tools/differential/snapshot.jsonl @@ -3286,6 +3286,7 @@ {"name": "bin Stromae Elizabeth Windsor Maria Teresa MP", "order": "FAMILY_FIRST", "given": "Stromae", "middle": "Elizabeth Windsor Maria Teresa", "family": "bin", "suffix": "MP", "ambiguities": ["PARTICLE_OR_GIVEN"], "initials": "S. E. W. M. T. b."} {"name": "bin Stromae, Elizabeth Windsor Maria Teresa MP", "given": "Elizabeth", "middle": "Windsor Maria Teresa", "family": "bin Stromae", "suffix": "MP", "initials": "E. W. M. T. S."} {"name": "bob v. de la macdole-eisenhower phd", "given": "bob", "middle": "v.", "family": "de la macdole-eisenhower", "suffix": "phd", "initials": "b. v. m."} +{"name": "carod i", "given": "carod", "family": "i", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "c. i."} {"name": "da Silva Sr. José Pedro Eduardo Jr., MP", "order": "FAMILY_FIRST", "given": "Silva", "middle": "Sr. José Pedro Eduardo", "family": "da", "suffix": "Jr., MP", "ambiguities": ["PARTICLE_OR_GIVEN"], "initials": "S. S. J. P. E. d."} {"name": "de", "given": "de", "initials": "d."} {"name": "de Anke van y", "family": "de Anke van y", "initials": "A."} @@ -3336,7 +3337,7 @@ {"name": "der, y van", "given": "y", "family": "van der", "ambiguities": ["PARTICLE_OR_GIVEN"], "initials": "y. v. d."} {"name": "dhr. Bertus Wevers Harry Robert de Jong jr. mp", "title": "dhr.", "given": "Bertus", "middle": "Wevers Harry Robert", "family": "de Jong", "suffix": "jr. mp", "initials": "B. W. H. R. J."} {"name": "doe, jane phd vd ma", "given": "jane", "family": "doe", "suffix": "phd vd ma", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "j. d."} -{"name": "doe, jane v phd do", "given": "jane", "family": "doe", "suffix": "v phd do", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "j. d."} +{"name": "doe, jane v phd do", "given": "jane", "middle": "v", "family": "doe", "suffix": "phd do", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "j. v. d."} {"name": "doe, john ma", "given": "john", "family": "doe", "suffix": "ma", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "j. d."} {"name": "donovan mcnabb-smith", "given": "donovan", "family": "mcnabb-smith", "initials": "d. m."} {"name": "dos Santos", "family": "dos Santos", "initials": "S."} @@ -3379,8 +3380,7 @@ {"name": "john e jones, III", "given": "john", "middle": "e", "family": "jones", "suffix": "III", "initials": "j. e. j."} {"name": "john e smith", "given": "john", "middle": "e", "family": "smith", "ambiguities": ["CONJUNCTION_OR_INITIAL"], "initials": "j. e. s."} {"name": "john e. smith", "given": "john", "middle": "e.", "family": "smith", "initials": "j. e. s."} -{"name": "john i", "given": "john", "family": "i", "ambiguities": ["CONJUNCTION_OR_INITIAL"], "initials": "j. i."} -{"name": "john nee smith i", "family": "john", "suffix": "i", "maiden": "smith", "initials": "j."} +{"name": "john i", "given": "john", "family": "i", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "j. i."} {"name": "john smith", "given": "john", "family": "smith", "initials": "j. s."} {"name": "john smith EdD", "given": "john", "family": "smith", "suffix": "EdD", "initials": "j. s."} {"name": "john smith MA", "given": "john", "family": "smith", "suffix": "MA", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "j. s."} @@ -3402,7 +3402,7 @@ {"name": "john smith ph.d.", "given": "john", "family": "smith", "suffix": "ph.d.", "initials": "j. s."} {"name": "john smith phd", "given": "john", "family": "smith", "suffix": "phd", "initials": "j. s."} {"name": "john smith psyd", "given": "john", "family": "smith", "suffix": "psyd", "initials": "j. s."} -{"name": "john smith v", "given": "john", "family": "smith", "suffix": "v", "initials": "j. s."} +{"name": "john smith v", "given": "john", "family": "smith", "suffix": "v", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "j. s."} {"name": "john smith vi", "given": "john", "family": "smith", "suffix": "vi", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "j. s."} {"name": "john smith x.y.z.", "given": "john", "family": "smith", "suffix": "x.y.z.", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "j. s."} {"name": "john smith xi", "given": "john", "middle": "smith", "family": "xi", "initials": "j. s. x."} @@ -3411,10 +3411,10 @@ {"name": "john smith, phd meng", "given": "john", "family": "smith", "suffix": "phd meng", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "j. s."} {"name": "john smith, v", "given": "john", "family": "smith", "suffix": "v", "initials": "j. s."} {"name": "john smith, vi", "given": "vi", "family": "john smith", "initials": "v. j. s."} -{"name": "john v", "given": "john", "suffix": "v", "ambiguities": ["GIVEN_OR_FAMILY"], "initials": "j."} +{"name": "john v", "given": "john", "family": "v", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "j. v."} {"name": "john van der berg ma", "given": "john", "family": "van der berg", "suffix": "ma", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "j. b."} {"name": "johnny y", "given": "johnny", "family": "y", "initials": "j. y."} -{"name": "jones, john e", "given": "john", "middle": "e", "family": "jones", "ambiguities": ["CONJUNCTION_OR_INITIAL"], "initials": "j. e. j."} +{"name": "jones, john e", "given": "john", "middle": "e", "family": "jones", "initials": "j. e. j."} {"name": "jose e maria santos", "given": "jose", "middle": "e maria", "family": "santos", "ambiguities": ["CONJUNCTION_OR_INITIAL"], "initials": "j. e. m. s."} {"name": "jose ortega-y-gasset", "given": "jose", "family": "ortega-y-gasset", "initials": "j. o."} {"name": "josep carod i rovira", "given": "josep", "middle": "carod i", "family": "rovira", "ambiguities": ["CONJUNCTION_OR_INITIAL"], "initials": "j. c. i. r."} @@ -3430,7 +3430,7 @@ {"name": "juan y garcia", "given": "juan", "middle": "y", "family": "garcia", "initials": "j. y. g."} {"name": "juan y-garcia", "given": "juan", "family": "y-garcia", "initials": "j. y."} {"name": "juan y-garcia-lopez", "given": "juan", "family": "y-garcia-lopez", "initials": "j. y."} -{"name": "larry james edward johnson v", "given": "larry", "middle": "james edward", "family": "johnson", "suffix": "v", "initials": "l. j. e. j."} +{"name": "larry james edward johnson v", "given": "larry", "middle": "james edward", "family": "johnson", "suffix": "v", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "l. j. e. j."} {"name": "lt. gen. john a. kenneth doe iv", "title": "lt. gen.", "given": "john", "middle": "a. kenneth", "family": "doe", "suffix": "iv", "initials": "j. a. k. d."} {"name": "m.d ph.d. Inger Marie Adele Synnøve von Wolf jr.", "given": "m.d", "middle": "ph.d. Inger Marie Adele Synnøve", "family": "von Wolf", "suffix": "jr.", "initials": "m. p. I. M. A. S. W."} {"name": "m.d ph.d. von Wolf Inger Marie Adele Synnøve", "order": "FAMILY_FIRST", "given": "ph.d.", "middle": "von Wolf Inger Marie Adele Synnøve", "family": "m.d", "initials": "p. W. I. M. A. S. m."} From ba7b25ed569514d8c63fe1b8ef7b0d127284c51d Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sun, 18 Oct 2026 13:49:32 -0700 Subject: [PATCH 2/9] Address #721's reviews: keep the stand-down, count a bare letter in front (#721) Both reviews found the stand-down this branch had removed as dead was not: a trailing period-marked title leaves a marked letter its part's last own word to classify but not the walk's, so 'john i dr.' carried conjunction-or-initial and suffix-or-name on one token. Restored, with S10's sentence and its precedence (P3's report is the one). The numeral fork's test of the word in front was case-dependent: a bare lowercase letter is never initial-shaped, so 'john j v' read suffix 'v' where 'JOHN J V' reads family 'V'. In a one-case name a bare cased letter in front now counts as written as an initial. The one-case grid gains the shapes both reviews found missing (a bare letter in front, a trailing title, a doubled numeral, a trailing period): master disagrees on 288 of 1,575 pairs, the branch on none, with a recorded control per guard. Also: classify builds `ends` only when a marked letter reaches the emitter, and asks `_ROMAN` at C level for the one-case initial, so no ordinary name pays a frame; the stale reasoning in classify, the public `initial` tag and CONJUNCTION_OR_INITIAL docstrings, P3's pointer, decisions/P3's exemption claim and pointer, and decisions/S10 (unreproducible figures replaced, `john nee smith i` recorded) are corrected. Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 2 +- docs/design/decisions/P3-connectives.md | 2 +- .../decisions/S10-trailing-roman-numeral.md | 5 +- docs/design/rules/P3-connectives.md | 4 +- .../rules/S10-trailing-roman-numeral.md | 10 +++- nameparser/_pipeline/_assign.py | 26 ++++++++-- nameparser/_pipeline/_classify.py | 49 +++++++++++-------- nameparser/_pipeline/_pieces.py | 2 +- nameparser/_pipeline/_vocab.py | 15 ++++-- nameparser/_types.py | 8 ++- tests/v2/test_properties.py | 35 +++++++++---- tools/differential/corpus_rules.jsonl | 2 + tools/differential/snapshot.jsonl | 2 + 13 files changed, 116 insertions(+), 46 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 68ac7ac3..d01ec1d4 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -173,7 +173,7 @@ The library has two layers: `nameparser/config/` (data) and `nameparser/parser.p **Sweep the forms a change can reach: comma shapes, then `name_order`.** No comma, a FULL name before the comma, a ONE-WORD name before the comma and NOTHING before the comma are four paths, and the last two are the misses — #429, #430 and #432 are all the one-word path disagreeing with the full-name path on inputs the full-name path reads correctly, and #613's PR review found `, PhD` and `, Jr.` (an empty surname field, as CSV data writes it) read as given and title by a decision that returned early on an empty head, after two review rounds whose every grid had a name before the comma. Put an empty head and a nickname-only head (`(Bob), PhD`) in every comma grid. Count the one-word path the way C1 counts, which is neither tokens nor pieces: a particle run and the one word it attaches to are one word (#575), a connective join and a bound given-name pair count as their words (#613, so `Ortega y Gasset, Dr.` is a whole name), and a suffix beside the surname counts as no name word at all. #603's review found the suffix half (`Vega y Lopez Jr., PhD Jones` lost its family where `Vega y Lopez, PhD Jones` kept it); #613 then reversed the connective half, and both now read given 'Vega', middle 'y', family 'Lopez' — so test a particle surname, a connective surname and a surname with a suffix, each before a comma. For orders the sweep already exists (`tests/v2/test_cases.py` runs every row under all three) but carries one assertion, R2's family partition, so it checks nothing a new change moves; coverage there has been vacuous before (PR #394's review found the suite passed with `name_order` discarded from grouping). Parse your change's names in each comma shape and each order, and read the ones you did not predict. And read a generated grid's moves by SHAPE, not by corpus membership: #614's first design passed the suite, the five gates and a fingerprint of every corpus name, because its grid moves were sorted into 'corpus name moved' and 'garbage', and the review found the moves that mattered among the 'garbage' -- a bare salutation (`Mr. and Mrs.` lost its connective to the family name) and a name with one trailing word (`Freiherr von Berg X.Y.Z.`). Pull out the head-plus-one-word slice of the grid and a salutation with nothing after it, and read every move there. Two more were the second review's (#614): no grid put TWO words that are both a title and a particle at the head (`Freiherr St van Berg MA`, where an index into the pieces was applied to a merged view of them -- `TITLES ∩ particles_ambiguous` is `{freiherr, st}`, so put both in every head list, each in its bare and its period-marked spelling -- #637's grid had `St` and `Freiherr` alone and missed that `St.` and `Freiherr.` are declined by the bound join and then attached by P6), and the grids compared ambiguity KINDS, so a report telling the caller the wrong field (`Freiherr von Berg Ma`, 'read as a given name' of a word in the family) passed every pin. #701's review added one: put a title/suffix dual (`Ms`, `Sr.`, `Cpl`) SECOND in a title run, behind an ordinary title. A stop that asks only "is this word suffix vocabulary" passes every row that opens with the dual and fails `Rev. Sr. Smith, Mary`; it was found by comparing `T Smith, John` against `T John Smith` over every title word, and a comma-versus-comma-less comparison is the cheap check for any rule that moves a word between the two slots. Compare each report's `detail` against the field its tokens land in -- since #626 `test_a_report_names_the_field_its_word_lands_in` (tests/v2/test_cases.py) does it over every non-locale case row under all three orders (a row carrying its own policy only as declared), for the phrasings its `_FIELD_CLAIMS` lists -- so a row with the shape is the test only where the report's wording is listed: a new emitter's wording goes in that list, and segment's report on a part past the second comma names no field (#629). #626's own first grid then missed an emitter for want of one comma shape (`, Ma Dr.`: an empty head, an ambiguous word, a trailing title, which H1 moves to the family): a trailing title belongs in every empty-head comma grid too. And #620's: an exclusion list in code being replaced is a set of rules, not noise -- #614's merged copy excluded every word the trailing read weighs from its units, the first #620 prototype dropped the list as a symptom of the copy, and its review found the list had been keeping positional counting's rule that a member in front counts as a word (`Freiherr von Berg MA X.Y.Z.` lost its suffix). Before deleting one, mutate each entry out and read what moves. -**Then sweep the spellings: a class reached by vocabulary is also reached by SHAPE.** A test that uses only the listed spelling of a word class walks only the vocabulary path, and the shape path is a separate branch that can be missing while every test passes. Roman numerals are the sharpest case: `ii`, `iii` and `iv` are suffix vocabulary; `i` and `v` are vocabulary too but initial-shaped, so `is_suffix_piece` vetoes them and only the numeral fork reads them; and `vi` through `x` are in no list at all, read by the fork's shape test alone (checked 2026-10-04 against `Lexicon.default()`) -- which stops at `x`: `_vocab._ROMAN` matches I to X, so `John Smith XI` reads `XI` as a name word (re-checked 2026-10-07, #614's /simplify, which found a docstring claiming "any length"). And the fork counts a ONE-letter numeral besides, since #718 (rules/S10): it is the suffix only where the name has a word to spare, so `John V` and `John Smith V` take different branches of one fork, a one-case name being counted like capitals whatever the letter -- since #721 a one-case one-letter numeral carries the `initial` tag its capital does, so `john v` reads as `JOHN V` does in every shape (a lower/upper grid over comma shapes, orders, a maiden clause and a particle chain found 171 disagreeing pairs on master and none after) -- put a head-plus-numeral row beside every words-to-spare row. So a change that handles `V` and `III` can still miss `VI` — #610 found exactly that, the maiden take's candidate loop admitting a shape-only numeral in one position where the peel reads it in two (`Jane Doe nee Smith VI Prof.` kept maiden `Smith VI` while `V` and `III` worked). The same split runs through the credentials (`MA` listed, `X.Y.Z.` by the dotted shape, `XYZ` by the caps shape, and `Ph. D.` merged by group into one flagged piece the peel's walk never holds -- #603 found #602's run-start predicate refusing it too, so `Smith, John Ph. D. Jones` kept middle 'Jones' where `Smith, John PhD Jones` read suffix 'PhD Jones') and the titles (`Prof.` period-marked and read at the end of a name, `Prof` bare and a name word there). When a change touches a word class, test one spelling from each path. The same holds for a separator's SPACING: a comma (or any punctuation) can stand alone, glue to the word before it or glue to the word after, and each is a different token stream. #653's first fix swept `(née, Jones)` and missed `(née , Jones)`, where the kept comma became a word after the marker and M3 read the clause as maiden ', Jones'; its review found it. Put all three spacings in any grid around a separator. +**Then sweep the spellings: a class reached by vocabulary is also reached by SHAPE.** A test that uses only the listed spelling of a word class walks only the vocabulary path, and the shape path is a separate branch that can be missing while every test passes. Roman numerals are the sharpest case: `ii`, `iii` and `iv` are suffix vocabulary; `i` and `v` are vocabulary too but initial-shaped, so `is_suffix_piece` vetoes them and only the numeral fork reads them; and `vi` through `x` are in no list at all, read by the fork's shape test alone (checked 2026-10-04 against `Lexicon.default()`) -- which stops at `x`: `_vocab._ROMAN` matches I to X, so `John Smith XI` reads `XI` as a name word (re-checked 2026-10-07, #614's /simplify, which found a docstring claiming "any length"). And the fork counts a ONE-letter numeral besides, since #718 (rules/S10): it is the suffix only where the name has a word to spare, so `John V` and `John Smith V` take different branches of one fork, a one-case name being counted like capitals whatever the letter -- since #721 a one-case one-letter numeral carries the `initial` tag its capital does, and a bare letter in front of the numeral is initial-shaped as its capital is, so `john v` reads as `JOHN V` does over `test_a_one_case_name_reads_as_its_capitals`'s grid (comma shapes, orders, a maiden clause, a particle chain, a bare letter in front, a trailing title: 288 disagreeing pairs on master, none after). Its first version claimed "every shape" with no bare letter in front and no trailing title in the grid, and #721's reviews found both broken; a grid's claim is the grid -- put a head-plus-numeral row beside every words-to-spare row. So a change that handles `V` and `III` can still miss `VI` — #610 found exactly that, the maiden take's candidate loop admitting a shape-only numeral in one position where the peel reads it in two (`Jane Doe nee Smith VI Prof.` kept maiden `Smith VI` while `V` and `III` worked). The same split runs through the credentials (`MA` listed, `X.Y.Z.` by the dotted shape, `XYZ` by the caps shape, and `Ph. D.` merged by group into one flagged piece the peel's walk never holds -- #603 found #602's run-start predicate refusing it too, so `Smith, John Ph. D. Jones` kept middle 'Jones' where `Smith, John PhD Jones` read suffix 'PhD Jones') and the titles (`Prof.` period-marked and read at the end of a name, `Prof` bare and a name word there). When a change touches a word class, test one spelling from each path. The same holds for a separator's SPACING: a comma (or any punctuation) can stand alone, glue to the word before it or glue to the word after, and each is a different token stream. #653's first fix swept `(née, Jones)` and missed `(née , Jones)`, where the kept comma became a word after the marker and M3 read the clause as maiden ', Jones'; its review found it. Put all three spacings in any grid around a separator. **And the encoding: a letter class written `[^\W\d_]` or tested with `isalpha()` is blind to a combining mark.** Python's `\w` and `isalpha()` do not match Mn, so a letter written as base plus accent (NFD) ends a word early; the vocabulary path is immune because `_normalize` NFC-folds, the shape path is not. #652: `Dr. inž. Janez Novak` read `inž.` as the given name when the ž was decomposed, and `Janez Novak, ŽŠČ` reversed, in two families that share the blind spot — the letter-shape regexes (`period_abbreviation`, `_DOTTED`, `_CHUNKED`, `_PAIRED_INITIALS`), now spelled with `_vocab._LETTER` (a letter plus the general combining blocks; `test_regex_sync` compares the two `_MARKS` strings and pins `period_abbreviation` against config's copy), and the caps predicates, which NFC-compose a non-ASCII word before `isalpha()` (`is_caps_word`, behind C-level `isalpha()`/`is_normalized` tests, so already-composed text pays no frame — measured 2026-10-12, `Smith, José` was 193 frames before and after; the first version called it for every non-ASCII word and cost one frame each, and no guard saw it because the reference name is ASCII). A width test must count letters, not code points (`composed_len`): admitting a decomposed word to the caps set put `John Smith, MÁ` on the pair exception's wrong side, and the review's grid found it where the rows (all three letters long) did not — put a two-letter word in every caps grid. The initial shape (`_INITIAL`, `\w\.`, v1-verbatim and public) is not widened but asked of the COMPOSED word, in `is_initial_shaped` and in `_render`'s two copies; and the single-letter connective tests count composed letters, since a decomposed `й` is two code points (#665, where `É. y É. Smith` and `John й Smith` read differently decomposed). Compose the WHOLE predicate, not one half: `is_initial` is a shape ANDed with the #320 script veto, and composing the shape alone made a decomposed `씨.` an initial, since decomposed hangul is conjoining jamo outside every script range (#665's first version; `in_initialless_script` composes too now). Not covered, deliberately: the marks only some scripts write (Indic vowel signs, Arabic and Hebrew points; rules/H2's Accepted clause). The guard is `test_a_decomposed_name_renders_as_its_composed_twin` over the CLDR corpus plus `test_a_decomposed_abbreviation_reads_as_its_composed_twin` (rows × three orders × EVERYWHERE); the corpus guard reaches a shape only if some corpus name does, which is how the caps shape was missed until review, so when adding a shape rule add a row there. Four of #652's rows and two of #665's pass with every fix reverted (measured) and are parity pins, and the test compares ambiguity kinds and `initials()` too, since #665's `John й Smith` differed in its report; `_DOTTED` has no observable input and is pinned by a direct call. diff --git a/docs/design/decisions/P3-connectives.md b/docs/design/decisions/P3-connectives.md index feadb514..80489e23 100644 --- a/docs/design/decisions/P3-connectives.md +++ b/docs/design/decisions/P3-connectives.md @@ -62,4 +62,4 @@ - 2026-10-07 #614 — CROSS-REFERENCE: P3's joins are made before S2's trailing run is read, so the run counts a joined part as one name word, as this rule says every count does (`Juan y Garcia X.Y.Z.` keeps family 'X.Y.Z.'), and a salutation keeps its connective (`Mr. and Mrs.` reads title 'Mr. and Mrs.'). #614's first design read before these joins too and broke both; #614's review reversed it (decisions/S2, this date). - Provenance: the single-letter-connective guard is v1's fix for Google Code issue 11 ("john e smith", 2013, commit 33676c9) — the "#11" citations that circulated pointed at a GitHub accident, not the real source. Recorded so the archaeology stays done. -- 2026-10-18, #721 — THE OFF SWITCH NO LONGER RESTORES THE ONE-CASE ROLE MOVERS. With `i` removed from the conjunctions, a one-case `i` is a one-letter roman numeral, which since #721 is the initial its capital is (rules/S10), so `rovira, i`, `john smith i jr` and `maier, amy i, jr.` read with the switch off exactly as with it on, and as their all-caps twins read either way — where 2.3.0 read them as suffixes, the disagreement with the capitals this paragraph's 2026-09-20 entry measured. The property the off-switch grid holds is unchanged, its exemption (2) — the readings disagree about the letter being an initial — covering it. And a connective letter ending its part is not reported (rules/P3, "unless nothing of its part stands to its right"): the fork had no other way to go. +- 2026-10-18, #721 — THE OFF SWITCH NO LONGER RESTORES THE ONE-CASE ROLE MOVERS. With `i` removed from the conjunctions, a one-case `i` is a one-letter roman numeral, which since #721 is the initial its capital is (rules/S10), so `rovira, i`, `john smith i jr` and `maier, amy i, jr.` read with the switch off exactly as with it on, and as their all-caps twins read either way — where 2.3.0 read them as suffixes, the disagreement with the capitals that decisions/P3's 2026-09-20 bullet "ACCEPTED, AND THE HONEST STATEMENT OF THE ONE-CASE HALF" measured. The off-switch grid's property is unchanged, and these rows no longer lean on its exemption (2): the two readings now agree on them (on: `initial`; off: `initial`, where 2.3.0's off reading was not), so they pass on equal fields. The exemption still guards the rows it was written for (#721's design-docs review measured the rows only it saves falling from thousands on master to a few hundred, all of them a connective `i` before a strict-comma numeral such as `I ROVIRA, V` — a dated snapshot). And a connective letter ending its part is not reported (rules/P3, "unless nothing of its part stands to its right"): the fork had no other way to go. diff --git a/docs/design/decisions/S10-trailing-roman-numeral.md b/docs/design/decisions/S10-trailing-roman-numeral.md index 271fff8c..d321047c 100644 --- a/docs/design/decisions/S10-trailing-roman-numeral.md +++ b/docs/design/decisions/S10-trailing-roman-numeral.md @@ -10,5 +10,6 @@ ONE STAND-DOWN, FOUND BY INV10 rather than designed. In a one-case name the connective `i` reaches the fork as a letter classify has already reported as `conjunction-or-initial`, read as an initial, and the new report put a second report on the same token: 138 rows of the connective grid (measured on this branch with the `reported_as_initial` check out of assign's numeral report, a dated snapshot) in tests/v2/test_properties.py (`AND ROVIRA I`, `HENRY I`) tripped INV10's kind pair. The two reports did not contradict each other — both read the letter as a name word — but they named one reading twice, so the numeral's report stands down where the initial reading is already reported, the shape of rules/S7's "a member its own capitals already made the credential reports nothing new". The guard was left untouched: its kind pair still means "two readings" everywhere else, and narrowing it to suit this change would have retaken its recorded control. `HENRY I` therefore reports `conjunction-or-initial` alone, and moved in test_assemble's contrast list from the withdrawn side to the side where the fork resolved to a name word. - 2026-10-18, #718 review — THE COUNT IS NOT ASKED IN THE GIVEN PART AFTER A FAMILY COMMA. The code review's grid (family-comma and empty-head shapes behind a title-particle, about 558,000 parses) found 2,880 parses where the first form moved a numeral from the suffix into the given or family name with no report: `Smith, Freiherr von Berg V` read given 'von Berg V', `, Freiherr von V` family 'von V', `Smith, St van V` given 'van V', where `Smith, von V` and `Smith, Freiherr John V` kept suffix V. The cause was not a choice: assign's walk over that part never counts the numeral (rules/S4: the given part's trailing slot "is read without it (S5)"), but the particle chain's bound there asks `tail_reading`, which the count had reached, so the chain took a word assign would have read as the suffix. The bound now reads with the count off (`trailing_run_start`, `count_numeral=False`), and all four read as they did on master (2.2.0 onward, since #424; 1.4.0 read the numeral into the name). The class is the one the AGENTS.md gotcha on group-side predictors of the given part names, met from the other side: the predictor did not change, the peel it asks did. The S4 analogue, `Smith, Freiherr von Berg Ma` → given 'von Berg Ma', is master's reading and is left alone here. - 2026-10-18, #718's PR review — TWO OF THE THREE REPORT SITES ARE ONE, AND THE CLAUSE ASKS THE CASE. The test-coverage review mutated each new emitter out in a copy of the tree. Removing the particle chain's numeral report moved no field and no kind — assign's pick loop reported the same `suffix-or-name`, worded from the final role ('read as a family name') — so the chain's branch was only choosing a sentence, and it is gone: a declined numeral stays one of the read's picks, and assign alone reports it, the decision having been the read's and not the chain's. The maiden clause's stand-down on a letter already reported as an initial could not fire at all (classify reads no connective inside a clause), so that call is gone too. And the clause's report on a numeral its take gave up did not ask the case: `Jane Doe nee Smith vi` and `x` reported where every other path had gone silent, the drift mechanisms.md#ONE-PREDICATE-PER-QUESTION names; the take now asks `numeral_settled_by_case` as assign does, `VI` still reporting beside them. Five rows the first draft labelled `fix(#718)` read field for field as 1.4.0 does and are `parity`, as are the two Freiherr rows added here; a restored 1.4.0 reading is parity by the table's own definition (rules/S4's `bare_surname_is_not_a_credential` is the precedent). -- 2026-10-18 (Derek), #721 — A ONE-CASE ONE-LETTER NUMERAL IS THE INITIAL ITS CAPITAL IS, AND A LETTER ENDING ITS PART IS NO CONNECTIVE. Two Accepted gaps of the entries above close. First, `john v` read suffix 'v' with no family while `JOHN V` and `john x` were counted: `V` carries the `initial` tag (a bare capital) and every reader of the trailing run honors it — the suffix veto, the particle chain, the credential run — while a lowercase `v` was a plain listed suffix word. A first fix routed a one-case `v` to the numeral fork at the walk's last position only, and a lower/upper invariant (every one-case name parses as its capitals, fields and report kinds, over 19 shapes × 7 numerals × 3 heads × 3 orders, 1,197 pairs) showed it too shallow: 171 pairs disagreed on master, 33 still did (`john v phd`, `john v, phd`, `freiherr von smith v`). The fix that holds tags the letter where its capital is tagged: in a name written in one case, a one-letter roman numeral is an `initial` (classify, reading the NAME-level one-case fact so a maiden clause's `v` is one too, as #516's caps branch reads it), and the grid then shows 0. Second, Derek on the issue: `john i` reporting `conjunction-or-initial` is wrong "because there's nothing for a conjunction to join to there". P3 already said a connective with nothing to its right connects nothing; classify's report now skips a letter that is the last own word of its part — before a comma, a marker's clause or the end. Such a letter is the numeral fork's, which reports `suffix-or-name` (`john i`, `HENRY I`, `carod i`), and the stand-down the entry above added for a letter already reported as an initial is gone: no declined numeral can carry that report any more, measured by disabling it (the same eight failures with and without, INV10 passing both ways). Two comma rows go silent rather than move — `rovira, i` and `maier, amy i, jr.` — since the comma path has never reported the numeral (decisions/C1, "Not done"), and `jones, john e`, a corpus name, loses its report the same way. - WHAT MOVES, the snapshot against master at 2bfa9533: six corpus names, `john v` (family 'v'), `john smith v` and `larry james edward johnson v` (gain the report), `doe, jane v phd do` (middle 'v', suffix 'phd do', as `DOE, JANE V PHD DO` and `Doe, Jane V PhD Do` read; rules/S7's example keeps its input, whose point — `do` anchored by the degree in front — is unchanged), `john i` (report kind), `jones, john e` (report dropped). And decisions/P3's off switch no longer restores 2.3.0's three one-case readings (`rovira, i` suffix 'i', and the two beside it): with `i` out of the conjunctions the letter is a one-letter numeral, so it is the initial `I` is there, switch on or off — those 2.3.0 readings were themselves a lower/upper disagreement. The switch still unjoins the link (`Josep Carod i Rovira`). +- 2026-10-18 (Derek), #721 — A ONE-CASE ONE-LETTER NUMERAL IS THE INITIAL ITS CAPITAL IS, AND A LETTER ENDING ITS PART IS NO CONNECTIVE. Two Accepted gaps of the entries above close. First, `john v` read suffix 'v' with no family while `JOHN V` and `john x` were counted: `V` carries the `initial` tag (a bare capital) and every reader of the trailing run honors it — the suffix veto, the particle chain, the credential run — while a lowercase `v` was a plain listed suffix word. A first prototype (not shipped) routed a one-case `v` to the numeral fork at the walk's last position only, and a lower/upper invariant — every one-case name parses as its capitals, fields and report kinds — showed it too shallow (`john v phd`, `john v, phd` and `freiherr von smith v` still disagreed). The fix that holds tags the letter where its capital is tagged: in a name written in one case, a one-letter roman numeral is an `initial` (classify, reading the NAME-level one-case fact so a maiden clause's `v` is one too, as #516's caps branch reads it). And #721's reviews found the numeral fork's own test of the word in front case-dependent: a bare lowercase letter is never initial-shaped, so `john j v` read suffix 'v' where `JOHN J V` reads family 'V' (on master too). In a one-case name a bare cased letter in front now counts as written as an initial. The invariant is `test_a_one_case_name_reads_as_its_capitals` in tests/v2/test_properties.py, its grid and its two recorded controls stated there: master disagreed on 288 of its 1,575 pairs, the branch on none. + Second, Derek on the issue: `john i` reporting `conjunction-or-initial` is wrong "because there's nothing for a conjunction to join to there". P3 already said a connective with nothing to its right connects nothing; classify's report now skips a letter that is the last own word of its part — before a comma, a marker's clause or the end. Such a letter is read as an initial and not reported, and where it is a roman numeral the numeral fork reports `suffix-or-name` instead (`john i`, `HENRY I`, `carod i`); a marked letter that is no numeral is simply silent (`john e`, `jones, john e`). Not all such letters are the walk's last word: a trailing title (rules/H5) is an own word to classify that the walk peels first, so in `john i dr.` the letter keeps P3's report and is still the numeral fork's declined pick. The stand-down the entry above added (a letter already reported as an initial reports nothing new) is what keeps that to one report, and it stays: a first draft of this change deleted it as dead on a mutation run over an intermediate tree, and both reviews then found `john i dr.`, `JOHN I DR.` and `henry i prof.` carrying both reports on one token, where master carries `conjunction-or-initial` alone. Two comma rows go silent rather than move — `rovira, i` and `maier, amy i, jr.` — since the comma path has never reported the numeral (decisions/C1, "Not done"). + WHAT MOVES, the snapshot against master at 2bfa9533: six corpus names, `john v` (family 'v'), `john smith v` and `larry james edward johnson v` (gain the report), `doe, jane v phd do` (middle 'v', suffix 'phd do', as `DOE, JANE V PHD DO` and `Doe, Jane V PhD Do` read; rules/S7's example keeps its input, whose point — `do` anchored by the degree in front — is unchanged), `john i` (report kind), `jones, john e` (report dropped). And a seventh reading the snapshot shows as a removal rather than a move, since the rules example that pinned it was deleted with S10's Accepted clause: `john nee smith i` read suffix 'i' and now reads maiden 'smith i', reported — the clause-free name is `john i`, whose letter has no word to spare, so M2's clause keeps it, as it keeps `john nee smith v`. And decisions/P3's off switch no longer restores 2.3.0's three one-case readings (that file's #721 entry). diff --git a/docs/design/rules/P3-connectives.md b/docs/design/rules/P3-connectives.md index 43ead296..e6b750ed 100644 --- a/docs/design/rules/P3-connectives.md +++ b/docs/design/rules/P3-connectives.md @@ -19,7 +19,9 @@ P3. Rationale: connective words ("y", "of the") bind name words into initial in a name written wholly in one case, is a call that could have gone the other way, and is reported — unless nothing of its part stands to its right, where it connects nothing and there is - no other way for the call to go; S10 reads such a letter. + no other way for the call to go: it is read as an initial and + not reported, and where it is a roman numeral S10's fork reports + it instead. The joined part is ONE name word wherever another rule counts them, so a rule taking "one name word" takes the whole join and never half of it — except C1's count of the words before a comma, diff --git a/docs/design/rules/S10-trailing-roman-numeral.md b/docs/design/rules/S10-trailing-roman-numeral.md index e1d93a43..91f20544 100644 --- a/docs/design/rules/S10-trailing-roman-numeral.md +++ b/docs/design/rules/S10-trailing-roman-numeral.md @@ -8,14 +8,18 @@ S10. Rationale: a roman numeral ending a name is its generation, the generation — the numerals the vocabulary lists, and those up to X it does not, by their shape alone — unless the word in front of it is written as an initial, which makes it a name word - too. A single letter written with a period is an initial, and a + too; in a name written wholly in one case a bare letter in front + is written as one. A single letter written with a period is an initial, and a numeral past X is a name word. A numeral of one letter needs words to spare, as a bare ambiguous acronym does (S4): as the second of two name words it is the family name, a title in front being no word to spare, and either reading is reported, as either reading of an ambiguous acronym is. A numeral read by its shape alone reports too; a numeral the vocabulary lists in two - or more letters is not in question and reports nothing. Written case + or more letters is not in question and reports nothing, and a + letter already reported as read as an initial reports nothing + new: where a title stands after a marked letter (P3), P3's + report is the one. Written case settles the fork in one way only: a numeral written in lower case in a name of mixed case is the generation and reports nothing, since nobody writes an initial that way. A name written @@ -45,6 +49,8 @@ S10. Rationale: a roman numeral ending a name is its generation, "john smith v" → ambiguities=("suffix-or-name",) "john i" → family="i" "john i" → ambiguities=("suffix-or-name",) + "john i dr." → ambiguities=("conjunction-or-initial",) + "john j v" → family="v" Accepted: a numeral with another suffix word behind it is not the last word of the name and is read as any word there is, so a one-letter numeral before a credential is the family name. diff --git a/nameparser/_pipeline/_assign.py b/nameparser/_pipeline/_assign.py index 9fdbcce0..f4a885b6 100644 --- a/nameparser/_pipeline/_assign.py +++ b/nameparser/_pipeline/_assign.py @@ -306,6 +306,21 @@ def _placed(read: TailRead, rest: list[int], peel.absorbed) +# rules/S10: "a letter already reported as read as an initial reports +# nothing new" -- asked only by the numeral fork's declined pick +# (#718). The maiden clause's report on the numeral it keeps cannot +# meet the case: classify reads no connective inside a clause +def _reported_as_initial(ambiguities: Sequence[PendingAmbiguity], + index: int) -> bool: + """Whether classify's connective-or-initial fork already reported + the token at `index`, reading it as an initial. Declining the + numeral reads the letter as that same name word, so a second + report would name one reading twice. A rare shape, so the scan + costs ordinary names nothing.""" + return any(a.kind is AmbiguityKind.CONJUNCTION_OR_INITIAL + and index in a.indices for a in ambiguities) + + def _assign_main(seg_idx: int, state: ParseState, tokens: list[WorkToken], ambiguities: list[PendingAmbiguity], @@ -535,11 +550,12 @@ def _assign_main(seg_idx: int, state: ParseState, # declined pick (#718), which carries neither credential tag # (the acronym fork's picks carry one) and is never a suffix if token.tags.isdisjoint(_AMBIGUOUS_CREDENTIAL_TAGS): - ambiguities.append(PendingAmbiguity( - AmbiguityKind.SUFFIX_OR_NAME, - f"{token.text!r} is a roman numeral, but the name has " - f"no other word to spare for it; read as ", piece, - field_tail=" rather than a generational suffix")) + if not _reported_as_initial(ambiguities, piece[0]): + ambiguities.append(PendingAmbiguity( + AmbiguityKind.SUFFIX_OR_NAME, + f"{token.text!r} is a roman numeral, but the name " + f"has no other word to spare for it; read as ", piece, + field_tail=" rather than a generational suffix")) continue declined = ("a name part" if token.role is Role.SUFFIX else "a post-nominal") diff --git a/nameparser/_pipeline/_classify.py b/nameparser/_pipeline/_classify.py index fbecafcd..0566fa54 100644 --- a/nameparser/_pipeline/_classify.py +++ b/nameparser/_pipeline/_classify.py @@ -63,8 +63,7 @@ ) from nameparser._types import AmbiguityKind, Role from nameparser._pipeline._vocab import ( - caps_shape_candidate, composed_len, is_initial, is_one_case, - is_single_letter_numeral, + _ROMAN, caps_shape_candidate, composed_len, is_initial, is_one_case, period_joined_vocab, suffix_as_written, tag_marker_runs, ) from nameparser._pipeline._pieces import own_words @@ -167,12 +166,13 @@ def _tags_for(token: WorkToken, n: str, state: ParseState, # there is the initial its capital is, and every reader that # vetoes 'V' vetoes 'v' too (#721). The NAME-level fact, as # #516's caps branch reads it: a capital 'V' in a maiden clause - # is an initial, so a one-case 'v' there is one too. `len` - # first, C-level, so only one-character words of a one-case - # name pay the call + # is an initial, so a one-case 'v' there is one too. One + # character already, so `_ROMAN` alone asks what + # `is_single_letter_numeral` would, at C level: no frame per + # one-letter word initial = is_initial(token.text) or ( one_case and len(token.text) == 1 - and is_single_letter_numeral(token.text)) + and _ROMAN.match(token.text) is not None) if n in lex.conjunctions and not initial: tags.add("conjunction") if initial: @@ -321,14 +321,12 @@ def classify(state: ParseState) -> ParseState: # rules/P3: "A connective with nothing to its right is connecting # nothing" -- so a letter ending its part (the last own word before # a comma, a marker's clause or the end of the name) offers the - # fork no connective reading, and it is not reported here (#721); - # the numeral fork reads such a letter instead (rules/S10). Built - # only for a one-case name, the one the emitter below can reach. - ends = frozenset( - own_seg[-1] for seg in state.segments - if (own_seg := [j for j in seg - if j < clause_at and tokens[j].role is None]) - ) if one_case else frozenset() + # fork no connective reading, and it is not reported here (#721): + # it is read as an initial, and where it is a roman numeral the + # numeral fork reports instead (rules/S10). Built only when a + # letter has passed the emitter's other tests, so a name that holds + # no marked letter pays nothing for it. + ends: set[int] | None = None for i, token in enumerate(tokens): if (token.role is Role.NICKNAME and AMBIGUOUS_ACRONYM_TAG in token.tags): @@ -347,11 +345,14 @@ def classify(state: ParseState) -> ParseState: # Of the two clauses beside it, one is load-bearing and one is # not. `conjunctions_ambiguous` is IMPLIED by the others for a # fresh parse -- the contrapositive of what it looks like: - # `is_initial` matches an ASCII capital only, so ONLY the - # fork's branch can tag a bare LOWERCASE letter "initial" - # ("jose e maria santos" tags a lowercase e), and it does so - # only for a conjunctions_ambiguous member. So for a letter of - # EITHER case, "initial" plus len 1 implies membership. It is + # `is_initial` matches an ASCII capital only, so a bare + # LOWERCASE letter is tagged "initial" only by the fork's + # branch ("jose e maria santos" tags a lowercase e), for a + # conjunctions_ambiguous member, or by the one-case numeral + # (#721), which is no connective and fails `conjunctions` + # below unless the caller made it one. So for a letter of + # EITHER case in `conjunctions`, "initial" plus len 1 implies + # membership. It is # kept only because `_tags_for` starts from `set(token.tags)`, so # this clause is what keeps the emitter honest if a runner ever # hands classify tokens it did not build; no such path exists @@ -369,10 +370,18 @@ def classify(state: ParseState) -> ParseState: and folded[i] in state.lexicon.conjunctions_ambiguous and folded[i] in state.lexicon.conjunctions and i < clause_at and token.role is None - and i not in ends and (len(token.text) == 1 or not token.text.isascii() and composed_len(token.text) == 1)): + if ends is None: + ends = set() + for seg in state.segments: + for j in reversed(seg): + if j < clause_at and tokens[j].role is None: + ends.add(j) + break + if i in ends: + continue ambiguities.append(PendingAmbiguity( AmbiguityKind.CONJUNCTION_OR_INITIAL, f"{token.text!r} is both a connective and an initial; " diff --git a/nameparser/_pipeline/_pieces.py b/nameparser/_pipeline/_pieces.py index e772fec0..0b48a63f 100644 --- a/nameparser/_pipeline/_pieces.py +++ b/nameparser/_pipeline/_pieces.py @@ -1159,7 +1159,7 @@ def peel_trailing(rest: Sequence[int], pieces: Sequence[Sequence[int]], if (k == len(rest) and k >= 2 and len(piece) == 1 and is_trailing_numeral_suffix( text := tokens[piece[0]].text, - tokens[pieces[rest[k - 2]][0]].text)): + tokens[pieces[rest[k - 2]][0]].text, one_case)): # rules/S10: "a numeral written in lower case in a name of # mixed case is the generation and reports nothing" -- # decided here once: taken as any suffix word is, and not diff --git a/nameparser/_pipeline/_vocab.py b/nameparser/_pipeline/_vocab.py index 52514325..31b9230a 100644 --- a/nameparser/_pipeline/_vocab.py +++ b/nameparser/_pipeline/_vocab.py @@ -256,7 +256,8 @@ def is_initial_shaped(text: str) -> bool: # reads as the generation" -- which the initial veto would otherwise # take: V and I are suffix vocabulary, X a numeral by shape alone, and # all three are bare capitals, which inside a name are middle initials. -def is_trailing_numeral_suffix(text: str, preceding: str) -> bool: +def is_trailing_numeral_suffix(text: str, preceding: str, + one_case: bool | None = None) -> bool: """assign's roman-numeral fork, shared with group's bound-given reserve (#401): a FINAL single-token piece that is a roman numeral reads as the suffix when the piece before it does not look like @@ -270,9 +271,17 @@ def is_trailing_numeral_suffix(text: str, preceding: str) -> bool: piece looks like part of an initial run, which is a question about layout, and #320 narrowed the tag to initials that can really stand in for a name. Reading the tag here made '씨.' stop - suppressing the fork and cost 'John 씨. V' its family name.""" + suppressing the fork and cost 'John 씨. V' its family name. + + `one_case` is ParseState.one_case: in a name written in one case + a bare cased letter in front is initial-shaped as its capital is + (rules/S10, #721's review: 'john j v' reads as 'JOHN J V'). A + caller passing None, the bound-given reserve's superset, gets the + shape test alone, which admits more and so stays a superset.""" return (_ROMAN.match(text) is not None - and not is_initial_shaped(preceding)) + and not is_initial_shaped(preceding) + and not (one_case and len(preceding) == 1 + and preceding.upper() != preceding.lower())) # rules/S10: "a numeral written in lower case in a name of mixed case is diff --git a/nameparser/_types.py b/nameparser/_types.py index c31e5a5d..246a8c25 100644 --- a/nameparser/_types.py +++ b/nameparser/_types.py @@ -88,7 +88,8 @@ def __add__(self, other: object) -> NoReturn: # type: ignore[override] #: word READ as an initial -- initial-shaped in a script that HAS #: initials ("J." or "А.", never "씨.", #320), or a marked #: single-letter connective in a name written in one case -#: (rules/P3, see CONJUNCTION_OR_INITIAL); +#: (rules/P3, see CONJUNCTION_OR_INITIAL), or a one-letter roman +#: numeral in a name written in one case, as its capital is (rules/S10); #: "joined" a continuation of the token before it -- within one #: merged piece the tag is role-blind and every view joins the pair #: with a space ("Ph." + "D."; 'Ph. D., Jr.' gives last_list @@ -625,6 +626,11 @@ class AmbiguityKind(StrEnum): #: doctrine, rules/P3): appending " née Jones" or a nickname #: changes no report, because the clause's words were never #: eligible for it. + #: A letter that ends its part -- nothing of the name after it + #: before a comma, a clause or the end, as in "JOSE E" or + #: "rovira, i" -- connects nothing, so it is read as an initial and + #: not reported; a roman numeral there is reported by the numeral + #: fork instead (SUFFIX_OR_NAME). #: ``detail`` names the token, the kind naming neither the field nor #: the letter: which field the reading lands in follows the name's #: shape and its ``name_order``, the PARTICLE_OR_GIVEN precedent. diff --git a/tests/v2/test_properties.py b/tests/v2/test_properties.py index 93f730c7..dddf6fb8 100644 --- a/tests/v2/test_properties.py +++ b/tests/v2/test_properties.py @@ -3344,21 +3344,26 @@ def test_every_one_letter_particle_veto_is_load_bearing( # after it -- with the numerals a listed and a shape-only spelling of # each length, three heads (one Catalan, one holding an ambiguous # acronym) and all three orders: -# 1,197 pairs, about 2,400 parses, 0.19s on py3.11 (2026-10-18). +# 1,575 pairs, about 3,150 parses, 0.24s on py3.11 (2026-10-18). _ONE_CASE_SHAPES = ( "{a} {n}", "{a} {b} {n}", "{a} {n}, phd", "{a} {b} {n}, phd", "{a} {n}, jr.", "{b}, {a} {n}", ", {a} {n}", "{a} nee {b} {n}", "{a} {b} nee {c} {n}", "dr {a} {n}", "dr {a} {b} {n}", "sir {a} {n}", "{a} van {b} {n}", "freiherr von {b} {n}", "abdul {a} {n}", - "{a} j. {n}", "({c}) {a} {n}", "{a} {b} jr {n}", "{a} {n} phd") + "{a} j. {n}", "({c}) {a} {n}", "{a} {b} jr {n}", "{a} {n} phd", + # #721's review: a bare letter in front, a trailing title, a + # doubled numeral and a trailing period + "{a} b {n}", "{a} e {n}", "{a} {n} prof.", "{a} {b} {n} prof.", + "{a} {b} {n} {n}", "{a} {b} {n}.") _ONE_CASE_NUMERALS = ("v", "i", "x", "vi", "iii", "ii", "iv") _ONE_CASE_HEADS = (("john", "smith", "jones"), ("carod", "puig", "rovira"), ("mary", "ma", "lee")) -#: the negative control: the pairs that disagree with classify's -#: one-case numeral initial taken out, measured 2026-10-18 -- the -#: same 171 master disagreed on before #721, so the tag is the whole -#: of the fix the grid sees. -_ONE_CASE_UNGUARDED = 171 +#: the negative controls, measured 2026-10-18 on this grid, which +#: master (2bfa9533) disagreed on 288 of: with classify's one-case +#: numeral initial taken out, and with the numeral fork's one-case +#: letter in front taken out -- each guard seen on its own +_ONE_CASE_UNGUARDED = 216 +_ONE_CASE_LETTER_IN_FRONT_UNGUARDED = 99 def _one_case_disagreements() -> list[str]: @@ -3383,6 +3388,18 @@ def test_a_one_case_name_reads_as_its_capitals() -> None: def test_the_one_case_grid_sees_the_numeral_initial( monkeypatch: pytest.MonkeyPatch) -> None: - monkeypatch.setattr("nameparser._pipeline._classify." - "is_single_letter_numeral", lambda text: False) + # classify's one use of `_ROMAN` is the one-case numeral initial + monkeypatch.setattr("nameparser._pipeline._classify._ROMAN", + re.compile(r"(?!)")) assert len(_one_case_disagreements()) == _ONE_CASE_UNGUARDED + + +def test_the_one_case_grid_sees_the_letter_in_front( + monkeypatch: pytest.MonkeyPatch) -> None: + from nameparser._pipeline import _vocab + monkeypatch.setattr( + "nameparser._pipeline._pieces.is_trailing_numeral_suffix", + lambda text, preceding, one_case=None: + _vocab.is_trailing_numeral_suffix(text, preceding)) + assert (len(_one_case_disagreements()) + == _ONE_CASE_LETTER_IN_FRONT_UNGUARDED) diff --git a/tools/differential/corpus_rules.jsonl b/tools/differential/corpus_rules.jsonl index 6fda5167..fab487b4 100644 --- a/tools/differential/corpus_rules.jsonl +++ b/tools/differential/corpus_rules.jsonl @@ -512,6 +512,8 @@ "john doe q.c." "john e smith" "john i" +"john i dr." +"john j v" "john smith EdD" "john smith MEng" "john smith bsc" diff --git a/tools/differential/snapshot.jsonl b/tools/differential/snapshot.jsonl index 1729e4d2..402dba19 100644 --- a/tools/differential/snapshot.jsonl +++ b/tools/differential/snapshot.jsonl @@ -3381,6 +3381,8 @@ {"name": "john e smith", "given": "john", "middle": "e", "family": "smith", "ambiguities": ["CONJUNCTION_OR_INITIAL"], "initials": "j. e. s."} {"name": "john e. smith", "given": "john", "middle": "e.", "family": "smith", "initials": "j. e. s."} {"name": "john i", "given": "john", "family": "i", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "j. i."} +{"name": "john i dr.", "title": "dr.", "given": "john", "family": "i", "ambiguities": ["CONJUNCTION_OR_INITIAL"], "initials": "j. i."} +{"name": "john j v", "given": "john", "middle": "j", "family": "v", "initials": "j. j. v."} {"name": "john smith", "given": "john", "family": "smith", "initials": "j. s."} {"name": "john smith EdD", "given": "john", "family": "smith", "suffix": "EdD", "initials": "j. s."} {"name": "john smith MA", "given": "john", "family": "smith", "suffix": "MA", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "j. s."} From 2e60a5471c63b71d4f42dd9c7c05af75a59be741 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sun, 18 Oct 2026 13:51:40 -0700 Subject: [PATCH 3/9] Declare classify's imported _ROMAN in the regex-sync roster (#721) Co-Authored-By: Claude Opus 5.5 --- tests/v2/test_regex_sync.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/tests/v2/test_regex_sync.py b/tests/v2/test_regex_sync.py index 0e55094c..7bce5a0a 100644 --- a/tests/v2/test_regex_sync.py +++ b/tests/v2/test_regex_sync.py @@ -142,6 +142,8 @@ def test_dotted_initial_is_the_period_alternative_of_initial() -> None: # SAME object, imported for the trailing read's weighed-word test # (#620), as _PERIOD_ABBREV is above ("_pieces", "_ROMAN"): "roman_numeral", + # SAME object, imported for the one-case numeral initial (#721) + ("_classify", "_ROMAN"): "roman_numeral", ("_post_rules", "_EAST_SLAVIC"): "east_slavic_patronymic", ("_post_rules", "_EAST_SLAVIC_CYR"): "east_slavic_patronymic_cyrillic", ("_post_rules", "_TURKIC"): "turkic_patronymic_marker", From 1f37d0ad55c43540018350638ffe1b7d9617fc3a Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sun, 18 Oct 2026 14:53:29 -0700 Subject: [PATCH 4/9] Report a numeral before a credential; keep #718's count (#721 review) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Derek on the PR: 'john i phd' reports suffix-or-name as 'john i, phd' does, and 'John Smith V PhD' keeps family V with the report. Assign, once the trailing read is known, withdraws classify's connective report on the part's last name word when only post-nominals or titles follow it (rules/P3: "unless no name word of its part stands to its right"), and reports a roman numeral there. Fields do not move. Two stricter rules Derek weighed are declined with evidence: 'Hardie Leannon V' (#36) and John W. Ingram V (#136; also written 'John W Ingram V' and 'John Ingram V') write the generation without a comma. Both spellings are now case rows. The review's other defect: the one-case test of the word in front took any cased letter, moving 'IVAN Ж V' and splitting NFD from composed; it now asks whether the letter's capital is initial-shaped. The grid gains a non-ASCII letter in front and 'vii' (master 315 of 1,872 pairs; controls 225 and 117). Comment-review fixes: classify's honorific-peel and own-words claims, the public `initial` tag and CONJUNCTION_OR_INITIAL docstrings, Lexicon and config wording, the predicate's stale caller list, a case note, and the facade test's name. Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 2 +- .../decisions/S10-trailing-roman-numeral.md | 3 + docs/design/rules/P3-connectives.md | 12 +-- .../rules/S10-trailing-roman-numeral.md | 12 ++- nameparser/_lexicon.py | 6 +- nameparser/_pipeline/_assign.py | 43 +++++++++- nameparser/_pipeline/_classify.py | 20 +++-- nameparser/_pipeline/_vocab.py | 16 ++-- nameparser/_types.py | 16 ++-- nameparser/config/conjunctions.py | 4 +- tests/v2/cases.py | 85 +++++++++++++++++-- tests/v2/test_cases.py | 8 +- tests/v2/test_facade.py | 15 ++-- tests/v2/test_properties.py | 14 +-- tests/v2/test_regex_sync.py | 3 + tools/differential/corpus_cjk.jsonl | 1 + tools/differential/corpus_rules.jsonl | 4 + tools/differential/snapshot.jsonl | 9 +- 18 files changed, 210 insertions(+), 63 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index d01ec1d4..bd128ad6 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -173,7 +173,7 @@ The library has two layers: `nameparser/config/` (data) and `nameparser/parser.p **Sweep the forms a change can reach: comma shapes, then `name_order`.** No comma, a FULL name before the comma, a ONE-WORD name before the comma and NOTHING before the comma are four paths, and the last two are the misses — #429, #430 and #432 are all the one-word path disagreeing with the full-name path on inputs the full-name path reads correctly, and #613's PR review found `, PhD` and `, Jr.` (an empty surname field, as CSV data writes it) read as given and title by a decision that returned early on an empty head, after two review rounds whose every grid had a name before the comma. Put an empty head and a nickname-only head (`(Bob), PhD`) in every comma grid. Count the one-word path the way C1 counts, which is neither tokens nor pieces: a particle run and the one word it attaches to are one word (#575), a connective join and a bound given-name pair count as their words (#613, so `Ortega y Gasset, Dr.` is a whole name), and a suffix beside the surname counts as no name word at all. #603's review found the suffix half (`Vega y Lopez Jr., PhD Jones` lost its family where `Vega y Lopez, PhD Jones` kept it); #613 then reversed the connective half, and both now read given 'Vega', middle 'y', family 'Lopez' — so test a particle surname, a connective surname and a surname with a suffix, each before a comma. For orders the sweep already exists (`tests/v2/test_cases.py` runs every row under all three) but carries one assertion, R2's family partition, so it checks nothing a new change moves; coverage there has been vacuous before (PR #394's review found the suite passed with `name_order` discarded from grouping). Parse your change's names in each comma shape and each order, and read the ones you did not predict. And read a generated grid's moves by SHAPE, not by corpus membership: #614's first design passed the suite, the five gates and a fingerprint of every corpus name, because its grid moves were sorted into 'corpus name moved' and 'garbage', and the review found the moves that mattered among the 'garbage' -- a bare salutation (`Mr. and Mrs.` lost its connective to the family name) and a name with one trailing word (`Freiherr von Berg X.Y.Z.`). Pull out the head-plus-one-word slice of the grid and a salutation with nothing after it, and read every move there. Two more were the second review's (#614): no grid put TWO words that are both a title and a particle at the head (`Freiherr St van Berg MA`, where an index into the pieces was applied to a merged view of them -- `TITLES ∩ particles_ambiguous` is `{freiherr, st}`, so put both in every head list, each in its bare and its period-marked spelling -- #637's grid had `St` and `Freiherr` alone and missed that `St.` and `Freiherr.` are declined by the bound join and then attached by P6), and the grids compared ambiguity KINDS, so a report telling the caller the wrong field (`Freiherr von Berg Ma`, 'read as a given name' of a word in the family) passed every pin. #701's review added one: put a title/suffix dual (`Ms`, `Sr.`, `Cpl`) SECOND in a title run, behind an ordinary title. A stop that asks only "is this word suffix vocabulary" passes every row that opens with the dual and fails `Rev. Sr. Smith, Mary`; it was found by comparing `T Smith, John` against `T John Smith` over every title word, and a comma-versus-comma-less comparison is the cheap check for any rule that moves a word between the two slots. Compare each report's `detail` against the field its tokens land in -- since #626 `test_a_report_names_the_field_its_word_lands_in` (tests/v2/test_cases.py) does it over every non-locale case row under all three orders (a row carrying its own policy only as declared), for the phrasings its `_FIELD_CLAIMS` lists -- so a row with the shape is the test only where the report's wording is listed: a new emitter's wording goes in that list, and segment's report on a part past the second comma names no field (#629). #626's own first grid then missed an emitter for want of one comma shape (`, Ma Dr.`: an empty head, an ambiguous word, a trailing title, which H1 moves to the family): a trailing title belongs in every empty-head comma grid too. And #620's: an exclusion list in code being replaced is a set of rules, not noise -- #614's merged copy excluded every word the trailing read weighs from its units, the first #620 prototype dropped the list as a symptom of the copy, and its review found the list had been keeping positional counting's rule that a member in front counts as a word (`Freiherr von Berg MA X.Y.Z.` lost its suffix). Before deleting one, mutate each entry out and read what moves. -**Then sweep the spellings: a class reached by vocabulary is also reached by SHAPE.** A test that uses only the listed spelling of a word class walks only the vocabulary path, and the shape path is a separate branch that can be missing while every test passes. Roman numerals are the sharpest case: `ii`, `iii` and `iv` are suffix vocabulary; `i` and `v` are vocabulary too but initial-shaped, so `is_suffix_piece` vetoes them and only the numeral fork reads them; and `vi` through `x` are in no list at all, read by the fork's shape test alone (checked 2026-10-04 against `Lexicon.default()`) -- which stops at `x`: `_vocab._ROMAN` matches I to X, so `John Smith XI` reads `XI` as a name word (re-checked 2026-10-07, #614's /simplify, which found a docstring claiming "any length"). And the fork counts a ONE-letter numeral besides, since #718 (rules/S10): it is the suffix only where the name has a word to spare, so `John V` and `John Smith V` take different branches of one fork, a one-case name being counted like capitals whatever the letter -- since #721 a one-case one-letter numeral carries the `initial` tag its capital does, and a bare letter in front of the numeral is initial-shaped as its capital is, so `john v` reads as `JOHN V` does over `test_a_one_case_name_reads_as_its_capitals`'s grid (comma shapes, orders, a maiden clause, a particle chain, a bare letter in front, a trailing title: 288 disagreeing pairs on master, none after). Its first version claimed "every shape" with no bare letter in front and no trailing title in the grid, and #721's reviews found both broken; a grid's claim is the grid -- put a head-plus-numeral row beside every words-to-spare row. So a change that handles `V` and `III` can still miss `VI` — #610 found exactly that, the maiden take's candidate loop admitting a shape-only numeral in one position where the peel reads it in two (`Jane Doe nee Smith VI Prof.` kept maiden `Smith VI` while `V` and `III` worked). The same split runs through the credentials (`MA` listed, `X.Y.Z.` by the dotted shape, `XYZ` by the caps shape, and `Ph. D.` merged by group into one flagged piece the peel's walk never holds -- #603 found #602's run-start predicate refusing it too, so `Smith, John Ph. D. Jones` kept middle 'Jones' where `Smith, John PhD Jones` read suffix 'PhD Jones') and the titles (`Prof.` period-marked and read at the end of a name, `Prof` bare and a name word there). When a change touches a word class, test one spelling from each path. The same holds for a separator's SPACING: a comma (or any punctuation) can stand alone, glue to the word before it or glue to the word after, and each is a different token stream. #653's first fix swept `(née, Jones)` and missed `(née , Jones)`, where the kept comma became a word after the marker and M3 read the clause as maiden ', Jones'; its review found it. Put all three spacings in any grid around a separator. +**Then sweep the spellings: a class reached by vocabulary is also reached by SHAPE.** A test that uses only the listed spelling of a word class walks only the vocabulary path, and the shape path is a separate branch that can be missing while every test passes. Roman numerals are the sharpest case: `ii`, `iii` and `iv` are suffix vocabulary; `i` and `v` are vocabulary too but initial-shaped, so `is_suffix_piece` vetoes them and only the numeral fork reads them; and `vi` through `x` are in no list at all, read by the fork's shape test alone (checked 2026-10-04 against `Lexicon.default()`) -- which stops at `x`: `_vocab._ROMAN` matches I to X, so `John Smith XI` reads `XI` as a name word (re-checked 2026-10-07, #614's /simplify, which found a docstring claiming "any length"). And the fork counts a ONE-letter numeral besides, since #718 (rules/S10): it is the suffix only where the name has a word to spare, so `John V` and `John Smith V` take different branches of one fork, a one-case name being counted like capitals whatever the letter -- since #721 a one-case one-letter numeral carries the `initial` tag its capital does, and a bare letter in front of the numeral is initial-shaped as its capital is, so `john v` reads as `JOHN V` does over `test_a_one_case_name_reads_as_its_capitals`'s grid (comma shapes, orders, a maiden clause, a particle chain, a bare letter in front, a trailing title, a non-ASCII letter in front: 315 disagreeing pairs on master, none after). Its first version claimed "every shape" with no bare letter in front and no trailing title in the grid, and #721's reviews found both broken; a grid's claim is the grid -- put a head-plus-numeral row beside every words-to-spare row. So a change that handles `V` and `III` can still miss `VI` — #610 found exactly that, the maiden take's candidate loop admitting a shape-only numeral in one position where the peel reads it in two (`Jane Doe nee Smith VI Prof.` kept maiden `Smith VI` while `V` and `III` worked). The same split runs through the credentials (`MA` listed, `X.Y.Z.` by the dotted shape, `XYZ` by the caps shape, and `Ph. D.` merged by group into one flagged piece the peel's walk never holds -- #603 found #602's run-start predicate refusing it too, so `Smith, John Ph. D. Jones` kept middle 'Jones' where `Smith, John PhD Jones` read suffix 'PhD Jones') and the titles (`Prof.` period-marked and read at the end of a name, `Prof` bare and a name word there). When a change touches a word class, test one spelling from each path. The same holds for a separator's SPACING: a comma (or any punctuation) can stand alone, glue to the word before it or glue to the word after, and each is a different token stream. #653's first fix swept `(née, Jones)` and missed `(née , Jones)`, where the kept comma became a word after the marker and M3 read the clause as maiden ', Jones'; its review found it. Put all three spacings in any grid around a separator. **And the encoding: a letter class written `[^\W\d_]` or tested with `isalpha()` is blind to a combining mark.** Python's `\w` and `isalpha()` do not match Mn, so a letter written as base plus accent (NFD) ends a word early; the vocabulary path is immune because `_normalize` NFC-folds, the shape path is not. #652: `Dr. inž. Janez Novak` read `inž.` as the given name when the ž was decomposed, and `Janez Novak, ŽŠČ` reversed, in two families that share the blind spot — the letter-shape regexes (`period_abbreviation`, `_DOTTED`, `_CHUNKED`, `_PAIRED_INITIALS`), now spelled with `_vocab._LETTER` (a letter plus the general combining blocks; `test_regex_sync` compares the two `_MARKS` strings and pins `period_abbreviation` against config's copy), and the caps predicates, which NFC-compose a non-ASCII word before `isalpha()` (`is_caps_word`, behind C-level `isalpha()`/`is_normalized` tests, so already-composed text pays no frame — measured 2026-10-12, `Smith, José` was 193 frames before and after; the first version called it for every non-ASCII word and cost one frame each, and no guard saw it because the reference name is ASCII). A width test must count letters, not code points (`composed_len`): admitting a decomposed word to the caps set put `John Smith, MÁ` on the pair exception's wrong side, and the review's grid found it where the rows (all three letters long) did not — put a two-letter word in every caps grid. The initial shape (`_INITIAL`, `\w\.`, v1-verbatim and public) is not widened but asked of the COMPOSED word, in `is_initial_shaped` and in `_render`'s two copies; and the single-letter connective tests count composed letters, since a decomposed `й` is two code points (#665, where `É. y É. Smith` and `John й Smith` read differently decomposed). Compose the WHOLE predicate, not one half: `is_initial` is a shape ANDed with the #320 script veto, and composing the shape alone made a decomposed `씨.` an initial, since decomposed hangul is conjoining jamo outside every script range (#665's first version; `in_initialless_script` composes too now). Not covered, deliberately: the marks only some scripts write (Indic vowel signs, Arabic and Hebrew points; rules/H2's Accepted clause). The guard is `test_a_decomposed_name_renders_as_its_composed_twin` over the CLDR corpus plus `test_a_decomposed_abbreviation_reads_as_its_composed_twin` (rows × three orders × EVERYWHERE); the corpus guard reaches a shape only if some corpus name does, which is how the caps shape was missed until review, so when adding a shape rule add a row there. Four of #652's rows and two of #665's pass with every fix reverted (measured) and are parity pins, and the test compares ambiguity kinds and `initials()` too, since #665's `John й Smith` differed in its report; `_DOTTED` has no observable input and is pinned by a direct call. diff --git a/docs/design/decisions/S10-trailing-roman-numeral.md b/docs/design/decisions/S10-trailing-roman-numeral.md index d321047c..57ccb53a 100644 --- a/docs/design/decisions/S10-trailing-roman-numeral.md +++ b/docs/design/decisions/S10-trailing-roman-numeral.md @@ -13,3 +13,6 @@ - 2026-10-18 (Derek), #721 — A ONE-CASE ONE-LETTER NUMERAL IS THE INITIAL ITS CAPITAL IS, AND A LETTER ENDING ITS PART IS NO CONNECTIVE. Two Accepted gaps of the entries above close. First, `john v` read suffix 'v' with no family while `JOHN V` and `john x` were counted: `V` carries the `initial` tag (a bare capital) and every reader of the trailing run honors it — the suffix veto, the particle chain, the credential run — while a lowercase `v` was a plain listed suffix word. A first prototype (not shipped) routed a one-case `v` to the numeral fork at the walk's last position only, and a lower/upper invariant — every one-case name parses as its capitals, fields and report kinds — showed it too shallow (`john v phd`, `john v, phd` and `freiherr von smith v` still disagreed). The fix that holds tags the letter where its capital is tagged: in a name written in one case, a one-letter roman numeral is an `initial` (classify, reading the NAME-level one-case fact so a maiden clause's `v` is one too, as #516's caps branch reads it). And #721's reviews found the numeral fork's own test of the word in front case-dependent: a bare lowercase letter is never initial-shaped, so `john j v` read suffix 'v' where `JOHN J V` reads family 'V' (on master too). In a one-case name a bare cased letter in front now counts as written as an initial. The invariant is `test_a_one_case_name_reads_as_its_capitals` in tests/v2/test_properties.py, its grid and its two recorded controls stated there: master disagreed on 288 of its 1,575 pairs, the branch on none. Second, Derek on the issue: `john i` reporting `conjunction-or-initial` is wrong "because there's nothing for a conjunction to join to there". P3 already said a connective with nothing to its right connects nothing; classify's report now skips a letter that is the last own word of its part — before a comma, a marker's clause or the end. Such a letter is read as an initial and not reported, and where it is a roman numeral the numeral fork reports `suffix-or-name` instead (`john i`, `HENRY I`, `carod i`); a marked letter that is no numeral is simply silent (`john e`, `jones, john e`). Not all such letters are the walk's last word: a trailing title (rules/H5) is an own word to classify that the walk peels first, so in `john i dr.` the letter keeps P3's report and is still the numeral fork's declined pick. The stand-down the entry above added (a letter already reported as an initial reports nothing new) is what keeps that to one report, and it stays: a first draft of this change deleted it as dead on a mutation run over an intermediate tree, and both reviews then found `john i dr.`, `JOHN I DR.` and `henry i prof.` carrying both reports on one token, where master carries `conjunction-or-initial` alone. Two comma rows go silent rather than move — `rovira, i` and `maier, amy i, jr.` — since the comma path has never reported the numeral (decisions/C1, "Not done"). WHAT MOVES, the snapshot against master at 2bfa9533: six corpus names, `john v` (family 'v'), `john smith v` and `larry james edward johnson v` (gain the report), `doe, jane v phd do` (middle 'v', suffix 'phd do', as `DOE, JANE V PHD DO` and `Doe, Jane V PhD Do` read; rules/S7's example keeps its input, whose point — `do` anchored by the degree in front — is unchanged), `john i` (report kind), `jones, john e` (report dropped). And a seventh reading the snapshot shows as a removal rather than a move, since the rules example that pinned it was deleted with S10's Accepted clause: `john nee smith i` read suffix 'i' and now reads maiden 'smith i', reported — the clause-free name is `john i`, whose letter has no word to spare, so M2's clause keeps it, as it keeps `john nee smith v`. And decisions/P3's off switch no longer restores 2.3.0's three one-case readings (that file's #721 entry). +- 2026-10-18 (Derek), #721's PR review — A NUMERAL BEFORE A CREDENTIAL IS REPORTED, AND THE TWO-WORD COUNT STANDS. Derek: `john i phd` should report `suffix-or-name` as `john i, phd` does, and `John Smith V PhD` keeps family 'V' with the report — so the Accepted clause's reading stands and only the report is added; the fields do not move (1.4.0 read the same). The part's last name word with nothing but the trailing run behind it is the site: assign, once the trailing read has said what that run is, withdraws classify's connective report on a marked letter there (rules/P3 now says "unless no name word of its part stands to its right") and reports a roman numeral there, worded for what follows it. So `john i dr.` reports the numeral once, `john e phd` and `jose e jr` report nothing, and `john smith i jr` reports `suffix-or-name` where it reported `conjunction-or-initial`. #718's stand-down is kept as a defensive check, and says so: measured on this commit's tree, the full suite and a 1,680-parse trailing-title sweep fail and pass alike without it. + Declined: two stricter readings Derek weighed on the PR, both measured over the snapshot corpus. (A) a one-letter numeral is the generation only when given, middle and family are all filled — 11 readings move, among them `Hardie Leannon V`; (B) a comma-less one-letter numeral is never the generation — 18 move, among them `John W. Ingram V`. Both are real people who write the numeral without a comma: #36 (2015) reported `Hardie Leannon V` reading last 'V' as a bug, #136 (2022) came from John W. Ingram V writing his name with and without the comma, and Derek found the same sheriff written `John W Ingram V` and, once, `John Ingram V` (Brunswick Beacon, his candidate profile). The two-word count reads all four as the generation; the rows `a_real_generation_with_a_middle_initial` and `a_real_generation_with_no_middle_name` pin the last two. + And the review's other defect: the one-case test of the word in front accepted any one-character cased letter, so `IVAN Ж V` and `JOSÉ É V` moved off their mixed-case reading and a decomposed `é` read otherwise than a composed one. It asks now whether the letter's capital is initial-shaped, which is the rule the docstring states (`john j v` as `JOHN J V`; `é`, `ж` and an uncased word as before). The grid gains a non-ASCII letter in front and a shape-only three-letter numeral; master disagrees on 315 of its 1,872 pairs, the branch on none. diff --git a/docs/design/rules/P3-connectives.md b/docs/design/rules/P3-connectives.md index e6b750ed..96c8c75d 100644 --- a/docs/design/rules/P3-connectives.md +++ b/docs/design/rules/P3-connectives.md @@ -17,11 +17,12 @@ P3. Rationale: connective words ("y", "of the") bind name words into marks as reading both ways. A letter the vocabulary marks as reading both ways, read as an initial in a name written wholly in one case, is a call that could - have gone the other way, and is reported — unless nothing of its - part stands to its right, where it connects nothing and there is + have gone the other way, and is reported — unless no name word + of its part stands to its right, nothing at all or only + post-nominals and titles, where it connects nothing and there is no other way for the call to go: it is read as an initial and - not reported, and where it is a roman numeral S10's fork reports - it instead. + not reported, and where it is a roman numeral ending the name + S10 reports it instead. The joined part is ONE name word wherever another rule counts them, so a rule taking "one name word" takes the whole join and never half of it — except C1's count of the words before a comma, @@ -79,6 +80,7 @@ P3. Rationale: connective words ("y", "of the") bind name words into "John Quincy Smith i" → suffix="i" "Carod i" → suffix="i" "carod i" → ambiguities=("suffix-or-name",) · boundary + "jose e phd" → ambiguities=() · boundary "Josep Lluis Carod i III" → suffix="i III" "Carod y de Rovira i" → middle="y" "John й Smith" → middle="й" @@ -154,5 +156,5 @@ P3. Rationale: connective words ("y", "of the") bind name words into same two words unjoined are two name words and H1 does not fire. P1's leading run is the second (#395, landed): its run takes the "Vega y Santos" join whole or stops before it. - history: decisions/P3 · interacts: H1, P1, M2, R1, R3, R4, C1, S6 · implemented: nameparser/_pipeline/_classify.py, nameparser/_pipeline/_comma.py, nameparser/_pipeline/_group.py, nameparser/_pipeline/_pieces.py, nameparser/_pipeline/_post_rules.py + history: decisions/P3 · interacts: H1, P1, M2, R1, R3, R4, C1, S6, S10 · implemented: nameparser/_pipeline/_assign.py, nameparser/_pipeline/_classify.py, nameparser/_pipeline/_comma.py, nameparser/_pipeline/_group.py, nameparser/_pipeline/_pieces.py, nameparser/_pipeline/_post_rules.py diff --git a/docs/design/rules/S10-trailing-roman-numeral.md b/docs/design/rules/S10-trailing-roman-numeral.md index 91f20544..03e6911b 100644 --- a/docs/design/rules/S10-trailing-roman-numeral.md +++ b/docs/design/rules/S10-trailing-roman-numeral.md @@ -18,8 +18,7 @@ S10. Rationale: a roman numeral ending a name is its generation, shape alone reports too; a numeral the vocabulary lists in two or more letters is not in question and reports nothing, and a letter already reported as read as an initial reports nothing - new: where a title stands after a marked letter (P3), P3's - report is the one. Written case + new. Written case settles the fork in one way only: a numeral written in lower case in a name of mixed case is the generation and reports nothing, since nobody writes an initial that way. A name written @@ -30,6 +29,8 @@ S10. Rationale: a roman numeral ending a name is its generation, "John Smith V" → suffix="V" "John Smith V" → ambiguities=("suffix-or-name",) "John W. Ingram V" → suffix="V" + "John W Ingram V" → suffix="V" + "John Ingram V" → suffix="V" "John Smith VI" → suffix="VI" "John Smith VI" → ambiguities=("suffix-or-name",) "John Smith III" → ambiguities=() @@ -49,10 +50,13 @@ S10. Rationale: a roman numeral ending a name is its generation, "john smith v" → ambiguities=("suffix-or-name",) "john i" → family="i" "john i" → ambiguities=("suffix-or-name",) - "john i dr." → ambiguities=("conjunction-or-initial",) + "john i dr." → ambiguities=("suffix-or-name",) "john j v" → family="v" Accepted: a numeral with another suffix word behind it is not the last word of the name and is read as any word there is, so a - one-letter numeral before a credential is the family name. + one-letter numeral before a credential is the family name, and + the reading is reported, as either reading of the fork is. "John Smith V PhD" → family="V" + "John Smith V PhD" → ambiguities=("suffix-or-name",) + "john i phd" → ambiguities=("suffix-or-name",) history: decisions/S10 · interacts: S2, S4, S7, C1, P2, P3, M2, H1, H3 · implemented: nameparser/_pipeline/_assign.py, nameparser/_pipeline/_classify.py, nameparser/_pipeline/_group.py, nameparser/_pipeline/_pieces.py, nameparser/_pipeline/_vocab.py diff --git a/nameparser/_lexicon.py b/nameparser/_lexicon.py index 06acc453..6331fc0c 100644 --- a/nameparser/_lexicon.py +++ b/nameparser/_lexicon.py @@ -644,8 +644,10 @@ class Lexicon: #: decided by the writing instead and never consults this set, and #: neither does a caseless letter (Arabic و), which has no case to #: read. A member additionally reports - #: :attr:`~nameparser.AmbiguityKind.CONJUNCTION_OR_INITIAL`; a - #: non-member reports nothing, its reading not being in doubt. + #: :attr:`~nameparser.AmbiguityKind.CONJUNCTION_OR_INITIAL` where a + #: name word of its part follows it (with none, it connects + #: nothing); a non-member reports nothing, its reading not being in + #: doubt. #: Full default list: #: :data:`~nameparser.config.conjunctions.CONJUNCTIONS_AMBIGUOUS`. #: Entries need not also be in ``conjunctions``; one that is not is diff --git a/nameparser/_pipeline/_assign.py b/nameparser/_pipeline/_assign.py index f4a885b6..c8186746 100644 --- a/nameparser/_pipeline/_assign.py +++ b/nameparser/_pipeline/_assign.py @@ -89,7 +89,8 @@ from nameparser._lexicon import Lexicon from nameparser._pipeline._vocab import ( - effective_script, is_suffix_lenient, resolve_script_set, + _ROMAN, effective_script, is_suffix_lenient, is_trailing_numeral_suffix, + numeral_settled_by_case, resolve_script_set, ) from nameparser._pipeline._pieces import ( anchor_in_reach, credential_at_the_given_slot, given_slot_anchors, @@ -308,8 +309,13 @@ def _placed(read: TailRead, rest: list[int], # rules/S10: "a letter already reported as read as an initial reports # nothing new" -- asked only by the numeral fork's declined pick -# (#718). The maiden clause's report on the numeral it keeps cannot -# meet the case: classify reads no connective inside a clause +# (#718). Defensive since #721: classify withdraws its report on a +# letter ending its part, and the check below on the part's last name +# word withdraws it before a trailing run, so no input reached it on +# #721's final tree (its full suite and a 1,680-parse trailing-title +# sweep fail and pass alike without it). The maiden clause's report +# on the numeral it keeps cannot meet the case: classify reads no +# connective inside a clause def _reported_as_initial(ambiguities: Sequence[PendingAmbiguity], index: int) -> bool: """Whether classify's connective-or-initial fork already reported @@ -540,6 +546,37 @@ def _assign_main(seg_idx: int, state: ParseState, f"decides it; read as ", tuple(head), field_tail=" by convention, which follows the read " "order")) + # rules/P3: "unless no name word of its part stands to its right" + # and rules/S10: "a one-letter numeral before a credential is the + # family name, and the reading is reported" -- the part's last name + # word with nothing but the trailing run behind it (#721, Derek: + # 'john i phd' reports as 'john i, phd' does). Decided here, where + # the trailing read has said what that run is; classify's own + # end-of-part test covers a letter with nothing at all behind it + if name_pieces and (suffix_pieces or titled_tail): + last = pieces[name_pieces[-1]] + word = tokens[last[0]].text + if len(last) == 1: + # a connective with only post-nominals or titles to its + # right connects nothing; reported only in a one-case name + if state.one_case: + ambiguities[:] = [ + a for a in ambiguities + if a.kind is not AmbiguityKind.CONJUNCTION_OR_INITIAL + or a.indices != (last[0],)] + # `_ROMAN` first, C-level, so an ordinary name word pays + # no frame for the rest + if (_ROMAN.match(word) is not None and len(name_pieces) >= 2 + and tuple(last) not in peeled.picks + and is_trailing_numeral_suffix( + word, tokens[pieces[name_pieces[-2]][0]].text, + state.one_case) + and not numeral_settled_by_case(word, state.one_case)): + ambiguities.append(PendingAmbiguity( + AmbiguityKind.SUFFIX_OR_NAME, + f"{word!r} is a roman numeral, but a post-nominal or " + f"title follows it; read as ", tuple(last), + field_tail=" rather than a generational suffix")) for piece in peeled.picks: # every pick is in rest, so the loops above just gave it a # role; whether it was read as a suffix or a name settles what diff --git a/nameparser/_pipeline/_classify.py b/nameparser/_pipeline/_classify.py index 0566fa54..139dde64 100644 --- a/nameparser/_pipeline/_classify.py +++ b/nameparser/_pipeline/_classify.py @@ -163,10 +163,13 @@ def _tags_for(token: WorkToken, n: str, state: ParseState, # the Spanish conjunction 'e' # rules/S10: "A name written wholly in one case says nothing # about any letter in it" -- so a one-letter roman numeral - # there is the initial its capital is, and every reader that - # vetoes 'V' vetoes 'v' too (#721). The NAME-level fact, as - # #516's caps branch reads it: a capital 'V' in a maiden clause - # is an initial, so a one-case 'v' there is one too. One + # there is the initial its capital is, and every reader of the + # tag after classify vetoes 'v' as it vetoes 'V' (#721) -- not + # the honorific peel upstream, which asks the shape itself + # ('田中さん v' still reads otherwise than '田中さん V'). + # `one_case` is the fact about the name's own words, applied to + # a clause's words too, as #516's caps branch applies it, since + # a capital 'V' in a clause is an initial as well. One # character already, so `_ROMAN` alone asks what # `is_single_letter_numeral` would, at C level: no frame per # one-letter word @@ -349,10 +352,11 @@ def classify(state: ParseState) -> ParseState: # LOWERCASE letter is tagged "initial" only by the fork's # branch ("jose e maria santos" tags a lowercase e), for a # conjunctions_ambiguous member, or by the one-case numeral - # (#721), which is no connective and fails `conjunctions` - # below unless the caller made it one. So for a letter of - # EITHER case in `conjunctions`, "initial" plus len 1 implies - # membership. It is + # (#721) -- which an own word reaches only when it is no + # single-letter connective, that taking the fork's branch + # first, while a clause word fails `i < clause_at` below. So + # for a letter of EITHER case in `conjunctions`, "initial" plus + # len 1 implies membership. It is # kept only because `_tags_for` starts from `set(token.tags)`, so # this clause is what keeps the emitter honest if a runner ever # hands classify tokens it did not build; no such path exists diff --git a/nameparser/_pipeline/_vocab.py b/nameparser/_pipeline/_vocab.py index 31b9230a..f8609b91 100644 --- a/nameparser/_pipeline/_vocab.py +++ b/nameparser/_pipeline/_vocab.py @@ -258,8 +258,9 @@ def is_initial_shaped(text: str) -> bool: # all three are bare capitals, which inside a name are middle initials. def is_trailing_numeral_suffix(text: str, preceding: str, one_case: bool | None = None) -> bool: - """assign's roman-numeral fork, shared with group's bound-given - reserve (#401): a FINAL single-token piece that is a roman numeral + """The trailing walk's roman-numeral fork, shared with + `trailing_candidates`' superset view and assign's report on a + numeral before a credential (#721): a FINAL single-token piece that is a roman numeral reads as the suffix when the piece before it does not look like part of an initial run -- for a one-letter numeral a necessary condition only, the caller's words-to-spare count deciding @@ -274,14 +275,15 @@ def is_trailing_numeral_suffix(text: str, preceding: str, suppressing the fork and cost 'John 씨. V' its family name. `one_case` is ParseState.one_case: in a name written in one case - a bare cased letter in front is initial-shaped as its capital is - (rules/S10, #721's review: 'john j v' reads as 'JOHN J V'). A - caller passing None, the bound-given reserve's superset, gets the + the word in front is read as its capital would be, so a bare 'j' + is initial-shaped as 'J' is (rules/S10, #721's review: 'john j v' + reads as 'JOHN J V'), while 'é', 'ж' or an uncased word, whose + capitals are not, stay as they were. A caller passing None, + `trailing_candidates`' superset for the maiden take, gets the shape test alone, which admits more and so stays a superset.""" return (_ROMAN.match(text) is not None and not is_initial_shaped(preceding) - and not (one_case and len(preceding) == 1 - and preceding.upper() != preceding.lower())) + and not (one_case and is_initial_shaped(preceding.upper()))) # rules/S10: "a numeral written in lower case in a name of mixed case is diff --git a/nameparser/_types.py b/nameparser/_types.py index 246a8c25..b6b1f6aa 100644 --- a/nameparser/_types.py +++ b/nameparser/_types.py @@ -89,7 +89,8 @@ def __add__(self, other: object) -> NoReturn: # type: ignore[override] #: initials ("J." or "А.", never "씨.", #320), or a marked #: single-letter connective in a name written in one case #: (rules/P3, see CONJUNCTION_OR_INITIAL), or a one-letter roman -#: numeral in a name written in one case, as its capital is (rules/S10); +#: numeral in a name whose own words are written in one case, as its +#: capital is (rules/S10); #: "joined" a continuation of the token before it -- within one #: merged piece the tag is role-blind and every view joins the pair #: with a space ("Ph." + "D."; 'Ph. D., Jr.' gives last_list @@ -626,11 +627,14 @@ class AmbiguityKind(StrEnum): #: doctrine, rules/P3): appending " née Jones" or a nickname #: changes no report, because the clause's words were never #: eligible for it. - #: A letter that ends its part -- nothing of the name after it - #: before a comma, a clause or the end, as in "JOSE E" or - #: "rovira, i" -- connects nothing, so it is read as an initial and - #: not reported; a roman numeral there is reported by the numeral - #: fork instead (SUFFIX_OR_NAME). + #: A letter with no name word after it in its part -- nothing + #: before a comma, a clause or the end, or only post-nominals and + #: titles, as in "JOSE E", "rovira, i" or "jose e phd" -- connects + #: nothing, so it is read as an initial and not reported. Where + #: such a letter is a roman numeral at the end of the name the + #: numeral fork reports it instead (SUFFIX_OR_NAME: "HENRY I", + #: "john i phd"); one ending a comma part reports nothing + #: ("rovira, i"). #: ``detail`` names the token, the kind naming neither the field nor #: the letter: which field the reading lands in follows the name's #: shape and its ``name_order``, the PARTICLE_OR_GIVEN precedent. diff --git a/nameparser/config/conjunctions.py b/nameparser/config/conjunctions.py index 25229b32..ebe552b5 100644 --- a/nameparser/config/conjunctions.py +++ b/nameparser/config/conjunctions.py @@ -56,7 +56,9 @@ # 'i' (Catalan/Polish) ships here beside 'e' for the same reason # (#397): a bare I initial is as common as a bare E, so a name # written wholly in one case reads the letter as an initial and - # reports the fork rather than joining in silence. + # reports the fork rather than joining in silence -- where a name + # word follows it in its part, there being nothing to join + # otherwise (#721). # # A caller edits the vocabulary rather than a switch: remove 'e' to # restore joining for Portuguese data, remove 'i' for Catalan or diff --git a/tests/v2/cases.py b/tests/v2/cases.py index 06fb90a6..363fb792 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -1582,11 +1582,11 @@ def _check_cjk_shape_purity(self) -> None: "case it leans nothing, so P6's attachment would take " "it (as in 'NASCIMENTO, EDSON ARANTES DO'); the degree " "in front outranks the attachment as the capitals do " - "(Derek, 2026-09-27). The roles are 1.4.0's; 2.3.0 " - "read family 'do doe', and 1.4.0 wrote the suffix " - "'v, phd, do'. The 'v' is the one-case initial its " - "capital is since #721, as 'DOE, JANE V PHD DO' reads " - "middle 'V' (rules/S10)", + "(Derek, 2026-09-27). 1.4.0 read suffix 'v, phd, do' " + "and 2.3.0 family 'do doe'. The 'v' is the one-case " + "initial its capital is since #721, as 'DOE, JANE V " + "PHD DO' reads middle 'V' (S10, the trailing roman " + "numeral), where 1.4.0 and #718 read it in the suffix", shape=2), Case("by_design_trailing_mc_reads_as_a_credential", "Donald Mc", {"given": "Donald", "suffix": "Mc"}, @@ -4520,7 +4520,7 @@ def _check_cjk_shape_purity(self) -> None: {"given": "john", "middle": "smith", "family": "i", "suffix": "jr"}, classification="fix(#397)", - ambiguities=("conjunction-or-initial",), + ambiguities=("suffix-or-name",), notes="the same movement with a generation behind it: the " "letter reads as an initial, so the peel stops before " "it and 'jr' alone is the suffix. 2.3.0 read family " @@ -4528,7 +4528,10 @@ def _check_cjk_shape_purity(self) -> None: "The ALL-CAPS twin 'JOHN SMITH I JR' read middle " "'SMITH', last 'I', suffix 'JR' at both of them -- " "again the reading this row now gives. No shape " - "tag, for the reason the row above gives"), + "tag, for the reason the row above gives. Since #721 " + "the letter, with only 'jr' behind it, connects nothing " + "(P3, the connectives rule), so the report is the " + "numeral's"), Case("a_one_case_link_in_a_comma_suffix_run_is_a_middle_initial", "maier, amy i, jr.", {"given": "amy", "middle": "i", "family": "maier", @@ -7557,6 +7560,74 @@ def _check_cjk_shape_purity(self) -> None: ambiguities=("suffix-or-name",), classification="fix(#274)", notes="the contrast that makes the silence above a decision " "about case"), + Case("a_real_generation_with_a_middle_initial", "John W Ingram V", + {"given": "John", "middle": "W", "family": "Ingram", + "suffix": "V"}, + ambiguities=("suffix-or-name",), + notes="a sheriff in North Carolina, consistently written this " + "way (Derek, #721); #136's reporter wrote the same name " + "'John W. Ingram V' and '..., V', expecting the suffix " + "both ways -- the evidence that kept #718's two-word " + "count against reading a comma-less letter as never " + "the generation"), + Case("a_real_generation_with_no_middle_name", "John Ingram V", + {"given": "John", "family": "Ingram", "suffix": "V"}, + ambiguities=("suffix-or-name",), + notes="the same sheriff as a local paper wrote him (Derek, " + "#721), two name words and the numeral: the shape #36's " + "'Hardie Leannon V' was reported as a bug for reading " + "otherwise in 2015"), + Case("a_lone_numeral_before_a_credential_reports", + "John Smith V PhD", + {"given": "John", "middle": "Smith", "family": "V", + "suffix": "PhD"}, + ambiguities=("suffix-or-name",), + notes="S10's Accepted clause: a numeral with a credential " + "behind it is the family name, and since #721 the " + "reading is reported (Derek). 1.4.0 read the same " + "fields"), + Case("a_one_case_link_before_a_credential_reports_the_numeral", + "john i phd", + {"given": "john", "family": "i", "suffix": "phd"}, + ambiguities=("suffix-or-name",), + notes="the letter has only a credential behind it, so it " + "connects nothing (P3) and the numeral is what reports, " + "as for 'john i, phd' (Derek, #721)"), + Case("a_one_case_link_before_a_title_reports_the_numeral", + "john i dr.", + {"title": "dr.", "given": "john", "family": "i"}, + ambiguities=("suffix-or-name",), classification="fix(#316)", + notes="the same with a trailing title (H5): one report on " + "the letter, the numeral's. #721's first form reported " + "both kinds on it. 1.4.0 read suffix 'dr.'"), + Case("a_one_case_non_numeral_link_before_a_credential_is_silent", + "jose e phd", + {"given": "jose", "family": "e", "suffix": "phd"}, + notes="nothing to join and no numeral fork: no report"), + Case("a_one_case_link_before_a_maiden_marker_is_silent", + "jose e nee smith", + {"given": "jose", "family": "e", "maiden": "smith"}, + classification="fix(#274)", + notes="the clause is one of the ends of a part, so the letter " + "before the marker connects nothing (P3, #721); 1.4.0 " + "had no maiden markers"), + Case("a_one_case_letter_ending_a_comma_part_is_silent", + "jones, john e", + {"given": "john", "middle": "e", "family": "jones"}, + notes="a corpus name: 'e' ends its part, so it connects " + "nothing, and being no numeral nothing else reports"), + Case("a_one_case_bare_letter_in_front_keeps_the_numeral_a_name", + "john j v", {"given": "john", "middle": "j", "family": "v"}, + classification="fix(#721)", + notes="read as 'JOHN J V' is: in a one-case name the bare 'j' " + "in front is the initial its capital is (S10). 1.4.0 " + "read last 'j', suffix 'v'"), + Case("an_uncased_word_in_front_is_no_initial", + "김 v", {"given": "김", "family": "v"}, + ambiguities=("suffix-or-name",), + notes="hangul has no capital, so the one-case test of the word " + "in front does not make it an initial; the count reads " + "the numeral as it reads '김 V'"), Case("a_lowercase_numeral_in_a_one_case_name_still_reports", "john smith i", {"given": "john", "family": "smith", "suffix": "i"}, diff --git a/tests/v2/test_cases.py b/tests/v2/test_cases.py index e8ee387a..4da1e123 100644 --- a/tests/v2/test_cases.py +++ b/tests/v2/test_cases.py @@ -191,9 +191,11 @@ def test_the_field_sweep_sees_the_claims_it_checks() -> None: # was read into, on those rows under each order; 1191 -> 1210 the # same day: the PR review's rows (the chain-taken, family-first, # lowercase and maiden-clause numerals); 1210 -> 1213 the same day: - # #721's one-case lowercase numerals, counted and reported. - assert claims == 1213, ( - f"the field sweep checks {claims} claims, recorded as 1213 on " + # #721's one-case lowercase numerals, counted and reported; 1213 + # -> 1234 the same day: #721's PR-review rows (the numeral before a + # credential or title, the real generations, the lone letters). + assert claims == 1234, ( + f"the field sweep checks {claims} claims, recorded as 1234 on " f"2026-10-18") diff --git a/tests/v2/test_facade.py b/tests/v2/test_facade.py index 80a2adfd..8d40bf5f 100644 --- a/tests/v2/test_facade.py +++ b/tests/v2/test_facade.py @@ -1050,17 +1050,16 @@ def test_initials_of_an_unpickled_or_copied_name_ask_the_vocabulary_too() -> Non assert str(built) == restored_cap -def test_the_v1_off_switch_restores_the_pre_397_fields() -> None: +def test_the_v1_off_switch_unjoins_the_link_only() -> None: """decisions/P3's off switch on the V1 SURFACE (#397 second - review). The bullet says deleting `i` from `C.conjunctions` - restores the parent's readings; the core-side switch is pinned by - the off-switch grid in tests/v2/test_properties.py and this is the - facade half, which has no `Lexicon` of its own to remove from. + review; named for what it restored, the pre-#397 fields, until + #721). The core-side switch is pinned by the off-switch grid in + tests/v2/test_properties.py and this is the facade half, which + has no `Lexicon` of its own to remove from. The three role movers of decisions/P3's one-case paragraph are - the rows, because they are the ones whose FIELDS move: every - value on the right below was what the released 2.3.0 wheel gives, - measured 2026-09-20 -- until #721, which made a one-case + the rows: the switch restored their 2.3.0 fields, measured + 2026-09-20 -- until #721, which made a one-case one-letter numeral the initial its capital is (rules/S10), so with `i` out of the conjunctions the letter now reads as 'I' does in each, switch on or off. What the switch still moves is the join, diff --git a/tests/v2/test_properties.py b/tests/v2/test_properties.py index dddf6fb8..cfacb73e 100644 --- a/tests/v2/test_properties.py +++ b/tests/v2/test_properties.py @@ -3344,7 +3344,7 @@ def test_every_one_letter_particle_veto_is_load_bearing( # after it -- with the numerals a listed and a shape-only spelling of # each length, three heads (one Catalan, one holding an ambiguous # acronym) and all three orders: -# 1,575 pairs, about 3,150 parses, 0.24s on py3.11 (2026-10-18). +# 1,872 pairs, about 3,750 parses, 0.29s on py3.11 (2026-10-18). _ONE_CASE_SHAPES = ( "{a} {n}", "{a} {b} {n}", "{a} {n}, phd", "{a} {b} {n}, phd", "{a} {n}, jr.", "{b}, {a} {n}", ", {a} {n}", "{a} nee {b} {n}", @@ -3354,16 +3354,18 @@ def test_every_one_letter_particle_veto_is_load_bearing( # #721's review: a bare letter in front, a trailing title, a # doubled numeral and a trailing period "{a} b {n}", "{a} e {n}", "{a} {n} prof.", "{a} {b} {n} prof.", - "{a} {b} {n} {n}", "{a} {b} {n}.") -_ONE_CASE_NUMERALS = ("v", "i", "x", "vi", "iii", "ii", "iv") + "{a} {b} {n} {n}", "{a} {b} {n}.", + # a non-ASCII letter in front, whose capital is no initial + "{a} é {n}") +_ONE_CASE_NUMERALS = ("v", "i", "x", "vi", "iii", "ii", "iv", "vii") _ONE_CASE_HEADS = (("john", "smith", "jones"), ("carod", "puig", "rovira"), ("mary", "ma", "lee")) #: the negative controls, measured 2026-10-18 on this grid, which -#: master (2bfa9533) disagreed on 288 of: with classify's one-case +#: master (2bfa9533) disagreed on 315 of: with classify's one-case #: numeral initial taken out, and with the numeral fork's one-case #: letter in front taken out -- each guard seen on its own -_ONE_CASE_UNGUARDED = 216 -_ONE_CASE_LETTER_IN_FRONT_UNGUARDED = 99 +_ONE_CASE_UNGUARDED = 225 +_ONE_CASE_LETTER_IN_FRONT_UNGUARDED = 117 def _one_case_disagreements() -> list[str]: diff --git a/tests/v2/test_regex_sync.py b/tests/v2/test_regex_sync.py index 7bce5a0a..03338891 100644 --- a/tests/v2/test_regex_sync.py +++ b/tests/v2/test_regex_sync.py @@ -144,6 +144,9 @@ def test_dotted_initial_is_the_period_alternative_of_initial() -> None: ("_pieces", "_ROMAN"): "roman_numeral", # SAME object, imported for the one-case numeral initial (#721) ("_classify", "_ROMAN"): "roman_numeral", + # SAME object, imported for the report on a numeral before a + # credential (#721) + ("_assign", "_ROMAN"): "roman_numeral", ("_post_rules", "_EAST_SLAVIC"): "east_slavic_patronymic", ("_post_rules", "_EAST_SLAVIC_CYR"): "east_slavic_patronymic_cyrillic", ("_post_rules", "_TURKIC"): "turkic_patronymic_marker", diff --git a/tools/differential/corpus_cjk.jsonl b/tools/differential/corpus_cjk.jsonl index 5bf09ad6..d18a8191 100644 --- a/tools/differential/corpus_cjk.jsonl +++ b/tools/differential/corpus_cjk.jsonl @@ -109,6 +109,7 @@ "高橋・一郎" "高橋一郎" "鵜殿" +"김 v" "김 민준" "김, 민준 MA" "김, 민준씨" diff --git a/tools/differential/corpus_rules.jsonl b/tools/differential/corpus_rules.jsonl index fab487b4..72b2e87c 100644 --- a/tools/differential/corpus_rules.jsonl +++ b/tools/differential/corpus_rules.jsonl @@ -204,6 +204,7 @@ "John . Smith" "John Doctor Smith" "John III" +"John Ingram V" "John J. V" "John Ma" "John PhD Smith" @@ -279,6 +280,7 @@ "John Smith, vd Ma" "John V" "John V, Dr." +"John W Ingram V" "John W. Ingram V" "John and Jane Smith" "John nee Prof. ba MA" @@ -513,6 +515,7 @@ "john e smith" "john i" "john i dr." +"john i phd" "john j v" "john smith EdD" "john smith MEng" @@ -542,6 +545,7 @@ "john v" "john van der berg ma" "jose e maria santos" +"jose e phd" "jose ortega-y-gasset" "josé garcía" "juan de la vega" diff --git a/tools/differential/snapshot.jsonl b/tools/differential/snapshot.jsonl index 402dba19..83f7ad15 100644 --- a/tools/differential/snapshot.jsonl +++ b/tools/differential/snapshot.jsonl @@ -1490,6 +1490,7 @@ {"name": "John Hámish Watson", "given": "John", "middle": "Hámish", "family": "Watson", "initials": "J. H. W."} {"name": "John I Smith", "given": "John", "middle": "I", "family": "Smith", "initials": "J. I. S."} {"name": "John III", "given": "John", "suffix": "III", "ambiguities": ["GIVEN_OR_FAMILY"], "initials": "J."} +{"name": "John Ingram V", "given": "John", "family": "Ingram", "suffix": "V", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. I."} {"name": "John J. V", "given": "John", "middle": "J.", "family": "V", "initials": "J. J. V."} {"name": "John Jack Andrew Kennedy", "given": "John", "middle": "Jack Andrew", "family": "Kennedy", "initials": "J. J. A. K."} {"name": "John Jones (Google Docs)", "given": "John", "family": "Jones", "nickname": "Google Docs", "initials": "J. J."} @@ -1547,7 +1548,7 @@ {"name": "John Smith Rev.", "title": "Rev.", "given": "John", "family": "Smith", "initials": "J. S."} {"name": "John Smith Sir", "given": "John", "middle": "Smith", "family": "Sir", "initials": "J. S. S."} {"name": "John Smith V", "given": "John", "family": "Smith", "suffix": "V", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. S."} -{"name": "John Smith V PhD", "given": "John", "middle": "Smith", "family": "V", "suffix": "PhD", "initials": "J. S. V."} +{"name": "John Smith V PhD", "given": "John", "middle": "Smith", "family": "V", "suffix": "PhD", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. S. V."} {"name": "John Smith V.", "given": "John", "middle": "Smith", "family": "V.", "initials": "J. S. V."} {"name": "John Smith VI", "given": "John", "family": "Smith", "suffix": "VI", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. S."} {"name": "John Smith X.Y.Z.", "given": "John", "family": "Smith", "suffix": "X.Y.Z.", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. S."} @@ -1612,6 +1613,7 @@ {"name": "John V", "given": "John", "family": "V", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. V."} {"name": "John V, Dr.", "title": "Dr.", "given": "John", "family": "V", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. V."} {"name": "John V. Smith", "given": "John", "middle": "V.", "family": "Smith", "initials": "J. V. S."} +{"name": "John W Ingram V", "given": "John", "middle": "W", "family": "Ingram", "suffix": "V", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. W. I."} {"name": "John W. Ingram", "given": "John", "middle": "W.", "family": "Ingram", "initials": "J. W. I."} {"name": "John W. Ingram V", "given": "John", "middle": "W.", "family": "Ingram", "suffix": "V", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. W. I."} {"name": "John W. Ingram, V", "given": "John", "middle": "W.", "family": "Ingram", "suffix": "V", "initials": "J. W. I."} @@ -3381,7 +3383,8 @@ {"name": "john e smith", "given": "john", "middle": "e", "family": "smith", "ambiguities": ["CONJUNCTION_OR_INITIAL"], "initials": "j. e. s."} {"name": "john e. smith", "given": "john", "middle": "e.", "family": "smith", "initials": "j. e. s."} {"name": "john i", "given": "john", "family": "i", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "j. i."} -{"name": "john i dr.", "title": "dr.", "given": "john", "family": "i", "ambiguities": ["CONJUNCTION_OR_INITIAL"], "initials": "j. i."} +{"name": "john i dr.", "title": "dr.", "given": "john", "family": "i", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "j. i."} +{"name": "john i phd", "given": "john", "family": "i", "suffix": "phd", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "j. i."} {"name": "john j v", "given": "john", "middle": "j", "family": "v", "initials": "j. j. v."} {"name": "john smith", "given": "john", "family": "smith", "initials": "j. s."} {"name": "john smith EdD", "given": "john", "family": "smith", "suffix": "EdD", "initials": "j. s."} @@ -3418,6 +3421,7 @@ {"name": "johnny y", "given": "johnny", "family": "y", "initials": "j. y."} {"name": "jones, john e", "given": "john", "middle": "e", "family": "jones", "initials": "j. e. j."} {"name": "jose e maria santos", "given": "jose", "middle": "e maria", "family": "santos", "ambiguities": ["CONJUNCTION_OR_INITIAL"], "initials": "j. e. m. s."} +{"name": "jose e phd", "given": "jose", "family": "e", "suffix": "phd", "initials": "j. e."} {"name": "jose ortega-y-gasset", "given": "jose", "family": "ortega-y-gasset", "initials": "j. o."} {"name": "josep carod i rovira", "given": "josep", "middle": "carod i", "family": "rovira", "ambiguities": ["CONJUNCTION_OR_INITIAL"], "initials": "j. c. i. r."} {"name": "josé garcía", "given": "josé", "family": "garcía", "initials": "j. g."} @@ -5715,6 +5719,7 @@ {"name": "공옥진", "given": "옥진", "family": "공", "initials": "옥. 공."} {"name": "권재진", "given": "재진", "family": "권", "initials": "재. 권."} {"name": "권중동", "given": "중동", "family": "권", "initials": "중. 권."} +{"name": "김 v", "given": "김", "family": "v", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "김. v."} {"name": "김 민준", "given": "민준", "family": "김", "initials": "민. 김."} {"name": "김, 민준 MA", "given": "민준", "family": "김", "suffix": "MA", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "민. 김."} {"name": "김, 민준씨", "given": "민준", "family": "김", "suffix": "씨", "initials": "민. 김."} From 3d3b4473726610f8fd16a986cf7869b8bbf0a0f3 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sun, 18 Oct 2026 14:58:36 -0700 Subject: [PATCH 5/9] Pin the non-ASCII letter in front, which the case-symmetric grid cannot see (#721 review) Co-Authored-By: Claude Opus 5.5 --- tests/v2/cases.py | 15 +++++++++++++++ tests/v2/test_cases.py | 7 ++++--- 2 files changed, 19 insertions(+), 3 deletions(-) diff --git a/tests/v2/cases.py b/tests/v2/cases.py index 363fb792..da9a60bd 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -7628,6 +7628,21 @@ def _check_cjk_shape_purity(self) -> None: notes="hangul has no capital, so the one-case test of the word " "in front does not make it an initial; the count reads " "the numeral as it reads '김 V'"), + Case("a_non_ascii_capital_in_front_is_no_initial", "IVAN Ж V", + {"given": "IVAN", "family": "Ж", "suffix": "V"}, + ambiguities=("suffix-or-name",), + notes="the one-case test of the word in front asks whether its " + "CAPITAL is initial-shaped, and 'Ж' is not (an initial is " + "a Latin capital), so this reads as 'Ivan Ж V' does. " + "#721's first form took any cased letter and read middle " + "'Ж', family 'V'"), + Case("a_non_ascii_lowercase_letter_in_front_is_no_initial", + "josé é v", + {"given": "josé", "family": "é", "suffix": "v"}, + ambiguities=("suffix-or-name",), + notes="the lowercase twin of the row above, and the one whose " + "decomposed spelling had read otherwise under #721's " + "first form"), Case("a_lowercase_numeral_in_a_one_case_name_still_reports", "john smith i", {"given": "john", "family": "smith", "suffix": "i"}, diff --git a/tests/v2/test_cases.py b/tests/v2/test_cases.py index 4da1e123..98624c88 100644 --- a/tests/v2/test_cases.py +++ b/tests/v2/test_cases.py @@ -193,9 +193,10 @@ def test_the_field_sweep_sees_the_claims_it_checks() -> None: # lowercase and maiden-clause numerals); 1210 -> 1213 the same day: # #721's one-case lowercase numerals, counted and reported; 1213 # -> 1234 the same day: #721's PR-review rows (the numeral before a - # credential or title, the real generations, the lone letters). - assert claims == 1234, ( - f"the field sweep checks {claims} claims, recorded as 1234 on " + # credential or title, the real generations, the lone letters); + # 1234 -> 1240: the two non-ASCII letter-in-front rows. + assert claims == 1240, ( + f"the field sweep checks {claims} claims, recorded as 1240 on " f"2026-10-18") From 6e84cb155a8347a1d5d1f3864db67b0d9242901c Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sun, 18 Oct 2026 15:18:43 -0700 Subject: [PATCH 6/9] Withdraw an unjoined connective's report once, over roles (#722 review) The last review found P3's broadened exception honored only where the part's last name word was one token on the no-comma path: a letter ending a particle chain ('john van e phd'), the given part after a family comma ('smith, john e phd') and an empty head kept their connective report, and 'freiherr von i dr.' reached #718's stand-down and lost the numeral's report. One role-based pass at the end of assign now withdraws classify's report on a marked letter that is its segment's last name-role word with only suffixes and titles after it; the numeral's report is emitted before, so the stand-down is gone. The maiden clause gains the numeral report's twin ('Jane Doe nee Smith V PhD'). Records: option B moves 19, not 18 (recipe stated); #36/#136 are not shown to be real people (the sheriff is); 1.4.0 parity holds for the capital spellings only; AGENTS.md's count names its commit; P3 no longer says such a letter "is read as an initial"; S10 records the split 'Ph. D.' reading. And Derek's correction that generational markers are common, so #718's lone-numeral reading rests on the count. Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 2 +- .../decisions/S10-trailing-roman-numeral.md | 2 + docs/design/rules/P3-connectives.md | 7 +- .../rules/S10-trailing-roman-numeral.md | 11 ++- nameparser/_pipeline/_assign.py | 93 +++++++++++-------- nameparser/_pipeline/_group.py | 17 +++- tests/v2/cases.py | 37 +++++++- tests/v2/test_cases.py | 7 +- tools/differential/corpus_rules.jsonl | 4 + tools/differential/snapshot.jsonl | 4 + 10 files changed, 133 insertions(+), 51 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index bd128ad6..3298f5eb 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -173,7 +173,7 @@ The library has two layers: `nameparser/config/` (data) and `nameparser/parser.p **Sweep the forms a change can reach: comma shapes, then `name_order`.** No comma, a FULL name before the comma, a ONE-WORD name before the comma and NOTHING before the comma are four paths, and the last two are the misses — #429, #430 and #432 are all the one-word path disagreeing with the full-name path on inputs the full-name path reads correctly, and #613's PR review found `, PhD` and `, Jr.` (an empty surname field, as CSV data writes it) read as given and title by a decision that returned early on an empty head, after two review rounds whose every grid had a name before the comma. Put an empty head and a nickname-only head (`(Bob), PhD`) in every comma grid. Count the one-word path the way C1 counts, which is neither tokens nor pieces: a particle run and the one word it attaches to are one word (#575), a connective join and a bound given-name pair count as their words (#613, so `Ortega y Gasset, Dr.` is a whole name), and a suffix beside the surname counts as no name word at all. #603's review found the suffix half (`Vega y Lopez Jr., PhD Jones` lost its family where `Vega y Lopez, PhD Jones` kept it); #613 then reversed the connective half, and both now read given 'Vega', middle 'y', family 'Lopez' — so test a particle surname, a connective surname and a surname with a suffix, each before a comma. For orders the sweep already exists (`tests/v2/test_cases.py` runs every row under all three) but carries one assertion, R2's family partition, so it checks nothing a new change moves; coverage there has been vacuous before (PR #394's review found the suite passed with `name_order` discarded from grouping). Parse your change's names in each comma shape and each order, and read the ones you did not predict. And read a generated grid's moves by SHAPE, not by corpus membership: #614's first design passed the suite, the five gates and a fingerprint of every corpus name, because its grid moves were sorted into 'corpus name moved' and 'garbage', and the review found the moves that mattered among the 'garbage' -- a bare salutation (`Mr. and Mrs.` lost its connective to the family name) and a name with one trailing word (`Freiherr von Berg X.Y.Z.`). Pull out the head-plus-one-word slice of the grid and a salutation with nothing after it, and read every move there. Two more were the second review's (#614): no grid put TWO words that are both a title and a particle at the head (`Freiherr St van Berg MA`, where an index into the pieces was applied to a merged view of them -- `TITLES ∩ particles_ambiguous` is `{freiherr, st}`, so put both in every head list, each in its bare and its period-marked spelling -- #637's grid had `St` and `Freiherr` alone and missed that `St.` and `Freiherr.` are declined by the bound join and then attached by P6), and the grids compared ambiguity KINDS, so a report telling the caller the wrong field (`Freiherr von Berg Ma`, 'read as a given name' of a word in the family) passed every pin. #701's review added one: put a title/suffix dual (`Ms`, `Sr.`, `Cpl`) SECOND in a title run, behind an ordinary title. A stop that asks only "is this word suffix vocabulary" passes every row that opens with the dual and fails `Rev. Sr. Smith, Mary`; it was found by comparing `T Smith, John` against `T John Smith` over every title word, and a comma-versus-comma-less comparison is the cheap check for any rule that moves a word between the two slots. Compare each report's `detail` against the field its tokens land in -- since #626 `test_a_report_names_the_field_its_word_lands_in` (tests/v2/test_cases.py) does it over every non-locale case row under all three orders (a row carrying its own policy only as declared), for the phrasings its `_FIELD_CLAIMS` lists -- so a row with the shape is the test only where the report's wording is listed: a new emitter's wording goes in that list, and segment's report on a part past the second comma names no field (#629). #626's own first grid then missed an emitter for want of one comma shape (`, Ma Dr.`: an empty head, an ambiguous word, a trailing title, which H1 moves to the family): a trailing title belongs in every empty-head comma grid too. And #620's: an exclusion list in code being replaced is a set of rules, not noise -- #614's merged copy excluded every word the trailing read weighs from its units, the first #620 prototype dropped the list as a symptom of the copy, and its review found the list had been keeping positional counting's rule that a member in front counts as a word (`Freiherr von Berg MA X.Y.Z.` lost its suffix). Before deleting one, mutate each entry out and read what moves. -**Then sweep the spellings: a class reached by vocabulary is also reached by SHAPE.** A test that uses only the listed spelling of a word class walks only the vocabulary path, and the shape path is a separate branch that can be missing while every test passes. Roman numerals are the sharpest case: `ii`, `iii` and `iv` are suffix vocabulary; `i` and `v` are vocabulary too but initial-shaped, so `is_suffix_piece` vetoes them and only the numeral fork reads them; and `vi` through `x` are in no list at all, read by the fork's shape test alone (checked 2026-10-04 against `Lexicon.default()`) -- which stops at `x`: `_vocab._ROMAN` matches I to X, so `John Smith XI` reads `XI` as a name word (re-checked 2026-10-07, #614's /simplify, which found a docstring claiming "any length"). And the fork counts a ONE-letter numeral besides, since #718 (rules/S10): it is the suffix only where the name has a word to spare, so `John V` and `John Smith V` take different branches of one fork, a one-case name being counted like capitals whatever the letter -- since #721 a one-case one-letter numeral carries the `initial` tag its capital does, and a bare letter in front of the numeral is initial-shaped as its capital is, so `john v` reads as `JOHN V` does over `test_a_one_case_name_reads_as_its_capitals`'s grid (comma shapes, orders, a maiden clause, a particle chain, a bare letter in front, a trailing title, a non-ASCII letter in front: 315 disagreeing pairs on master, none after). Its first version claimed "every shape" with no bare letter in front and no trailing title in the grid, and #721's reviews found both broken; a grid's claim is the grid -- put a head-plus-numeral row beside every words-to-spare row. So a change that handles `V` and `III` can still miss `VI` — #610 found exactly that, the maiden take's candidate loop admitting a shape-only numeral in one position where the peel reads it in two (`Jane Doe nee Smith VI Prof.` kept maiden `Smith VI` while `V` and `III` worked). The same split runs through the credentials (`MA` listed, `X.Y.Z.` by the dotted shape, `XYZ` by the caps shape, and `Ph. D.` merged by group into one flagged piece the peel's walk never holds -- #603 found #602's run-start predicate refusing it too, so `Smith, John Ph. D. Jones` kept middle 'Jones' where `Smith, John PhD Jones` read suffix 'PhD Jones') and the titles (`Prof.` period-marked and read at the end of a name, `Prof` bare and a name word there). When a change touches a word class, test one spelling from each path. The same holds for a separator's SPACING: a comma (or any punctuation) can stand alone, glue to the word before it or glue to the word after, and each is a different token stream. #653's first fix swept `(née, Jones)` and missed `(née , Jones)`, where the kept comma became a word after the marker and M3 read the clause as maiden ', Jones'; its review found it. Put all three spacings in any grid around a separator. +**Then sweep the spellings: a class reached by vocabulary is also reached by SHAPE.** A test that uses only the listed spelling of a word class walks only the vocabulary path, and the shape path is a separate branch that can be missing while every test passes. Roman numerals are the sharpest case: `ii`, `iii` and `iv` are suffix vocabulary; `i` and `v` are vocabulary too but initial-shaped, so `is_suffix_piece` vetoes them and only the numeral fork reads them; and `vi` through `x` are in no list at all, read by the fork's shape test alone (checked 2026-10-04 against `Lexicon.default()`) -- which stops at `x`: `_vocab._ROMAN` matches I to X, so `John Smith XI` reads `XI` as a name word (re-checked 2026-10-07, #614's /simplify, which found a docstring claiming "any length"). And the fork counts a ONE-letter numeral besides, since #718 (rules/S10): it is the suffix only where the name has a word to spare, so `John V` and `John Smith V` take different branches of one fork, a one-case name being counted like capitals whatever the letter -- since #721 a one-case one-letter numeral carries the `initial` tag its capital does, and a bare letter in front of the numeral is initial-shaped as its capital is, so `john v` reads as `JOHN V` does over `test_a_one_case_name_reads_as_its_capitals`'s grid (comma shapes, orders, a maiden clause, a particle chain, a bare letter in front, a trailing title, a non-ASCII letter in front: 315 disagreeing pairs at 2bfa9533, before #721, and none after). Its first version claimed "every shape" with no bare letter in front and no trailing title in the grid, and #721's reviews found both broken; a grid's claim is the grid -- put a head-plus-numeral row beside every words-to-spare row. So a change that handles `V` and `III` can still miss `VI` — #610 found exactly that, the maiden take's candidate loop admitting a shape-only numeral in one position where the peel reads it in two (`Jane Doe nee Smith VI Prof.` kept maiden `Smith VI` while `V` and `III` worked). The same split runs through the credentials (`MA` listed, `X.Y.Z.` by the dotted shape, `XYZ` by the caps shape, and `Ph. D.` merged by group into one flagged piece the peel's walk never holds -- #603 found #602's run-start predicate refusing it too, so `Smith, John Ph. D. Jones` kept middle 'Jones' where `Smith, John PhD Jones` read suffix 'PhD Jones') and the titles (`Prof.` period-marked and read at the end of a name, `Prof` bare and a name word there). When a change touches a word class, test one spelling from each path. The same holds for a separator's SPACING: a comma (or any punctuation) can stand alone, glue to the word before it or glue to the word after, and each is a different token stream. #653's first fix swept `(née, Jones)` and missed `(née , Jones)`, where the kept comma became a word after the marker and M3 read the clause as maiden ', Jones'; its review found it. Put all three spacings in any grid around a separator. **And the encoding: a letter class written `[^\W\d_]` or tested with `isalpha()` is blind to a combining mark.** Python's `\w` and `isalpha()` do not match Mn, so a letter written as base plus accent (NFD) ends a word early; the vocabulary path is immune because `_normalize` NFC-folds, the shape path is not. #652: `Dr. inž. Janez Novak` read `inž.` as the given name when the ž was decomposed, and `Janez Novak, ŽŠČ` reversed, in two families that share the blind spot — the letter-shape regexes (`period_abbreviation`, `_DOTTED`, `_CHUNKED`, `_PAIRED_INITIALS`), now spelled with `_vocab._LETTER` (a letter plus the general combining blocks; `test_regex_sync` compares the two `_MARKS` strings and pins `period_abbreviation` against config's copy), and the caps predicates, which NFC-compose a non-ASCII word before `isalpha()` (`is_caps_word`, behind C-level `isalpha()`/`is_normalized` tests, so already-composed text pays no frame — measured 2026-10-12, `Smith, José` was 193 frames before and after; the first version called it for every non-ASCII word and cost one frame each, and no guard saw it because the reference name is ASCII). A width test must count letters, not code points (`composed_len`): admitting a decomposed word to the caps set put `John Smith, MÁ` on the pair exception's wrong side, and the review's grid found it where the rows (all three letters long) did not — put a two-letter word in every caps grid. The initial shape (`_INITIAL`, `\w\.`, v1-verbatim and public) is not widened but asked of the COMPOSED word, in `is_initial_shaped` and in `_render`'s two copies; and the single-letter connective tests count composed letters, since a decomposed `й` is two code points (#665, where `É. y É. Smith` and `John й Smith` read differently decomposed). Compose the WHOLE predicate, not one half: `is_initial` is a shape ANDed with the #320 script veto, and composing the shape alone made a decomposed `씨.` an initial, since decomposed hangul is conjoining jamo outside every script range (#665's first version; `in_initialless_script` composes too now). Not covered, deliberately: the marks only some scripts write (Indic vowel signs, Arabic and Hebrew points; rules/H2's Accepted clause). The guard is `test_a_decomposed_name_renders_as_its_composed_twin` over the CLDR corpus plus `test_a_decomposed_abbreviation_reads_as_its_composed_twin` (rows × three orders × EVERYWHERE); the corpus guard reaches a shape only if some corpus name does, which is how the caps shape was missed until review, so when adding a shape rule add a row there. Four of #652's rows and two of #665's pass with every fix reverted (measured) and are parity pins, and the test compares ambiguity kinds and `initials()` too, since #665's `John й Smith` differed in its report; `_DOTTED` has no observable input and is pinned by a direct call. diff --git a/docs/design/decisions/S10-trailing-roman-numeral.md b/docs/design/decisions/S10-trailing-roman-numeral.md index 57ccb53a..b9ecf8ee 100644 --- a/docs/design/decisions/S10-trailing-roman-numeral.md +++ b/docs/design/decisions/S10-trailing-roman-numeral.md @@ -16,3 +16,5 @@ - 2026-10-18 (Derek), #721's PR review — A NUMERAL BEFORE A CREDENTIAL IS REPORTED, AND THE TWO-WORD COUNT STANDS. Derek: `john i phd` should report `suffix-or-name` as `john i, phd` does, and `John Smith V PhD` keeps family 'V' with the report — so the Accepted clause's reading stands and only the report is added; the fields do not move (1.4.0 read the same). The part's last name word with nothing but the trailing run behind it is the site: assign, once the trailing read has said what that run is, withdraws classify's connective report on a marked letter there (rules/P3 now says "unless no name word of its part stands to its right") and reports a roman numeral there, worded for what follows it. So `john i dr.` reports the numeral once, `john e phd` and `jose e jr` report nothing, and `john smith i jr` reports `suffix-or-name` where it reported `conjunction-or-initial`. #718's stand-down is kept as a defensive check, and says so: measured on this commit's tree, the full suite and a 1,680-parse trailing-title sweep fail and pass alike without it. Declined: two stricter readings Derek weighed on the PR, both measured over the snapshot corpus. (A) a one-letter numeral is the generation only when given, middle and family are all filled — 11 readings move, among them `Hardie Leannon V`; (B) a comma-less one-letter numeral is never the generation — 18 move, among them `John W. Ingram V`. Both are real people who write the numeral without a comma: #36 (2015) reported `Hardie Leannon V` reading last 'V' as a bug, #136 (2022) came from John W. Ingram V writing his name with and without the comma, and Derek found the same sheriff written `John W Ingram V` and, once, `John Ingram V` (Brunswick Beacon, his candidate profile). The two-word count reads all four as the generation; the rows `a_real_generation_with_a_middle_initial` and `a_real_generation_with_no_middle_name` pin the last two. And the review's other defect: the one-case test of the word in front accepted any one-character cased letter, so `IVAN Ж V` and `JOSÉ É V` moved off their mixed-case reading and a decomposed `é` read otherwise than a composed one. It asks now whether the letter's capital is initial-shaped, which is the rule the docstring states (`john j v` as `JOHN J V`; `é`, `ж` and an uncased word as before). The grid gains a non-ASCII letter in front and a shape-only three-letter numeral; master disagrees on 315 of its 1,872 pairs, the branch on none. +- 2026-10-18 (Derek), #722's last review — ONE WITHDRAWAL, AT THE END OF ASSIGN, AND CORRECTIONS TO THE ENTRY ABOVE. Two reviews of the round above found P3's broadened exception ("unless no name word of its part stands to its right") implemented only where the part's last name word is one token on the no-comma path: `john van e phd` (the letter ending a particle chain), `smith, john e phd` (the given part after a family comma) and `, john i phd` kept `conjunction-or-initial`, and `freiherr von i dr.` reached #718's stand-down and lost the numeral's report to the connective's. The withdrawal is one role-based pass at the end of the stage now, over every segment: a marked letter that is its segment's last name-role word with only suffix- and title-roled words after it loses classify's report. The numeral's own report is emitted before it, so one token never carries both, and the stand-down is gone — not "kept as a defensive check", which the entry above said on a measurement that reached no particle chain. The maiden clause gains the numeral report's twin (`Jane Doe nee Smith V PhD`, maiden 'Smith V', reported), the same fork taken a stage earlier. Corrections to the entry above, each measured on this commit's tree: option B moves 19 readings, not 18, and A 11, over the snapshot's 5,711 names with the stated patches to `_pieces.peel_trailing`'s count (A: `k - 3 >= second`; B: the count dropped) — the first figures were over an earlier snapshot plus fifteen named inputs. #36 and #136 do not say their names are real people's: #36 reports 'Hardie Leannon V' (possibly generated test data), #136 a reporter writing 'John W. Ingram, V' and 'John W. Ingram V'. The real person is the sheriff Derek found, written 'John W Ingram V' and 'John Ingram V'. And "1.4.0 read the same" holds for the capital spellings only: 1.4.0 read `john smith v phd` and `john smith i jr` with the letter in the suffix. + THE PREMISE BEHIND #718, REVISED. #718's entry quotes Derek that a generational V is "mostly archaic". On this PR he found otherwise: searching shows many people named John Ingram with every variation of generational marker and middle initial, written without a comma unless a credential follows ("MBA", "MD"). Nothing decided here moves — the two-word count reads each of those as the generation — but #718's lone-numeral reading (`John V`, family 'V') now rests on the count alone, a name of two words having no word to spare for a generation, not on the generation being rare. diff --git a/docs/design/rules/P3-connectives.md b/docs/design/rules/P3-connectives.md index 96c8c75d..299968ac 100644 --- a/docs/design/rules/P3-connectives.md +++ b/docs/design/rules/P3-connectives.md @@ -20,9 +20,8 @@ P3. Rationale: connective words ("y", "of the") bind name words into have gone the other way, and is reported — unless no name word of its part stands to its right, nothing at all or only post-nominals and titles, where it connects nothing and there is - no other way for the call to go: it is read as an initial and - not reported, and where it is a roman numeral ending the name - S10 reports it instead. + no other way for the call to go: it is not reported, and where + S10 reads it as a numeral S10's report is the one. The joined part is ONE name word wherever another rule counts them, so a rule taking "one name word" takes the whole join and never half of it — except C1's count of the words before a comma, @@ -81,6 +80,8 @@ P3. Rationale: connective words ("y", "of the") bind name words into "Carod i" → suffix="i" "carod i" → ambiguities=("suffix-or-name",) · boundary "jose e phd" → ambiguities=() · boundary + "smith, john e phd" → ambiguities=() · boundary + "john van e phd" → ambiguities=() · boundary "Josep Lluis Carod i III" → suffix="i III" "Carod y de Rovira i" → middle="y" "John й Smith" → middle="й" diff --git a/docs/design/rules/S10-trailing-roman-numeral.md b/docs/design/rules/S10-trailing-roman-numeral.md index 03e6911b..fcbe7459 100644 --- a/docs/design/rules/S10-trailing-roman-numeral.md +++ b/docs/design/rules/S10-trailing-roman-numeral.md @@ -16,9 +16,9 @@ S10. Rationale: a roman numeral ending a name is its generation, being no word to spare, and either reading is reported, as either reading of an ambiguous acronym is. A numeral read by its shape alone reports too; a numeral the vocabulary lists in two - or more letters is not in question and reports nothing, and a - letter already reported as read as an initial reports nothing - new. Written case + or more letters is not in question and reports nothing; a marked + letter (P3) that is such a numeral reports this way and not as a + connective. Written case settles the fork in one way only: a numeral written in lower case in a name of mixed case is the generation and reports nothing, since nobody writes an initial that way. A name written @@ -59,4 +59,9 @@ S10. Rationale: a roman numeral ending a name is its generation, "John Smith V PhD" → family="V" "John Smith V PhD" → ambiguities=("suffix-or-name",) "john i phd" → ambiguities=("suffix-or-name",) + "Jane Doe nee Smith V PhD" → ambiguities=("suffix-or-name",) + Accepted: a credential written split across two words is read + as one with the numeral in front of it, the run taking both, as + 1.4.0 read it. + "John Smith V Ph. D." → suffix="V Ph. D." history: decisions/S10 · interacts: S2, S4, S7, C1, P2, P3, M2, H1, H3 · implemented: nameparser/_pipeline/_assign.py, nameparser/_pipeline/_classify.py, nameparser/_pipeline/_group.py, nameparser/_pipeline/_pieces.py, nameparser/_pipeline/_vocab.py diff --git a/nameparser/_pipeline/_assign.py b/nameparser/_pipeline/_assign.py index c8186746..21999412 100644 --- a/nameparser/_pipeline/_assign.py +++ b/nameparser/_pipeline/_assign.py @@ -7,7 +7,9 @@ group's joins left no name word in front of it. Produces: tokens with roles set on every main-stream token, and the folded-middle tag on the particles P6's attachment joins to the family -after a family comma (#613). +after a family comma (#613); its own ambiguities (below), less +classify's CONJUNCTION_OR_INITIAL on a letter that only suffixes and +titles follow in its part (#721). Reads: Policy.name_order (#270), is_suffix_lenient on the trailing piece of a two-part comma name, and Policy.script_orders (#271, which overrides it when every name piece is written wholly in one script, or @@ -100,7 +102,8 @@ tail_reading, trailing_titles, ) from nameparser._pipeline._state import ( - AMBIGUOUS_ACRONYM_TAG, ParseState, PendingAmbiguity, Structure, + AMBIGUOUS_ACRONYM_TAG, NAME_ROLES, ParseState, PendingAmbiguity, + Structure, WorkToken, _AMBIGUOUS_CREDENTIAL_TAGS, _NEVER_FLIPPED, copy_with, ) from nameparser._policy import Policy, Script @@ -307,24 +310,40 @@ def _placed(read: TailRead, rest: list[int], peel.absorbed) -# rules/S10: "a letter already reported as read as an initial reports -# nothing new" -- asked only by the numeral fork's declined pick -# (#718). Defensive since #721: classify withdraws its report on a -# letter ending its part, and the check below on the part's last name -# word withdraws it before a trailing run, so no input reached it on -# #721's final tree (its full suite and a 1,680-parse trailing-title -# sweep fail and pass alike without it). The maiden clause's report -# on the numeral it keeps cannot meet the case: classify reads no -# connective inside a clause -def _reported_as_initial(ambiguities: Sequence[PendingAmbiguity], - index: int) -> bool: - """Whether classify's connective-or-initial fork already reported - the token at `index`, reading it as an initial. Declining the - numeral reads the letter as that same name word, so a second - report would name one reading twice. A rare shape, so the scan - costs ordinary names nothing.""" - return any(a.kind is AmbiguityKind.CONJUNCTION_OR_INITIAL - and index in a.indices for a in ambiguities) +# rules/P3: "unless no name word of its part stands to its right" -- +# a marked letter whose part holds nothing after it but suffixes and +# titles connects nothing, so classify's connective-or-initial report +# on it is withdrawn here, once every segment's roles are known: the +# no-comma name, a particle chain ending in the letter ('john van e +# phd') and the given part after a family comma ('smith, john e phd') +# alike (#721's reviews). A letter with nothing at all after it never +# carried the report (classify). Where the letter is also a numeral the +# fork read, its own report (S10) is the one left, so one token never +# carries both -- which is why #718's stand-down is gone. Called only +# for a one-case name, the one classify can report on. +def _withdraw_unjoined_connectives(segments: Sequence[Sequence[int]], + tokens: Sequence[WorkToken], + ambiguities: list[PendingAmbiguity], + ) -> None: + if not any(a.kind is AmbiguityKind.CONJUNCTION_OR_INITIAL + for a in ambiguities): + return + unjoined = set() + for seg in segments: + last, after = None, False + for j in seg: + role = tokens[j].role + if role in NAME_ROLES: + last, after = j, False + elif role is Role.SUFFIX or role is Role.TITLE: + after = after or last is not None + if last is not None and after: + unjoined.add(last) + if unjoined: + ambiguities[:] = [ + a for a in ambiguities + if a.kind is not AmbiguityKind.CONJUNCTION_OR_INITIAL + or len(a.indices) != 1 or a.indices[0] not in unjoined] def _assign_main(seg_idx: int, state: ParseState, @@ -546,26 +565,21 @@ def _assign_main(seg_idx: int, state: ParseState, f"decides it; read as ", tuple(head), field_tail=" by convention, which follows the read " "order")) - # rules/P3: "unless no name word of its part stands to its right" - # and rules/S10: "a one-letter numeral before a credential is the + # rules/S10: "a one-letter numeral before a credential is the # family name, and the reading is reported" -- the part's last name # word with nothing but the trailing run behind it (#721, Derek: - # 'john i phd' reports as 'john i, phd' does). Decided here, where - # the trailing read has said what that run is; classify's own - # end-of-part test covers a letter with nothing at all behind it + # 'John Smith V PhD' keeps family 'V' and reports, as 'john i, phd' + # does). Decided here, where the trailing read has said what that + # run is if name_pieces and (suffix_pieces or titled_tail): last = pieces[name_pieces[-1]] word = tokens[last[0]].text if len(last) == 1: - # a connective with only post-nominals or titles to its - # right connects nothing; reported only in a one-case name - if state.one_case: - ambiguities[:] = [ - a for a in ambiguities - if a.kind is not AmbiguityKind.CONJUNCTION_OR_INITIAL - or a.indices != (last[0],)] # `_ROMAN` first, C-level, so an ordinary name word pays - # no frame for the rest + # no frame for the rest; the connective report the letter + # may carry is withdrawn at the end of the stage, once + # every segment's roles are known + # (`_withdraw_unjoined_connectives`) if (_ROMAN.match(word) is not None and len(name_pieces) >= 2 and tuple(last) not in peeled.picks and is_trailing_numeral_suffix( @@ -587,12 +601,11 @@ def _assign_main(seg_idx: int, state: ParseState, # declined pick (#718), which carries neither credential tag # (the acronym fork's picks carry one) and is never a suffix if token.tags.isdisjoint(_AMBIGUOUS_CREDENTIAL_TAGS): - if not _reported_as_initial(ambiguities, piece[0]): - ambiguities.append(PendingAmbiguity( - AmbiguityKind.SUFFIX_OR_NAME, - f"{token.text!r} is a roman numeral, but the name " - f"has no other word to spare for it; read as ", piece, - field_tail=" rather than a generational suffix")) + ambiguities.append(PendingAmbiguity( + AmbiguityKind.SUFFIX_OR_NAME, + f"{token.text!r} is a roman numeral, but the name has " + f"no other word to spare for it; read as ", piece, + field_tail=" rather than a generational suffix")) continue declined = ("a name part" if token.role is Role.SUFFIX else "a post-nominal") @@ -1572,6 +1585,8 @@ def reads_as_a_suffix(m: int, titled: frozenset[int]) -> bool: and not any("vocab:suffix" in tokens[i].tags for i in piece)) else Role.SUFFIX) + if state.one_case: + _withdraw_unjoined_connectives(state.segments, tokens, ambiguities) return copy_with(state, tokens=tuple(tokens), order=order, ambiguities=tuple(ambiguities)) diff --git a/nameparser/_pipeline/_group.py b/nameparser/_pipeline/_group.py index b71b8aae..dbec8ef1 100644 --- a/nameparser/_pipeline/_group.py +++ b/nameparser/_pipeline/_group.py @@ -81,7 +81,8 @@ import nameparser._pipeline._comma as _comma from nameparser._pipeline._vocab import D, PH from nameparser._pipeline._vocab import ( - comma_follows_marker, delimiter_cores, + comma_follows_marker, delimiter_cores, is_trailing_numeral_suffix, + numeral_settled_by_case, ) from nameparser._types import AmbiguityKind, Role @@ -360,6 +361,20 @@ def _maiden_take(pieces: Sequence[Sequence[int]], f"but the name has no other word to spare for it; the " f"maiden marker's clause keeps it rather than reading it as " f"a generational suffix", tuple(last))) + elif (released and len(last) == 1 and end - 2 >= lo + and is_trailing_numeral_suffix( + word.text, tokens[pieces[end - 2][0]].text, one_case) + and not numeral_settled_by_case(word.text, one_case)): + # rules/S10: "a one-letter numeral before a credential is the + # family name, and the reading is reported" -- the clause's + # twin of assign's report (#721's review): the clause keeps the + # numeral its released post-nominal stands behind + ambiguities.append(PendingAmbiguity( + AmbiguityKind.SUFFIX_OR_NAME, + f"{word.text!r} ending the maiden name is a roman numeral, " + f"but a post-nominal follows it; the maiden marker's clause " + f"keeps it rather than reading it as a generational suffix", + tuple(last))) return (list(range(m, lo)), [(j, Role.MAIDEN) for j in range(lo, end)] + released) diff --git a/tests/v2/cases.py b/tests/v2/cases.py index da9a60bd..dc3c55ad 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -6025,10 +6025,13 @@ def _check_cjk_shape_purity(self) -> None: {"given": "Jane", "family": "Doe", "suffix": "MA", "maiden": "Smith V"}, classification="fix(#533)", - ambiguities=("suffix-or-name",), + ambiguities=("suffix-or-name", "suffix-or-name"), notes="both stops read the TRAILING word, so the numeral is " "not the word either fork asks about: the acronym fork " "stops at 'MA' and 'V' stays maiden text behind it. " + "Since #721 the kept 'V', a numeral with a post-nominal " + "behind it, reports as 'John Smith V MA' reports its " + "own (S10, the trailing roman numeral). " "The mirror image, 'Jane Smith née Jones Ma V', keeps " "maiden 'Jones Ma' and reads suffix 'V'", shape=2), @@ -7643,6 +7646,38 @@ def _check_cjk_shape_purity(self) -> None: notes="the lowercase twin of the row above, and the one whose " "decomposed spelling had read otherwise under #721's " "first form"), + Case("a_one_case_link_ending_a_particle_chain_before_a_credential", + "john van e phd", + {"given": "john", "family": "van e", "suffix": "phd"}, + notes="the letter ends the chain the particle made, with only " + "a credential behind it: it connects nothing (P3), so " + "nothing reports. #721's first withdrawal tested a " + "one-token piece and missed it"), + Case("a_one_case_link_before_a_credential_after_a_family_comma", + "smith, john e phd", + {"given": "john", "middle": "e", "family": "smith", + "suffix": "phd"}, + notes="the given part after a family comma: the same rule, " + "decided over roles at the end of the stage, so it " + "reaches the comma's own walk too"), + Case("a_chained_one_case_numeral_before_a_title_reports_once", + "freiherr von i dr.", + {"title": "freiherr dr.", "family": "von i"}, + ambiguities=("particle-or-given", "suffix-or-name"), + classification="fix(#316)", + notes="a letter ending a particle chain with only a title " + "behind it: the numeral's report, the connective's " + "withdrawn -- where #718's stand-down had kept the " + "connective's and dropped the numeral's. 1.4.0 read " + "first 'freiherr von', last 'i', suffix 'dr.'"), + Case("a_numeral_kept_by_a_maiden_clause_before_a_credential_reports", + "Jane Doe nee Smith V PhD", + {"given": "Jane", "family": "Doe", "suffix": "PhD", + "maiden": "Smith V"}, + ambiguities=("suffix-or-name",), classification="fix(#274)", + notes="the clause's twin of 'John Smith V PhD': the clause " + "keeps the numeral a post-nominal stands behind, and " + "reports it (S10). 1.4.0 had no maiden markers"), Case("a_lowercase_numeral_in_a_one_case_name_still_reports", "john smith i", {"given": "john", "family": "smith", "suffix": "i"}, diff --git a/tests/v2/test_cases.py b/tests/v2/test_cases.py index 98624c88..49663bf0 100644 --- a/tests/v2/test_cases.py +++ b/tests/v2/test_cases.py @@ -194,9 +194,10 @@ def test_the_field_sweep_sees_the_claims_it_checks() -> None: # #721's one-case lowercase numerals, counted and reported; 1213 # -> 1234 the same day: #721's PR-review rows (the numeral before a # credential or title, the real generations, the lone letters); - # 1234 -> 1240: the two non-ASCII letter-in-front rows. - assert claims == 1240, ( - f"the field sweep checks {claims} claims, recorded as 1240 on " + # 1234 -> 1240: the two non-ASCII letter-in-front rows; 1240 -> + # 1243: the end-of-stage withdrawal's rows (#722's last review). + assert claims == 1243, ( + f"the field sweep checks {claims} claims, recorded as 1243 on " f"2026-10-18") diff --git a/tools/differential/corpus_rules.jsonl b/tools/differential/corpus_rules.jsonl index 72b2e87c..e19b471d 100644 --- a/tools/differential/corpus_rules.jsonl +++ b/tools/differential/corpus_rules.jsonl @@ -171,6 +171,7 @@ "Jane Doe nee Smith PhD Smith" "Jane Doe nee Smith Prof." "Jane Doe nee Smith Prof. MA" +"Jane Doe nee Smith V PhD" "Jane Doe nee Smith V Prof." "Jane Doe nee Smith VI Prof." "Jane Doe nee Smith X.Y.Z." @@ -234,6 +235,7 @@ "John Smith R.A.I." "John Smith Sir" "John Smith V" +"John Smith V Ph. D." "John Smith V PhD" "John Smith V." "John Smith VI" @@ -544,6 +546,7 @@ "john smith, vi" "john v" "john van der berg ma" +"john van e phd" "jose e maria santos" "jose e phd" "jose ortega-y-gasset" @@ -563,6 +566,7 @@ "née Jones" "qc mp" "smith, de ma" +"smith, john e phd" "tran lac" "van Berg Jan de" "van Gogh" diff --git a/tools/differential/snapshot.jsonl b/tools/differential/snapshot.jsonl index 83f7ad15..8aded66a 100644 --- a/tools/differential/snapshot.jsonl +++ b/tools/differential/snapshot.jsonl @@ -1356,6 +1356,7 @@ {"name": "Jane Doe nee Smith Prof. MA", "title": "Prof.", "given": "Jane", "family": "Doe", "suffix": "MA", "maiden": "Smith", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. D."} {"name": "Jane Doe nee Smith V", "given": "Jane", "family": "Doe", "suffix": "V", "maiden": "Smith", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. D."} {"name": "Jane Doe nee Smith V MA", "given": "Jane", "family": "Doe", "suffix": "MA", "maiden": "Smith V", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. D."} +{"name": "Jane Doe nee Smith V PhD", "given": "Jane", "family": "Doe", "suffix": "PhD", "maiden": "Smith V", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. D."} {"name": "Jane Doe nee Smith V Prof.", "title": "Prof.", "given": "Jane", "family": "Doe", "suffix": "V", "maiden": "Smith", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. D."} {"name": "Jane Doe nee Smith VI Prof.", "title": "Prof.", "given": "Jane", "family": "Doe", "suffix": "VI", "maiden": "Smith", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. D."} {"name": "Jane Doe nee Smith X.Y.Z.", "given": "Jane", "family": "Doe", "suffix": "X.Y.Z.", "maiden": "Smith", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. D."} @@ -1548,6 +1549,7 @@ {"name": "John Smith Rev.", "title": "Rev.", "given": "John", "family": "Smith", "initials": "J. S."} {"name": "John Smith Sir", "given": "John", "middle": "Smith", "family": "Sir", "initials": "J. S. S."} {"name": "John Smith V", "given": "John", "family": "Smith", "suffix": "V", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. S."} +{"name": "John Smith V Ph. D.", "given": "John", "family": "Smith", "suffix": "V Ph. D.", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. S."} {"name": "John Smith V PhD", "given": "John", "middle": "Smith", "family": "V", "suffix": "PhD", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. S. V."} {"name": "John Smith V.", "given": "John", "middle": "Smith", "family": "V.", "initials": "J. S. V."} {"name": "John Smith VI", "given": "John", "family": "Smith", "suffix": "VI", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. S."} @@ -3418,6 +3420,7 @@ {"name": "john smith, vi", "given": "vi", "family": "john smith", "initials": "v. j. s."} {"name": "john v", "given": "john", "family": "v", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "j. v."} {"name": "john van der berg ma", "given": "john", "family": "van der berg", "suffix": "ma", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "j. b."} +{"name": "john van e phd", "given": "john", "family": "van e", "suffix": "phd", "initials": "j. e."} {"name": "johnny y", "given": "johnny", "family": "y", "initials": "j. y."} {"name": "jones, john e", "given": "john", "middle": "e", "family": "jones", "initials": "j. e. j."} {"name": "jose e maria santos", "given": "jose", "middle": "e maria", "family": "santos", "ambiguities": ["CONJUNCTION_OR_INITIAL"], "initials": "j. e. m. s."} @@ -3489,6 +3492,7 @@ {"name": "scott l. werner", "given": "scott", "middle": "l.", "family": "werner", "initials": "s. l. w."} {"name": "señora María García", "title": "señora", "given": "María", "family": "García", "initials": "M. G."} {"name": "smith, de ma", "family": "smith de ma", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "s. m."} +{"name": "smith, john e phd", "given": "john", "middle": "e", "family": "smith", "suffix": "phd", "initials": "j. e. s."} {"name": "st.repr. Gregers Werle Marve Almar von Fleksnes jr.", "title": "st.repr.", "given": "Gregers", "middle": "Werle Marve Almar", "family": "von Fleksnes", "suffix": "jr.", "initials": "G. W. M. A. F."} {"name": "st.repr. von Fleksnes Gregers Werle Marve Almar", "order": "FAMILY_FIRST", "title": "st.repr.", "given": "Fleksnes", "middle": "Gregers Werle Marve Almar", "family": "von", "ambiguities": ["PARTICLE_OR_GIVEN"], "initials": "F. G. W. M. A. v."} {"name": "test", "given": "test", "ambiguities": ["GIVEN_OR_FAMILY"], "initials": "t."} From 2fc28d8fa0ef5795ddd2e503e5835a6e45ceb832 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sun, 18 Oct 2026 15:22:11 -0700 Subject: [PATCH 7/9] Ask whether any connective report exists inline, so a one-case name pays no frame (#722 review) Co-Authored-By: Claude Opus 5.5 --- nameparser/_pipeline/_assign.py | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/nameparser/_pipeline/_assign.py b/nameparser/_pipeline/_assign.py index 21999412..ba4a0496 100644 --- a/nameparser/_pipeline/_assign.py +++ b/nameparser/_pipeline/_assign.py @@ -325,9 +325,6 @@ def _withdraw_unjoined_connectives(segments: Sequence[Sequence[int]], tokens: Sequence[WorkToken], ambiguities: list[PendingAmbiguity], ) -> None: - if not any(a.kind is AmbiguityKind.CONJUNCTION_OR_INITIAL - for a in ambiguities): - return unjoined = set() for seg in segments: last, after = None, False @@ -1585,8 +1582,14 @@ def reads_as_a_suffix(m: int, titled: frozenset[int]) -> bool: and not any("vocab:suffix" in tokens[i].tags for i in piece)) else Role.SUFFIX) + # a plain loop, so a one-case name with no connective report pays + # no frame for the pass if state.one_case: - _withdraw_unjoined_connectives(state.segments, tokens, ambiguities) + for amb in ambiguities: + if amb.kind is AmbiguityKind.CONJUNCTION_OR_INITIAL: + _withdraw_unjoined_connectives(state.segments, tokens, + ambiguities) + break return copy_with(state, tokens=tuple(tokens), order=order, ambiguities=tuple(ambiguities)) From 053f0e72487bf9400d00e8a4a822fdffb220d9da Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sun, 18 Oct 2026 17:10:18 -0700 Subject: [PATCH 8/9] Ask P3's "connects nothing" once, in assemble over final roles (/simplify) /simplify pass over PR #722. Reuse, simplification and altitude reviews agreed: rules/P3's exception was answered twice, by classify's positional end-of-part set and by assign's end-of-stage pass. Both go; assemble, which already withdrew this report kind where the letter was roled SUFFIX or TITLE, withdraws it where no given, middle or family word follows the letter in its segment. Same readings, and the position cut's one wrong silence goes: 'jose e nee' and 'smith, e nee smith', where the marker is a name word, report again as master did. The two reports on a numeral a post-nominal follows share `_vocab.reads_as_a_reported_numeral`; `is_trailing_numeral_suffix`'s `one_case` is required, the superset passing None explicitly. History moved out of comments, case notes and test docstrings; the facade off-switch rows collapse to one field set; the v/j asymmetry is an S10 Open question. Co-Authored-By: Claude Opus 5.5 --- .../decisions/S10-trailing-roman-numeral.md | 2 + docs/design/rules/P3-connectives.md | 2 +- nameparser/_pipeline/_assemble.py | 23 ++++- nameparser/_pipeline/_assign.py | 85 ++++--------------- nameparser/_pipeline/_classify.py | 39 ++------- nameparser/_pipeline/_group.py | 8 +- nameparser/_pipeline/_pieces.py | 2 +- nameparser/_pipeline/_vocab.py | 30 +++++-- tests/v2/cases.py | 38 +++++---- tests/v2/test_cases.py | 8 +- tests/v2/test_facade.py | 43 +++------- tests/v2/test_properties.py | 12 +-- 12 files changed, 119 insertions(+), 173 deletions(-) diff --git a/docs/design/decisions/S10-trailing-roman-numeral.md b/docs/design/decisions/S10-trailing-roman-numeral.md index b9ecf8ee..14a4710f 100644 --- a/docs/design/decisions/S10-trailing-roman-numeral.md +++ b/docs/design/decisions/S10-trailing-roman-numeral.md @@ -18,3 +18,5 @@ And the review's other defect: the one-case test of the word in front accepted any one-character cased letter, so `IVAN Ж V` and `JOSÉ É V` moved off their mixed-case reading and a decomposed `é` read otherwise than a composed one. It asks now whether the letter's capital is initial-shaped, which is the rule the docstring states (`john j v` as `JOHN J V`; `é`, `ж` and an uncased word as before). The grid gains a non-ASCII letter in front and a shape-only three-letter numeral; master disagrees on 315 of its 1,872 pairs, the branch on none. - 2026-10-18 (Derek), #722's last review — ONE WITHDRAWAL, AT THE END OF ASSIGN, AND CORRECTIONS TO THE ENTRY ABOVE. Two reviews of the round above found P3's broadened exception ("unless no name word of its part stands to its right") implemented only where the part's last name word is one token on the no-comma path: `john van e phd` (the letter ending a particle chain), `smith, john e phd` (the given part after a family comma) and `, john i phd` kept `conjunction-or-initial`, and `freiherr von i dr.` reached #718's stand-down and lost the numeral's report to the connective's. The withdrawal is one role-based pass at the end of the stage now, over every segment: a marked letter that is its segment's last name-role word with only suffix- and title-roled words after it loses classify's report. The numeral's own report is emitted before it, so one token never carries both, and the stand-down is gone — not "kept as a defensive check", which the entry above said on a measurement that reached no particle chain. The maiden clause gains the numeral report's twin (`Jane Doe nee Smith V PhD`, maiden 'Smith V', reported), the same fork taken a stage earlier. Corrections to the entry above, each measured on this commit's tree: option B moves 19 readings, not 18, and A 11, over the snapshot's 5,711 names with the stated patches to `_pieces.peel_trailing`'s count (A: `k - 3 >= second`; B: the count dropped) — the first figures were over an earlier snapshot plus fifteen named inputs. #36 and #136 do not say their names are real people's: #36 reports 'Hardie Leannon V' (possibly generated test data), #136 a reporter writing 'John W. Ingram, V' and 'John W. Ingram V'. The real person is the sheriff Derek found, written 'John W Ingram V' and 'John Ingram V'. And "1.4.0 read the same" holds for the capital spellings only: 1.4.0 read `john smith v phd` and `john smith i jr` with the letter in the suffix. THE PREMISE BEHIND #718, REVISED. #718's entry quotes Derek that a generational V is "mostly archaic". On this PR he found otherwise: searching shows many people named John Ingram with every variation of generational marker and middle initial, written without a comma unless a credential follows ("MBA", "MD"). Nothing decided here moves — the two-word count reads each of those as the generation — but #718's lone-numeral reading (`John V`, family 'V') now rests on the count alone, a name of two words having no word to spare for a generation, not on the generation being rare. +- 2026-10-18, #722's /simplify pass — P3'S "CONNECTS NOTHING" IS ASKED IN ONE PLACE, OVER FINAL ROLES. The last two rounds answered rules/P3's "unless no name word of its part stands to its right" twice: classify skipped its report on a letter that was its part's last own word by position (`ends`), and assign withdrew it at the end of the stage for a letter followed only by suffix and title roles. Both go; assemble, which already withdrew this report where the letter was roled SUFFIX or TITLE (rules/A1), withdraws it too where no given, middle or family word follows the letter in its segment — the roles being final there and nothing downstream moving them. Measured, the one site reads everything the two did, and restores one reading the position cut had wrongly silenced: where no clause forms and the marker is read as a name word, a name word does follow the letter, so `jose e nee` (family 'nee') and `smith, e nee smith` report `conjunction-or-initial` again, as master did. The two reports on a numeral a post-nominal follows (assign's, and the maiden take's twin) share one predicate, `_vocab.reads_as_a_reported_numeral`. + Open: "a name written wholly in one case says nothing about any letter in it" is applied to the roman-numeral letters only. classify tags a one-case lowercase `v`, `i` or `x` the initial its capital is, and the numeral fork reads a bare letter in front as its capital would; other letters keep their lowercase reading (`j nee smith` reads family 'j' where `v nee smith` reads given 'v'; a lone `j` reports given-or-family). Generalizing to every letter is a behavior change — #722's altitude review prototyped it and found roles, initials and reports moving on its grid (`jose j y lopez` → middle 'j y') and two tests failing — so it is a question for Derek, not a cleanup. diff --git a/docs/design/rules/P3-connectives.md b/docs/design/rules/P3-connectives.md index 299968ac..01dd0e06 100644 --- a/docs/design/rules/P3-connectives.md +++ b/docs/design/rules/P3-connectives.md @@ -157,5 +157,5 @@ P3. Rationale: connective words ("y", "of the") bind name words into same two words unjoined are two name words and H1 does not fire. P1's leading run is the second (#395, landed): its run takes the "Vega y Santos" join whole or stops before it. - history: decisions/P3 · interacts: H1, P1, M2, R1, R3, R4, C1, S6, S10 · implemented: nameparser/_pipeline/_assign.py, nameparser/_pipeline/_classify.py, nameparser/_pipeline/_comma.py, nameparser/_pipeline/_group.py, nameparser/_pipeline/_pieces.py, nameparser/_pipeline/_post_rules.py + history: decisions/P3 · interacts: H1, P1, M2, R1, R3, R4, C1, S6, S10 · implemented: nameparser/_pipeline/_assemble.py, nameparser/_pipeline/_classify.py, nameparser/_pipeline/_comma.py, nameparser/_pipeline/_group.py, nameparser/_pipeline/_pieces.py, nameparser/_pipeline/_post_rules.py diff --git a/nameparser/_pipeline/_assemble.py b/nameparser/_pipeline/_assemble.py index 363509b6..58d1e03f 100644 --- a/nameparser/_pipeline/_assemble.py +++ b/nameparser/_pipeline/_assemble.py @@ -2,7 +2,8 @@ Consumes: tokens (all roles set), dropped, ambiguities (by index; one carrying ``field_tail`` is worded from its word's final role, -#626). +#626), and segments, for the part a withdrawn connective report's +letter ends (rules/P3, #721). Produces: a validated ParsedName -- the constructor re-checks every invariant (span order/bounds, ambiguity subset), so a pipeline bug that would produce an invalid result dies HERE, not in a renderer @@ -20,7 +21,7 @@ """ from __future__ import annotations -from nameparser._pipeline._state import FIELD_READING, ParseState +from nameparser._pipeline._state import FIELD_READING, NAME_ROLES, ParseState from nameparser._types import ( Ambiguity, AmbiguityKind, ParsedName, Role, Token, ) @@ -74,6 +75,7 @@ def assemble(state: ParseState) -> ParsedName: if contentless: final = {} ambiguities = [] + segment_of: dict[int, tuple[int, ...]] | None = None for pending in state.ambiguities: materialized = tuple(final[i] for i in pending.indices if i in final) @@ -114,6 +116,23 @@ def assemble(state: ParseState) -> ParsedName: and any(t.role in (Role.SUFFIX, Role.TITLE) for t in materialized)): continue + # rules/P3: "unless no name word of its part stands to its + # right" -- nothing after the letter in its segment, or only + # suffixes, titles, a maiden clause or a nickname: it connects + # nothing, so the report goes (#721). Asked here for the reason + # above: the roles are final. Where the letter is also a + # numeral the fork read, its own report (rules/S10) is the one + # left, so one token never carries both. Built only for a + # connective report, so other names pay nothing. + if pending.kind is AmbiguityKind.CONJUNCTION_OR_INITIAL: + if segment_of is None: + segment_of = {j: seg for seg in state.segments + for j in seg} + i = pending.indices[0] + if not any(j > i and j in final + and final[j].role in NAME_ROLES + for j in segment_of.get(i, ())): + continue detail = pending.detail if pending.field_tail is not None: # A report naming a field, in a name emptied above (no diff --git a/nameparser/_pipeline/_assign.py b/nameparser/_pipeline/_assign.py index ba4a0496..da2b5841 100644 --- a/nameparser/_pipeline/_assign.py +++ b/nameparser/_pipeline/_assign.py @@ -7,9 +7,7 @@ group's joins left no name word in front of it. Produces: tokens with roles set on every main-stream token, and the folded-middle tag on the particles P6's attachment joins to the family -after a family comma (#613); its own ambiguities (below), less -classify's CONJUNCTION_OR_INITIAL on a letter that only suffixes and -titles follow in its part (#721). +after a family comma (#613). Reads: Policy.name_order (#270), is_suffix_lenient on the trailing piece of a two-part comma name, and Policy.script_orders (#271, which overrides it when every name piece is written wholly in one script, or @@ -91,8 +89,8 @@ from nameparser._lexicon import Lexicon from nameparser._pipeline._vocab import ( - _ROMAN, effective_script, is_suffix_lenient, is_trailing_numeral_suffix, - numeral_settled_by_case, resolve_script_set, + _ROMAN, effective_script, is_suffix_lenient, + reads_as_a_reported_numeral, resolve_script_set, ) from nameparser._pipeline._pieces import ( anchor_in_reach, credential_at_the_given_slot, given_slot_anchors, @@ -102,8 +100,7 @@ tail_reading, trailing_titles, ) from nameparser._pipeline._state import ( - AMBIGUOUS_ACRONYM_TAG, NAME_ROLES, ParseState, PendingAmbiguity, - Structure, + AMBIGUOUS_ACRONYM_TAG, ParseState, PendingAmbiguity, Structure, WorkToken, _AMBIGUOUS_CREDENTIAL_TAGS, _NEVER_FLIPPED, copy_with, ) from nameparser._policy import Policy, Script @@ -310,39 +307,6 @@ def _placed(read: TailRead, rest: list[int], peel.absorbed) -# rules/P3: "unless no name word of its part stands to its right" -- -# a marked letter whose part holds nothing after it but suffixes and -# titles connects nothing, so classify's connective-or-initial report -# on it is withdrawn here, once every segment's roles are known: the -# no-comma name, a particle chain ending in the letter ('john van e -# phd') and the given part after a family comma ('smith, john e phd') -# alike (#721's reviews). A letter with nothing at all after it never -# carried the report (classify). Where the letter is also a numeral the -# fork read, its own report (S10) is the one left, so one token never -# carries both -- which is why #718's stand-down is gone. Called only -# for a one-case name, the one classify can report on. -def _withdraw_unjoined_connectives(segments: Sequence[Sequence[int]], - tokens: Sequence[WorkToken], - ambiguities: list[PendingAmbiguity], - ) -> None: - unjoined = set() - for seg in segments: - last, after = None, False - for j in seg: - role = tokens[j].role - if role in NAME_ROLES: - last, after = j, False - elif role is Role.SUFFIX or role is Role.TITLE: - after = after or last is not None - if last is not None and after: - unjoined.add(last) - if unjoined: - ambiguities[:] = [ - a for a in ambiguities - if a.kind is not AmbiguityKind.CONJUNCTION_OR_INITIAL - or len(a.indices) != 1 or a.indices[0] not in unjoined] - - def _assign_main(seg_idx: int, state: ParseState, tokens: list[WorkToken], ambiguities: list[PendingAmbiguity], @@ -568,26 +532,19 @@ def _assign_main(seg_idx: int, state: ParseState, # 'John Smith V PhD' keeps family 'V' and reports, as 'john i, phd' # does). Decided here, where the trailing read has said what that # run is - if name_pieces and (suffix_pieces or titled_tail): - last = pieces[name_pieces[-1]] - word = tokens[last[0]].text - if len(last) == 1: - # `_ROMAN` first, C-level, so an ordinary name word pays - # no frame for the rest; the connective report the letter - # may carry is withdrawn at the end of the stage, once - # every segment's roles are known - # (`_withdraw_unjoined_connectives`) - if (_ROMAN.match(word) is not None and len(name_pieces) >= 2 - and tuple(last) not in peeled.picks - and is_trailing_numeral_suffix( - word, tokens[pieces[name_pieces[-2]][0]].text, - state.one_case) - and not numeral_settled_by_case(word, state.one_case)): - ambiguities.append(PendingAmbiguity( - AmbiguityKind.SUFFIX_OR_NAME, - f"{word!r} is a roman numeral, but a post-nominal or " - f"title follows it; read as ", tuple(last), - field_tail=" rather than a generational suffix")) + # `_ROMAN` first, C-level, so an ordinary name word pays no frame + if (len(name_pieces) >= 2 and (suffix_pieces or titled_tail) + and len(last := pieces[name_pieces[-1]]) == 1 + and _ROMAN.match(word := tokens[last[0]].text) is not None + and tuple(last) not in peeled.picks + and reads_as_a_reported_numeral( + word, tokens[pieces[name_pieces[-2]][0]].text, + state.one_case)): + ambiguities.append(PendingAmbiguity( + AmbiguityKind.SUFFIX_OR_NAME, + f"{word!r} is a roman numeral, but a post-nominal or title " + f"follows it; read as ", tuple(last), + field_tail=" rather than a generational suffix")) for piece in peeled.picks: # every pick is in rest, so the loops above just gave it a # role; whether it was read as a suffix or a name settles what @@ -1582,14 +1539,6 @@ def reads_as_a_suffix(m: int, titled: frozenset[int]) -> bool: and not any("vocab:suffix" in tokens[i].tags for i in piece)) else Role.SUFFIX) - # a plain loop, so a one-case name with no connective report pays - # no frame for the pass - if state.one_case: - for amb in ambiguities: - if amb.kind is AmbiguityKind.CONJUNCTION_OR_INITIAL: - _withdraw_unjoined_connectives(state.segments, tokens, - ambiguities) - break return copy_with(state, tokens=tuple(tokens), order=order, ambiguities=tuple(ambiguities)) diff --git a/nameparser/_pipeline/_classify.py b/nameparser/_pipeline/_classify.py index 139dde64..5e9b8f78 100644 --- a/nameparser/_pipeline/_classify.py +++ b/nameparser/_pipeline/_classify.py @@ -1,7 +1,6 @@ """Stage: classify. -Consumes: tokens, segments (where a part ends, for the connective -fork's report, #721), comma_offsets (with token roles, the two halves of +Consumes: tokens, comma_offsets (with token roles, the two halves of the structural-boundary test the marker pass applies -- see _vocab.tag_marker_runs), and one_case where an earlier stage recorded it -- segment writes it lazily where C2's flag on a part past the @@ -107,8 +106,7 @@ def _tags_for(token: WorkToken, n: str, state: ParseState, clause cut, never from the name's own writing (#516 review round, a second reviewer's finding). `single_letter_connective` below keeps `one_case_own`: that fork is genuinely about the OWN-WORDS - span, and reads a clause's word as no evidence on purpose. The - one-case numeral initial reads `one_case` too (#721).""" + span, and reads a clause's word as no evidence on purpose.""" lex = state.lexicon tags = set(token.tags) if marker_tag is not None: @@ -321,15 +319,6 @@ def classify(state: ParseState) -> ParseState: # rather than at the escape itself, which runs before tokenize and # so has no token index to point at. ambiguities = list(state.ambiguities) - # rules/P3: "A connective with nothing to its right is connecting - # nothing" -- so a letter ending its part (the last own word before - # a comma, a marker's clause or the end of the name) offers the - # fork no connective reading, and it is not reported here (#721): - # it is read as an initial, and where it is a roman numeral the - # numeral fork reports instead (rules/S10). Built only when a - # letter has passed the emitter's other tests, so a name that holds - # no marked letter pays nothing for it. - ends: set[int] | None = None for i, token in enumerate(tokens): if (token.role is Role.NICKNAME and AMBIGUOUS_ACRONYM_TAG in token.tags): @@ -352,12 +341,11 @@ def classify(state: ParseState) -> ParseState: # LOWERCASE letter is tagged "initial" only by the fork's # branch ("jose e maria santos" tags a lowercase e), for a # conjunctions_ambiguous member, or by the one-case numeral - # (#721) -- which an own word reaches only when it is no - # single-letter connective, that taking the fork's branch - # first, while a clause word fails `i < clause_at` below. So - # for a letter of EITHER case in `conjunctions`, "initial" plus - # len 1 implies membership. It is - # kept only because `_tags_for` starts from `set(token.tags)`, so + # (#721), which never tags an own word that is a single-letter + # connective (the fork's branch takes those first). So for a + # letter of EITHER case in `conjunctions`, "initial" plus len 1 + # implies membership. It is kept only because `_tags_for` + # starts from `set(token.tags)`, so # this clause is what keeps the emitter honest if a runner ever # hands classify tokens it did not build; no such path exists # today. `conjunctions` is the clause doing real work: it keeps @@ -368,8 +356,8 @@ def classify(state: ParseState) -> ParseState: # clause's words were never eligible for the fork, so they must # never be eligible to report either. Emitted at the decision # site (mechanisms.md#AMBIGUITY-AT-THE-DECISION-SITE), per - # token -- 'e and e e' reports twice, the last 'e' connecting - # nothing (#721). + # token -- 'e and e e' emits twice; assemble withdraws the + # report on a letter no name word follows (rules/P3, #721). if (one_case and "initial" in token.tags and folded[i] in state.lexicon.conjunctions_ambiguous and folded[i] in state.lexicon.conjunctions @@ -377,15 +365,6 @@ def classify(state: ParseState) -> ParseState: and (len(token.text) == 1 or not token.text.isascii() and composed_len(token.text) == 1)): - if ends is None: - ends = set() - for seg in state.segments: - for j in reversed(seg): - if j < clause_at and tokens[j].role is None: - ends.add(j) - break - if i in ends: - continue ambiguities.append(PendingAmbiguity( AmbiguityKind.CONJUNCTION_OR_INITIAL, f"{token.text!r} is both a connective and an initial; " diff --git a/nameparser/_pipeline/_group.py b/nameparser/_pipeline/_group.py index dbec8ef1..3383bf12 100644 --- a/nameparser/_pipeline/_group.py +++ b/nameparser/_pipeline/_group.py @@ -81,8 +81,7 @@ import nameparser._pipeline._comma as _comma from nameparser._pipeline._vocab import D, PH from nameparser._pipeline._vocab import ( - comma_follows_marker, delimiter_cores, is_trailing_numeral_suffix, - numeral_settled_by_case, + comma_follows_marker, delimiter_cores, reads_as_a_reported_numeral, ) from nameparser._types import AmbiguityKind, Role @@ -362,9 +361,8 @@ def _maiden_take(pieces: Sequence[Sequence[int]], f"maiden marker's clause keeps it rather than reading it as " f"a generational suffix", tuple(last))) elif (released and len(last) == 1 and end - 2 >= lo - and is_trailing_numeral_suffix( - word.text, tokens[pieces[end - 2][0]].text, one_case) - and not numeral_settled_by_case(word.text, one_case)): + and reads_as_a_reported_numeral( + word.text, tokens[pieces[end - 2][0]].text, one_case)): # rules/S10: "a one-letter numeral before a credential is the # family name, and the reading is reported" -- the clause's # twin of assign's report (#721's review): the clause keeps the diff --git a/nameparser/_pipeline/_pieces.py b/nameparser/_pipeline/_pieces.py index 0b48a63f..e3f14252 100644 --- a/nameparser/_pipeline/_pieces.py +++ b/nameparser/_pipeline/_pieces.py @@ -1541,7 +1541,7 @@ def trailing_candidates(lo: int, pieces: Sequence[Sequence[int]], not tokens[piece[0]].tags.isdisjoint(_TRAILING_WORD_TAGS) or (titles_behind and is_trailing_numeral_suffix( tokens[piece[0]].text, - tokens[pieces[k - 1][0]].text))))): + tokens[pieces[k - 1][0]].text, None))))): break # a group-flagged suffix piece is outside the peel's walk # altogether (`peel_walk` drops it), so it is behind the diff --git a/nameparser/_pipeline/_vocab.py b/nameparser/_pipeline/_vocab.py index f8609b91..c5fc5c82 100644 --- a/nameparser/_pipeline/_vocab.py +++ b/nameparser/_pipeline/_vocab.py @@ -257,10 +257,9 @@ def is_initial_shaped(text: str) -> bool: # take: V and I are suffix vocabulary, X a numeral by shape alone, and # all three are bare capitals, which inside a name are middle initials. def is_trailing_numeral_suffix(text: str, preceding: str, - one_case: bool | None = None) -> bool: - """The trailing walk's roman-numeral fork, shared with - `trailing_candidates`' superset view and assign's report on a - numeral before a credential (#721): a FINAL single-token piece that is a roman numeral + one_case: bool | None) -> bool: + """The trailing walk's roman-numeral fork: a FINAL single-token + piece that is a roman numeral reads as the suffix when the piece before it does not look like part of an initial run -- for a one-letter numeral a necessary condition only, the caller's words-to-spare count deciding @@ -278,9 +277,9 @@ def is_trailing_numeral_suffix(text: str, preceding: str, the word in front is read as its capital would be, so a bare 'j' is initial-shaped as 'J' is (rules/S10, #721's review: 'john j v' reads as 'JOHN J V'), while 'é', 'ж' or an uncased word, whose - capitals are not, stay as they were. A caller passing None, - `trailing_candidates`' superset for the maiden take, gets the - shape test alone, which admits more and so stays a superset.""" + capitals are not, stay as they were. None (the maiden take's + superset view passes it) gets the shape test alone, which admits + more and so stays a superset.""" return (_ROMAN.match(text) is not None and not is_initial_shaped(preceding) and not (one_case and is_initial_shaped(preceding.upper()))) @@ -289,8 +288,9 @@ def is_trailing_numeral_suffix(text: str, preceding: str, # rules/S10: "a numeral written in lower case in a name of mixed case is # the generation and reports nothing" -- the one writing that settles # the fork, since nobody writes an initial in lower case beside -# capitals. Asked by the walk alone, which takes such a numeral as any -# suffix word and records no call for a reporter to report. +# capitals. Asked by the walk, which takes such a numeral as any suffix +# word and records no call, and by the report on a numeral before a +# credential (`reads_as_a_reported_numeral`). def numeral_settled_by_case(text: str, one_case: bool | None) -> bool: """Whether a numeral the trailing fork reads is lower case in a name written in more than one case. `one_case` is @@ -298,6 +298,18 @@ def numeral_settled_by_case(text: str, one_case: bool | None) -> bool: return one_case is False and text.islower() +# rules/S10: "either reading is reported" -- the one question the two +# reports on a numeral a post-nominal follows ask (assign's, and the +# maiden take's for a numeral its clause keeps): the fork would read +# this numeral, and its writing does not settle it (#721) +def reads_as_a_reported_numeral(text: str, preceding: str, + one_case: bool | None) -> bool: + """Whether a numeral the walk left a name word, with `preceding` + in front of it, is one the fork would weigh and so reports.""" + return (is_trailing_numeral_suffix(text, preceding, one_case) + and not numeral_settled_by_case(text, one_case)) + + # #544: a single-letter roman numeral ('V', 'v', 'I.'), in any case, # is excluded for its SHAPE -- one letter is how a middle initial is # written, and rules/S3 retires single-character matches for the diff --git a/tests/v2/cases.py b/tests/v2/cases.py index dc3c55ad..eaed1afb 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -7601,8 +7601,7 @@ def _check_cjk_shape_purity(self) -> None: {"title": "dr.", "given": "john", "family": "i"}, ambiguities=("suffix-or-name",), classification="fix(#316)", notes="the same with a trailing title (H5): one report on " - "the letter, the numeral's. #721's first form reported " - "both kinds on it. 1.4.0 read suffix 'dr.'"), + "the letter, the numeral's. 1.4.0 read suffix 'dr.'"), Case("a_one_case_non_numeral_link_before_a_credential_is_silent", "jose e phd", {"given": "jose", "family": "e", "suffix": "phd"}, @@ -7636,30 +7635,25 @@ def _check_cjk_shape_purity(self) -> None: ambiguities=("suffix-or-name",), notes="the one-case test of the word in front asks whether its " "CAPITAL is initial-shaped, and 'Ж' is not (an initial is " - "a Latin capital), so this reads as 'Ivan Ж V' does. " - "#721's first form took any cased letter and read middle " - "'Ж', family 'V'"), + "a Latin capital), so this reads as 'Ivan Ж V' does"), Case("a_non_ascii_lowercase_letter_in_front_is_no_initial", "josé é v", {"given": "josé", "family": "é", "suffix": "v"}, ambiguities=("suffix-or-name",), - notes="the lowercase twin of the row above, and the one whose " - "decomposed spelling had read otherwise under #721's " - "first form"), + notes="the lowercase twin of the row above; its decomposed " + "spelling reads the same (rules/H2's encoding note)"), Case("a_one_case_link_ending_a_particle_chain_before_a_credential", "john van e phd", {"given": "john", "family": "van e", "suffix": "phd"}, notes="the letter ends the chain the particle made, with only " "a credential behind it: it connects nothing (P3), so " - "nothing reports. #721's first withdrawal tested a " - "one-token piece and missed it"), + "nothing reports"), Case("a_one_case_link_before_a_credential_after_a_family_comma", "smith, john e phd", {"given": "john", "middle": "e", "family": "smith", "suffix": "phd"}, - notes="the given part after a family comma: the same rule, " - "decided over roles at the end of the stage, so it " - "reaches the comma's own walk too"), + notes="the given part after a family comma: only a " + "credential follows the letter in its part (P3)"), Case("a_chained_one_case_numeral_before_a_title_reports_once", "freiherr von i dr.", {"title": "freiherr dr.", "family": "von i"}, @@ -7667,9 +7661,8 @@ def _check_cjk_shape_purity(self) -> None: classification="fix(#316)", notes="a letter ending a particle chain with only a title " "behind it: the numeral's report, the connective's " - "withdrawn -- where #718's stand-down had kept the " - "connective's and dropped the numeral's. 1.4.0 read " - "first 'freiherr von', last 'i', suffix 'dr.'"), + "withdrawn (P3). 1.4.0 read first 'freiherr von', last " + "'i', suffix 'dr.'"), Case("a_numeral_kept_by_a_maiden_clause_before_a_credential_reports", "Jane Doe nee Smith V PhD", {"given": "Jane", "family": "Doe", "suffix": "PhD", @@ -7678,6 +7671,19 @@ def _check_cjk_shape_purity(self) -> None: notes="the clause's twin of 'John Smith V PhD': the clause " "keeps the numeral a post-nominal stands behind, and " "reports it (S10). 1.4.0 had no maiden markers"), + Case("a_one_case_link_before_a_marker_read_as_a_name_reports", + "jose e nee", + {"given": "jose", "middle": "e", "family": "nee"}, + ambiguities=("conjunction-or-initial",), + notes="no clause forms, so the marker is the family name and a " + "name word follows the letter: P3 reports it, as 1.4.0's " + "successors did -- the contrast with 'jose e nee smith'"), + Case("a_lowercase_shape_numeral_before_a_credential_is_settled", + "John x PhD", + {"given": "John", "family": "x", "suffix": "PhD"}, + notes="lower case beside capitals settles the numeral (S10), " + "so the report on a numeral a post-nominal follows does " + "not fire"), Case("a_lowercase_numeral_in_a_one_case_name_still_reports", "john smith i", {"given": "john", "family": "smith", "suffix": "i"}, diff --git a/tests/v2/test_cases.py b/tests/v2/test_cases.py index 49663bf0..8e37449b 100644 --- a/tests/v2/test_cases.py +++ b/tests/v2/test_cases.py @@ -190,12 +190,8 @@ def test_the_field_sweep_sees_the_claims_it_checks() -> None: # #718's declined-numeral report, which names the field the letter # was read into, on those rows under each order; 1191 -> 1210 the # same day: the PR review's rows (the chain-taken, family-first, - # lowercase and maiden-clause numerals); 1210 -> 1213 the same day: - # #721's one-case lowercase numerals, counted and reported; 1213 - # -> 1234 the same day: #721's PR-review rows (the numeral before a - # credential or title, the real generations, the lone letters); - # 1234 -> 1240: the two non-ASCII letter-in-front rows; 1240 -> - # 1243: the end-of-stage withdrawal's rows (#722's last review). + # lowercase and maiden-clause numerals); 1210 -> 1243 the same day: + # #721's rows (#722). assert claims == 1243, ( f"the field sweep checks {claims} claims, recorded as 1243 on " f"2026-10-18") diff --git a/tests/v2/test_facade.py b/tests/v2/test_facade.py index 8d40bf5f..5a386edd 100644 --- a/tests/v2/test_facade.py +++ b/tests/v2/test_facade.py @@ -1052,44 +1052,29 @@ def test_initials_of_an_unpickled_or_copied_name_ask_the_vocabulary_too() -> Non def test_the_v1_off_switch_unjoins_the_link_only() -> None: """decisions/P3's off switch on the V1 SURFACE (#397 second - review; named for what it restored, the pre-#397 fields, until - #721). The core-side switch is pinned by the off-switch grid in + review). The core-side switch is pinned by the off-switch grid in tests/v2/test_properties.py and this is the facade half, which has no `Lexicon` of its own to remove from. - The three role movers of decisions/P3's one-case paragraph are - the rows: the switch restored their 2.3.0 fields, measured - 2026-09-20 -- until #721, which made a one-case - one-letter numeral the initial its capital is (rules/S10), so with - `i` out of the conjunctions the letter now reads as 'I' does in - each, switch on or off. What the switch still moves is the join, - below. + The switch moves the Catalan join and leaves the three one-case + role movers of decisions/P3's paragraph alone: with `i` out of the + conjunctions it is a one-letter roman numeral, the initial its + capital is (rules/S10), so each reads as with the switch on. A local `Constants`, never the shared `CONSTANTS`: a removal on the singleton would leak into every later test in the process. """ constants = Constants() constants.conjunctions.remove("i") - for text, on_fields, off_fields in ( - ("rovira, i", - {"first": "i", "last": "rovira", "suffix": ""}, - {"first": "i", "last": "rovira", "suffix": ""}), - ("john smith i jr", - {"first": "john", "middle": "smith", "last": "i", - "suffix": "jr"}, - {"first": "john", "middle": "smith", "last": "i", - "suffix": "jr"}), - ("maier, amy i, jr.", - {"first": "amy", "middle": "i", "last": "maier", - "suffix": "jr."}, - {"first": "amy", "middle": "i", "last": "maier", - "suffix": "jr."})): - on = HumanName(text) - off = HumanName(text, constants) - for field, value in on_fields.items(): - assert getattr(on, field) == value, (text, field) - for field, value in off_fields.items(): - assert getattr(off, field) == value, (text, field) + for text, fields in ( + ("rovira, i", {"first": "i", "last": "rovira", "suffix": ""}), + ("john smith i jr", {"first": "john", "middle": "smith", + "last": "i", "suffix": "jr"}), + ("maier, amy i, jr.", {"first": "amy", "middle": "i", + "last": "maier", "suffix": "jr."})): + for name in (HumanName(text), HumanName(text, constants)): + for field, value in fields.items(): + assert getattr(name, field) == value, (text, field) # and the join itself, the reading the bullet leads with joined = HumanName("Josep Carod i Rovira") assert joined.last == "Carod i Rovira" diff --git a/tests/v2/test_properties.py b/tests/v2/test_properties.py index cfacb73e..45ac067b 100644 --- a/tests/v2/test_properties.py +++ b/tests/v2/test_properties.py @@ -3336,9 +3336,9 @@ def test_every_one_letter_particle_veto_is_load_bearing( # --- #721: a name written in one case reads as its capitals --------- # S10 (the trailing roman numeral) says a name written wholly in one -# case says nothing about any letter in it -- so a wholly lower-case name parses exactly as the -# same name in capitals, fields (compared case-folded) and report -# kinds alike. The grid crosses the shapes a one-letter numeral meets +# case says nothing about any letter in it -- so a wholly lower-case +# name parses exactly as the same name in capitals, fields (compared +# case-folded) and report kinds alike. The grid crosses the shapes a one-letter numeral meets # -- every comma shape, a maiden clause, a title, a particle chain, a # bound given name, an initial, a nickname and a credential before or # after it -- with the numerals a listed and a shape-only spelling of @@ -3351,8 +3351,8 @@ def test_every_one_letter_particle_veto_is_load_bearing( "{a} {b} nee {c} {n}", "dr {a} {n}", "dr {a} {b} {n}", "sir {a} {n}", "{a} van {b} {n}", "freiherr von {b} {n}", "abdul {a} {n}", "{a} j. {n}", "({c}) {a} {n}", "{a} {b} jr {n}", "{a} {n} phd", - # #721's review: a bare letter in front, a trailing title, a - # doubled numeral and a trailing period + # a bare letter in front, a trailing title, a doubled numeral and + # a trailing period "{a} b {n}", "{a} e {n}", "{a} {n} prof.", "{a} {b} {n} prof.", "{a} {b} {n} {n}", "{a} {b} {n}.", # a non-ASCII letter in front, whose capital is no initial @@ -3402,6 +3402,6 @@ def test_the_one_case_grid_sees_the_letter_in_front( monkeypatch.setattr( "nameparser._pipeline._pieces.is_trailing_numeral_suffix", lambda text, preceding, one_case=None: - _vocab.is_trailing_numeral_suffix(text, preceding)) + _vocab.is_trailing_numeral_suffix(text, preceding, None)) assert (len(_one_case_disagreements()) == _ONE_CASE_LETTER_IN_FRONT_UNGUARDED) From f199dd375230a0368951428cab43bddd4ae4967d Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sun, 18 Oct 2026 17:25:34 -0700 Subject: [PATCH 9/9] Describe the one-case single-letter question as what it is, and point at #723 Co-Authored-By: Claude Opus 5.5 --- docs/design/decisions/S10-trailing-roman-numeral.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/design/decisions/S10-trailing-roman-numeral.md b/docs/design/decisions/S10-trailing-roman-numeral.md index 14a4710f..28691d24 100644 --- a/docs/design/decisions/S10-trailing-roman-numeral.md +++ b/docs/design/decisions/S10-trailing-roman-numeral.md @@ -19,4 +19,4 @@ - 2026-10-18 (Derek), #722's last review — ONE WITHDRAWAL, AT THE END OF ASSIGN, AND CORRECTIONS TO THE ENTRY ABOVE. Two reviews of the round above found P3's broadened exception ("unless no name word of its part stands to its right") implemented only where the part's last name word is one token on the no-comma path: `john van e phd` (the letter ending a particle chain), `smith, john e phd` (the given part after a family comma) and `, john i phd` kept `conjunction-or-initial`, and `freiherr von i dr.` reached #718's stand-down and lost the numeral's report to the connective's. The withdrawal is one role-based pass at the end of the stage now, over every segment: a marked letter that is its segment's last name-role word with only suffix- and title-roled words after it loses classify's report. The numeral's own report is emitted before it, so one token never carries both, and the stand-down is gone — not "kept as a defensive check", which the entry above said on a measurement that reached no particle chain. The maiden clause gains the numeral report's twin (`Jane Doe nee Smith V PhD`, maiden 'Smith V', reported), the same fork taken a stage earlier. Corrections to the entry above, each measured on this commit's tree: option B moves 19 readings, not 18, and A 11, over the snapshot's 5,711 names with the stated patches to `_pieces.peel_trailing`'s count (A: `k - 3 >= second`; B: the count dropped) — the first figures were over an earlier snapshot plus fifteen named inputs. #36 and #136 do not say their names are real people's: #36 reports 'Hardie Leannon V' (possibly generated test data), #136 a reporter writing 'John W. Ingram, V' and 'John W. Ingram V'. The real person is the sheriff Derek found, written 'John W Ingram V' and 'John Ingram V'. And "1.4.0 read the same" holds for the capital spellings only: 1.4.0 read `john smith v phd` and `john smith i jr` with the letter in the suffix. THE PREMISE BEHIND #718, REVISED. #718's entry quotes Derek that a generational V is "mostly archaic". On this PR he found otherwise: searching shows many people named John Ingram with every variation of generational marker and middle initial, written without a comma unless a credential follows ("MBA", "MD"). Nothing decided here moves — the two-word count reads each of those as the generation — but #718's lone-numeral reading (`John V`, family 'V') now rests on the count alone, a name of two words having no word to spare for a generation, not on the generation being rare. - 2026-10-18, #722's /simplify pass — P3'S "CONNECTS NOTHING" IS ASKED IN ONE PLACE, OVER FINAL ROLES. The last two rounds answered rules/P3's "unless no name word of its part stands to its right" twice: classify skipped its report on a letter that was its part's last own word by position (`ends`), and assign withdrew it at the end of the stage for a letter followed only by suffix and title roles. Both go; assemble, which already withdrew this report where the letter was roled SUFFIX or TITLE (rules/A1), withdraws it too where no given, middle or family word follows the letter in its segment — the roles being final there and nothing downstream moving them. Measured, the one site reads everything the two did, and restores one reading the position cut had wrongly silenced: where no clause forms and the marker is read as a name word, a name word does follow the letter, so `jose e nee` (family 'nee') and `smith, e nee smith` report `conjunction-or-initial` again, as master did. The two reports on a numeral a post-nominal follows (assign's, and the maiden take's twin) share one predicate, `_vocab.reads_as_a_reported_numeral`. - Open: "a name written wholly in one case says nothing about any letter in it" is applied to the roman-numeral letters only. classify tags a one-case lowercase `v`, `i` or `x` the initial its capital is, and the numeral fork reads a bare letter in front as its capital would; other letters keep their lowercase reading (`j nee smith` reads family 'j' where `v nee smith` reads given 'v'; a lone `j` reports given-or-family). Generalizing to every letter is a behavior change — #722's altitude review prototyped it and found roles, initials and reports moving on its grid (`jose j y lopez` → middle 'j y') and two tests failing — so it is a question for Derek, not a cleanup. + Open: "a name written wholly in one case says nothing about any letter in it" is applied to the roman-numeral letters only. In a one-case name a bare capital is tagged an initial and a bare lowercase letter is not, unless it is `v`, `i` or `x` (this PR) or a marked connective (P3), so the lowercase and all-caps spellings of a one-case name read differently wherever a reader honors the tag — before #721 too: M4 makes a lone name word before a marker the family but not an initial (`j nee smith` family 'j', `J NEE SMITH` given 'J'), O5 is silenced by one (a lone `j` reports given-or-family, `J` does not), and the connective join defers to one (`jose j y lopez` family 'j y lopez', `JOSE J Y LOPEZ` middle 'J Y', family 'LOPEZ'). Tagging every one-case single letter as its capital is tagged is a behavior change beyond this PR: #723.