diff --git a/AGENTS.md b/AGENTS.md index c2d2941f..350fa3ae 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -186,7 +186,7 @@ The library has two layers: `nameparser/config/` (data) and `nameparser/parser.p **Sweep the forms a change can reach: comma shapes, then `name_order`.** No comma, a FULL name before the comma, a ONE-WORD name before the comma and NOTHING before the comma are four paths, and the last two are the misses — #429, #430 and #432 are all the one-word path disagreeing with the full-name path on inputs the full-name path reads correctly, and #613's PR review found `, PhD` and `, Jr.` (an empty surname field, as CSV data writes it) read as given and title by a decision that returned early on an empty head, after two review rounds whose every grid had a name before the comma. Put an empty head and a nickname-only head (`(Bob), PhD`) in every comma grid. Count the one-word path the way C3 counts, which is neither tokens nor pieces: a particle run and the one word it attaches to are one word (#575), a connective join and a bound given-name pair count as their words (#613, so `Ortega y Gasset, Dr.` is a whole name), and a suffix beside the surname counts as no name word at all. #603's review found the suffix half (`Vega y Lopez Jr., PhD Jones` lost its family where `Vega y Lopez, PhD Jones` kept it); #613 then reversed the connective half, and both now read given 'Vega', middle 'y', family 'Lopez' — so test a particle surname, a connective surname and a surname with a suffix, each before a comma. For orders the sweep already exists (`tests/v2/test_cases.py` runs every row under all three) but carries one assertion, R2's family partition, so it checks nothing a new change moves; coverage there has been vacuous before (PR #394's review found the suite passed with `name_order` discarded from grouping). Parse your change's names in each comma shape and each order, and read the ones you did not predict. And read a generated grid's moves by SHAPE, not by corpus membership: #614's first design passed the suite, the five gates and a fingerprint of every corpus name, because its grid moves were sorted into 'corpus name moved' and 'garbage', and the review found the moves that mattered among the 'garbage' -- a bare salutation (`Mr. and Mrs.` lost its connective to the family name) and a name with one trailing word (`Freiherr von Berg X.Y.Z.`). Pull out the head-plus-one-word slice of the grid and a salutation with nothing after it, and read every move there. Two more were the second review's (#614): no grid put TWO words that are both a title and a particle at the head (`Freiherr St van Berg MA`, where an index into the pieces was applied to a merged view of them -- `TITLES ∩ particles_ambiguous` is `{freiherr, st}`, so put both in every head list, each in its bare and its period-marked spelling -- #637's grid had `St` and `Freiherr` alone and missed that `St.` and `Freiherr.` are declined by the bound join and then attached by P6), and the grids compared ambiguity KINDS, so a report telling the caller the wrong field (`Freiherr von Berg Ma`, 'read as a given name' of a word in the family) passed every pin. #701's review added one: put a title/suffix dual (`Ms`, `Sr.`, `Cpl`) SECOND in a title run, behind an ordinary title. A stop that asks only "is this word suffix vocabulary" passes every row that opens with the dual and fails `Rev. Sr. Smith, Mary`; it was found by comparing `T Smith, John` against `T John Smith` over every title word, and a comma-versus-comma-less comparison is the cheap check for any rule that moves a word between the two slots. Compare each report's `detail` against the field its tokens land in -- since #626 `test_a_report_names_the_field_its_word_lands_in` (tests/v2/test_cases.py) does it over every non-locale case row under all three orders (a row carrying its own policy only as declared), for the phrasings its `_FIELD_CLAIMS` lists -- so a row with the shape is the test only where the report's wording is listed: a new emitter's wording goes in that list, and segment's report on a part past the second comma names no field (#629). #626's own first grid then missed an emitter for want of one comma shape (`, Ma Dr.`: an empty head, an ambiguous word, a trailing title, which H1 moves to the family): a trailing title belongs in every empty-head comma grid too. And #620's: an exclusion list in code being replaced is a set of rules, not noise -- #614's merged copy excluded every word the trailing read weighs from its units, the first #620 prototype dropped the list as a symptom of the copy, and its review found the list had been keeping positional counting's rule that a member in front counts as a word (`Freiherr von Berg MA X.Y.Z.` lost its suffix). Before deleting one, mutate each entry out and read what moves. The converse holds for replacing a hand copy WITH the shared predicate: it is a behavior change wherever the two disagreed, and #735's /simplify shipped one as "no reading moves" because the suite and the snapshot were its only check (`St Sir de Mesnil` family-first, a token test replaced by P1's piece test); diff the two predicates' answers over a grid before calling the swap a refactor. -**Then sweep the spellings: a class reached by vocabulary is also reached by SHAPE.** A test that uses only the listed spelling of a word class walks only the vocabulary path, and the shape path is a separate branch that can be missing while every test passes. Roman numerals are the sharpest case: `ii`, `iii` and `iv` are suffix vocabulary; `i` and `v` are vocabulary too but initial-shaped, so `is_suffix_piece` vetoes them and only the numeral fork reads them; and `vi` through `x` are in no list at all, read by the fork's shape test alone (checked 2026-10-04 against `Lexicon.default()`) -- which stops at `x`: `_vocab._ROMAN` matches I to X, so `John Smith XI` reads `XI` as a name word (re-checked 2026-10-07, #614's /simplify, which found a docstring claiming "any length"). And the fork counts a ONE-letter numeral besides, since #718 (rules/S10): it is the suffix only where the name has a word to spare, so `John V` and `John Smith V` take different branches of one fork, a one-case name being counted like capitals whatever the letter -- since #721 a one-case one-letter numeral carries the `initial` tag its capital does, and since #723 every one-case bare letter does (rules/T4), and a bare letter in front of the numeral is initial-shaped as its capital is, so `john v` reads as `JOHN V` does over `test_a_one_case_name_reads_as_its_capitals`'s grid (comma shapes, orders, a maiden clause, a particle chain, a bare letter in front, a trailing title, a non-ASCII letter in front: 315 of its then 1,872 pairs disagreed at 2bfa9533, before #721, and none after). Its first version claimed "every shape" with no bare letter in front and no trailing title in the grid, and #721's reviews found both broken; a grid's claim is the grid -- put a head-plus-numeral row beside every words-to-spare row. So a change that handles `V` and `III` can still miss `VI` — #610 found exactly that, the maiden take's candidate loop admitting a shape-only numeral in one position where the peel reads it in two (`Jane Doe nee Smith VI Prof.` kept maiden `Smith VI` while `V` and `III` worked). #656's sweep found the same class from the other side: C12 takes the word ending the name before a comma as EVIDENCE, and its first draft asked the suffix-word tag, so `John Smith VI, AD` missed the rule `III` reached, and then counted the whole part, so `Smith VI, AD` fired with `VI` as its own name word and `Smith V, AD` with a `V` that S10 reads as the family. A rule that weighs a numeral as a generation must ask the predicate that READS it as one (the trailing fork and its words-to-spare count), not the vocabulary that lists some of its spellings. The same split runs through the credentials (`MA` listed, `X.Y.Z.` by the dotted shape, `XYZ` by the caps shape, and `Ph. D.` merged by group into one flagged piece the peel's walk never holds -- #603 found #602's run-start predicate refusing it too, so `Smith, John Ph. D. Jones` kept middle 'Jones' where `Smith, John PhD Jones` read suffix 'PhD Jones') and the titles (`Prof.` period-marked and read at the end of a name, `Prof` bare and a name word there, and `Mr. and Mrs.` a run P3 joins, carrying a piece tag a one-word vocabulary test never sees -- PR #735's review found C12's hand copy of C13's title test (C8's until its 2026-10-19 split) reading `John Smith Jr., Mr. and Mrs. AD` as all suffix; ask C13's predicates in C13's order, `not is_suffix_piece` then `is_title_piece`, not the token tags, since `is_title_piece` alone reads the duals `Ms` and `Sr.` as titles). When a change touches a word class, test one spelling from each path. The same holds for a separator's SPACING: a comma (or any punctuation) can stand alone, glue to the word before it or glue to the word after, and each is a different token stream. #653's first fix swept `(née, Jones)` and missed `(née , Jones)`, where the kept comma became a word after the marker and M3 read the clause as maiden ', Jones'; its review found it. Put all three spacings in any grid around a separator. And a separator glued to a word hides the word from every vocabulary test, not only the one being changed: the semicolon delimiter's first draft read a glued word's stem in C1's run test alone and put 365 of ClinicalTrials.gov's reversed names right, missing the ones C13, C4 and C6 read off a word's TAGS (`MBBS;`, `MA;`); tagging the stem put 412 right, and reading it again in the comma decision's shape tests, which read a word's text rather than its tags (`MBBS;`), 431 (decisions/C10, 2026-10-24). A rule over a glued spelling reads the stem everywhere the word is read, its tags and every test of its text, and is swept against the space twin -- with a declared word delimiter too, which a substring test finds inside other words ('Stand' under ' and ', the review of the same change). A full stop INSIDE a word has four spellings of its own: initials run together (`J.R.R.`), joined words each closed by a stop (`Lt.Gov.`, `Msc.Ed.`), a joined word or name closed by none (`St.John`, `J.Smith`), and an abbreviation's own ending, closed or not (Italian `Sig.ra`, Latvian `Dr.philol.`). Shape tells the last from none of the others, so a rule over the shape decides which it costs; #838 found the ending only through the snapshot's CLDR row `Dut.ra Prof.ra`, and its first narrowing traded `St.John` for `Sig.ra` while the snapshot listed the `Dr.philol.` move among moves read as intended, until the design-docs review named it. Initials run together are also how some SURNAMES are written (Nepali `B.K.`, `K.C.`), and a caseless script writes its abbreviations, degrees included, as dotted letters (Thai `พ.บ.`): #864 found S3's position reading taking the first from 2.4.0 on, visible only once #866's Nepal candidates gave the corpora a population that writes them, and its first fix moved CLDR's Thai degrees into the name. A rule over dotted letters reads both before it ships. +**Then sweep the spellings: a class reached by vocabulary is also reached by SHAPE.** A test that uses only the listed spelling of a word class walks only the vocabulary path, and the shape path is a separate branch that can be missing while every test passes. Roman numerals are the sharpest case: `ii`, `iii` and `iv` are suffix vocabulary; `i` and `v` are vocabulary too but initial-shaped, so `is_suffix_piece` vetoes them and only the numeral fork reads them; and `vi` through `x` are in no list at all, read by the fork's shape test alone (checked 2026-10-04 against `Lexicon.default()`) -- which stops at `x`: `_vocab._ROMAN` matches I to X, so `John Smith XI` reads `XI` as a name word (re-checked 2026-10-07, #614's /simplify, which found a docstring claiming "any length"). And the fork counts a ONE-letter numeral besides, since #718 (rules/S10): it is the suffix only where the name has a word to spare, so `John V` and `John Smith V` take different branches of one fork, a one-case name being counted like capitals whatever the letter -- since #721 a one-case one-letter numeral carries the `initial` tag its capital does, and since #723 every one-case bare letter does (rules/T4), and a bare letter in front of the numeral is initial-shaped as its capital is, so `john v` reads as `JOHN V` does over `test_a_one_case_name_reads_as_its_capitals`'s grid (comma shapes, orders, a maiden clause, a particle chain, a bare letter in front, a trailing title, a non-ASCII letter in front: 315 of its then 1,872 pairs disagreed at 2bfa9533, before #721, and none after). Its first version claimed "every shape" with no bare letter in front and no trailing title in the grid, and #721's reviews found both broken; a grid's claim is the grid -- put a head-plus-numeral row beside every words-to-spare row. So a change that handles `V` and `III` can still miss `VI` — #610 found exactly that, the maiden take's candidate loop admitting a shape-only numeral in one position where the peel reads it in two (`Jane Doe nee Smith VI Prof.` kept maiden `Smith VI` while `V` and `III` worked). #656's sweep found the same class from the other side: C12 takes the word ending the name before a comma as EVIDENCE, and its first draft asked the suffix-word tag, so `John Smith VI, AD` missed the rule `III` reached, and then counted the whole part, so `Smith VI, AD` fired with `VI` as its own name word and `Smith V, AD` with a `V` that S10 reads as the family. A rule that weighs a numeral as a generation must ask the predicate that READS it as one (the trailing fork, `_vocab.reads_as_generation`, and its words-to-spare count), not the vocabulary that lists some of its spellings. The same split runs through the credentials (`MA` listed, `X.Y.Z.` by the dotted shape, `XYZ` by the caps shape, and `Ph. D.` merged by group into one flagged piece the peel's walk never holds -- #603 found #602's run-start predicate refusing it too, so `Smith, John Ph. D. Jones` kept middle 'Jones' where `Smith, John PhD Jones` read suffix 'PhD Jones') and the titles (`Prof.` period-marked and read at the end of a name, `Prof` bare and a name word there, and `Mr. and Mrs.` a run P3 joins, carrying a piece tag a one-word vocabulary test never sees -- PR #735's review found C12's hand copy of C13's title test (C8's until its 2026-10-19 split) reading `John Smith Jr., Mr. and Mrs. AD` as all suffix; ask C13's predicates in C13's order, `not is_suffix_piece` then `is_title_piece`, not the token tags, since `is_title_piece` alone reads the duals `Ms` and `Sr.` as titles). When a change touches a word class, test one spelling from each path. The same holds for a separator's SPACING: a comma (or any punctuation) can stand alone, glue to the word before it or glue to the word after, and each is a different token stream. #653's first fix swept `(née, Jones)` and missed `(née , Jones)`, where the kept comma became a word after the marker and M3 read the clause as maiden ', Jones'; its review found it. Put all three spacings in any grid around a separator. And a separator glued to a word hides the word from every vocabulary test, not only the one being changed: the semicolon delimiter's first draft read a glued word's stem in C1's run test alone and put 365 of ClinicalTrials.gov's reversed names right, missing the ones C13, C4 and C6 read off a word's TAGS (`MBBS;`, `MA;`); tagging the stem put 412 right, and reading it again in the comma decision's shape tests, which read a word's text rather than its tags (`MBBS;`), 431 (decisions/C10, 2026-10-24). A rule over a glued spelling reads the stem everywhere the word is read, its tags and every test of its text, and is swept against the space twin -- with a declared word delimiter too, which a substring test finds inside other words ('Stand' under ' and ', the review of the same change). A full stop INSIDE a word has four spellings of its own: initials run together (`J.R.R.`), joined words each closed by a stop (`Lt.Gov.`, `Msc.Ed.`), a joined word or name closed by none (`St.John`, `J.Smith`), and an abbreviation's own ending, closed or not (Italian `Sig.ra`, Latvian `Dr.philol.`). Shape tells the last from none of the others, so a rule over the shape decides which it costs; #838 found the ending only through the snapshot's CLDR row `Dut.ra Prof.ra`, and its first narrowing traded `St.John` for `Sig.ra` while the snapshot listed the `Dr.philol.` move among moves read as intended, until the design-docs review named it. Initials run together are also how some SURNAMES are written (Nepali `B.K.`, `K.C.`), and a caseless script writes its abbreviations, degrees included, as dotted letters (Thai `พ.บ.`): #864 found S3's position reading taking the first from 2.4.0 on, visible only once #866's Nepal candidates gave the corpora a population that writes them, and its first fix moved CLDR's Thai degrees into the name. A rule over dotted letters reads both before it ships. **And the encoding: a letter class written `[^\W\d_]` or tested with `isalpha()` is blind to a combining mark.** Python's `\w` and `isalpha()` do not match Mn, so a letter written as base plus accent (NFD) ends a word early; the vocabulary path is immune because `_normalize` NFC-folds, the shape path is not. #652: `Dr. inž. Janez Novak` read `inž.` as the given name when the ž was decomposed, and `Janez Novak, ŽŠČ` reversed, in two families that share the blind spot — the letter-shape regexes (`period_abbreviation`, `_DOTTED`, `_CHUNKED`, `_PAIRED_INITIALS`), now spelled with `_vocab._LETTER` (a letter plus the general combining blocks; `test_regex_sync` compares the two `_MARKS` strings and pins `period_abbreviation` against config's copy), and the caps predicates, which NFC-compose a non-ASCII word before `isalpha()` (`is_caps_word`, behind C-level `isalpha()`/`is_normalized` tests, so already-composed text pays no frame — measured 2026-10-12, `Smith, José` was 193 frames before and after; the first version called it for every non-ASCII word and cost one frame each, and no guard saw it because the reference name is ASCII). A width test must count letters, not code points (`composed_len`): admitting a decomposed word to the caps set put `John Smith, MÁ` on the pair exception's wrong side, and the review's grid found it where the rows (all three letters long) did not — put a two-letter word in every caps grid. The initial shape (`_INITIAL`, `\w\.`, v1-verbatim and public) is not widened but asked of the COMPOSED word, in `is_initial_shaped` and in `_render`'s two copies; and the single-letter connective tests count composed letters, since a decomposed `й` is two code points (#665, where `É. y É. Smith` and `John й Smith` read differently decomposed). Compose the WHOLE predicate, not one half: `is_initial` is a shape ANDed with the #320 script veto, and composing the shape alone made a decomposed `씨.` an initial, since decomposed hangul is conjoining jamo outside every script range (#665's first version; `in_initialless_script` composes too now). Not covered, deliberately: the marks only some scripts write (Indic vowel signs, Arabic and Hebrew points), declined in decisions/nfd-shapes because widening the class would move readings nobody had measured; rules/H2's Accepted clause records the one consequence it names, that a period-marked abbreviation in Bengali or Devanagari never reads as a title by shape. The guard is `test_a_decomposed_name_renders_as_its_composed_twin` over the CLDR corpus plus `test_a_decomposed_abbreviation_reads_as_its_composed_twin` (rows × three orders × EVERYWHERE); the corpus guard reaches a shape only if some corpus name does, which is how the caps shape was missed until review, so when adding a shape rule add a row there. Four of #652's rows and two of #665's pass with every fix reverted (measured) and are parity pins, and the test compares ambiguity kinds and `initials()` too, since #665's `John й Smith` differed in its report; `_DOTTED` has no observable input and is pinned by a direct call. diff --git a/docs/design/decisions/C12-suffix-word-ending-the-name-before-the-comma.md b/docs/design/decisions/C12-suffix-word-ending-the-name-before-the-comma.md index c20dd487..3933ece1 100644 --- a/docs/design/decisions/C12-suffix-word-ending-the-name-before-the-comma.md +++ b/docs/design/decisions/C12-suffix-word-ending-the-name-before-the-comma.md @@ -16,7 +16,7 @@ Open: #774 — a roman numeral S10 reads is lost into the family before dotted p - 2026-10-18 (Derek), [#656](https://github.com/derek73/python-nameparser/issues/656) (its declined `mbchb` superseded by #797, decisions/suffix-acronym-collisions) — A SUFFIX WORD ENDING A WHOLE NAME BEFORE THE COMMA IS EVIDENCE THAT THE COMMA IS A SUFFIX COMMA, AND THE PART AFTER IT DECIDES ONLY WHETHER IT HOLDS A NAME. Until this rule only the part after the comma decided (C1), so an unlisted credential there that no shape rule admitted reversed the whole name into the listing form: `John Smith II, AD` read given 'AD', family 'John Smith'. Derek's ground, in the #720/#656 session: a generation is part of the name, more so than a credential, and is commonly written before the comma that starts the credentials (`John J Ingram III, MD`), and `II` counts as such a word though it is also initial-shaped. The evidence is the suffix WORD (`vocab:suffix-word`: every member of `suffix_words`, 170 on 2026-10-18 by `len(Lexicon.default().suffix_words)` — the generations, `esq`, `ret`, `vet`, `2`, and the honorifics, `rinpoche`, जी, 先生, 씨 among them), not an acronym, because a listing with a long surname puts an acronym there as readily (`von bergen wessels MD, pennie`, a corpus.jsonl row, keeps its given name). DECLINED: the generation alone deciding. CLDR's own renderings hold the counter-case, a two-word surname with a generation before a listing comma: `Excm. Sr. Lloret Palol II, Josep Antoni Carles Joan` and `Excma. Dra. Van Dalen Hoogeveen II, Maria da Graça César Martín`, both keep the listing form and their given names before and after this rule (their title and family fields disagree with CLDR for other reasons, the title run left in the family). So the part after the comma still decides, by a narrower question than C1's: a word in it written as a name (`_vocab.written_as_a_name`, unlisted, no period) or an initial keeps the listing form. That needs the name's case contrast, which is C6's test (`_comma._name_contrast`) reused rather than restated; without it a given name written in capitals is indistinguishable from `AD`, so a one-case name and a caseless script decline (the Arabic, Gujarati, Kannada and Malayalam CLDR renderings of this shape stay as they read). - THE NUMERALS, found by the sweep before review. The evidence is reached by vocabulary AND by shape: `ii`–`iv`, `v` and `i` carry the suffix-word tag, but `VI` to `X` are numerals by shape alone, so a tag-only test read `John Smith III, AD` and left `John Smith VI, AD` in the listing form, #610's class. The numeral path asks `_vocab.is_trailing_numeral_suffix`, the trailing fork's own predicate, behind a C-level `ROMAN.match` filter in `decide` so the listing form pays no frame. Both paths then count the words IN FRONT of the generation as C3 does (`_whole_name(seg0[:-1])`): `Smith VI` is one word and a numeral, as `Smith III` is, where counting the whole part had made the shape-only numeral its own name word. And a one-letter numeral is evidence only where S10 reads it as the generation, a name word to spare in front of it (`name_word_count`, a title being none): `Smith V, AD` and `St. Berg V, AD` read the V as a name word and keep the listing form. + THE NUMERALS, found by the sweep before review. The evidence is reached by vocabulary AND by shape: `ii`–`iv`, `v` and `i` carry the suffix-word tag, but `VI` to `X` are numerals by shape alone, so a tag-only test read `John Smith III, AD` and left `John Smith VI, AD` in the listing form, #610's class. The numeral path asks `_vocab.is_trailing_numeral_suffix` (`_vocab.reads_as_generation` since #752), the trailing fork's own predicate, behind a C-level `ROMAN.match` filter in `decide` so the listing form pays no frame. Both paths then count the words IN FRONT of the generation as C3 does (`_whole_name(seg0[:-1])`): `Smith VI` is one word and a numeral, as `Smith III` is, where counting the whole part had made the shape-only numeral its own name word. And a one-letter numeral is evidence only where S10 reads it as the generation, a name word to spare in front of it (`name_word_count`, a title being none): `Smith V, AD` and `St. Berg V, AD` read the V as a name word and keep the listing form. THE HONORIFICS, found by the design-docs review: the tag reaches every suffix word, so `Tenzin Gyatso Rinpoche, AD` reads suffix 'Rinpoche, AD' where the parent read given 'AD'. Kept as stated rather than narrowed to generations: an honorific written after the name stands in the same place a generation does, and no vocabulary set separates the generations from the rest. Dotted paired initials were the review's other find: `Smith Jones Jr., J.A.` read suffix 'Jr., J.A.', since the veto asked only the `initial` tag; `is_paired_initials` joins it, as C5 keeps such a pair the given name. DECLINED: adding `mbchb` to SUFFIX_ACRONYMS, approved in the session before the rule was measured. `MBChB` ends in a capital, so it is not written as a name and the rule reads `van den Berg jnr., MBChB PhD` without it; a vocabulary member would also reach `Smith, MBChB`, which nothing here measured. Measured 2026-10-18 on this change against its parent: five corpus names move, every one a CLDR rendering of the shape and each to the agreeing reading on family, given, suffix and title where CLDR states them (`Sig. Fabio Massimo Carlo Alberto Martini II, AD` in both orders, `prof. dr. Ada Cornelia César Martín von Brühl Jr, univ. bacc`, `Prof. dr. Cornelia Margarita Sofia Aletta van den Berg jnr., MBChB PhD`, `Skeggjason Hr. Geir Jón Magnús Brynjar yngri, þingmaður`); the last still disagrees on its given names, CLDR's `Geir Jón Magnús Brynjar` against `Hr. Geir Jón Magnús Brynjar`, a title inside a surname-first name, which is not this rule. No other corpus name moves, read off `snapshot.py`'s report. The issue's comment counted 25 disagreements of the shape on 2026-10-13; the `Jr, MP` renderings among them read correctly on the parent already, MP being listed. diff --git a/docs/design/decisions/C9-one-character-suffix-word-inside-a-comma-part.md b/docs/design/decisions/C9-one-character-suffix-word-inside-a-comma-part.md index c5b4f596..94aa3594 100644 --- a/docs/design/decisions/C9-one-character-suffix-word-inside-a-comma-part.md +++ b/docs/design/decisions/C9-one-character-suffix-word-inside-a-comma-part.md @@ -16,7 +16,7 @@ Open: [#819](https://github.com/derek73/python-nameparser/issues/819) (whether r - 2026-08-25 #430 / #432 (its accepted `Smith, John PSM I I.` reading superseded by #602, decisions/S8) — one boundary, fixed as one: inside a comma segment a ONE-CHARACTER suffix word — the only kind a reader could take for an initial — is read by WHAT STANDS BEFORE IT. Behind another suffix it describes that suffix and continues the run, period included — `Smith, PSM I` and `Smith, PSM I.` are suffix 'PSM I'/'PSM I.' where they read given 'PSM' plus suffix 'I', an initial in that slot being no shape anyone writes. Behind a name word the period is decisive, marking an abbreviation and so name material — `Smith, John V.` is middle 'V.' where it read suffix 'V.', which is what `Smith, John B.` always did; the only thing that made V differ is that V is also suffix vocabulary. The bare form is the boundary and does not move: `Smith, John V` stays suffix 'V' (v1 parity, #144), which is why the test is the PERIOD and not the numeral. SCOPE, stated as the code reads rather than as the intent was argued: the class is one-character SUFFIX VOCABULARY, not "a roman numeral". It sweeps in `2` (`Smith, John 2.` is middle '2.'), misses `X` and `IX` (roman, but not vocabulary — `Smith, PSM X` is unchanged), and never touched `II`/`III`/`IV` at all, which are two characters and so were never initial-shaped and never vetoed. Anyone adding a one-character suffix in a script that has initials extends it. And the strict knob is consulted: under `lenient_comma_suffixes=False` the initial veto stands and the run ends at the numeral, which the first draft silently overrode by reading no policy at all. #430's cause was rules/S2's initial veto in `is_suffix_piece`, which keeps a numeral out of a suffix reading — right after a name, wrong inside a credential run — so the segment did not look like a run at all and fell to the given-and-suffix walk. Two wrong answers came out of that one cause, and a fix verified on `Smith, PSM I` alone would have left the other: where the run's first word is TITLES vocabulary too (`Smith, MD I`, and `Smith, Jr. I` through the period-abbreviation inference) the leading-title peel took it instead, giving title 'MD' plus given 'I'. THREE readers, one reading, which is the part worth copying. The gate that says a segment is a credential run, assign's router that assigns its roles, and group's one-entry join that renders it must agree PIECE FOR PIECE, and #429 shipped the inverse of its own fix by deriving that agreement twice. `segment_suffix_reading` returns the per-piece verdicts and all three read it (mechanisms.md#ONE-PREDICATE-PER-QUESTION); the intermediate state where only the gate and the router had been converted rendered `Smith, PSM I` as 'PSM, I', the render splitting a run whose roles were already right — the same tell as #429, one predicate along. TWO readers since 2026-09-06: #436/#437 took the render join out of group, and `segment_suffix_reading`'s remaining readers are assign's gate and assign's router, one call site between them. The predicate is not moved — one call site is no reason to relocate what two sites will contest again — and what is worth carrying away is unchanged, since the reason the three had to agree is the reason the two still do. - Declined, and the v1 suite caught it: asking `is_trailing_numeral_suffix` for #432's half. It answers the period question — `_ROMAN` is anchored and admits no period — but it also refuses a numeral standing behind an initial-shaped piece, which is a NO-COMMA rule and the opposite of this path's parity: `Chang, Andy C I` is first 'Andy', middle 'C', suffix 'I', and asking that predicate here made the numeral a middle. decisions.md's #401/#421 entry records the same fork declining to transfer to this walk under LENIENT. The half that applies is the period alone. + Declined, and the v1 suite caught it: asking `is_trailing_numeral_suffix` (`_vocab.reads_as_generation` since #752) for #432's half. It answers the period question — `_ROMAN` is anchored and admits no period — but it also refuses a numeral standing behind an initial-shaped piece, which is a NO-COMMA rule and the opposite of this path's parity: `Chang, Andy C I` is first 'Andy', middle 'C', suffix 'I', and asking that predicate here made the numeral a middle. decisions.md's #401/#421 entry records the same fork declining to transfer to this walk under LENIENT. The half that applies is the period alone. BLAST RADIUS, over the 1069 corpus names against 56fb505c (PR #443): seven carry a one-character numeral behind a word inside a comma segment, every one bare, and none moves; the only movers are the three names this change added as examples, so the case table is the whole coverage. `Berg, abdul V` cannot reach the walk at all, the bound-given join making `abdul V` one piece, which is why the LENIENT mirror #401/#421 declined does not collide here. Two things the /simplify round settled by measurement rather than argument. The walk asks `is_suffix_piece(pieces[m - 1])` — a VOCABULARY test — where the reading carries the verdict it just recorded, and unifying them looks obviously right until you run it: the walk assigns `PSM` to GIVEN in `Smith, PSM I.`, so a reading-style flag says "behind a name" and sends the numeral to middle, losing the strict parity restored two commits earlier. The two sites ask different questions on purpose, and the vocabulary answer is the load-bearing one. The cost is a reading the walk cannot reach: `Smith, John PSM I I.` moves from middle 'I' + suffix 'PSM, I.' to middle 'I I.' + suffix 'PSM', two consecutive one-character numerals behind a credential — accepted, both readings being defensible for a shape nobody writes, and recorded here rather than chased at the price of two correct pins. The walk is also strict-blind, pre-existing and unchanged: it calls is_suffix_lenient unconditionally, so `Smith, John V` reads suffix 'V' under `lenient_comma_suffixes=False` as under the default. The reading consults the knob and the walk does not, which is now conspicuous where it was merely unexamined. Not this change's to fix. diff --git a/docs/design/decisions/M2-maiden-marker-rule.md b/docs/design/decisions/M2-maiden-marker-rule.md index 0aade2a4..b1ba7947 100644 --- a/docs/design/decisions/M2-maiden-marker-rule.md +++ b/docs/design/decisions/M2-maiden-marker-rule.md @@ -108,7 +108,7 @@ Open: [#820](https://github.com/derek73/python-nameparser/issues/820) — whethe - 2026-09-26 #535 (its release question, its joins, its given-part reach, its family-comma numeral reading `Doe nee Smith V, Jane` and the further splits #601 closed superseded in part by #601, the 2026-10-04 entry, which keeps the first-suffix-word stop there) — THE MAIDEN WALK READS THE TRAILING TITLE CHAIN, AND EVERY STOP ASKS ONE RELEASE QUESTION OF THE SPAN IT GIVES UP (Derek, 2026-09-26). This resolves the NOT TRANSPARENT HERE clause of the 2026-09-19 #533 entry above, which recorded `Jane Doe nee Smith MA Prof.` and `Jane Doe nee Smith Prof. MA` disagreeing and deferred the pair as one decision about what "trailing" means inside a clause. BOTH HALVES WERE TAKEN TOGETHER: a trailing title ends the clause (`Jane Doe nee Smith Prof.` reads title 'Prof.', maiden 'Smith', where 2.3.0 read maiden 'Smith Prof.'), and a credential or numeral standing in front of that title gets the stop it gets with the title absent (`… Smith MA Prof.` gives suffix 'MA', `… Smith V Prof.` suffix 'V'). Only the first half would have left the pair disagreeing in the other direction; H5's transparency is a statement about the peel and the chain read together to their fixed point, so the walk now reads the end of the clause the way assign reads the end of the name — `tail_reading`, the shared predicate mechanisms.md#ONE-PREDICATE-PER-QUESTION names — and over the forms the transparency property test below exercises, the two spellings give one answer. They split where something in or ahead of the clause stands to take the title — for example a particle chain (ahead of the clause or inside it), a bound-given join, a name left with no name word, or, after a family comma, a title word in front of a class member with a title behind it, which the given part's chain stops short of exactly as it does with no marker (`Doe, Jane nee Smith Rev. MA Prof.` reads maiden 'Smith Rev.', title 'Prof.', suffix 'MA', where `… Rev. Prof. MA` reads title 'Rev. Prof.', maiden 'Smith', suffix 'MA'; bare `Doe, Jane Rev. MA Prof.` reads middle 'Rev.') — and the Accepted pairs recorded here and in rules/M2 are examples of that, not a complete list. `tests/v2/test_properties.py::test_a_trailing_title_is_transparent_to_the_maiden_clause` holds that as an invariant over the two inputs, over heads and runs where the credential is given up and nothing ahead would take the title, with its negative control at e0f1a2fa in its docstring. ACCEPTED, THE TRANSPARENCY BOUNDARY: where the clause KEEPS the credential the spellings differ, because a clause is one contiguous run and a title inside the kept text cannot leave without the words behind it — `Doe nee Smith ba Prof.` reads title 'Prof.', maiden 'Smith ba', and `Doe nee Smith Prof. ba` maiden 'Smith Prof. ba'; `abdul nee Smith MA Prof.` reads title 'Prof.', family 'abdul', maiden 'Smith MA', and `abdul nee Smith Prof. MA` given 'abdul', maiden 'Smith Prof. MA' (2.3.0 kept every word in all four). The report differs with them: `Doe, Dr. nee Smith MA` and `… Prof. MA` report `suffix-or-name` on the kept MA, `… MA Prof.` reports nothing, the kept member no longer ENDING the clause, which is what M2's report is asked of. rules/M2 carries the first pair as an Accepted example. ACCEPTED, FURTHER SPLITS, measured 2026-09-26. Where something ahead of the clause would take the title, the spellings differ even with the credential given up in both, because the first-suffix-word stop ends the clause before the title check runs and the title check alone declines: `Jane van der Berg nee Smith PhD Prof.` reads title 'Prof.', maiden 'Smith', suffix 'PhD', and `… Prof. PhD` maiden 'Smith Prof.', suffix 'PhD' — H5's accepted particle-chain boundary inherited, the bare `Jane van der Berg Smith PhD Prof.` / `… Prof. PhD` splitting the same way — and `Berg, abdul nee Smith PhD Prof.` (bound-given join) and `Doe, Dr. nee Smith PhD Prof.` (no name word left) split likewise; the parent d9d80492 and 2.3.0 read all of these as the tree does, so only the claim is new. And a released particle with a title behind it is withdrawn: `Jane Doe nee Smith DO Prof.` reads title 'Prof.', maiden 'Smith DO', where `… Prof. DO` reads title 'Prof.', maiden 'Smith', suffix 'DO' (2.3.0 maiden 'Smith DO Prof.' and 'Smith Prof. DO'); `do` likewise. A particle INSIDE the clause does it from the other side, its chain able to take the credential behind it: `Jane Doe nee Smith do MA Prof.` reads title 'Prof.', maiden 'Smith do MA', and `… do Prof. MA` title 'Prof.', maiden 'Smith do', suffix 'MA' (2.3.0 kept every word in both). rules/M2 carries a pair of each shape as Accepted examples. ACCEPTED, H5'S REACH: `Jane Doe nee Smith King.` reads title 'King.', maiden 'Smith' (2.3.0 maiden 'Smith King.'), H5's accepted `Mary Jane King.` cost now reaching the last word of a birth name as it reaches the last word of a current one; rules/M2 carries it as an Accepted example. READER SCOPE IS UNCHANGED: the chain is consulted only where a trailing rule reads the clause's words (no comma, the part before a suffix comma, the given part after a family comma); before a family comma and in a tail segment the walk reads the peel alone and the clause keeps the title (`Doe nee Smith Prof., Jane` keeps maiden 'Smith Prof.'). THE FIRST-WORD FLOOR MOVED INTO THE CHAIN. `trailing_titles` and `tail_reading` take a `floor` (1 everywhere else), and the walk passes the position just past the marker's first word, so the chain never takes that word: `Jane Doe nee King.` keeps its maiden name (TITLES holds borne surnames and the marker announced one), and `Jane Doe nee Prof. Dr.` keeps 'Prof.' and gives up 'Dr.'. The first draft applied the floor as a clamp on the title stop after the chain had run, and that is measurably wrong rather than merely inelegant: a chain allowed to take the first word has already spliced it out of the count the re-peel reads, so `Jane Doe nee King. ba` kept 'ba' in the clause where `Jane Doe nee Smith ba` gives it up. `test_a_title_first_word_counts_as_a_word` is the invariant (a title-vocabulary first word against an ordinary one, the credential behind it read alike) and its docstring records the clamp's failure count. ONE RELEASE CHECK FOR THREE STOPS: the numeral, credential and title stops each asked `_release_reads_off`, per reader, whether the name the take would leave reads the whole span a stop gives up as titles or suffixes; #601 replaced the release question with the clause-free reading bound as roles. THE JOINS: a released span a P2 chain or the P5 join would absorb was withdrawn, release machinery #601 removed. THREE NUMERAL-STOP READINGS moved with it, each the release invariant at the numeral stop and mostly after a family comma (`Berg, abdul nee Smith V`, `Doe, Jane nee Smith V, PhD`, `Doe, Jane nee V, PhD`), where #601 made the marker an ordinary word. A LINK THE WALK STOPS AT asked the release question too, where the title chain refused a link the exception would have joined (`Jane Doe nee Smith i DO Prof.`), scoped by a 411,936-parse link grid (6,195 fixed, 0 new violations, measured 2026-09-26); release machinery #601 removed. A RELEASED TITLE THAT IS ALSO A PARTICLE was kept after a family comma (`Doe, Jane nee Smith St.`), P6 attaching it to the family; a given-part reach #601 removed. A FOURTH MEASUREMENT, of the M2 invariant over a twelve-head fuzz (107,352 parses) against e0f1a2fa: 588 texts fixed and 12 newly violating, all `Dr. nee Smith i MA|V` behind a title, #548's class. ACCEPTED: `Jane Doe (nee Smith Prof.)` reads title 'Prof.', maiden 'Smith' — bracket content ending in a period is suffix-shaped (S1), so the brackets are dropped and the clause is read bare, outside the reach of the delimiter precedence; rules/M2 carries it as an Accepted example. MEASURED 2026-09-26, the tree against its parent d9d80492, over three generated grids, and these are dated snapshots. Recipe: grid A is each head in {`Doe, Jane`, `Doe, J.`, `Doe, Prof.`, `Berg, abdul`, `Doe, Jane van`, `Jane Doe`, `J. Doe`, `Dr.`, `abdul`, `Jane van der Berg`, `John`}, then ` nee `, then every sequence of one to three words (repetition allowed) from {Smith, Jones, Prof., Dr., Sir, MA, DO, PhD, V, III, i, do, van, Ma, M.A., King., Rev., ba}, then either nothing or `, PhD`, each text as written, upper-cased and lower-cased, duplicates removed — 327,936 texts; grid B, aimed at the bound-given join, is the same construction over heads {`Dr. abdul`, `abdul rahman`, `Dr. abdul rahman`, `Berg, Dr. abdul`, `Berg, abdul rahman`, `abu`, `Berg, abu`, `Mr. abu`, `Dr.`, `Berg, Dr.`, `abdul`, `Berg, abdul`}, words {Smith, Prof., MA, V, do, van, PhD, Ma, ba, III, Jr., M.A.} and tails {nothing, `, PhD`, `, Jr., MD`}, plus these fourteen texts, as written only: `Jane Doe nee Ph. D. Prof.`, `Jane Doe nee Ph. D. Smith Prof.`, `Jane Doe z domu King. ba`, `Jane Doe z domu Smith Prof.`, `Jane Doe nee King. Prof. ba`, `Jane Doe nee King. Prof.`, `Doe, Jane nee Smith V,`, `Doe, Jane nee Smith V, ` (with the trailing space), `Doe, Jane, nee Smith V`, `Doe, Jane nee Smith V Prof., PhD`, `Doe, Jane nee Smith Prof. V, PhD`, `Doe, Jane nee Smith MA, PhD`, `Doe, Jane nee Smith V (Jr.)`, `Doe, Jane nee Smith V "Bo"` — 173,057 texts. The invariant tested is M2's: in a parse with a non-empty maiden field, every token written after the ` nee ` marker is roled maiden, title or suffix (so the two `z domu` texts are parsed and not checked). Grid C, a four-word probe of the title chain after a family comma, is each head in {`Doe, Jane`, `Jane Doe`, `Doe, J.`, `John Smith`, `Doe, Jane Mary`}, then ` nee `, then every sequence of four words (repetition allowed) from {Smith, Jones, Prof., Dr., King., MA, PhD, V, Jr., ba, do}, then either nothing or `, PhD`, as written only — 146,410 texts. Per grid (newly violating at the tree, fixed, violating at both), measured on the tree as narrowed: A 0, 2,012 and 17,854; B 0, 1,222 and 11,149, one of those last the check itself counting the quoted nickname in `Doe, Jane nee Smith V "Bo"`; C 0, 1,368 and 15,534. In grids A and B, every other remaining violation has the clause ending immediately before a word of the unambiguous suffix vocabulary written the way the plain suffix-word stop takes it (PhD, III, M.A., Jr., and a lower-case i or v; never a bare capital, which reads as an initial), which is where that stop ends a clause without asking a release question. DEFERRED: the walk's plain suffix-word stop — the one ending the clause at the first suffix word, as distinct from the trailing numeral, credential and title stops — asks no release question at all, and where the words behind it cannot read as post-nominals they land in a name part: on degenerate heads the stop word itself does (`Dr. nee Smith PhD Prof.` reads family 'PhD', `Doe nee Smith Jr. Prof., Jane` family 'Doe Prof.'), and with an ordinary head the words behind it do (`Doe, Jane nee Smith PhD Smith` reads middle 'Smith'). Unchanged here, and what "the clause keeps it" should mean where the kept words would follow a credential the clause itself ended at is its own question; rules/M2 carries `Doe nee Smith Jr. Prof., Jane` as an Accepted example meanwhile. The trailing numeral's stop has the same gap before a family comma, where it is made over the peel alone with no release question: `Doe nee Smith V, Jane` reads family 'Doe V', maiden 'Smith' (so did 2.2.0, 2.3.0 and the parent d9d80492; 2.0.0 and 2.1.0 read maiden 'Smith V'), and rules/M2 carries it beside the other. Open: #548. COST, `sys.setprofile` call events in one parse after a warm-up on CPython 3.11.16, d9d80492 → tree (PR #550): `John Smith`, `Jane Doe Prof.` and `John Smith MA` unchanged, `Jane Doe nee Smith` 244 → 246, `… MA` 377 → 388, `… Prof.` 276 → 372, the one shape that now runs the chain and a release check. - 2026-09-27 (Derek), #544 (its boundary across the clause superseded in part by #601) — S2'S COMPANY CLAUSE DOES NOT REACH ACROSS A CLAUSE, recorded as an Accepted boundary rather than repaired. #601 superseded it: a marker behind a credential is an ordinary word now, and the credential's run takes the clause in (rules/M2's Accepted `Jane Doe Jr. nee Smith`). Inside the clause the company reads as it does anywhere: `Jane Doe nee Smith PhD MEng` ends the clause at 'PhD' and reads suffix 'PhD MEng', maiden 'Smith'. -- 2026-10-04 (Derek), #601 (its `Jane Doe, PhD née Smith` reading superseded in part by #603, decisions/C13: suffix 'PhD née Smith') — A MARKER COUNTS ONLY BEHIND A SURNAME, AND IT TAKES THE WORDS AFTER IT UP TO THE TRAILING RUN THE NAME WOULD END WITH IF THE CLAUSE WERE NOT WRITTEN. This SUPERSEDES the release machinery of the #533 and #535 entries above (the release question, the per-stop checks, the anchored-head boundary of the #544 entry) and the given-part and suffix-comma reach the 2026-07-03 rule had; the #399, #420, #434 and delimiter entries stand. The trigger was #548, `Dr. nee Smith PhD Prof.`, where 2.3.0 read family 'PhD', maiden 'Smith': a clause whose head was nothing but a title, and a release check asking whether each word it gave up could still read as a post-nominal. #548 was closed into this issue and #602 (S2's run) because both halves of its answer were rules rather than repairs. THE HEAD RULE (Derek): a marker is one only in the name before any comma or the surname part before a family comma, behind at least one name word past the leading title run, and not straight behind an unambiguous suffix word or a connective. Anywhere else it is an ordinary word — in the given part after a family comma (`Doe, Jane nee Smith` reads middle 'nee Smith', 1.4.0's reading) and in a part after a suffix comma (`Smith, John, PhD née Jones` reads suffix 'PhD née Jones', also 1.4.0's). Derek's framing was that a maiden marker always follows a family name, and that a marker following something the parser has decided is a title, suffix, particle or connective is not a marker. THE PARTICLE HALF WAS DROPPED BY MEASUREMENT: prototype variant 1 refused a marker behind particle vocabulary and so declined `Anh Do née Tran` and `Mai Le née Nguyen`, Do, Le and Van being particles that are also the commonest Vietnamese surnames; a particle counts as the surname (`Jane van nee Smith` reads family 'van', maiden 'Smith'). THE BOUND (Derek): the clause takes its words up to the run of post-nominals and titles that the trailing rules read at the end of the name with the clause removed, and the words it gives up take the roles that reading gives them — the TRAILING rule's reading of the clause-free name, not the full parse of it, so a head the full parse would read differently (H1's lone title, a P5 join) cannot change where the clause ends. The first word after the marker is always taken, the marker having announced a name — a lone roman numeral included, so `Jane Smith née V` reads maiden 'V' where the clause-free `Jane Smith V` reads suffix 'V', accepted rather than given a numeral exception (rules/M2's Accepted block) — unless it is an unambiguous suffix word, where the marker declines (`Jane Smith nee PhD` reads family 'nee', suffix 'PhD', as before). Before a family comma the clause keeps the 2026-07-03 stop at the first suffix word, since no trailing rule reads that part's end. VARIANTS, prototyped 2026-10-03 behind an environment switch on one tree (the numbers are in #601's comments): v2/v3 grew the clause until the existing release check accepted it and reached 0 invariant violations but inherited the check's caution about particles (`Anh Do geb. de la Vega PhD Prof.` kept 'PhD Prof.' in the maiden name); v4 read the run once over the name as written without binding it, and is the recorded negative control in tests/v2/test_properties.py (350 of the clause grid's 4,482 parses fail under it, measured 2026-10-04); v5/v6 bound that reading as roles but still counted the clause's words among the words to spare, so `John nee Smith ba` read 'ba' as a credential where `John ba` reads a name; v7, the clause-free reading bound as roles, was adopted. Against #535's grids v7 left 0 violations of the M2 invariant where the tree then had 15,586 (grid A, 327,936 texts) and 8,713 (grid B, 173,057), measured 2026-10-03 with an invariant check written to #535's recipe. WHAT THE PROTOTYPE MISSED, found implementing it: the numeral fork. A trailing roman numeral is a suffix only by its shape against the word before it, and v7 read the clause's last word without that test, so `John née Jones Smith VI` kept 'VI' in the maiden name; the take now asks `is_trailing_numeral_suffix` of the last piece, and cases.py's `a_numeral_the_clause_free_name_reads_ends_the_clause` pins it. And the head's title run is measured over the words before the marker, a lone title excepted: measured over the whole segment, `Lord Chancellor née Jones` counted 'Chancellor' as a title and refused the marker. TRIAGE, 2026-10-04: the prototype's 68 failing case rows (40 facade failures among them) were re-pinned as `fix(#601)` with their group named in the note, bar the head-rule bug above; tests/v2/test_parser.py's corpus-append invariant became two-sided (appending ` née X` moves no other field, OR the take records why it declined), with a named set for the suffixes that themselves hold a marker. DEGENERATE INPUT, NO READING DESIGNED FOR (Derek, 2026-10-04: garbage in, garbage out). Two names that read worse by eye moved, and they are recorded here so the move is not mistaken for a regression, not because either reading is wanted: `Berg, Jane van der nee Smith DO` reads middle 'van der nee Smith', family 'Berg' where 2.3.0 read family 'van der Berg', maiden 'Smith DO' — the marker in a given part being a word, the particles no longer trail the given name (P6); and `Jane Doe, PhD née Smith` reads given 'PhD', middle 'née Smith', family 'Jane Doe' where 2.3.0 read given 'Jane', family 'Doe', suffix 'PhD', maiden 'Smith', which 2.3.0 reached only through the clause taking the marker (the comma part opening with a credential is #603's shape). Measured 2026-10-04 against 2.3.0. COST: the take is one function and no release helper survives; the #533 review's comma-head grids could no longer reach a take at all and were replaced by counting heads (tests/v2/test_properties.py's `_maiden_clause_grid`, 4,482 parses). +- 2026-10-04 (Derek), #601 (its `Jane Doe, PhD née Smith` reading superseded in part by #603, decisions/C13: suffix 'PhD née Smith') — A MARKER COUNTS ONLY BEHIND A SURNAME, AND IT TAKES THE WORDS AFTER IT UP TO THE TRAILING RUN THE NAME WOULD END WITH IF THE CLAUSE WERE NOT WRITTEN. This SUPERSEDES the release machinery of the #533 and #535 entries above (the release question, the per-stop checks, the anchored-head boundary of the #544 entry) and the given-part and suffix-comma reach the 2026-07-03 rule had; the #399, #420, #434 and delimiter entries stand. The trigger was #548, `Dr. nee Smith PhD Prof.`, where 2.3.0 read family 'PhD', maiden 'Smith': a clause whose head was nothing but a title, and a release check asking whether each word it gave up could still read as a post-nominal. #548 was closed into this issue and #602 (S2's run) because both halves of its answer were rules rather than repairs. THE HEAD RULE (Derek): a marker is one only in the name before any comma or the surname part before a family comma, behind at least one name word past the leading title run, and not straight behind an unambiguous suffix word or a connective. Anywhere else it is an ordinary word — in the given part after a family comma (`Doe, Jane nee Smith` reads middle 'nee Smith', 1.4.0's reading) and in a part after a suffix comma (`Smith, John, PhD née Jones` reads suffix 'PhD née Jones', also 1.4.0's). Derek's framing was that a maiden marker always follows a family name, and that a marker following something the parser has decided is a title, suffix, particle or connective is not a marker. THE PARTICLE HALF WAS DROPPED BY MEASUREMENT: prototype variant 1 refused a marker behind particle vocabulary and so declined `Anh Do née Tran` and `Mai Le née Nguyen`, Do, Le and Van being particles that are also the commonest Vietnamese surnames; a particle counts as the surname (`Jane van nee Smith` reads family 'van', maiden 'Smith'). THE BOUND (Derek): the clause takes its words up to the run of post-nominals and titles that the trailing rules read at the end of the name with the clause removed, and the words it gives up take the roles that reading gives them — the TRAILING rule's reading of the clause-free name, not the full parse of it, so a head the full parse would read differently (H1's lone title, a P5 join) cannot change where the clause ends. The first word after the marker is always taken, the marker having announced a name — a lone roman numeral included, so `Jane Smith née V` reads maiden 'V' where the clause-free `Jane Smith V` reads suffix 'V', accepted rather than given a numeral exception (rules/M2's Accepted block) — unless it is an unambiguous suffix word, where the marker declines (`Jane Smith nee PhD` reads family 'nee', suffix 'PhD', as before). Before a family comma the clause keeps the 2026-07-03 stop at the first suffix word, since no trailing rule reads that part's end. VARIANTS, prototyped 2026-10-03 behind an environment switch on one tree (the numbers are in #601's comments): v2/v3 grew the clause until the existing release check accepted it and reached 0 invariant violations but inherited the check's caution about particles (`Anh Do geb. de la Vega PhD Prof.` kept 'PhD Prof.' in the maiden name); v4 read the run once over the name as written without binding it, and is the recorded negative control in tests/v2/test_properties.py (350 of the clause grid's 4,482 parses fail under it, measured 2026-10-04); v5/v6 bound that reading as roles but still counted the clause's words among the words to spare, so `John nee Smith ba` read 'ba' as a credential where `John ba` reads a name; v7, the clause-free reading bound as roles, was adopted. Against #535's grids v7 left 0 violations of the M2 invariant where the tree then had 15,586 (grid A, 327,936 texts) and 8,713 (grid B, 173,057), measured 2026-10-03 with an invariant check written to #535's recipe. WHAT THE PROTOTYPE MISSED, found implementing it: the numeral fork. A trailing roman numeral is a suffix only by its shape against the word before it, and v7 read the clause's last word without that test, so `John née Jones Smith VI` kept 'VI' in the maiden name; the take now asks `is_trailing_numeral_suffix` (`_vocab.reads_as_generation` since #752) of the last piece, and cases.py's `a_numeral_the_clause_free_name_reads_ends_the_clause` pins it. And the head's title run is measured over the words before the marker, a lone title excepted: measured over the whole segment, `Lord Chancellor née Jones` counted 'Chancellor' as a title and refused the marker. TRIAGE, 2026-10-04: the prototype's 68 failing case rows (40 facade failures among them) were re-pinned as `fix(#601)` with their group named in the note, bar the head-rule bug above; tests/v2/test_parser.py's corpus-append invariant became two-sided (appending ` née X` moves no other field, OR the take records why it declined), with a named set for the suffixes that themselves hold a marker. DEGENERATE INPUT, NO READING DESIGNED FOR (Derek, 2026-10-04: garbage in, garbage out). Two names that read worse by eye moved, and they are recorded here so the move is not mistaken for a regression, not because either reading is wanted: `Berg, Jane van der nee Smith DO` reads middle 'van der nee Smith', family 'Berg' where 2.3.0 read family 'van der Berg', maiden 'Smith DO' — the marker in a given part being a word, the particles no longer trail the given name (P6); and `Jane Doe, PhD née Smith` reads given 'PhD', middle 'née Smith', family 'Jane Doe' where 2.3.0 read given 'Jane', family 'Doe', suffix 'PhD', maiden 'Smith', which 2.3.0 reached only through the clause taking the marker (the comma part opening with a credential is #603's shape). Measured 2026-10-04 against 2.3.0. COST: the take is one function and no release helper survives; the #533 review's comma-head grids could no longer reach a take at all and were replaced by counting heads (tests/v2/test_properties.py's `_maiden_clause_grid`, 4,482 parses). - 2026-10-04 #610 — THE MAIDEN TAKE'S CANDIDATE WORDS ARE ONE FUNCTION BESIDE THE PEEL, AND SHARING IT FIXED A GAP THE HAND COPY HAD. `_maiden_take` chose the clause words to read its clause-free view over with a filter of its own (`_clause_tail_word` plus a numeral clause), a hand copy of the peel's and the chain's admissions whose contract — a superset of what `tail_reading` takes — nothing stated or tested. It is now `_pieces.trailing_candidates`, written beside `peel_trailing`, and tests/v2/pipeline/test_pieces.py pins the superset at the take itself, on the unjoined pieces the take is handed, over a generated grid (37,044 clauses through `group`). The grid found the copy wrong twice. The numeral fork reads a shape-only roman numeral (`VI`, in no wordlist) as the WALK's last piece, and two things leave it last without standing at the end of the name: a period-marked title behind it, which the chain takes first, and the merged `Ph. D.`, which group flags and the walk never holds. The copy admitted the numeral only as the clause's last word, so `Jane Doe nee Smith VI Prof.` kept maiden 'Smith VI' where `John Smith VI Prof.` reads suffix 'VI', and `Jane Doe nee Smith VI Ph. D.` the same way. Both now end the clause at the numeral, as rules/M2's statement already said they should. Recorded negative control: the loop as #601 shipped it misses 432 of that grid's clauses, every one this shape. The test's first version read plain names through all of `group`, on pieces the joins had already merged, which review caught: there the old loop missed 288 names, and a first draft of the fix, covering the title half only, still missed 111 — the `Ph. D.` half, which that grid is what found. Against master 8a8459a8 over the 95,813-text equivalence grid the change moved 10 readings, all of this class. The same issue lifted P6's particle-tail walk into `_pieces.particle_tail`, read off roles, so post_rules' attachment and assign's given-part credential run (#602) ask one walk; that half moved none of the 95,813 readings. - 2026-10-18, #731 — THE CANDIDATE TEST IS A SUPERSET WITH ONE STATED EXCEPTION. SUPERSEDED by 2026-10-18 #731 reverted (PR #733) — the entry below; the full text is in git and in PR #732. - 2026-10-18 (Derek), #731 reverted — the #731 entry above is superseded: `trailing_candidates` takes no case fact again and is a superset of what `tail_reading` takes, without the exception. Its statement keeps naming S10. History: decisions/T4's revert entry. diff --git a/docs/design/decisions/P5-bound-given-names.md b/docs/design/decisions/P5-bound-given-names.md index 85e4ebe8..cea6965b 100644 --- a/docs/design/decisions/P5-bound-given-names.md +++ b/docs/design/decisions/P5-bound-given-names.md @@ -48,7 +48,7 @@ Open: none in this file. - 2026-08-22 #401 / #421 — the reserve and the join ask the same question, and it is the one assign answers. Two defects, one cause: P5 says the join "needs a name word to spare" and joins "the word after it", and neither site knew what a name word was. The reserve counted the pieces that were not suffix pieces, and `_is_suffix_piece` vetoes on the initial tag — right for assign's middle-initial question, wrong for the reserve's: a bare `V` carries both `vocab:suffix` and `initial`, so `abdul Smith V` counted three name words, joined, and assign then read the `V` as the suffix it always was — given 'abdul Smith', family '' (#401; shipped since 1.x — 1.4.0 read first 'abdul Smith', last 'V' — every bound word, only the single-letter numerals V, I, X). The join never asked at all: with a word to spare it took a suffix piece as "the word after" — `abdul Jr Smith Berg` read given 'abdul Jr' (#421; 1.4.0 parity) — and where the suffix was the split credential, merge() unioned the piece tags, the joined piece became a suffix piece, and assign sent the bound word to the suffix field: `abdul Ph. D. Smith Berg` read suffix 'abdul Ph. D.', a 2.0 regression (v1's fix_phd extracted the credential before parsing, so position never mattered). - Fixed with one predicate, `reads_as_suffix`: a suffix piece, or the trailing roman numeral assign's fork takes — and that fork is now `_vocab.is_trailing_numeral_suffix`, lifted out of assign so the two stages share its definition instead of the reserve carrying a copy (the `_ROMAN` pattern moved with it; the regex-sync pin follows it). The reserve counts a piece out when it reads as a suffix, and the join declines the piece it would take when it does — the same decline it already made for a marker. The mirror is assign's fork condition for condition: the numeral is the LAST piece and not the first name piece — not "the piece before it is not a title", which the first draft wrote and the design-docs review falsified, `jr` being title vocabulary as well as a suffix, so `abdul Smith Jr V` lost its family exactly as before — and the piece before it is read as the join would leave it, the joined pair's first token being the bound word (`abdul J. V`, the design-docs review's other find, otherwise suppressed the fork for the reserve alone and read given 'abdul J.', family ''; the reserve now declines, and assign, seeing the unjoined pieces, reads given 'abdul', middle 'J.', family 'V', exactly as it reads `John J. V`). Two more the code review found: assign drops the group-flagged credential pieces (the Ph. D. merge) from its walk at any position before the trailing peel, so a numeral can be last in that walk without being the last piece — `abdul Smith V Ph. D.` lost its family until "last" and "the piece before" were read over the pieces assign keeps — pinned at the piece level only, because a trailing split credential is the shape the 1.4.0 ledger's `[[never]]` entry refuses to explain any diff on, and v1's reading of that name carried the #401 family loss; the example stayed out of the rules corpus rather than the never-entry being narrowed for one bound-word spelling; and the post-comma walk has no numeral fork at all (a trailing numeral there is read by a lenient last-of-two rule, or as a middle initial with a third part), so a mirror applied under LENIENT declined `Berg, abdul V` and `Berg, abdul V, jr` against every baseline with nothing classifying it — the numeral branch is scoped to STRICT, and after a family comma the reserve counts suffix pieces alone, as it always did. So `abdul Smith V Jr` still joins and assign reads family 'V', and `abdul V Smith` still joins an initial, which is a name word. What is NOT mirrored: S2's other positional fork, the bare ambiguous acronym. Where the join leaves two pieces no family is lost — `abdul Smith Ma` reads family 'Ma' where `John Smith Ma` reads suffix 'Ma', assign keeping the acronym as the family of a two-piece name — but with a suffix word between, the join leaves three, assign peels the acronym and then the suffix, and the family is gone: `abdul Smith Jr Ma` reads given 'abdul Smith', family '', suffix 'Jr, Ma' where `John Smith Jr Ma` keeps family 'Smith'. Pre-existing and byte-identical before this fix, and the whole of what remains of #401's shape on the main walk: a review sweep of the STRICT path found 132 family-losing shapes, every one ending in a bare ambiguous acronym. Left for #425 rather than widened into here, and framed there as sharing assign's peel over a post-join VIEW rather than mirroring a second fork by hand: with a view the count is the view's length, the remap and the first-piece carve-out fall away, and the other two sites that stop at a suffix with the same initial veto — P2's chain and M2's walk, which take a trailing numeral the same way (#424) — would ask the question once. The walk was also computed once rather than per piece after review measured the per-piece rebuild as quadratic (13x growth for 4x the input against the benchmark's 6x bound, on a shape no benchmark unit led with); `bound_given` is that unit now. Readings: `abdul Smith V` → given 'abdul', family 'Smith', suffix 'V', the SUFFIX_OR_NAME report unchanged; `abdul Jr Smith Berg` → given 'abdul', middle 'Jr Smith', where `John Jr Smith Berg` puts it; `abdul Ph. D. Smith Berg` → given 'abdul', middle 'Smith', suffix 'Ph. D.' — the SUFFIX field is 1.4.0's again, the given is not: v1's fix_phd extracted the credential before parsing and the join then took 'Smith', reading given 'abdul Smith' — which the issue reports correctly, and then mislabels its own proposed reading as "1.4.0's reading restored". The decline is chosen over reaching past the credential for the next name word, because P5 joins "the word after it": a join that skipped a suffix would read a shape no one writes as if the suffix were not there, and the marker decline already sets the precedent that the join takes the word after it or nothing. A judgement, recorded as one; `Berg, abdul Jr Smith` → suffix 'Jr'; and `Sir abdul V`, the shape #369's licence had widened #401 onto, reads suffix 'V'. #369's licence-local suffix clause is retired, subsumed by the general decline. `bound_given_join_sees_only_the_surviving_name` re-pins to given 'abd', middle 'Jr Smith', as the #418 entry anticipated. No differential corpus name has either shape — measured: the gate was byte-identical at all three baselines before the rules.md examples went in — so the examples carry the shapes into the rules corpus, where they move against every baseline and are classified there. The 1.4.0 `fix(#401)` rule lists three fields where the 2.x rules list two, because 1.4.0 did not even read the V as a suffix; a draft with two fields handed the name to the fields-only suffix-routing catch-all, whose capped printout hid it, and this entry briefly called #401 a 2.0 regression on that inert measurement — the design-docs review caught it. EXPLAINED NOTHING is not byte-identical. + Fixed with one predicate, `reads_as_suffix`: a suffix piece, or the trailing roman numeral assign's fork takes — and that fork is now `_vocab.is_trailing_numeral_suffix` (`_vocab.reads_as_generation` since #752), lifted out of assign so the two stages share its definition instead of the reserve carrying a copy (the `_ROMAN` pattern moved with it; the regex-sync pin follows it). The reserve counts a piece out when it reads as a suffix, and the join declines the piece it would take when it does — the same decline it already made for a marker. The mirror is assign's fork condition for condition: the numeral is the LAST piece and not the first name piece — not "the piece before it is not a title", which the first draft wrote and the design-docs review falsified, `jr` being title vocabulary as well as a suffix, so `abdul Smith Jr V` lost its family exactly as before — and the piece before it is read as the join would leave it, the joined pair's first token being the bound word (`abdul J. V`, the design-docs review's other find, otherwise suppressed the fork for the reserve alone and read given 'abdul J.', family ''; the reserve now declines, and assign, seeing the unjoined pieces, reads given 'abdul', middle 'J.', family 'V', exactly as it reads `John J. V`). Two more the code review found: assign drops the group-flagged credential pieces (the Ph. D. merge) from its walk at any position before the trailing peel, so a numeral can be last in that walk without being the last piece — `abdul Smith V Ph. D.` lost its family until "last" and "the piece before" were read over the pieces assign keeps — pinned at the piece level only, because a trailing split credential is the shape the 1.4.0 ledger's `[[never]]` entry refuses to explain any diff on, and v1's reading of that name carried the #401 family loss; the example stayed out of the rules corpus rather than the never-entry being narrowed for one bound-word spelling; and the post-comma walk has no numeral fork at all (a trailing numeral there is read by a lenient last-of-two rule, or as a middle initial with a third part), so a mirror applied under LENIENT declined `Berg, abdul V` and `Berg, abdul V, jr` against every baseline with nothing classifying it — the numeral branch is scoped to STRICT, and after a family comma the reserve counts suffix pieces alone, as it always did. So `abdul Smith V Jr` still joins and assign reads family 'V', and `abdul V Smith` still joins an initial, which is a name word. What is NOT mirrored: S2's other positional fork, the bare ambiguous acronym. Where the join leaves two pieces no family is lost — `abdul Smith Ma` reads family 'Ma' where `John Smith Ma` reads suffix 'Ma', assign keeping the acronym as the family of a two-piece name — but with a suffix word between, the join leaves three, assign peels the acronym and then the suffix, and the family is gone: `abdul Smith Jr Ma` reads given 'abdul Smith', family '', suffix 'Jr, Ma' where `John Smith Jr Ma` keeps family 'Smith'. Pre-existing and byte-identical before this fix, and the whole of what remains of #401's shape on the main walk: a review sweep of the STRICT path found 132 family-losing shapes, every one ending in a bare ambiguous acronym. Left for #425 rather than widened into here, and framed there as sharing assign's peel over a post-join VIEW rather than mirroring a second fork by hand: with a view the count is the view's length, the remap and the first-piece carve-out fall away, and the other two sites that stop at a suffix with the same initial veto — P2's chain and M2's walk, which take a trailing numeral the same way (#424) — would ask the question once. The walk was also computed once rather than per piece after review measured the per-piece rebuild as quadratic (13x growth for 4x the input against the benchmark's 6x bound, on a shape no benchmark unit led with); `bound_given` is that unit now. Readings: `abdul Smith V` → given 'abdul', family 'Smith', suffix 'V', the SUFFIX_OR_NAME report unchanged; `abdul Jr Smith Berg` → given 'abdul', middle 'Jr Smith', where `John Jr Smith Berg` puts it; `abdul Ph. D. Smith Berg` → given 'abdul', middle 'Smith', suffix 'Ph. D.' — the SUFFIX field is 1.4.0's again, the given is not: v1's fix_phd extracted the credential before parsing and the join then took 'Smith', reading given 'abdul Smith' — which the issue reports correctly, and then mislabels its own proposed reading as "1.4.0's reading restored". The decline is chosen over reaching past the credential for the next name word, because P5 joins "the word after it": a join that skipped a suffix would read a shape no one writes as if the suffix were not there, and the marker decline already sets the precedent that the join takes the word after it or nothing. A judgement, recorded as one; `Berg, abdul Jr Smith` → suffix 'Jr'; and `Sir abdul V`, the shape #369's licence had widened #401 onto, reads suffix 'V'. #369's licence-local suffix clause is retired, subsumed by the general decline. `bound_given_join_sees_only_the_surviving_name` re-pins to given 'abd', middle 'Jr Smith', as the #418 entry anticipated. No differential corpus name has either shape — measured: the gate was byte-identical at all three baselines before the rules.md examples went in — so the examples carry the shapes into the rules corpus, where they move against every baseline and are classified there. The 1.4.0 `fix(#401)` rule lists three fields where the 2.x rules list two, because 1.4.0 did not even read the V as a suffix; a draft with two fields handed the name to the fields-only suffix-routing catch-all, whose capped printout hid it, and this entry briefly called #401 a 2.0 regression on that inert measurement — the design-docs review caught it. EXPLAINED NOTHING is not byte-identical. - 2026-08-22 #425 (the view the reserve compared, and the #369 precedent applied to `Sheik abdul Ma`, superseded by #614) — the reserve runs assign's peel over the view the join would leave. #423 had the reserve re-derive assign's trailing peel by hand — the flagged-credential drop, the walk's start, the numeral fork's conditions, the preceding piece remapped to "as the join would leave it" — with each seam held by a pin, and the bare-ambiguous-acronym fork not mirrored at all: `abdul Smith Jr Ma` read given 'abdul Smith', family '', suffix 'Jr, Ma' where `John Smith Jr Ma` keeps family 'Smith', the whole residue of #401's shape on the main walk (every family-losing bound-word shape in the sweep below contains one). The altitude review's framing is taken: the peel is one function, `_peel_trailing` over the list `_peel_walk` builds (the walk's input defined once too, after the /simplify pass found it spelled at three sites), housed in group beside the piece predicates because assign imports those and group cannot import assign; assign calls it and reports what it returns (the numeral fork's piece, the acronym picks, in the order it always reported them — byte-identical over the suite and the three gates); and the reserve builds the VIEW merge() would leave — the pair one piece, its tags the union — and calls the same function. `BoundJoin`'s "the value IS the threshold" trick survives restated as post-join name pieces: STRICT 2, LENIENT 1. The remap and the `k == first_name_k` carve-out are gone, the joined piece being two tokens and never a suffix piece. - 2026-10-07 #614 — THE RESERVE READS NO VIEW: S2's trailing run is read once, after P3's joins and before the bound-given join is tried, and split off, so the words to spare are the name pieces it left, and the join can move no suffix reading because nothing in group reads the run again (rules/P5 amended; decisions/S2, this date). SUPERSEDES the two-view comparison, which survives only where the run is not read there (a run with a name piece inside it), and with it two pinned readings: `abdul J. V` and `Sir abdul J. V` now join, given 'abdul J.' with family 'V' -- unjoined the initial in front suppresses the numeral fork, and that is the one reading taken now, where the joined view had read the V as a suffix and declined; 1.4.0 read the same, 2.0 and 2.1 suffix 'V', 2.2 and 2.3 middle 'J.' -- and `Sheik abdul Ma` reports the pick the read weighed (`suffix-or-name`), superseding the #425 bullet's application of the #369 precedent to the pair's suffix fork: the read decides 'Ma' before the join, so the fork is a fork. #369's own precedent, a bound word that is also a particle read as the bound word with no PARTICLE_OR_GIVEN report, is untouched (`Sheik Abu Bakar` reports nothing). Accepted by Derek as a class with the rest of #614's moves. diff --git a/docs/design/decisions/S10-trailing-roman-numeral.md b/docs/design/decisions/S10-trailing-roman-numeral.md index 3b87409c..7a527c67 100644 --- a/docs/design/decisions/S10-trailing-roman-numeral.md +++ b/docs/design/decisions/S10-trailing-roman-numeral.md @@ -20,7 +20,7 @@ Open: none in this file. THE LOWERCASE HALF. A numeral in lower case in a mixed-case name is not how anyone writes an initial, so it is the generation and reports nothing (Derek). Measured before the change, the reports had no rationale: `John Smith v` and `John Smith i` were silent because the listed letters reach the vocabulary path, `John Smith vi` and `John Smith x` reported because the shape fork reports everything it takes, and the one-case `john smith i` reported because the connective tag (P8, #479) makes a one-case `i` an initial. Now the mixed-case lower case is uniformly silent and a one-case name is counted like capitals, `v` excepted (the ACCEPTED paragraph below). `john smith i` keeps its report, by Derek's call. Declined: dropping the shape fork for single capital letters outright. Prototyped in a scratch edit, it moved 26 readings and cost the real generational names the count keeps: `John W. Ingram V` read middle 'W. Ingram', family 'V', and `John van der Berg V` folded the numeral into the particle chain. A variant that dropped the fork for I and X only, on top of the count, moved four more, among them the v1 test `Larry James Johnson I`. ACCEPTED, and recorded in rules/S10: a one-case lowercase `v` is read as the vocabulary reads it, so `john v` still loses its family and `john smith v` reports nothing, where `JOHN V` and `john x` keep their family and `JOHN SMITH V` reports — the listed lowercase letter is taken as a suffix word before the numeral fork is reached, and closing that means touching the suffix-vocabulary path, which is not this change. Not `i`, though it is listed too: a one-case `i` is P8's connective, which classify reads as an initial, so it reaches the fork and is counted (`john i` → family 'i'; `john smith i` reports). And a one-letter numeral with a credential behind it is not the last word and is read as a name word (`John Smith V PhD` → family V, 1.4.0 parity), where `John Smith III PhD` reads suffix 'III PhD'. - Recompute: the moved names are this commit's diff of tools/differential/snapshot.jsonl (or `uv run python tools/differential/snapshot.py` on the parent tree with this commit's code). The Declined paragraph's figures are a prototype's, over every cases.py row and snapshot name parsed before and after the patch in one process, on master at 958d252e: its 26 replace `_pieces.is_trailing_numeral_suffix` with a wrapper returning False for a text that is one capital `I`, `V` or `X` (periods stripped) and the original's answer otherwise; its "four more" apply that wrapper to `I` and `X` alone on top of the shipped count. Both are dated snapshots by Counting claims; nothing re-runs them. + Recompute: the moved names are this commit's diff of tools/differential/snapshot.jsonl (or `uv run python tools/differential/snapshot.py` on the parent tree with this commit's code). The Declined paragraph's figures are a prototype's, over every cases.py row and snapshot name parsed before and after the patch in one process, on master at 958d252e: its 26 replace `_pieces.is_trailing_numeral_suffix` (`_pieces.reads_as_generation` since #752) with a wrapper returning False for a text that is one capital `I`, `V` or `X` (periods stripped) and the original's answer otherwise; its "four more" apply that wrapper to `I` and `X` alone on top of the shipped count. Both are dated snapshots by Counting claims; nothing re-runs them. - 2026-10-18 (Derek), #718 (its particle-chain report and its stand-down superseded by #718's PR review and #722's last review) — THE DECLINED NUMERAL REPORTS TOO, BOTH DIRECTIONS AS S4'S FORK DOES. The count's first form reported only the generation, so `John V` read family V in silence while `Jack Ma` reports keeping its family; Derek: report both, as `Jack Ma` does. A declined numeral is a pick of the walk, and its branches are taken in three places, so three sites report it (mechanisms.md#AMBIGUITY-AT-THE-DECISION-SITE): assign where it stays a lone name word (`John V` — "read as a family name rather than a generational suffix"), the particle chain where it joins the run (`Freiherr von Richthofen V`), and a maiden clause that keeps it as its last word (`John née Jones Smith V`, `abdul nee Smith V`). Each wording is its own, since the acronym pick's ("both a post-nominal and an ordinary name") misdescribes a numeral. ONE STAND-DOWN, FOUND BY INV10 rather than designed: where classify had already reported a one-case connective `i` as an initial, the numeral's report stood down rather than put a second report on the token, INV10's kind pair being left to mean "two readings" everywhere else; #722's last review (below) replaced the stand-down with one withdrawal. - 2026-10-18, #718 review — THE COUNT IS NOT ASKED IN THE GIVEN PART AFTER A FAMILY COMMA. The code review's grid (family-comma and empty-head shapes behind a title-particle, about 558,000 parses) found 2,880 parses where the first form moved a numeral from the suffix into the given or family name with no report: `Smith, Freiherr von Berg V` read given 'von Berg V', `, Freiherr von V` family 'von V', `Smith, St van V` given 'van V', where `Smith, von V` and `Smith, Freiherr John V` kept suffix V. The cause was not a choice: assign's walk over that part never counts the numeral (rules/S4: the given part's trailing slot "is read without it (S5)"), but the particle chain's bound there asks `tail_reading`, which the count had reached, so the chain took a word assign would have read as the suffix. The bound now reads with the count off (`trailing_run_start`, `count_numeral=False`), and all four read as they did on master (2.2.0 onward, since #424; 1.4.0 read the numeral into the name). The class is the one the AGENTS.md gotcha on group-side predictors of the given part names, met from the other side: the predictor did not change, the peel it asks did. The S4 analogue, `Smith, Freiherr von Berg Ma` → given 'von Berg Ma', is master's reading and is left alone here. diff --git a/docs/design/decisions/S9-unlisted-words-that-join-the-ambiguous-class-by-shape.md b/docs/design/decisions/S9-unlisted-words-that-join-the-ambiguous-class-by-shape.md index 152a6044..0bf14899 100644 --- a/docs/design/decisions/S9-unlisted-words-that-join-the-ambiguous-class-by-shape.md +++ b/docs/design/decisions/S9-unlisted-words-that-join-the-ambiguous-class-by-shape.md @@ -32,7 +32,7 @@ Open: none in this file. - 2026-09-14 (Derek), corrected 2026-09-18 (its default superseded by #564, below; its ELEVEN-list roster by decisions/S6's 2026-09-18 whole-PR review round, which checks every wordlist; the three readings its THE RUN IS THE CAPS CLASS'S ALONE paragraph pinned superseded by #544, decisions/S7) — #516's ALL-CAPS HALF IS A SWITCH TOO, `Policy.unlisted_caps_suffixes`, DEFAULT OFF, and the asymmetry is the decision. An unlisted all-caps word of two or more letters, with no period ANYWHERE (`isalpha()` rejects a period anywhere, not merely an interior one, so a single TRAILING period, H2's own shape, is unaffected either way), in a name `one_case` says is written in more than one case, joins the class by shape (`_vocab.caps_shape_candidate`). UNLISTED MEANS IN NO WORDLIST AT ALL, not merely "no whole-token suffix vocabulary" — found only by measuring after a narrower, tag-based guard shipped and reversed: `titles`, `given_name_titles`, `particles`, `particles_ambiguous`, `conjunctions`, `conjunctions_ambiguous`, `bound_given_names`, `suffix_acronyms`, `suffix_acronyms_ambiguous`, `suffix_words` and `maiden_markers` — ELEVEN lists, checked by direct membership rather than by tag, since `segment` calls the shape test before `classify` has tagged anything. Measured with the narrower guard alone (`John Smith DE`, `John Smith NEE` wrongly moved to suffix), and stated here as a recipe rather than a fixed count, since a raw per-list membership tally and a de-duplicated one (a word appearing in two lists, `abu`/`أبو`/`ابو` across `bound_given_names` and `particles_ambiguous`, is one leak, not two) give different totals: `len([w for w in getattr(lexicon, name) if w.isascii() and len(w) >= 2 and w.isalpha()])` per excluded list, run over the shipped `Lexicon.default()`, gives on the order of dozens for `particles` and `particles_ambiguous` and single digits for `conjunctions` and `bound_given_names` — every one of them a wordlist entry the caps shape test's earlier, narrower guard did not check at all. The one figure that reproduces exactly, because ASCII case is what the test needs and every other maiden marker is a non-Latin or non-ASCII spelling the shape test's own `isupper()`/`isalpha()` gate excludes on its own: FOUR maiden markers are ASCII (`geb`, `geboren`, `geborene`, `nee`), out of `len(lexicon.maiden_markers)` total (17, measured 2026-09-18) — `'NEE'`/`'GEB'` are two of those four, capitalized, which is what the narrower guard let through. ACCEPTED GAP, recorded rather than chased: an all-caps German title spelled with `SS` for `ß` (`GROSSFÜRST`) is unlisted by this same lookup too, for a reason predating this switch entirely — `_normalize` uses `lower()`, never `casefold()`, specifically so it does not corrupt `großfürst` to the misspelling `grossfürst`, and classify's own existing `vocab:title` check has never recognized this spelling either. Left for whoever next generalizes the ß-fold question. THE RUN IS THE CAPS CLASS'S ALONE. The multi-token shape `'LEED AP'` needs a RUN test (`John Smith, LEED AP` → suffix `'LEED AP'`), but the listed and dotted halves are single-token classes by construction — the listed set is inherently bare words and the dotted half never needed more than one token — so testing the run over either of them asks a question this design never posed. An un-narrowed run test measured three reversals before this was caught: `John Smith, Ed Ma` (both listed, Title-case, which #289's OWN lean reads as NAME), `John Smith, ma do` (both listed, lower) and `John Smith, X.Y.Z. A.B.` (both dotted) all wrongly moved to suffix as a run. Narrowed to the caps branch alone; `Smith, LEED AP` still declines and reports, and all three reversals are now pinned identical OFF/ON. - `JACK VI` AND `JOHN SMITH VI` DO NOT MOVE, AND NOT FOR THE REASON A FIRST DRAFT GAVE. `VI` passes the caps shape test and carries both `SHAPE_ACRONYM_TAG` and `vocab:suffix-ambiguous` with the switch on — the predicate reaches it, contrary to what an early write-up of this entry said. What actually holds it unchanged is rules/S2's OWN roman-numeral fork (`_pieces.is_trailing_numeral_suffix`), which claims a bare `'VI'` piece and decrements the peel's walk BEFORE the bare-ambiguous/shape branches beneath it are ever reached — a mechanism this switch does not touch and never held `VI` back through non-membership. + `JACK VI` AND `JOHN SMITH VI` DO NOT MOVE, AND NOT FOR THE REASON A FIRST DRAFT GAVE. `VI` passes the caps shape test and carries both `SHAPE_ACRONYM_TAG` and `vocab:suffix-ambiguous` with the switch on — the predicate reaches it, contrary to what an early write-up of this entry said. What actually holds it unchanged is rules/S2's OWN roman-numeral fork (`_pieces.is_trailing_numeral_suffix`, `_vocab.reads_as_generation` since #752), which claims a bare `'VI'` piece and decrements the peel's walk BEFORE the bare-ambiguous/shape branches beneath it are ever reached — a mechanism this switch does not touch and never held `VI` back through non-membership. THE COST IS A TWO-WORD CONTRAPOSITIVE, not a flat one. Off by default because French and Korean records write the SURNAME in capitals — `Jean DUPONT`, `Minjun KIM` and `Jean Pierre DUPONT` all read family at 1.4.0, at 2.3 and at this default — and `isupper()` is script-agnostic, so the same decline holds in ANY script with a case contrast (`Иван ИВАНОВ`, `Jean ÉCOLE`, measured). Turning it on does not cost every candidate alike: at TWO words there are no words to spare, so `Jean DUPONT` and `Minjun KIM` report the fork and KEEP their family — the class is considered and declined, the mirror of the listed lean's own words-to-spare rule. At THREE words there ARE, and `Jean Pierre DUPONT` gives family `'Pierre'`, suffix `'DUPONT'` — the one genuine cost, and a swallowed family name is the worse of the two failures #vocabulary-collisions C-i already argues from. Default-off is 1.4.0 parity for the whole class regardless; on, nothing but the report changes at two words, and only the three-word shape actually moves. Off, NOTHING is reported either. - OUT OF SCOPE, each its own issue or entry: no tokenizer split (`J.R.R.`, `A.B.` stay single tokens); the trailing-period-only shape is NOT the signal (`John Smith Xyz.` → family, and AGENTS.md's trailing-abbreviation paragraph stands for bare words and sentence-final stops); no new `AmbiguityKind`; no locale packs; #492 stays deferred. NEITHER SWITCH REACHES THE v1 SHIM, and that is a consequence stated rather than repaired: `lenient_comma_suffixes` has no v1 manager either and `_build_snapshot()` simply does not translate it. Switch A is DEFAULT ON, so `HumanName` changes with it and a v1 caller cannot turn it off — the deliberate reading of the bridge, where the facade tracks the parser's defaults and 2.x behavior changes reach it (the #528 precedent). - 2026-09-18 — RULES.MD STATEMENTS NEVER NAME A POLICY FIELD, AND THIS ARC MADE THAT LOAD-BEARING. rules.md's own preamble binds statements to be implementation-free ("no stage names, no function names, no regexes"), and no existing statement names a `Policy` field either — C1 says "strict mode" and leaves `strict-comma-suffixes` to the annotation slot. S2's and S3's switch clauses follow that precedent rather than setting a new one: the prose describes `unlisted_dotted_suffixes` and `unlisted_caps_suffixes` as caller configuration and points at the example lines, and the ANNOTATIONS carry the exact field names (`unlisted_dotted_suffixes-off`, `unlisted_caps_suffixes-on`, registered in `tests/v2/rules_doc.py`'s `POLICIES` beside `strict-comma-suffixes`). Recorded here rather than as a silent edit to the preamble or to two rules' prose: if a future design wants a field named directly in a statement, that is a preamble amendment for someone to propose, not a convention this entry changes on its own authority. diff --git a/docs/design/decisions/vocabulary-exclusions.md b/docs/design/decisions/vocabulary-exclusions.md index 0dcf449d..c45ca159 100644 --- a/docs/design/decisions/vocabulary-exclusions.md +++ b/docs/design/decisions/vocabulary-exclusions.md @@ -30,7 +30,7 @@ Excluded (SUFFIX_ACRONYMS — esq, removed 2026-09-08, #316/#489 bundle): Excluded (SUFFIX_ACRONYMS — ph, removed 2026-09-13, #459/#521): -- ph is OUT of SUFFIX_ACRONYMS and must not be put back by a sweep that finds "Ph. D." parsing and assumes the acronym set is what carries it. The split spelling is merged by group's own rule (the `PH`/`D` regexes of `_pipeline/_vocab.py`, v1's fix_phd; see #phd-merge, whose "merged back by vocabulary" means that regex stage and not a lexicon lookup), which never consults acronym membership, and the single-token spellings `phd`/`Ph.D.` are the `phd` entry's. The fragment arrived in the 2019-12-11 bulk Wikipedia post-nominal import (af5bdab, #93) and was never reviewed. It did two things. One was to let the merged `Ph. D.` token pass the acronym test on its first piece, which nothing read until #459's all-caps repair did — with `ph` in the set that repair read `john smith ph. d.` as `John Smith PH. D.` on the default path, not only under force, and that is what forced the removal. The other, found by the docs review of the fix-up (2026-09-22) and not at the removal, was to read a bare trailing `Ph.` — no `D.` behind it — as a credential: `John Smith Ph.` gave family Smith, suffix Ph. on every release from 1.4.0 through 2.3.0 (wheels measured), and out of the set it falls to the positional read, middle Smith, family Ph., suffix empty; `Smith, John Ph.` moves the same way, to middle Ph. The cost is that CLASS and not one spelling, measured 2026-09-22 on HEAD and on the five wheels with `Lexicon.default().add(suffix_acronyms={"ph"})` as the comparator: the acronym test strips periods, so the undotted `John Smith Ph`/`PH`/`ph` move identically; inside a credential run `Ph.` no longer joins it, so `John Smith Ph. MD` reads family Ph., suffix MD, and `John Smith MD Ph.` reads middle `Smith MD`, family Ph., the real credential leaving the suffix field with the fragment; `John Smith Ph. Jr.` keeps only the Jr.; and the comma form `Smith, Ph.` reads title Ph. where 2.2.0 and 2.3.0 read suffix. `John Smith P.H.` does not move, the by-shape class reading it either way. Every shape in the class is a bare `ph` with no `D.` behind it, which nobody writes — the same criterion as esq's, asked of the machinery rather than a surname: the entry described the merge, not a word. Classified a behavior change, a 2.x parity break, on all five ledgers (`change(suffix-acronym-collisions) ph leaves the acronym set`), with the cases.py row `removed_ph_fragment_leaves_a_bare_trailing_ph_a_name_word` written to carry the plain shape; the ledger rule is literal to that one name, so a future row on any other shape of the class arrives unexplained at every baseline and is classified by widening that rule, not filed as a regression. No corpus name had any shape of the class, which is why the removal measured as 0 role movers at review time (2026-09-11, 1143 names), the corpus-blindness AGENTS.md warns of. Under `capitalized()` the 13 `PH. D.` shapes the repair had moved under `force=True` go back to `Ph. D.`; `john smith phd` and `ph.d.` still give `Ph.D.`. +- ph is OUT of SUFFIX_ACRONYMS and must not be put back by a sweep that finds "Ph. D." parsing and assumes the acronym set is what carries it. The split spelling is merged by group's own rule (`split_pair` in `_pipeline/_vocab.py`, which asks two patterns, v1's fix_phd; see #phd-merge, whose "merged back by vocabulary" means that regex stage and not a lexicon lookup), which never consults acronym membership, and the single-token spellings `phd`/`Ph.D.` are the `phd` entry's. The fragment arrived in the 2019-12-11 bulk Wikipedia post-nominal import (af5bdab, #93) and was never reviewed. It did two things. One was to let the merged `Ph. D.` token pass the acronym test on its first piece, which nothing read until #459's all-caps repair did — with `ph` in the set that repair read `john smith ph. d.` as `John Smith PH. D.` on the default path, not only under force, and that is what forced the removal. The other, found by the docs review of the fix-up (2026-09-22) and not at the removal, was to read a bare trailing `Ph.` — no `D.` behind it — as a credential: `John Smith Ph.` gave family Smith, suffix Ph. on every release from 1.4.0 through 2.3.0 (wheels measured), and out of the set it falls to the positional read, middle Smith, family Ph., suffix empty; `Smith, John Ph.` moves the same way, to middle Ph. The cost is that CLASS and not one spelling, measured 2026-09-22 on HEAD and on the five wheels with `Lexicon.default().add(suffix_acronyms={"ph"})` as the comparator: the acronym test strips periods, so the undotted `John Smith Ph`/`PH`/`ph` move identically; inside a credential run `Ph.` no longer joins it, so `John Smith Ph. MD` reads family Ph., suffix MD, and `John Smith MD Ph.` reads middle `Smith MD`, family Ph., the real credential leaving the suffix field with the fragment; `John Smith Ph. Jr.` keeps only the Jr.; and the comma form `Smith, Ph.` reads title Ph. where 2.2.0 and 2.3.0 read suffix. `John Smith P.H.` does not move, the by-shape class reading it either way. Every shape in the class is a bare `ph` with no `D.` behind it, which nobody writes — the same criterion as esq's, asked of the machinery rather than a surname: the entry described the merge, not a word. Classified a behavior change, a 2.x parity break, on all five ledgers (`change(suffix-acronym-collisions) ph leaves the acronym set`), with the cases.py row `removed_ph_fragment_leaves_a_bare_trailing_ph_a_name_word` written to carry the plain shape; the ledger rule is literal to that one name, so a future row on any other shape of the class arrives unexplained at every baseline and is classified by widening that rule, not filed as a regression. No corpus name had any shape of the class, which is why the removal measured as 0 role movers at review time (2026-09-11, 1143 names), the corpus-blindness AGENTS.md warns of. Under `capitalized()` the 13 `PH. D.` shapes the repair had moved under `force=True` go back to `Ph. D.`; `john smith phd` and `ph.d.` still give `Ph.D.`. Excluded (BOUND_GIVEN_NAMES): diff --git a/docs/design/decisions/vocabulary-model.md b/docs/design/decisions/vocabulary-model.md index 1b5129b4..88f344bb 100644 --- a/docs/design/decisions/vocabulary-model.md +++ b/docs/design/decisions/vocabulary-model.md @@ -21,7 +21,7 @@ The vocabulary was a set of base sets plus marker subsets (`particles` and `part - `cased` replaces `capitalization_exceptions` and, with `same_as`, a canonical spelling, and a reading's `native` spelling is for surfacing; `acronym` is a reading property replacing the acronym/word split, one suffix kind per word; `expands` records the meaning an entry was added for; `addresses` takes `family` and `given`, `full` and `any` being refused until a later phase — 2026-10-15/16; 2026-10-18 #737. - The vocabulary is per-word TOML in `nameparser/vocab/.toml`, one file per BCP 47 tag plus `und` and `mul`, at `schema = 1` (a compiler refuses a newer one), every reading a table and no `language` key; `tools/vocab/build.py` compiles it into a committed `nameparser/config/_compiled.py`, the parser never reads the TOML, and parsing it at import was declined — 2026-10-18 #737. - The marker sets are explicit properties on readings (`may_be_initial`, `attached`, `addresses`, `after_given`, `cased`), and `after_given` stands on a salutation reading only (decisions/H8) — 2026-10-18 #737; 2026-10-24 #835. -- The two markers that record a clash with a name are derived from name readings: an acronym suffix reading beside a `given` or `family` reading is an ambiguous acronym, a particle reading beside a `given` reading a particle that may be given, wherever in the files each reading sits. The build refuses `may_be_name` and `may_be_given = true`, naming the reading to write; a particle whose word has no `given` reading still states `may_be_given = false`, never defaulted; a name reading may stand without a count, with a `note` saying why; and a count kept as evidence against marking such a word goes in the clashing reading's `note` (`ld`). `may_be_initial` is not derived — 2026-10-25 #752. +- The two markers that record a clash with a name are derived from name readings: an acronym suffix reading beside a `given` or `family` reading is an ambiguous acronym, a particle reading beside a `given` reading a particle that may be given, wherever in the files each reading sits. The build refuses `may_be_name` and `may_be_given = true`, naming the reading to write; a particle whose word has no `given` reading still states `may_be_given = false`, never defaulted; a name reading may stand without a count, with a `note` saying why; and a count kept as evidence against marking such a word goes in the clashing reading's `note` (`ld`). `may_be_initial` is not derived — 2026-10-25 #752, A CLASH WITH A NAME IS A NAME READING. - Exclusions are data: an `excluded` key refuses a kind, a category or a behavioral property, in the file of the language whose reading it refuses, and the build refuses the excluded thing in any file, quoting the reason — 2026-10-18 #737. - A per-word fact that moves into the TOML is replaced in decisions/ by a query that finds it (decisions/README.md, edit kind 4) — 2026-10-18 #737. - The script is read from the word, not the file: a script subtag is only for a difference the text cannot show — 2026-10-18 #737. @@ -181,3 +181,10 @@ Open: #752 (phase 2, derived ambiguity; its sub-phases 2a to 2m, each its own PR - **`may_be_initial` stays a property.** Its clash is with the initial, a shape every cased single letter has, and which letters a tradition writes both ways is a judgment about the letter, not a name anyone bears. - **Measured (2026-10-25, against 632685c8):** `tools/differential/snapshot.py` matches; `cldr_agreement.py` matches every record; `snapshot.py --local` for ctgov-officials, fec-individuals and tse-donors, recorded from the base with `--tree`, matches; `local_agreement.py psc-individuals` matches the record. Negative control: dropping `jd`'s given reading takes it out of `SUFFIX_ACRONYMS_AMBIGUOUS`, and dropping `van`'s, with `may_be_given = false` written, puts it in `NON_GIVEN_NAME_PARTICLES` (tests/v2/test_vocab.py). - **Recompute** rather than quoting: `uv run python tools/vocab/query.py --kind given --count`; the markers' sizes, `uv run python -c "from nameparser.config import suffixes, particles; print(len(suffixes.SUFFIX_ACRONYMS_AMBIGUOUS), len(particles.PARTICLES - particles.NON_GIVEN_NAME_PARTICLES))"`. + +- **2026-10-25 #752 — one predicate per question (neutral).** Three questions that #752's later changes widen had their answers copied between sites, and each now has a shared predicate in `_pipeline/_vocab.py` for the widening to land in (mechanisms.md#ONE-PREDICATE-PER-QUESTION). Two are whole: every site asks them. The numeral's is not yet: the bullet below names the sites found still recognizing a numeral on their own. No reading moves. + - **`split_pair(first, second)`**, the split credential (`Ph.` `D.`, decisions/phd-merge): group's merge, `_comma._pieces` and `is_wholly_suffix` each held their own copy of the two-pattern test. The test runs per adjacent pair, so each caller asks `SPLIT_PAIR_OPENS`, the predicate's own first-word pattern, C-level before the call, and a name holding no `Ph.` pays no frame. The split-credential change widens both together (item 6 of the 2026-10-23 #752 entry above). + - **`reads_as_generation(text, preceding, one_case)`**, the numeral: `is_trailing_numeral_suffix` renamed, the predicate every numeral rule is to ask (item 8 of the 2026-10-23 #752 entry above, #784, #774, #819). Today it is the fork's SHAPE half alone. rules/C12's evidence asks a broader question beside its call: whether the word ending the part before the comma is any suffix word, by the suffix-word tag (`John Smith Jr., AD`, `John Smith Esq., AD`), or reads as a generation by the predicate (`John Smith VI, AD`); the tag stays at the call site because `Jr.` and `Esq.` are no numerals. The listed numerals belong in the predicate too, or each numeral rule tests the vocabulary itself again, the hand copy #656's first C12 draft was (it asked the suffix-word tag and missed `VI`). The shape test already takes the listed `ii` to `iv`, so reading the vocabulary inside it adds only the one-letter `i` and `v`, and that moves readings, which this PR may not: answered yes on a grid of 3,927 readings (17 heads, 11 numerals, 7 tails, three orders; the second review, 2026-10-25 on 30dc7aac), `i` and `v` moved 375 readings at the peel's fork, 348 of them in their fields (`John J. V` from middle 'J.', family 'V' to family 'J.', suffix 'V', with a new report), and 144 reports at `reads_as_a_reported_numeral` with no field moving (`John J. V MD` gains a `suffix-or-name`), the ones those sites meet carrying the `initial` tag (`I`, `V`, and their one-case spellings; a lower-case `v` in a name of mixed case carries none, and the peel takes it as a suffix word). Adding `ii` to `iv` alone moved nothing. That half is left to the change that has every numeral rule ask the predicate. Until it lands, these sites recognize a numeral on their own, each keeping the answer it had (found 2026-10-25 by two reviews and a grep of `_pipeline/` for the numeral pattern, the suffix-word tag and the suffix predicates; a list found by search, not a proof that no other site exists): the predicate's own missing `i` and `v`, above; rules/S8's run start asks the vocabulary alone (`John Smith III MD` → suffix 'III MD', `John Smith VI MD` → family 'VI', #784); rules/C1's listing form asks the suffix-word tag (`John Smith III, M.J.` keeps suffix 'III' where `John Smith VI, M.J.` and `John Smith V, M.J.` read family 'John Smith VI' and 'John Smith V', #774); the peel takes `ii` to `iv` through `is_suffix_piece` before its fork asks the predicate; rules/C9's walk over the given part after a family comma asks `is_suffix_lenient`, the vocabulary alone (`Smith, John V` → suffix 'V', `Smith, John VI` and `Smith, John X` → middle, where `John Smith VI` reads suffix 'VI'); and after a suffix word in that part a listed one-letter numeral is taken by its tags (`_pieces._numeral_behind_the_initial_veto`), so `Smith, John PhD V` reads suffix 'PhD V' with no report where `Smith, John PhD VI` reports `suffix-or-name`. + - **`is_cased_paired_initials(text)`**, rules/S3's paired initials in a script with capitals: classify asked the script half inline (#872, decisions/S3). This is the lift the 2026-10-24 #872 entry above names, for classify and the split merge to share. One overlap is left to the split-credential PR to state: item 6 refuses a split credential whose first chunk is a single letter, so for a split `A. B.` that length test answers before this predicate is asked. rules/C5's comma reading keeps the bare `is_paired_initials`, since after a comma a caseless pair reads as a name too (`John Smith, พ.บ.` → given 'พ.บ.'), and moving C5 onto this predicate would change that reading. + - **Measured neutral** against the branch, first on 632685c8 and again on 2dd0414b once phase 2a had merged: a grid of 20,364 texts (22 heads covering no comma, a whole name, one word and nothing before a comma, a nickname-only head, a maiden clause, a family comma with a title-particle head and a quoted clause before the pair; 33 spellings including `MA`, `M.D.`, `X.Y.Z.`, `XYZ`, `Ph. D.` and its case variants, `V`, `VI`, `III`, `XI`, `M.J.`, `K.C.` and `พ.บ.`; each text also in lower and upper case), under all three orders, read identically (fields, ambiguity kinds and details, token roles and tags, `initials()`, `capitalized()`), and the same harness reported 45,871, 18, 3,974 and 2,986 differing readings with `split_pair`, C12's call to the numeral predicate, the cased half and the peel's call to it each knocked out. The old and new predicates gave the same answer on 19,769 direct calls. The committed snapshot, every CLDR, GND, Wikidata and conventions agreement record, psc-individuals' agreement record and the ctgov-officials, fec-individuals and tse-donors local snapshots (each recorded from the base with `--tree`) all match, on both bases. psc-individuals' record was itself stale against 632685c8 (3 regressions, 5 progress, 5 changed before this change), so it was re-recorded from each base first. + - **Frames**, counted with `sys.setprofile` on both trees: `_CALL_BASELINE`'s reference name and `Smith, John` cost what they did (305 and 158). A word matching `SPLIT_PAIR_OPENS` (`Ph`, `ph.`) with a word after it pays one frame at each site that tests it, whether or not a `D.` follows (`John Smith Ph. D.` 233 → 234, `Smith, John Ph. D.` 239 → 241; the review counted `ph ph ph ph ph ph` 282 → 286), so the cost grows only with the number of such words. C12's evidence keeps its tag test ahead of the call, so it costs what it did (`John Smith Jr., AD` 372). That is a rare shape that no guard counts. diff --git a/docs/design/mechanisms.md b/docs/design/mechanisms.md index 246264f0..41d9d0e0 100644 --- a/docs/design/mechanisms.md +++ b/docs/design/mechanisms.md @@ -160,7 +160,7 @@ This catalog converts discovery into a one-time cost — it does not remove disc **How it works.** A hand-written mirror agrees with its original only until one of them moves, and the drift is invisible in both directions: each site keeps passing its own tests while they disagree about an input neither covers. Every instance below was found as a defect before it was found as a pattern. The destination follows the LAYER, not the topic: a predicate over token text goes to `_vocab`, one over pieces and tags to `_pieces`, both leaves the stages sit on. The piece layer got its own module only in #439; until then those predicates collected in `_group`, the one place both stages could reach, so the sharing between group and assign predates the module (decisions/parse-cost measured #439 as a pure relocation). The cost is a second evaluation of the same predicate, measured for #429 at 1.2–2.2% of a family-comma parse and 0% of every other — the right price there over plumbing a state field the two sites would not otherwise share. Folding two copies into one, take from each caller only what genuinely differs between them, and ask whether a parameter one caller had been filling with a guard is a question the other caller must answer too (#617, decisions/P3). The census of which stage calls which predicate, and how each instance was found, was kept here until #759 and is in git history (`git show fe85bb3d:docs/design/mechanisms.md`); the decisions entries carry each instance's argument. -**Lives in.** nameparser/_pipeline/_vocab.py over token text: `is_wholly_suffix` (#319, decisions/W2), `is_trailing_numeral_suffix` (#401/#421, decisions/P5), `maiden_marker_run` (#434, decisions/M2), `delimiter_cores` (#436/#437, decisions/R1), `in_initialless_script` (#322/#323, decisions/cjk-full-stops), `caps_shape_candidate` (#516, decisions/S9). nameparser/_pipeline/_pieces.py over pieces and tags: `is_suffix_piece`, `leading_titles` and `is_leading_title` (#424), `is_title_piece`, `peel_walk`, `trailing_start`, `tail_reading` (the peel-and-chain fixed point that answers where the name pieces end, decisions/H5), `segment_suffix_reading` (#430, decisions/R1), `particle_tail` and `trailing_candidates` (#610, decisions/M2), `join_connectives` (#617, decisions/P3). nameparser/_pipeline/_post_rules.py: `suffix_entries` (#511, decisions/R1), the one instance in a stage rather than a leaf, being a pass over a whole ParseState. tests/v2/test_layering.py holds each module's contract, and a piece predicate growing a dependency on a STAGE shows up there as a widened entry. +**Lives in.** nameparser/_pipeline/_vocab.py over token text: `is_wholly_suffix` (#319, decisions/W2), `reads_as_generation` (#401/#421 as `is_trailing_numeral_suffix`, decisions/P5; since #752 the predicate every numeral rule is to ask, today its shape half alone: the listed one-letter `i` and `v`, S8's run start, C1's listing form, the peel's vocabulary path, C9's family-comma walk and the one-letter numeral behind a suffix word there still recognize a numeral on their own until #784, #774 and #819, decisions/vocabulary-model), `split_pair` (#752, the split credential `Ph.` `D.` that group's merge, `_comma._pieces` and `is_wholly_suffix` each tested with their own copy, each asking `SPLIT_PAIR_OPENS` C-level first since the test runs per adjacent pair, decisions/phd-merge), `is_cased_paired_initials` (#752, rules/S3's paired initials in a script with capitals, which classify asked inline; C5 asks the bare `is_paired_initials`, decisions/S3), `maiden_marker_run` (#434, decisions/M2), `delimiter_cores` (#436/#437, decisions/R1), `in_initialless_script` (#322/#323, decisions/cjk-full-stops), `caps_shape_candidate` (#516, decisions/S9). nameparser/_pipeline/_pieces.py over pieces and tags: `is_suffix_piece`, `leading_titles` and `is_leading_title` (#424), `is_title_piece`, `peel_walk`, `trailing_start`, `tail_reading` (the peel-and-chain fixed point that answers where the name pieces end, decisions/H5), `segment_suffix_reading` (#430, decisions/R1), `particle_tail` and `trailing_candidates` (#610, decisions/M2), `join_connectives` (#617, decisions/P3). nameparser/_pipeline/_post_rules.py: `suffix_entries` (#511, decisions/R1), the one instance in a stage rather than a leaf, being a pass over a whole ParseState. tests/v2/test_layering.py holds each module's contract, and a piece predicate growing a dependency on a STAGE shows up there as a widened entry. **Reach for it when.** You are about to write a condition that mirrors, matches or "does what X does" — or you find a comment saying one does. Grep for the other site's predicate and call it instead. diff --git a/nameparser/_pipeline/_assign.py b/nameparser/_pipeline/_assign.py index b6305eca..91fb4cdc 100644 --- a/nameparser/_pipeline/_assign.py +++ b/nameparser/_pipeline/_assign.py @@ -615,7 +615,7 @@ def _reads_as_a_trailing_suffix(piece: Sequence[int], is_suffix_piece said no, and a lenient single token it refuses is one carrying the `initial` tag, so the shape is already implied. - NOT is_trailing_numeral_suffix, though it answers the period half: + NOT reads_as_generation, though it answers the period half: it also refuses a numeral behind an initial-shaped piece, which is a no-comma rule and the opposite of this path's v1 parity -- 'Chang, Andy C I' is first Andy, middle C, suffix I, and asking diff --git a/nameparser/_pipeline/_classify.py b/nameparser/_pipeline/_classify.py index f4ae3cd6..de1d1f13 100644 --- a/nameparser/_pipeline/_classify.py +++ b/nameparser/_pipeline/_classify.py @@ -68,8 +68,8 @@ from nameparser._types import AmbiguityKind, Role from nameparser._pipeline._vocab import ( BUILTIN_SUFFIX_DELIMITERS, _INITIAL, caps_shape_candidate, composed_len, - delimiter_cores, glued_stem, is_initial, is_one_case, is_paired_initials, - period_joined_vocab, suffix_as_written, tag_marker_runs, + delimiter_cores, glued_stem, is_cased_paired_initials, is_initial, + is_one_case, period_joined_vocab, suffix_as_written, tag_marker_runs, ) from nameparser._pipeline._pieces import own_words @@ -270,11 +270,10 @@ def _tags_for(token: WorkToken, n: str, state: ParseState, # tag alone, as with the switch off, so the trailing slots # read them as a name word and report the fork; the comma # restores the class tag where C5 reads them - # (decisions/S3 (#864)). Cased: a text with no case mapping - # is in a script without capitals ('พ.บ.'). + # (decisions/S3 (#864)); `is_cased_paired_initials` asks the + # script half ('พ.บ.' is no initial) if (state.policy.unlisted_dotted_suffixes - and not (is_paired_initials(token.text) - and token.text.upper() != token.text.lower())): + and not is_cased_paired_initials(token.text)): tags.add(AMBIGUOUS_ACRONYM_TAG) elif (state.policy.unlisted_caps_suffixes is CapsSuffixes.EVERYWHERE and token.role is None diff --git a/nameparser/_pipeline/_comma.py b/nameparser/_pipeline/_comma.py index 7d5147dc..681a9eaa 100644 --- a/nameparser/_pipeline/_comma.py +++ b/nameparser/_pipeline/_comma.py @@ -40,11 +40,11 @@ Structure, WorkToken, copy_with, ) from nameparser._pipeline._vocab import ( - D, PH, ROMAN, caps_shape_candidate, claimed_as_non_name, delimiter_cores, - composed_len, glued_stem, glues, is_caps_word, is_paired_initials, - is_single_letter_numeral, - is_trailing_numeral_suffix, is_wholly_suffix, name_word_count, - surname_unit_facts, unit_ends, written_as_a_name, + ROMAN, SPLIT_PAIR_OPENS, caps_shape_candidate, claimed_as_non_name, + composed_len, delimiter_cores, glued_stem, glues, is_caps_word, + is_paired_initials, is_single_letter_numeral, is_wholly_suffix, + name_word_count, reads_as_generation, split_pair, surname_unit_facts, + unit_ends, written_as_a_name, ) from nameparser._policy import CapsSuffixes from nameparser._types import AmbiguityKind, Role @@ -52,16 +52,17 @@ def _pieces(seg: Sequence[int], tokens: Sequence[WorkToken], ) -> tuple[list[list[int]], list[set[str]]]: - """One piece per token but the Ph./D. pair, which group's first - merge builds -- the part as it stands before any join. Every piece - a fresh list, as `merge_pieces` requires.""" + """One piece per token but a split credential (`split_pair`), + which group's first merge builds (group alone declines a pair at + the head of the string, rules/S2) -- the part as it stands before + any join. Every piece a fresh list, as `merge_pieces` requires.""" pieces: list[list[int]] = [] ptags: list[set[str]] = [] k = 0 while k < len(seg): a = seg[k] - if (k + 1 < len(seg) and PH.fullmatch(tokens[a].text) - and D.fullmatch(tokens[seg[k + 1]].text)): + if (k + 1 < len(seg) and SPLIT_PAIR_OPENS.fullmatch(tokens[a].text) + and split_pair(tokens[a].text, tokens[seg[k + 1]].text)): pieces.append([a, seg[k + 1]]) ptags.append({"suffix"}) k += 2 @@ -233,9 +234,10 @@ def _behind_a_generation(state: ParseState, seg0: Sequence[int], # a numeral by shape alone (VI to X) is the generation the trailing # fork reads, as the listed ones are (#610's class: 'VI' beside 'V' # and 'III'), and not one it reads as an initial's neighbour - # ('John J VI, AD') + # ('John J VI, AD'). C12's evidence is broader than a generation: + # any suffix word ('Jr.', 'Esq.') counts, by its tag if ("vocab:suffix-word" not in tokens[seg0[-1]].tags - and not is_trailing_numeral_suffix( + and not reads_as_generation( last, tokens[seg0[-2]].text, state.one_case)): return None # the generation set aside, the name's own words in front of it are diff --git a/nameparser/_pipeline/_group.py b/nameparser/_pipeline/_group.py index 1a51dae1..4d5a553a 100644 --- a/nameparser/_pipeline/_group.py +++ b/nameparser/_pipeline/_group.py @@ -79,15 +79,15 @@ WorkToken, _AMBIGUOUS_CREDENTIAL_TAGS, copy_with, ) import nameparser._pipeline._comma as _comma -from nameparser._pipeline._vocab import D, PH from nameparser._pipeline._vocab import ( - comma_follows_marker, delimiter_cores, reads_as_a_reported_numeral, + SPLIT_PAIR_OPENS, comma_follows_marker, delimiter_cores, + reads_as_a_reported_numeral, split_pair, ) from nameparser._types import AmbiguityKind, Role -# the credential-pair regexes live in _vocab, whose own -# is_wholly_suffix merges the same pair, as `_comma._pieces` does -- -# and that predicate's callers stay in sync with it: `_comma.decide`'s +# the split credential is `_vocab.split_pair`, which is_wholly_suffix +# asks to merge the same pair, as `_comma._pieces` does -- so that +# predicate's callers read what this stage merges: `_comma.decide`'s # reading of the part after the comma, segment's C2 flag on a tail # part, and script_segment's decline of a wholly-suffix post-comma # run, all of which see the merged reading @@ -641,8 +641,8 @@ def _group_segment(seg: tuple[int, ...], additional: int, a, b = pieces[k], pieces[k + 1] if (not (opens_the_name and k == 0 and a[0] == 0) and len(a) == 1 and len(b) == 1 - and PH.fullmatch(tokens[a[0]].text) - and D.fullmatch(tokens[b[0]].text)): + and SPLIT_PAIR_OPENS.fullmatch(tokens[a[0]].text) + and split_pair(tokens[a[0]].text, tokens[b[0]].text)): merge_pieces(pieces, ptags, k, k + 2, add={"suffix"}) else: k += 1 diff --git a/nameparser/_pipeline/_pieces.py b/nameparser/_pipeline/_pieces.py index 65e20c7c..aa0f49da 100644 --- a/nameparser/_pipeline/_pieces.py +++ b/nameparser/_pipeline/_pieces.py @@ -63,8 +63,8 @@ ) from nameparser._pipeline._vocab import ( _PERIOD_ABBREV, _ROMAN, Lean, ambiguous_lean, in_initialless_script, - is_single_letter_numeral, is_trailing_numeral_suffix, - numeral_settled_by_case, tag_marker_runs, + is_single_letter_numeral, numeral_settled_by_case, + reads_as_generation, tag_marker_runs, ) @@ -1165,7 +1165,7 @@ def peel_trailing(rest: Sequence[int], pieces: Sequence[Sequence[int]], # pick, since "either reading is reported", and no branch below # can take a numeral if (k == len(rest) and k >= 2 and len(piece) == 1 - and is_trailing_numeral_suffix( + and reads_as_generation( text := tokens[piece[0]].text, tokens[pieces[rest[k - 2]][0]].text, one_case)): # rules/S10: "a numeral written in lower case in a name of @@ -1558,7 +1558,7 @@ def trailing_candidates(lo: int, pieces: Sequence[Sequence[int]], or is_suffix_piece(piece, ptags[k], tokens) or (len(piece) == 1 and ( not tokens[piece[0]].tags.isdisjoint(_TRAILING_WORD_TAGS) - or (titles_behind and is_trailing_numeral_suffix( + or (titles_behind and reads_as_generation( tokens[piece[0]].text, tokens[pieces[k - 1][0]].text, None))))): break diff --git a/nameparser/_pipeline/_vocab.py b/nameparser/_pipeline/_vocab.py index fe9981a1..78a61e75 100644 --- a/nameparser/_pipeline/_vocab.py +++ b/nameparser/_pipeline/_vocab.py @@ -73,12 +73,35 @@ # sync by hand. _PERIOD_NOT_AT_END = re.compile(r".*\..+$", re.I) -# The fix_phd credential pair ('Ph.' + 'D.' as adjacent tokens), shared -# by is_wholly_suffix below and group's merge (v1 extracted the -# credential pre-parse; the predicate and the stage must agree on the -# pattern). -PH = re.compile(r"^ph\.?$", re.IGNORECASE) -D = re.compile(r"^d\.?$", re.IGNORECASE) +# The fix_phd credential pair ('Ph.' + 'D.' as adjacent tokens), the +# two halves of `split_pair` below. +_PH = re.compile(r"^ph\.?$", re.IGNORECASE) +_D = re.compile(r"^d\.?$", re.IGNORECASE) +#: `split_pair`'s necessary half on its FIRST word, asked C-level by +#: each caller ahead of the call, since the split test runs per +#: adjacent pair and an ordinary name word must pay no frame for it. +#: The same compiled object the predicate asks, so the two cannot +#: drift while the predicate's first-word test is `_PH`; a change to +#: that test must change this with it, or the callers' pre-test hides +#: the new spellings from the predicate. +SPLIT_PAIR_OPENS = _PH + + +# mechanisms.md#ONE-PREDICATE-PER-QUESTION ("exactly one predicate answers it and every other site calls that one") +# -- the split credential's one test, asked by group's merge, +# `_comma._pieces` and `is_wholly_suffix`, three sites that each kept a +# copy of the pattern pair until #752. decisions/phd-merge, +# decisions/vocabulary-model (#752) +def split_pair(first: str, second: str) -> bool: + """Whether two adjacent words are one credential written split + across a space, and so read as one unit: 'Ph.' then 'D.', with or + without their periods, in any case -- v1's fix_phd, the one split + spelling the parser merges (decisions/phd-merge). Where the merge + may happen (never at the head of the string, rules/S2) is the + caller's question, not this one's. Callers ask + `SPLIT_PAIR_OPENS` of `first` before the call.""" + return (_PH.fullmatch(first) is not None + and _D.fullmatch(second) is not None) # The codepoint table lives in _policy beside Script -- one copy # importable from the pipeline and the locale packs alike; everything @@ -252,7 +275,7 @@ def is_initial_shaped(text: str) -> bool: # v1 regexes.py "roman_numeral", pinned by tests/v2/test_regex_sync.py. _ROMAN = re.compile(r'^(X|IX|IV|V?I{0,3})$', re.I) #: The same pattern, shared rather than copied: `_comma` filters on -#: it before asking `is_trailing_numeral_suffix` (rules/C12). +#: it before asking `reads_as_generation` (rules/C12). ROMAN = _ROMAN @@ -263,17 +286,34 @@ def is_initial_shaped(text: str) -> bool: # is_initial_shaped tests on `preceding`. The words-to-spare count, # rules/S10's Except bullet "A numeral of one letter needs words to # spare", is the callers'. decisions/S10 (#718, #721) -def is_trailing_numeral_suffix(text: str, preceding: str, - one_case: bool | None) -> bool: - """The trailing walk's roman-numeral fork: a FINAL single-token - piece that is a roman numeral - reads as the suffix when the piece before it does not look like - part of an initial run -- for a one-letter numeral a necessary - condition only, the caller's words-to-spare count deciding +# mechanisms.md#ONE-PREDICATE-PER-QUESTION ("exactly one predicate answers it and every other site calls that one") +# -- the numeral's predicate, which every numeral rule is to ask; +# today its shape half, and the sites still recognizing a numeral on +# their own are named in the decisions entry (decisions/vocabulary- +# model, #752) +def reads_as_generation(text: str, preceding: str, + one_case: bool | None) -> bool: + """Whether a word ending a name reads as the generation: the + trailing walk's roman-numeral fork, a FINAL single-token piece + that is a roman numeral reading as the suffix when the piece + before it does not look like part of an initial run -- for a + one-letter numeral a necessary condition only, the caller's words-to-spare count deciding (rules/S10, #718). `preceding` is that piece's first token; the callers establish that `text` is last and that a name piece precedes it. + It answers by SHAPE alone. That covers the listed 'ii' to 'iv', + which the shape test takes as it takes 'VI'; of the listed + numerals only the one-letter 'i' and 'v' would be new here, and + the vocabulary half is the change that has every numeral rule ask + this predicate (#784, #774, #819). It cannot read them yet without + moving readings: answering yes for 'i' and 'v' moves the peel's + fork ('John J. V' family 'V' → suffix 'V') and the report on a + numeral before a credential ('John J. V MD' gains one), since the + ones those sites meet carry the `initial` tag. Other sites still recognize a + numeral without this predicate, each keeping the answer it had + before #752; decisions/vocabulary-model lists those found. + is_initial_shaped, not is_initial: this asks whether the preceding piece looks like part of an initial run, which is a question about layout, and #320 narrowed the tag to initials that can @@ -313,7 +353,7 @@ def reads_as_a_reported_numeral(text: str, preceding: str, one_case: bool | None) -> bool: """Whether a numeral the walk left a name word, with `preceding` in front of it, is one the fork would weigh and so reports.""" - return (is_trailing_numeral_suffix(text, preceding, one_case) + return (reads_as_generation(text, preceding, one_case) and not numeral_settled_by_case(text, one_case)) @@ -483,6 +523,24 @@ def is_paired_initials(text: str) -> bool: return _PAIRED_INITIALS.fullmatch(text) is not None +# rules/S3: "Paired initials are the exception, as C5 defines them, in a +# script that writes capitals" -- the script half of that clause, which +# classify asked inline until #752 (decisions/S3 (#864), +# decisions/vocabulary-model (#752)). C5's comma reading asks the bare +# `is_paired_initials`: after a comma a caseless pair reads as a name +# too ('John Smith, พ.บ.' given 'พ.บ.'), and moving it here would change +# that reading. +def is_cased_paired_initials(text: str) -> bool: + """Paired initials (`is_paired_initials`) in a script that writes + capitals: a text with a case mapping, so 'M.J.' and 'm.j.' but not + Thai 'พ.บ.', which is how that script writes an abbreviation. + The same compiled pattern as `is_paired_initials`, asked here + rather than through it so a dotted token pays one frame, not + two.""" + return (_PAIRED_INITIALS.fullmatch(text) is not None + and text.upper() != text.lower()) + + def suffix_as_written(n: str, text: str, lexicon: Lexicon) -> bool: """Counts as a suffix as written, with NO initial veto (the veto differs by caller): unambiguous suffix vocabulary, or an ambiguous @@ -1178,8 +1236,8 @@ def is_wholly_suffix(texts: Sequence[str], lexicon: Lexicon, the opposite. An adjacent Ph./D. pair counts as ONE unit (v1's fix_phd extracted - the credential pre-parse, so 'Smith, Ph. D.' read as suffix-comma); - keep in sync with group's _PH/_D merge and `_comma._pieces`. + the credential pre-parse, so 'Smith, Ph. D.' read as suffix-comma): + `split_pair`, which group's merge and `_comma._pieces` ask too. `one_case` is ParseState.one_case, and it admits the LEAN: a bare ambiguous acronym written in capitals inside a mixed-case name is @@ -1253,7 +1311,8 @@ def counts_as_suffix(text: str) -> bool: merged = list(texts) k = 0 while k < len(merged) - 1: - if PH.fullmatch(merged[k]) and D.fullmatch(merged[k + 1]): + if (SPLIT_PAIR_OPENS.fullmatch(merged[k]) + and split_pair(merged[k], merged[k + 1])): merged[k:k + 2] = ["phd"] else: k += 1 diff --git a/tests/v2/cases.py b/tests/v2/cases.py index 2e53f6ce..8ee5889b 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -402,8 +402,8 @@ def _check_cjk_shape_purity(self) -> None: "visible. The fragment let the merged 'Ph. D.' token pass " "the acronym test on its first piece, and #459's all-caps " "repair would have read 'john smith ph. d.' as " - "'John Smith PH. D.'; the merge itself keys on the PH/D " - "regexes of _pipeline/_vocab.py and never needed the " + "'John Smith PH. D.'; the merge itself keys on " + "_pipeline/_vocab.py's split_pair and never needed the " "entry. What the entry uniquely carried was a bare " "trailing 'Ph.' -- no 'D.' behind it -- as a credential, " "a spelling nobody writes; out of the set it falls to the " @@ -1481,6 +1481,26 @@ def _check_cjk_shape_purity(self) -> None: "family 'Ma'), which #602 retires; 2.3.0 read this same " "way, and 1.4.0 middle 'Smith PhD', last 'Prof.', suffix " "'Ma'"), + # the split credential's periods are optional, each half on its own + # (v1's fix_phd, `_vocab.split_pair`): every spelling is one suffix + # piece, at the end of a name, behind a family comma and after a + # suffix comma (#752's test review: no row held a periodless pair) + Case("a_split_degree_without_its_periods_merges", + "John Smith Ph D", + {"given": "John", "family": "Smith", "suffix": "Ph D"}, + classification="parity"), + Case("a_split_degree_with_one_period_merges", + "John Smith Ph. D", + {"given": "John", "family": "Smith", "suffix": "Ph. D"}, + classification="parity"), + Case("a_split_degree_without_its_periods_merges_in_the_given_part", + "Smith, John Ph D", + {"given": "John", "family": "Smith", "suffix": "Ph D"}, + classification="parity"), + Case("a_split_degree_in_capitals_merges_after_a_suffix_comma", + "John Smith, PH D", + {"given": "John", "family": "Smith", "suffix": "PH D"}, + classification="parity"), Case("a_split_degree_in_front_is_out_of_the_walk", "John Smith Ph. D. MEng", {"given": "John", "middle": "Smith", "family": "MEng", @@ -4332,7 +4352,7 @@ def _check_cjk_shape_purity(self) -> None: # #516 review round, F5: the spec's stated mechanism for why # 'Jack VI' does not move was wrong -- 'VI' DOES reach the caps # predicate and carries both tags with the switch on. It stays - # unchanged because `_pieces.is_trailing_numeral_suffix` (the + # unchanged because `_vocab.reads_as_generation` (the # roman-numeral fork) claims it downstream, before the lean/count # this switch adds is ever consulted -- unrelated to this switch, # and the reason these controls read identically on or off. diff --git a/tests/v2/pipeline/test_classify.py b/tests/v2/pipeline/test_classify.py index b364c007..f1c519e8 100644 --- a/tests/v2/pipeline/test_classify.py +++ b/tests/v2/pipeline/test_classify.py @@ -17,7 +17,7 @@ from nameparser._pipeline._tokenize import tokenize from nameparser._pipeline._vocab import ( ambiguous_class_candidate, ambiguous_class_member, - caps_shape_candidate, is_paired_initials, surname_unit_facts, + caps_shape_candidate, is_cased_paired_initials, surname_unit_facts, surname_unit_tags, ) from nameparser._policy import Policy @@ -680,7 +680,7 @@ def test_ambiguous_class_candidate_agrees_with_the_tag( # on by design: the candidate (segment's, before the comma has # been read) admits them, and classify holds the class tag back, # which `_comma.decide` restores at the comma C5 reads (#864) - paired = (is_paired_initials(word) and word.upper() != word.lower() + paired = (is_cased_paired_initials(word) and not ambiguous_class_member(word, lex)) assert (candidate and not paired) == ("vocab:suffix-ambiguous" in tags) shape_member = candidate and not ambiguous_class_member(word, lex) diff --git a/tests/v2/pipeline/test_vocab.py b/tests/v2/pipeline/test_vocab.py index b9c3ae2f..8678e1f6 100644 --- a/tests/v2/pipeline/test_vocab.py +++ b/tests/v2/pipeline/test_vocab.py @@ -11,7 +11,8 @@ ambiguous_class_candidate, ambiguous_class_member, ambiguous_lean, caps_shape_candidate, comma_follows_marker, delimiter_cores, glued_stem, effective_script, is_initial, is_initial_shaped, is_one_case, - is_single_letter_numeral, + SPLIT_PAIR_OPENS, is_cased_paired_initials, is_paired_initials, + is_single_letter_numeral, split_pair, is_suffix_lenient, is_suffix_strict, is_title_shaped, is_wholly_suffix, maiden_marker_run, name_word_count, period_joined_vocab, written_as_a_name, resolve_script_set, single_script, @@ -253,6 +254,8 @@ def test_is_wholly_suffix() -> None: # too, but only because _SUFFIX_LEX is synthetic -- under # Lexicon.default(), 'ph' IS in suffix_acronyms.) assert is_wholly_suffix(["Ph.", "D."], lex, pol) + # either period may be left off ('Ph D', 'PH D'), v1's fix_phd + assert is_wholly_suffix(["Ph", "D"], lex, pol) assert not is_wholly_suffix(["Ph."], lex, pol) assert is_wholly_suffix(["Ph.", "D.", "Jr."], lex, pol) # and the pair is found wherever it sits, not only at the head of @@ -873,6 +876,31 @@ def test_a_listed_member_written_in_period_closed_chunks_is_dotted( assert not is_suffix_strict(text, lex), text +def test_split_pair_opens_is_a_necessary_condition_of_split_pair() -> None: + # every caller asks SPLIT_PAIR_OPENS C-level before split_pair, so a + # split_pair widened past it would never be reached for the new + # opener (#752); the periods are optional and the case free + words = ["Ph.", "Ph", "PH.", "ph", "pH", "D.", "D", "d", "Pharm.", + "M.", "MD", "Phd", "Ph..", "", "Smith"] + for first in words: + for second in words: + if split_pair(first, second): + assert SPLIT_PAIR_OPENS.fullmatch(first), (first, second) + assert all(split_pair(a, b) for a in ("Ph.", "Ph", "PH", "ph.") + for b in ("D.", "D", "d")) + assert not split_pair("Ph.", "D..") and not split_pair("Phd", "D.") + + +def test_cased_paired_initials_is_the_bare_shape_in_a_cased_script() -> None: + for text in ("M.J.", "m.j.", "M.J", "Ж.Ш."): + assert is_cased_paired_initials(text), text + # Thai writes an abbreviation as dotted letters and has no case + # (rules/S3); the bare predicate, which C5 asks, still takes it + assert is_paired_initials("พ.บ.") + assert not is_cased_paired_initials("พ.บ.") + assert not is_cased_paired_initials("X.Y.Z.") + + def test_is_single_letter_numeral() -> None: """#544: the one-letter shape C4's run and the anchor both leave out -- one letter, periods allowed, that is a roman numeral.""" diff --git a/tests/v2/test_policy.py b/tests/v2/test_policy.py index 4c0b134c..65f282ab 100644 --- a/tests/v2/test_policy.py +++ b/tests/v2/test_policy.py @@ -890,7 +890,7 @@ def test_unlisted_caps_suffixes_on_reads_an_all_caps_word() -> None: # the boundaries: one case, one letter, a digit, and vocabulary. # 'John Smith X' is NOT the single-letter control -- 'X' is a bare # roman numeral (the trailing numeral fork, - # `_vocab.is_trailing_numeral_suffix`) and reads as suffix + # `_vocab.reads_as_generation`) and reads as suffix # 'X' with the switch either way, unrelated to this one; 'Z' is # not vocabulary at all and stays unaffected by the switch, which # is the actual boundary (a single capital never satisfies the diff --git a/tests/v2/test_properties.py b/tests/v2/test_properties.py index b8372875..0a8dbae0 100644 --- a/tests/v2/test_properties.py +++ b/tests/v2/test_properties.py @@ -3516,9 +3516,9 @@ def test_the_one_case_grid_sees_the_letter_in_front( monkeypatch: pytest.MonkeyPatch) -> None: from nameparser._pipeline import _vocab monkeypatch.setattr( - "nameparser._pipeline._pieces.is_trailing_numeral_suffix", + "nameparser._pipeline._pieces.reads_as_generation", lambda text, preceding, one_case=None: - _vocab.is_trailing_numeral_suffix(text, preceding, None)) + _vocab.reads_as_generation(text, preceding, None)) assert (len(_one_case_disagreements()) == _ONE_CASE_LETTER_IN_FRONT_UNGUARDED) diff --git a/tests/v2/test_regex_sync.py b/tests/v2/test_regex_sync.py index 253a3cc9..f7e2e74d 100644 --- a/tests/v2/test_regex_sync.py +++ b/tests/v2/test_regex_sync.py @@ -129,7 +129,11 @@ def test_dotted_initial_is_the_period_alternative_of_initial() -> None: # is_leading_title's H2 shape test), so it is visible under both # module names and the completeness scan below sees it twice. ("_vocab", "_PERIOD_ABBREV"): "period_abbreviation", - ("_group", "_D"): None, + # `split_pair`'s two halves, the Ph./D. split credential (#752; + # declared under _group until then, which had stopped holding them); + # no config key to mirror + ("_vocab", "_PH"): None, + ("_vocab", "_D"): None, ("_vocab", "_DOTTED"): None, # #544: the chunked sibling of _DOTTED ('M.Eng.'), S3's period gate # for a member with a lower-case tail; no config key to mirror @@ -137,7 +141,6 @@ def test_dotted_initial_is_the_period_alternative_of_initial() -> None: # #563: paired initials ('M.J.'), C5's exception to C4's name-word # count after a comma; no config key to mirror ("_vocab", "_PAIRED_INITIALS"): None, - ("_group", "_PH"): None, ("_vocab", "_ROMAN"): "roman_numeral", # SAME object, imported for the trailing read's weighed-word test # (#620), as _PERIOD_ABBREV is above @@ -251,3 +254,15 @@ def test_comma_char_matches_the_pipeline_comma_set() -> None: from nameparser._pipeline._state import COMMA_CHARS assert set(_render._COMMA_CHAR.pattern.strip("[]")) == set(COMMA_CHARS) + + +def test_every_declared_pattern_still_exists() -> None: + """The converse of the completeness check: an entry naming a + pattern its module no longer holds pins nothing. `_group._PH` and + `_group._D` stood here after both moved to `_vocab`, found when + #752 renamed them there; an entry with no config key was never read + by the parametrized test above, so nothing failed.""" + stale = sorted(where for where in _SOURCES + if not isinstance(getattr(_MODULES[where[0]], where[1], + None), re.Pattern)) + assert not stale, f"_SOURCES entries naming no compiled pattern: {stale}"