diff --git a/AGENTS.md b/AGENTS.md index e0f65aa8..b78166cf 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -306,7 +306,7 @@ Add a dedicated `copy.deepcopy()` round-trip test for it too (see `test_regexes_ **A group-side reading of a family comma's given part that asks `tail_reading` disagrees with the given part's own walk, and the disagreement is accepted.** `tail_reading` reads the part as the comma-less name reads it, plus the bare salutations rules/H8 adds after a family comma, which a predictor of this part asks for with `salutations=True` (#835); assign's walk over the given part after a family comma reads by its own predicates -- a shape-only numeral VI to X stays a name word there (`Smith, John van der Berg VI` reads middle 'van der Berg VI'), V before a third comma part is read strictly (#144), and P6 attaches a period-marked title-particle (`St.`) the comma-less reading takes as a title. So a bound or a decline group computes with `tail_reading` acts where the walk then reads no title: #623's chain bound (`Smith, John Mc Prof. VI`) and #637's join decline (`Smith, abdul Prof. VI` → middle 'Prof. VI', `Smith, abdul St.` → family 'St. Smith'). Both were recorded rather than narrowed (Derek: garbage input; decisions/H5 and #P5, rules/P5's Accepted clause). Before writing another group-side predictor of the given part, sweep those three shapes and decide, rather than rediscovering the gap in review. And a change to the trailing peel ITSELF reaches the given part through those predictors without touching them: #718's words-to-spare count for a one-letter numeral, asked through `trailing_run_start`, made the chain take the V in `Smith, Freiherr von Berg V` (given 'von Berg V', unreported) while assign's walk, which never counts there (S4: "read without it"), read it as the suffix -- found by review on a grid of family-comma and empty-head shapes behind a title-particle; the predictor now passes `count_numeral=False`. When a change moves what the peel takes, put a family comma with a title-particle head in its grid. -**A comparison that runs both sides in one tree reports 0 differences, which is what success looks like.** `tools/differential/compare.py` has been hardened against this — it generates its baseline worker into a temp dir, strips `PYTHONPATH` from the child, and aborts unless both the version AND the resolved path check out on each side; its README's three invocation traps are the analysis behind that design, worth reading before changing the harness. **The exposure is ad-hoc comparisons you write yourself**, where the same collapse has a cause the harness cannot disarm for you: **the shell's working directory persists between tool calls**, so a two-tree comparison written as two `cd`s silently runs both halves in whichever tree it landed in. Pin absolute paths, and assert `nameparser.__file__` on both sides the way `compare.py` does. The general rule outlives any particular trap: before believing a null result, prove the harness can report a difference — a clean run and a broken harness are the same output. The proof has to come from the input class the null result is about: #683 moved `tools/name_usage.py` to a POST and checked it with `shehu`, which is ASCII and so cannot show that the server decodes a Cyrillic body. A misdecoded query still runs and matches nothing, so the Cyrillic zeros it was meant to vouch for proved nothing until `Иван` (17,119 given) went through the same path. The converse trap is a mutation that moves nothing: it shows the code is unexercised by your grids, not that it is dead. #638 was filed calling group's chain re-ask dead on a grid with no title-particle head before a particle run and a dotted acronym; the next review found 400 moves (`Smith, Freiherr van Berg X.Y.Z.`) and the issue closed the same day. Before calling code dead, look for the input that reaches it. A CACHE of readings is the same trap one step removed: a wrong key copies one tree's readings in as another's, and the check then reports nothing. #859 first keyed `snapshot.py --local`'s kept baselines by `git rev-parse HEAD:nameparser`, which answers for the enclosing checkout, so a gitignored copy inside one (`build/lib`) took the checkout's key; two reviewers found it. Key a cached reading by what the import reads -- the files on disk, hashed again after the read -- never by what version control says should be there. +**A comparison that runs both sides in one tree reports 0 differences, which is what success looks like.** `tools/differential/compare.py` has been hardened against this — it generates its baseline worker into a temp dir, strips `PYTHONPATH` from the child, and aborts unless both the version AND the resolved path check out on each side; its README's three invocation traps are the analysis behind that design, worth reading before changing the harness. **The exposure is ad-hoc comparisons you write yourself**, where the same collapse has a cause the harness cannot disarm for you: **the shell's working directory persists between tool calls**, so a two-tree comparison written as two `cd`s silently runs both halves in whichever tree it landed in. Pin absolute paths, and assert `nameparser.__file__` on both sides the way `compare.py` does. The general rule outlives any particular trap: before believing a null result, prove the harness can report a difference — a clean run and a broken harness are the same output. The proof has to come from the input class the null result is about: #683 moved `tools/name_usage.py` to a POST and checked it with `shehu`, which is ASCII and so cannot show that the server decodes a Cyrillic body. A misdecoded query still runs and matches nothing, so the Cyrillic zeros it was meant to vouch for proved nothing until `Иван` (17,119 given) went through the same path. The converse trap is a mutation that moves nothing: it shows the code is unexercised by your grids, not that it is dead. #638 was filed calling group's chain re-ask dead on a grid with no title-particle head before a particle run and a dotted acronym; the next review found 400 moves (`Smith, Freiherr van Berg X.Y.Z.`) and the issue closed the same day. Before calling code dead, look for the input that reaches it. A CACHE of readings is the same trap one step removed: a wrong key copies one tree's readings in as another's, and the check then reports nothing. #859 first keyed `snapshot.py --local`'s kept baselines by `git rev-parse HEAD:nameparser`, which answers for the enclosing checkout, so a gitignored copy inside one (`build/lib`) took the checkout's key; two reviewers found it. Key a cached reading by what the import reads -- the files on disk, hashed again after the read -- never by what version control says should be there. And a switch that changes no file defeats any file key: #752's spike set its options by environment variable, so after the first run every option copied that run's cached local readings until the variables joined the key (2026-10-24). **`SUFFIX_ACRONYMS` and `SUFFIX_WORDS` are DISJOINT by construction (since 2026-10-18; an import-time assert from 2026-09-08) — a word has one suffix kind, and its reading is an acronym or a word (`acronym`), so the vocabulary build refuses a word classified twice; the fix is to choose, not to relax the check** — the two branches normalize differently (the word test strips only edge periods, the acronym test strips all of them), so a word in both is matched by two rules and which one fired is unreadable from outside. `esq` was the single overlap and left `SUFFIX_ACRONYMS`, which is what made the assert possible; the whole algebra that had made the overlap safe, and the criterion that decided which set a word belongs to, are in the `suffix-acronym-collisions` entry of `docs/design/decisions/` if anyone wants them. diff --git a/docs/design/decisions/vocabulary-model.md b/docs/design/decisions/vocabulary-model.md index 8369d462..57c9b63e 100644 --- a/docs/design/decisions/vocabulary-model.md +++ b/docs/design/decisions/vocabulary-model.md @@ -11,10 +11,12 @@ The vocabulary was a set of base sets plus marker subsets (`particles` and `part - `e`, `y` and `i` stay in the default; choosing a language is to settle their one-case fork, which no shipped pack does yet — 2026-10-15/16. - A patronymic field is to come in 3.0, and the particle kinds already draw the `clan`/`patronymic` line for it: a patronymic is followed by a parent's name, a clan prefix by a family name the lineage shares — 2026-10-15/16; 2026-10-19 #749. - Gender is recorded only where a word gives a definitive clue: a feminine form or one half of a fixed pair, never an unmarked noun — 2026-10-15/16; 2026-10-19 #750, title kinds. -- Title-phrase modifiers are to count as titles only next to a head title; the rule is parked, and until it is designed the `modifier` kind is read as every title is (`Vice John Smith` → title 'Vice', while `John Smith Vice` keeps family 'Vice') — 2026-10-15/16; 2026-10-19 #750, title kinds. +- Title-phrase modifiers are to count as titles only next to a head title, at the head of the name only and never in a comma part; until phase 2 implements it the `modifier` kind is read as every title is (`Vice John Smith` → title 'Vice', while `John Smith Vice` keeps family 'Vice') — 2026-10-15/16; 2026-10-19 #750, title kinds; 2026-10-23 #752. - Name readings are hand-declared, and only on a word some language file claims as vocabulary or refuses, so the vocabulary carries no surname list (the segmentation lists, Korean and the ZH pack's, are outside the per-word model) — 2026-10-15/16; 2026-10-19. - Bearer counts are structured provenance (source and count), never used to compute a reading — 2026-10-15/16. -- `Dean Smith` reads given `Dean`. The contested-head gate is to cover only words outside `titles` (`dean`, `justice`, `major`), and a title that is also a given name is decided one word at a time — 2026-10-15/16. +- Phase 2 is to put vocabulary, declared name and shape readings in one pool, derive `may_be_given` and `may_be_name` from declared name readings (a count-free one carrying a `note`, `ld` left ungated, `may_be_initial` kept), and decide every clash by one gate of four steps, floor, form, structure and count, asked at the sites that decide today; the reading not taken is to surface as `Ambiguity.alternative` in 2.x and the reading taken as `Token.reading` in 3.0 — 2026-10-23 #752. +- Phase 2's title batches go in order of lone-head breakage in PSC, `buddha`, `siddha` and `gyani` in the default gate and `guru` left to a `ne` pack — 2026-10-24 #752. +- `Dean Smith` reads given `Dean`. The contested-head gate is to cover the words outside `titles` (`dean`, `justice`, `major`), and a title that is also a given name gets a declared given reading one word at a time, in measured batches — 2026-10-15/16; 2026-10-23 #752. - Order and shape are separate: a language supplies only a default order, and the shapes (where titles, suffixes, particles and commas sit) belong to each order and live in the core; vocabulary readings are to settle a clash and name readings only to keep the gate on — 2026-10-15/16. - `cased` replaces `capitalization_exceptions` and, with `same_as`, a canonical spelling, and a reading's `native` spelling is for surfacing; `acronym` is a reading property replacing the acronym/word split, one suffix kind per word; `expands` records the meaning an entry was added for; `addresses` takes `family` and `given`, `full` and `any` being refused until a later phase — 2026-10-15/16; 2026-10-18 #737. - The vocabulary is per-word TOML in `nameparser/vocab/.toml`, one file per BCP 47 tag plus `und` and `mul`, at `schema = 1` (a compiler refuses a newer one), every reading a table and no `language` key; `tools/vocab/build.py` compiles it into a committed `nameparser/config/_compiled.py`, the parser never reads the TOML, and parsing it at import was declined — 2026-10-18 #737. @@ -32,7 +34,7 @@ The vocabulary was a set of base sets plus marker subsets (`particles` and `part Not standing: the plans the classifications carried out, phase 1's mechanical conversion of the unclassified categories into `und.toml` (2026-10-18 #737) and the shared readings left in place until their categories were classified (2026-10-19 #749), both done by 2026-10-20 #751, the conjunctions; and #747's reason for exempting `und.toml`, its entries keeping the Python lists' order for its vetting comments to point along (2026-10-19 #747), overtaken once it held two entries and no such comment. -Open: #752 (phase 2, derived ambiguity), #789 (whether the dotted `Wm.`, `Chas.`, `Geo.`, `Ma.` and `Jo.` opening the given part read given, outranking rules/H2's shape; phase 2's), #753 (phase 3, language packs), #754 (phase 4, order, shapes and the output model), #766 (whether `bon`, `dí` and three other words stay in the vocabulary). +Open: #752 (phase 2, derived ambiguity; its sub-phases 2a to 2m, each its own PR, are the 2026-10-23 #752 entry's and the RFC's), #789 (sub-phase 2k: the dotted `Wm.`, `Chas.`, `Geo.`, `Ma.` and `Jo.` opening the given part read given over rules/H2's shape, decided 2026-10-23 #752), #753 (phase 3, language packs), #754 (phase 4, order, shapes and the output model), #766 (whether `bon`, `dí` and three other words stay in the vocabulary). ## Log @@ -48,7 +50,7 @@ Open: #752 (phase 2, derived ambiguity), #789 (whether the dotted `Wm.`, `Chas.` 7. `e`/`y`/`i` stay in the default, at least to start; choosing a language settles the one-case fork. 8. A patronymic field belongs in 3.0. 9. Gender is recorded only where a word gives a definitive clue. - 10. Title-phrase modifiers count as titles only next to a head title (parked, not filed). + 10. Title-phrase modifiers count as titles only next to a head title (parked, not filed). (Unparked and narrowed to the head of the name, 2026-10-23 #752, below.) 11. Name readings are hand-declared, and only for words some language file claims as vocabulary, so nobody maintains surname lists. (Widened 2026-10-19, below: or refuses.) 12. Bearer counts are structured provenance (source + count), never used to compute a reading: two data sets deciding one reading would make the decision unclear. A new census flags changed names for review. 13. `Dean Smith` reads given `Dean`, as today. @@ -58,7 +60,7 @@ Open: #752 (phase 2, derived ambiguity), #789 (whether the dotted `Wm.`, `Chas.` 17. `cased` and `same_as` replace a "canonical" spelling, and `cased` replaces `capitalization_exceptions`; both default to the word, and a reading's `native` spelling is for surfacing. 18. `acronym` is a reading property, replacing the `suffix_acronyms`/`suffix_words` split; `expands` records the meaning an entry was added for, not every meaning. 19. `addresses_given` generalizes to `addresses = family | given | full | any`. - 20. A pack's preferred reading can outrank a reading derived by a SHAPE rule (period abbreviation, initials, capitals), not only a vocabulary reading. + 20. A pack's preferred reading can outrank a reading derived by a SHAPE rule (period abbreviation, initials, capitals), not only a vocabulary reading. (Applied in the default too, 2026-10-23 #752, below: a declared `given` reading outranks the leading-abbreviation title's shape.) Phasing, fixed then: (1) the data format and a compiler to today's sets, snapshot-neutral; (2) derived ambiguity, with the gate extended to every reading kind — the step that changes behavior, with its own design; (3) pack semantics, needing 1 and 2; (4) the rules menu, order and shapes, and the output model. @@ -130,3 +132,42 @@ Open: #752 (phase 2, derived ambiguity), #789 (whether the dotted `Wm.`, `Chas.` - **2026-10-24 #835 — `after_given`, the salutations that may end a given part.** A behavioral title property, allowed on a `salutation` reading only (a build check, since an office or rank word is borne as a middle name), compiles into `AFTER_GIVEN_TITLES` and `Lexicon.after_given_titles`, read by rules/H8. A salutation borne as a name refuses it with an exclusion carrying its counts. The rule is keyed on a property rather than on the kind, which would make a kind compile into a behavior where every other behavior is a property; the vetting and the measurement are decisions/H8's. Adding the field changes `Lexicon`'s pickle layout, as 2.1 and 2.3 did. - **Recompute** rather than quoting: `uv run python tools/vocab/query.py --kind salutation`, `--excluded after_given`. + +- **2026-10-23 #752 (Derek) — PHASE 2'S DESIGN IS AGREED: READINGS FROM THREE SOURCES IN ONE POOL, THE TWO NAME MARKERS DERIVED, AND ONE GATE OF FOUR STEPS (FLOOR, FORM, STRUCTURE, COUNT) ASKED AT THE SITES THAT DECIDE TODAY.** Derek agreed every recommendation of the design but the one on `;` (2026-10-23); that one and two points the re-measurement raised he decided on 2026-10-24 (the 2026-10-24 #752 entry: the `;` change, the title order and #867/#869's words). The design overview is docs/design/vocabulary-model-rfc.md's Phase 2 section; this entry is the record. Measurements are dated snapshots: each was taken with a spike, a throwaway patch of switches over master `638685b4` (2026-10-24, Python 3.11.17; an earlier round on `48ed448d`), which is not kept, so a figure here is evidence for the decision and each sub-phase PR re-measures on its own base. The measures, main one first: the PSC register's fielded names (`uv run python tools/differential/local_agreement.py psc-individuals`, 1,039,171 records scored, 1,004,775 agreeing on every field on `638685b4`), counted as records fixed, broken and changed; the committed corpora (`snapshot.py`); the CLDR and GND agreement records (`cldr_agreement.py`); the local snapshots of ctgov-officials, tse-donors and fec-individuals (`snapshot.py --local`); and spelling consistency, how many persons' two spellings go from reading differently to reading alike and back (mechanisms.md#PAIRED-SPELLINGS-PER-CHANGE). Local-corpus figures are counts and token shapes: `Aa` a word in Title case, `AA` in capitals, `A.` a dotted letter. + 1. **A name reading may be declared without a count, with a required `note` saying why.** Deriving the markers needs a `given` reading on 38 of the 40 may-be-given particles and a `given` or `family` reading on 7 of the 44 may-be-name credentials (`ba`, `do`, `ed`, `jd`, `lac`, `ma`, `meng`), and 12 of those particles carry no vetting note at all (`aan`, `aen`, `bat`, `da`, `dal`, `den`, `santa`, `tho`, `thoe`, `vande`, `vander`, `vel`). The note keeps the vocab README's unexamined test working. Recompute: `uv run python -c "import tomllib,glob; from nameparser import Lexicon as X; L=X.default(); n={w['text'] for f in glob.glob('nameparser/vocab/*.toml') for w in tomllib.load(open(f,'rb'))['word'] if 'given' in w or 'family' in w}; print(sorted(L.suffix_acronyms_ambiguous-n), len(L.particles_ambiguous-n))"`. + 2. **`ld` keeps no name reading; its count moves into the credential reading's `note`, and it stays ungated.** Its `given` reading records the evidence for NOT marking it (45 SSA births against 11,703 NPPES providers writing the credential, #797); a name reading that would turn the gate on contradicts decision 15's "name readings only keep the gate on" when the evidence is against the gate. Literal derivation would have moved 0 committed entries, 3 ctgov and 5 FEC readings, nothing in PSC. + 3. **A particle with no given reading still states `may_be_given = false`, and `may_be_given = true` is refused with a message naming the `given` reading to write instead**, so a new particle cannot fall to the unsafe side silently (#293's safe-by-default; decisions/vocabulary-collisions C-i). + 4. **"One gate for six rules" is four clashes and two floors.** The title run's floor (rules/H3) and bound given names (rules/P5) reach every member of their category and read only a count; no second reading is weighed, so they are step 1, unchanged. The acronym gate's nine decision sites stay nine, each keeping the order it reads its cues in (C4 the count before the case, S6 the case before the count); what becomes one is the predicate each asks. + 5. **Joined credentials land: a word joined by `-` or `/` whose every chunk is listed and none ambiguous reads as a credential, and an unlisted word of capitals so joined, with a chunk of two or more letters, joins the ambiguous class after a comma, gated by C6's case contrast.** The stricter chunk test is because a hyphen also joins names (`Smith-Jones`). Together they moved 1,782 distinct ctgov readings (1,254 and 759, overlapping on 231), nothing in PSC, the committed corpora or the agreement records; FEC is one case, so the contrast never admits a word there. + 6. **A split credential merges only where its first chunk has two or more letters, and the merged piece is a member of the ambiguous class, decided by C4's count and S4's words to spare; it starts S8's run (#827), landing after the numeral predicate answers the numeral before a credential.** FEC decided the opening letter: it writes a middle initial before a trailing salutation (`SMITH, JOHN A. MS.`), and `A. MS.` folds to the listed `ams` (merging any pair ending its part, but for two single letters, moved 5,122 FEC names, `A. MS.` 2,246 of them). A two-letter opening chunk needs the count: the 30 FEC merges it made all stood behind a one-word family (`SAM E.` → `same`), against 351 of 352 ctgov merges behind two or more name words. Accepted cost: `M. D.` and `M D` stay unmerged. Left to the split-credential PR: rules/S8 refuses a member of the ambiguous class as a run start, so how a merged piece can be both is that PR's to state (today's `Ph. D.` merge is unambiguous, and S8's #827 row covers it). + 7. **A word taken into the credential's run by position alone (`John Smith PhD Jones`) gets no new ambiguity kind.** It is not a clash between two readings; the surfaced reading's source tells it from a word with a declared name reading (item 13). This answers #785's fourth question; its first three (case drops the report and starts the run, the one-case row keeps its report) land through the pool, since retagging the word was measured to erase its other readings (`Doe, John DO` became family `DO Doe` once `do` stopped being an ambiguous acronym, its particle reading meeting P6). + 8. **Every numeral rule asks one predicate, answering for `i` to `x` the way S10's trailing fork does (shape, the count for one letter, an initial before it, the period, the case), with three readings stated:** a lone letter ending the given part ahead of a further comma is the initial (#819; `Smith, John PSM I, Jr.` the accepted cost), a lone letter in a given part after a family comma stays the initial, and a Title-case numeral in a name of mixed case leans to the name. It answers #784 (`John Smith VI MD` reads family `VI` where `III` starts the run) and #774. + 9. **Generation readings are gated by their kind's cue, and a reading with none is `default = false`, compiled to nothing by default and turned on by a pack in phase 3.** `fils` is gated by case and the count. The Brazilian agnomens `Filho`, `Filha`, `Neto`, `Neta`, `Netto`, `Sobrinho` and `Sobrinha` become gated generation readings in `pt.toml` (`pt-BR` where only Brazil writes one), with the count as their only cue: they are written in the case of the name around them. In tse-donors, 405 filed/registry pairs differ only by a trailing agnomen; master gives both spellings the same last family word in 1, and a gate emulated as an ambiguous acronym, so carrying S6's Title-case lean beside the count, in all 405, keeping every PSC surname (`Ana Neto`) and FEC given name (`SMITH, NETO`) the plain reading broke, by the count and the lean. The count-only gate the design chooses was not measured: it is to be measured before 2h lands. They are not part of the surname: a surname is what a family shares, and an agnomen marks the bearer against a relative of the same name, as `Júnior` does. Spanish `H.` (*hijo*), German regnal `I.` and Czech `st.` are `default = false`. When 2h lands it replaces decisions/generational-words' Standing line that the agnomens stay family names. + 10. **Decision 10 is unparked at the head of the name only: a modifier is a title only beside a head title there, never in a comma part,** after `resident` and `rt` move to `occupation` and `pro` to `academic`. At the head it fixed 177 PSC records and broke 7; in a comma part it reverses names whose part after the comma is the person's role (247 of 367 ctgov comma-part moves, the part being exactly `Resident` 120 times and `resident` 50, counted as the family taking master's given name). + 11. **A contested title's phrase partner is a modifier or a style before it, `of` after it, or a head title after it that the word `attaches` to;** a salutation is never one, and nor is a title before the word that is neither modifier nor style (#853). `attaches` is a property of a title reading naming the title kinds it qualifies, `chief` attaching to `office` keeping `Chief Justice`. + 12. **#853 lands first, as one rule: a title word with a declared given reading, standing after another title that is not its partner, is the given name. Given readings are then declared word by word, each batch its own measured PR;** `lama`, `lord`, `sir`, `lady`, `hon`, `baron` and `professor` stay titles, since PSC's filers type them in the title box; `title-or-name` is reported only where the rule decided against a declared name reading. Measured with nine words out of `titles` (`sheikh`, `shaikh`, `wing`, `prince`, `princess`, `sultan`, `sultana`, `mahdi`, `king`): the word after another title fixed 644 PSC records and broke none; alone at the head it fixed 152 and set 24 aside (title-box filings the scorer stops scoring once the word is no longer listed). That run took the words out of `titles`; the #853 rule is to be measured on its own before it lands. The recommended order, by forename against title-box filings in PSC, was `sultan`, `mahdi`, `sultana`, `wing`, `sheikh`/`shaikh`, `princess`, `prince`, `king`; it and two additions were re-decided in the 2026-10-24 #752 entry. `md` was settled outside phase 2 by #860. + 13. **The reading not taken surfaces as `Ambiguity.alternative` in a 2.x minor, with the pickle guard widened to accept a field added later and every narrow-guard skew still refused; `Token.reading` waits for 3.0.** The parser holds tags, not readings, so a token's reading needs a per-word reading index compiled for parse time, the first parse-time use of the TOML's descriptive data, and phase 1's import cost has to be re-taken. + 14. **Abbreviated given names outrank the leading-abbreviation title (#789's second half): `given` readings with `same_as` the full name, the English set, `ma` and `jo` in `fil.toml`, and the dotted `wm.`.** `ma` and `jo` fixed 86 PSC records (`Mrs Ma. Liza …` filings) and broke none; the English set moved 2 FEC names and nothing else. It changes rules/H2's own example, `Smith, Ma. John` → given `Ma.`. This is decision 20 applied in the default; decision 15, that name readings only keep the gate on, is about a clash between vocabulary readings that a pack settles, not a name reading against a shape reading. The first half, bare `wm` out of the titles, is Derek's 2026-10-22 split (the 2026-10-22 #789 entry), shipped in #873. + 15. **The reserved names: `attaches` and `default` gain their meaning in phase 2; `bare_only`, `leading_only`, `trailing_only` and `separators` stay reserved,** their messages naming the phase that needs them: the gate's slots carry position, rules/P7 settles `bare_only` by shape, and `separators` belongs with `attached`'s other side in phase 4. + 16. **The bracket escape reads the derived clash (#810): a bracketed member of the ambiguous class escapes the nickname reading where its writing settles it, with interior periods, or in capitals in a name of mixed case, the case asked of the whole name.** `(Ed)` stays a nickname in every case, and so does content standing inside the name rather than after it (`Jean 'JD' Smith`, whose `JD` an escape asking the word's own case moved to the middle name). Recorded in rules/S1 when 2l lands. + 17. **PSC's fielded names are the main measure for every phase-2 behavior PR, which reports fixed, broken and changed records beside the committed snapshot,** since it is the one corpus whose filers separated the fields; spelling consistency is read beside it, never instead. + + The rules: rules/S8 carries #827's row, which S8's present wording already covers. Every other behavior settled here changes a rule's statement (rules/H2's own example, rules/S1's Except for the ambiguous class, the numeral wording of rules/S8 and rules/S10, and the front of the name), so its rules amendment lands with its sub-phase PR under the same-PR amendment rule rather than as `deviates:` rows now. + + Declined, with evidence: + - THE RFC'S PARTNER WORDING, "after another title word or a modifier, before a phrase link or before a head title": with `dean`, `justice` and `major` given readings it broke 1,314 PSC records (`Mr Dean Smith` → title `Mr Dean`). + - ANY TITLE AFTER THE WORD AS A PARTNER: 99 of the 106 FEC names it moved were a given name and its trailing salutation read as a phrase (`SMITH, DEAN MR.`), so only a head the word `attaches` to is kept (item 11). + - LISTING `vi` TO `x` AS SUFFIX WORDS: 152 FEC moves, about 130 wrong (`SMITH, JOHN X`, a middle initial read as a generation), and S10's report silenced on every spelling. + - RETAGGING A CAPITALS CREDENTIAL AS UNAMBIGUOUS (#785): it erased the word's other readings (item 7). + - THE AGNOMENS AS PLAIN GENERATION READINGS: 32 PSC records broken, `Ana Neto` among them. + - MERGING ANY ADJACENT PAIR INTO A CREDENTIAL: 5,890 FEC moves, the middle initial before a salutation (item 6). + - DECISION 10 IN A COMMA PART: it reverses the names whose comma part is the role (item 10). + +- **2026-10-24 #752 (Derek) — THE `;` CHANGE IS A 2.x CHANGE OUTSIDE PHASE 2, AND THE TITLE BATCHES GO IN ORDER OF LONE-HEAD BREAKAGE, WITH `buddha`, `siddha` AND `gyani` IN THE DEFAULT GATE AND `guru` LEFT TO A `ne` PACK.** + - **`;`.** Derek decided to investigate adding `;` to the default suffix delimiters and making a trailing delimiter count, as a 2.x change outside phase 2. It shipped as #871: of the 494 ctgov names holding a `;`, 443 moved, 438 of them a name read reversed (its given name taken from after the comma) now read with its given name from before it, and nothing moved in PSC, FEC or tse. + - **The order.** Derek: "use lone-head breakage". Ranked from the nine-word measurement without `md` (the 2026-10-23 #752 entry's item 12), fewest title-box filings set aside first, ties by more fixed: `wing` (25 fixed, 0 set aside); `mahdi` with `bard` (19, 0; `bard` 1, 0); `buddha`, `siddha`, `gyani`; `king` (12, 0; its cost is the royals the committed corpora hold, `King Henry`, which PSC does not); `sultana` (8, 0); `sultan` (19, 1); `shaikh` (7, 1); `princess` (10, 4); `prince` (25, 6); `sheikh` (27, 12). `sheikh` and `shaikh` are separate batches. It replaces the order by forename against title-box filings recorded in the 2026-10-23 #752 entry's item 12. + - **#867 and #869's words.** `bard` (an occupation title from the 2014 list; the Norwegian given name `Bård`) joins `mahdi`: with both out of `titles`, PSC 79 fixed, 0 broken; FEC 20 names read right; committed 0. Derek agreed that `buddha`, `siddha` and `gyani` go in the default gate, which nepal-candidates measured at 120 romanized/Devanagari pairs brought to agree (172 names open with the four words) at no committed, PSC or FEC cost. `guru` goes to a `ne` pack in phase 3, beside #870's `श्री`: its title reading is #346's renunciate class, its Devanagari twin `गुरु` must move with it (44 nepal pairs go apart when only the Latin spelling moves), and the committed corpora's `Guru Nanak` and `Guru Gobind Singh` read it as a title. + +- **2026-10-24 #872 — THE RE-MEASUREMENT ON `638685b4` AND THE PRS THAT MERGED AFTER IT ADD THE PAIRED-INITIALS PREDICATE TO THE 2026-10-23 #752 ENTRY'S SPLIT CREDENTIALS, CHANGING NO AGREED DECISION.** + - **Paired initials.** #872 made the split-credential judgment for the joined spelling (`A.B.`, two dotted single letters in a script with capitals, outside the slot right after a comma), so the split merge asks #872's `is_paired_initials` for that case once its inline script condition is lifted into a predicate that classify and the split merge share. C5's comma reading keeps the bare predicate: `John Smith, พ.บ.` reads given `พ.บ.` today, and moving it would change that. + - Marking `sa` possibly a name, #857's first proposal and measured in the same re-measurement, was declined for its removal: decisions/C8's 2026-10-25 #857 entry. diff --git a/docs/design/mechanisms.md b/docs/design/mechanisms.md index 7ec0e874..246264f0 100644 --- a/docs/design/mechanisms.md +++ b/docs/design/mechanisms.md @@ -418,6 +418,16 @@ A second axis cuts across the jobs: whether the names are real or invented, labe **Reach for it when.** A source is proposed, before choosing between a builder and a tool column; and when a corpus would hold real people's names nobody vetted for publication, which is the case for a local corpus rather than a committed one. +### PAIRED-SPELLINGS-PER-CHANGE — count both directions, and compare whole fields + +**Problem shape.** A change is measured over a source that holds two or more spellings of one person (tse-donors' filed and registry names, nepal-candidates' romanization and Devanagari, a JRC-Names group), and the question is whether it makes the parser read one person one way. + +**Contract statement.** Per change, count the persons whose spellings go from reading differently to reading alike AND the persons who go the other way, never a net figure; compare the readings on what the two spellings can share (whole given and family fields with accents folded and an initial matching the word it begins, or, across scripts, the structure alone); and read the "now apart" moves by shape before calling any a regression. + +**How it works.** Agreement is not correctness, so this is a second measure beside a SCORE (mechanisms.md#NAME-DATA-JOBS), never in place of one. Its trap is a comparison that agreed for the wrong reason. #752's first version compared the family's LAST WORD as written, and gating the Brazilian agnomens counted 1,043 tse pairs "now apart" (2026-10-24): on master both spellings had agreed on `FILHO` as the family, and once the agnomen read as a suffix the comparison reached the real surname, which the two spellings write differently. Comparing whole fields after folding cut the agnomen gate's apart count to 304, and reading those by shape showed what was left: a word differing between the spellings, an initial, a length difference, and 47 differing only by accent or case, which point at the one real defect: the spike asked a case cue the design does not. Across scripts the same measure found what no single-script corpus could: taking `Guru` out of the titles pushed 44 Nepali pairs apart because its Devanagari twin stayed a title. + +**Reach for it when.** A change moves a word between fields in a population that holds the same names spelled twice; and when a vocabulary word has a twin in another script, which a pair source shows and a single-script snapshot cannot. + ### Field notes — the traps themselves - Test a function INSIDE a stage on the inputs its caller hands it, not on the stage's finished output. #610's first sweep test built pieces by running `group` whole and handed them to the maiden take's candidate function, but the take runs before any join, so the test read `de VI Prof.` as one piece where production hands it three: a guard checking a structure production never passes in, green for the wrong reason. Review moved the oracle to the take itself by wrapping it (tests/v2/pipeline/test_pieces.py), and recorded the old loop's misses there as the negative control (432), which is what shows the moved harness can still see a difference. diff --git a/docs/design/rules/S8-credentials-trailing-run.md b/docs/design/rules/S8-credentials-trailing-run.md index 6c6f0b84..a3c31211 100644 --- a/docs/design/rules/S8-credentials-trailing-run.md +++ b/docs/design/rules/S8-credentials-trailing-run.md @@ -9,6 +9,7 @@ "Smith, John PhD Jones" → suffix="PhD Jones" "John Smith PhD Prof. Ma" → suffix="PhD Ma" "Eric H. Holder Jr. Attorney General" → title="Attorney General" + "John Smith Ph. D. Jones" → suffix="Ph. D. Jones" deviates: #827 (today: suffix="Ph. D.") ``` - **How the name words are counted.** A lone particle is not a name word for the count, 'de Mesnil' being one surname, unless the credential stands right behind it, leaving it nothing to join: then it counts as a name word of its own ('John von PhD Jones'). A particle run with the words the chain joins to it is one name word, 'der la' and 'la Smith Secretary' each counting once (P2). ``` diff --git a/docs/design/vocabulary-model-rfc.md b/docs/design/vocabulary-model-rfc.md index 6b9022f9..6056e48f 100644 --- a/docs/design/vocabulary-model-rfc.md +++ b/docs/design/vocabulary-model-rfc.md @@ -23,7 +23,7 @@ Four phases, in a fixed order: | Phase | What | Changes parsing? | State | |---|---|---|---| | 1 | The per-word TOML format, compiled to today's sets | no | shipped in 2.6 (#737, #744); #749–#751 gave the words in `und.toml` their languages, leaving two particles of unrecorded origin | -| 2 | Ambiguity derived from clashing readings; the ambiguity gate extended to every reading kind | yes, the first phase that does | #752 | +| 2 | Ambiguity derived from clashing readings; the ambiguity gate extended to every reading kind | yes, the first phase that does | #752, design agreed 2026-10-23 | | 3 | Language packs that prefer or restrict their language | yes, for callers who choose one | #753, needs 1 and 2 | | 4 | Order and shape; the rules a pack can choose; the output model | yes | #754, mostly independent of 2 and 3 | @@ -54,55 +54,109 @@ The reference for the format is nameparser/vocab/README.md; the decisions behind ## Phase 2: derived ambiguity and a general gate -Phase 2 is the first phase that changes how names are parsed, and it gets its own design (#752). What follows is the shape that design starts from. +Phase 2 is the first phase that changes how names are parsed. Its design was agreed on 2026-10-23 (#752); the decisions, the evidence for each and the options declined are decisions/vocabulary-model's entry of that date. What follows is the design as agreed. + +### Three sources of readings + +Every clash the parser decides is between readings of one token, and phase 2 puts them in one pool: + +- **Vocabulary readings**, the TOML kinds: titles, suffixes, particles, conjunctions, bound given names, birth markers. +- **Declared name readings**, `given` and `family`, written by hand on a word some file claims or refuses (decision 11), with dated counts as provenance that never compute a reading (decision 12). Nobody maintains surname lists. +- **Shape readings**, which today are rules standing around the vocabulary: the initial ([rules/T4](rules/T4-bare-letter-as-an-initial.md), a bare letter as an initial; [rules/T5](rules/T5-written-shape-of-an-initial.md), the written shape of an initial), the leading abbreviation title ([rules/H2](rules/H2-leading-abbreviation-title.md)), the dotted acronym ([rules/S3](rules/S3-credentials-written-with-interior-periods.md)), the capitals acronym ([rules/S9](rules/S9-unlisted-words-that-join-the-ambiguous-class-by-shape.md), unlisted words that join the ambiguous class by shape), the roman numeral past the listed ones ([rules/S10](rules/S10-trailing-roman-numeral.md)), paired initials ([rules/C5](rules/C5-paired-initials-after-the-comma.md), paired initials after the comma), and the joined and split spellings of a credential that phase 2 adds. + +Shape readings join the pool because of decision 20, which lets a pack's preferred reading outrank a shape-derived one, and which phase 2 applies in the default as well (an abbreviated given name over the leading-abbreviation title, below): a reading can outrank a shape-derived one only if the shape's answer is a reading the gate weighs, not a verdict already written into a tag. Settling a clash by rewriting a tag also erases the word's other readings: retagging a capitals `DO` as an unambiguous credential let its particle reading take it into the family (`Doe, John DO`), which is why the case cue for credentials (#785) waits for the pool. ### Ambiguity from readings -A word whose readings clash is ambiguous, so the markers can be derived instead of curated: +A **clash** is a token whose readings would put it in different fields. The two markers that are clashes with a NAME become derived: -- a particle that may be a given name is a word with a particle reading and a `given` reading; -- an acronym that may be a name is a word with a credential reading and a `given` or `family` reading. +- an acronym that may be a name is a word with a credential reading and a `given` or `family` reading; +- a particle that may be a given name is a word with a particle reading and a `given` reading. -Name readings are declared by hand, and only for words some file claims as vocabulary or refuses (decision 11, widened 2026-10-19): `fils`'s `family` reading is the evidence for its refused generation reading. Nobody maintains surname lists. Each carries its bearer counts as dated provenance (source, count, date), and **counts never compute a reading** (decision 12): two data sets deciding one reading would make the decision unreadable. Counts are evidence for the person who declares a reading, can be cited in an ambiguity report, and flag a word for review when a new census changes its count. +The swap is snapshot-neutral by construction once the missing name readings are written, because no word sits in one marker set while needing to stay out of the other (decisions/vocabulary-model, 2026-10-18, with the recompute). The missing readings are declared on the flagged words, a few of them without a count, which a required `note` then explains. After the swap the build refuses the flags themselves, naming the reading to write instead, and a particle with no given reading still states `may_be_given = false`, so a new particle cannot fall to the unsafe side silently. -The swap is measured to be safe as a starting point: deriving both markers from name readings reproduces today's sets exactly if the readings are declared on exactly the flagged words, because no word sits in one marker set while needing to stay out of the other (decisions/vocabulary-model, 2026-10-18, with the recompute). After the swap, the counts are what expose the weak flags. +`may_be_initial` is not derived. Its clash is with the initial, a shape reading every cased single letter has, and which letters a tradition writes both ways is a judgment about the letter, not a name anyone bears: `e` and `i` may be read as an initial in a name written in one case, `y` may not. -Not every marker derives. `e` and `i` may be read as an initial in a name written in one case, and `y` may not, though all three are single-letter conjunctions; `may_be_initial` stays an explicit property. +### One gate, four steps -### The gate, for every reading kind +Every clash is decided by one ordered question, asked at each site that reaches it today: -Today the gates are separate rules, each written for one class of word and each reading its own evidence: +1. **Floor.** The name keeps a word to address: a title run gives its last word back ([rules/H3](rules/H3-title-runs-floor.md), the title run's floor), the second of two name words stays the family ([rules/S4](rules/S4-ambiguous-acronym-needs-words-to-spare.md), [rules/S10](rules/S10-trailing-roman-numeral.md)), and a bound given name keeps a family word in reserve ([rules/P5](rules/P5-bound-given-names.md)). +2. **Form.** A cue the reading's kind declares, read off the word's writing, settles the clash, and a clash settled by form is not reported. +3. **Structure.** The slot's own rule, in the order each slot reads it today: the ambiguous class at the comma ([rules/C4](rules/C4-ambiguous-credential-class-at-the-comma.md), the ambiguous credential class at the comma), the company of an unambiguous credential ([rules/S7](rules/S7-company-of-an-unambiguous-credential.md)), the credential's trailing run ([rules/S8](rules/S8-credentials-trailing-run.md)), a particle's leading position ([rules/P4](rules/P4-particle-at-the-front-of-the-name.md)), and the title phrase at the front of the name. +4. **Count.** Words to spare (S4, S10). -- an acronym that is also a name (`ma`, `do`): its periods, its capitals and the words left beside it ([rules/S3](rules/S3-credentials-written-with-interior-periods.md), credentials written with interior periods; [rules/S4](rules/S4-ambiguous-acronym-needs-words-to-spare.md), the ambiguous acronym needs words to spare; [rules/S6](rules/S6-case-signal-at-the-suffix-slot.md), the case signal at the suffix slot); -- a one-letter roman numeral ending a name: a word to spare, so `John V` reads family `V` and `John Smith V` suffix `V`, both reported ([rules/S10](rules/S10-trailing-roman-numeral.md), the trailing roman numeral); -- a title that may be the name: a title run with nothing but post-nominals behind it gives its last word back, so `Dr King Jr` reads family `King` and reports `title-or-name` ([rules/H3](rules/H3-title-runs-floor.md), the title run's floor); -- a particle that may be a given name: its position, so `Van Johnson` keeps `Van` as the given name and reports the fork ([rules/P4](rules/P4-particle-at-the-front-of-the-name.md), a particle at the front of the name); -- a bound given name: a name word to spare, so `abdul rahman` stays two names and `abdul rahman khan` joins them ([rules/P5](rules/P5-bound-given-names.md), bound given names); -- a single-letter conjunction that may be an initial: the name's case, so `JOHN E SMITH` reads `E` as a middle initial and reports it ([rules/P8](rules/P8-single-letter-connective-against-the-initial-it-spells.md), a single-letter connective against the initial it spells). +A clash not settled at step 2 is reported at the decision site (mechanisms.md#AMBIGUITY-AT-THE-DECISION-SITE), in the kind the slot reports today. -A listed word that none of these reaches is read wherever its position allows, and a word too risky for that is kept out. That is why `fils`, a whole-word generation word that S10's count does not reach, is excluded rather than marked. Phase 2 makes the gate a property of a reading, so that any reading that can clash with a name is gated by the evidence its form offers, rather than a rule written for each class of word; the rules above are the evidence it starts from. +**Where one gate does not reproduce the six rules.** H3 and P5 weigh no second reading: they reach every member of their category and read only a count, so they stay floors, step 1, and "one gate for six" means four clashes and two floors. The ambiguous acronym is reached at nine sites, each reading its cues in its own order, and that order is behavior, so the nine sites stay; what becomes one is the predicate each asks (mechanisms.md#ONE-PREDICATE-PER-QUESTION). Everything else reproduces once the name readings are declared, except `ld`, whose name reading is moved into its credential reading's `note` so that it stays ungated. -`fils` shows what that buys. Emulated by adding it as an ambiguous acronym (measured 2026-10-19, master `e9855df0`): +### What each kind reads -| Input | Today (`fils` excluded) | Gated | -|---|---|---| -| `Alexandre Dumas fils` | middle `Dumas`, family `fils` | family `Dumas`, suffix `fils`, reported | -| `ALEXANDRE DUMAS FILS` | middle `DUMAS`, family `FILS` | family `DUMAS`, suffix `FILS`, reported | -| `Marie Fils` | family `Fils` | family `Fils`, reported | -| `Jean Pierre Fils` | family `Fils` | family `Fils`, reported | -| `Dumas, Alexandre, fils` | suffix `fils` | suffix `fils` | +| Reading kind | Clashes with | Form cues (step 2) | Then | +|---|---|---|---| +| credential, acronym | a name reading | its periods; capitals in a name of mixed case, asked of the whole name; Title case leans to the name; the joined (`MD/PhD`) and split (`Pharm. D.`) spellings | the comma count, the company, the run, the count | +| generation, whole word (`fils`) | a family reading | lower case in a name of mixed case settles the generation; Title case leans to the name | the count | +| generation, agnomen (`Filho`, `Neto`) | a family or given reading | none: an agnomen is written in the case of the name around it | the count | +| generation, roman numeral | the initial, for one letter; a name word | a period makes it an initial; lower case in a name of mixed case settles the generation; Title case leans to the name; an initial before it makes it a name word | the count | +| generation, initial-shaped (`H.` for *hijo*, regnal `I.`) | the initial | none: written with its period it is exactly an initial | no cue: `default = false`, for a pack | +| particle | a given reading | none: `Van` and `van` are both written | the leading position, the lone particle's fold | +| conjunction, one letter | the initial | the name's case ([rules/P8](rules/P8-single-letter-connective-against-the-initial-it-spells.md)); in one case, `may_be_initial` | — | +| title at the front | a given or family reading | none | the phrase partner, the title before it | +| bound given name | none (a floor) | — | — | + +A reading whose writing offers no cue is one the gate could only guess, so it is `default = false`: recorded in the TOML, compiled to nothing by default, turned on by a pack in phase 3. Spanish `H.`, German regnal `I.` and Czech `st.` are the first. + +**The numerals ask one predicate.** `i` to `v` are vocabulary and `vi` to `x` are shape, and today some rules ask only the vocabulary, so `John Smith III MD` and `John Smith VI MD` read differently (#784, #774). One predicate answers for every spelling the way the trailing numeral rule does (shape, the count for one letter, an initial in front, the period, the case), and every rule that asks either recognizer asks it. Listing `vi` to `x` instead was measured and declined: a middle initial `X` would read as a generation. + +**`fils` and the agnomens show what kind-specific cues buy.** `fils` is gated by its case and the count, so `Alexandre Dumas fils` is to read suffix `fils` while `Marie Fils` keeps her family name, reported. The Brazilian agnomens are to be gated by the count alone: case would say nothing, and the plain generation reading breaks real surnames (`Ana Neto`) and given names. + +**A credential's spelling is a form cue.** A credential joined by `-` or `/` is to read as one when every chunk is listed and none is ambiguous, since a hyphen also joins names (`Smith-Jones`); an unlisted word of capitals joined that way is to join the ambiguous class after a comma, gated by the name's case contrast. Today `John Smith, MD/PhD` reads given `MD/PhD`. A credential split across a space (`Pharm. D.`, generalizing the `Ph. D.` merge) is to merge only where its first chunk has two or more letters, the merged piece a member of the ambiguous class, decided by the comma count and words to spare: an opening single letter is a middle initial (`SMITH, JOHN A. MS.` holds an initial and a salutation, not a credential `AMS`). #872 made the same judgment for the joined spelling (`A.B.`), in scripts with capitals and outside the slot right after a comma. The merged piece is also to start the credential's run (#827), once the numeral predicate answers the numeral before a credential; since [rules/S8](rules/S8-credentials-trailing-run.md) refuses a member of the ambiguous class as a run start, how the two combine is the split-credential PR's to state (today's `Ph. D.` merge is unambiguous, and S8 already covers it). + +### The front of the name -Every row is right or unchanged, and the forks are reported. But the emulation borrows the acronym gate's cues, and capitals are an acronym's cue, not a generation word's: it suits `fils` because French writes the word in lowercase. What evidence each kind's gate reads is phase 2's central question: +The front of a name has no words-to-spare cue: a title addresses the name after it. What it has is the contested-head rule. **A title word that also has a name reading is a title only inside a title phrase:** -- a lowercase word after the name (`fils`) has its case; -- an initial-shaped one (Spanish `H.` for *hijo*, German regnal `I.`) has the count S10 already gives a bare one-letter numeral, against a period, which marks a one-letter word as an initial ([rules/T5](rules/T5-written-shape-of-an-initial.md), the written shape of an initial); -- a reading with no reliable cue at all may need to be pack-only (`default = false`, phase 3). +- **The title before it is not a partner.** A contested word after another title is the given name (`Mr Sultan Khan`, `Ms Princess Ada Obi`, #853), unless that title is its partner; today `Mr Sultan Khan` reads title `Mr Sultan`. +- **Its partner** is a modifier or a style before it (`Her Royal Highness Princess Anne`), `of` after it (`Dean of Students`), or a head title after it that the word `attaches` to (`Major General`). A salutation is never a partner (`SMITH, DEAN MR.` is a given name and a salutation). +- **Modifiers** (decision 10, unparked) are titles only beside a head title, at the head of the name only, never in a comma part, where a lone title word is the person's role. -**A gate at the front of the name** is the same question for titles. `dean`, `justice` and `major` are title words borne as given names, left out of `titles` today. The candidate rule is that such a word is a title only inside a title phrase: after a modifier or another title (`Chief Justice`), before a phrase link (`Dean of Students`), or with a head title after it (`Major General`). Today all three of those read the word as a given name (measured 2026-10-19), while `Dean Martin` correctly does. Two decisions bound it: `Dean Smith` keeps reading given `Dean` (decision 13), and the rule covers only words outside `titles` today, so a title that is also a given name (`prince`, `sir`) is decided one word at a time and never re-decided silently by recording a given reading (decision 16). Title-phrase modifiers (`vice`, `rear`, `grand`) are the same rule from the other side: a title only next to a head title (decision 10, parked). +Which title words get a given reading is decided one word at a time (decision 16), in measured batches, and `Dean Smith` keeps reading given `Dean` (decision 13). A title that only ever stands in front stays a title (`lord`, `sir`, `lady`), and a word that opens names in one tradition but is a title class in another (`guru`, with its Devanagari twin) waits for a language pack. Where the rule reads a contested title as a name, the parse reports `title-or-name`. + +**Abbreviated given names** use the same pool from the other side (#789): a `given` reading with `same_as` the full name (`chas` → Charles, Filipino `ma` → María) is to outrank the leading-abbreviation title rule's shape, so that `Ma. Cristina Santos Reyes` reads given `Ma.` where today it reads title `Ma.`. + +### Brackets + +A bracketed or quoted credential is to escape the nickname reading where its writing settles it (#810): with interior periods, or in capitals in a name of mixed case, the case asked of the whole name. Today [rules/S1](rules/S1-bracketed-credential-is-not-a-nickname.md) keeps every member of the ambiguous class written without periods a nickname (`John Smith (MA)` → nickname `MA`), and S1 is amended when the escape lands. `(Ed)` stays a nickname, and so does content standing inside the name rather than after it (`Jean 'JD' Smith`). ### Readings on the parse -A caller should be able to see the reading behind a parse: its language, kind, native spelling (`do` → `Đỗ`) and counts. The intended split is that **an `Ambiguity` carries the reading the parse did not take**, and **a token carries the reading it did**, so `Nguyen Van Do`, which forks nothing, still exposes `Đỗ` without a report nobody would act on. The API shape is open (#752). +A caller should be able to see the reading behind a parse. **An `Ambiguity` carries the reading the parse did not take** (`Ambiguity.alternative`: its kind, the field it would have put the token in, and whether it came from the vocabulary, a declared name or a shape), and **a token carries the reading it did** (`Token.reading`), so `Nguyen Van Do`, which forks nothing, can still expose `Đỗ` without a report nobody would act on. A word the credential's run takes by position alone is told from a word with a declared name reading by that source, so it needs no new ambiguity kind. `Ambiguity.alternative` is planned for a 2.x minor, with the pickle guard widened to accept a field added later; `Token.reading` waits for 3.0, since it needs a per-word reading index compiled for parse time. + +### The phase 1 reserved names + +Phase 2 gives two of them meaning: `attaches`, the phrase partner above, a property of a title reading naming the title kinds it qualifies; and `default = false`, for readings with no cue. `leading_only` and `trailing_only` are not needed, the gate's slots carrying position; `bare_only` is settled by shape already ([rules/P7](rules/P7-one-letter-particle-against-the-initial-it-spells.md), a one-letter particle against the initial it spells); `separators` belongs with `attached` and its other side, phase 4's glued words. Those four stay reserved, their messages naming the phase that needs them. + +### The order of work + +Each sub-phase is one PR, measured on its own base: the committed snapshot read by shape, the PSC register's fielded names scored for fixed, broken and changed records (`local_agreement.py psc-individuals`, the main measure, since its filers separated the fields), the local snapshots it reaches, and the rules it moves amended in the same diff. + +| Sub-phase | What | Needs | +|---|---|---| +| 2a | the derived markers: the name readings declared, the flags refused | — (neutral) | +| 2b | one predicate per question: the split-credential merge, the numeral, paired initials in a script with capitals | — (neutral) | +| 2c | joined credentials, every chunk listed | 2b | +| 2d | joined unlisted capitals after a comma | 2b | +| 2e | split credentials, and the merged piece starting the run (#827) | 2b, 2f | +| 2f | every numeral rule asks one predicate (#784, #774, #819) | 2b | +| 2g | capitals settle a listed credential (#785), through the pool | 2a, 2b | +| 2h | generation readings gated by kind: `fils`, the agnomens, `default = false` | 2b | +| 2i | the front of the name: the phrase partner, the contested heads, #853's word after a title, modifiers at the head | 2b | +| 2j onward | titles that are names, one batch per word or group, in the agreed order | 2i | +| 2k | abbreviated given names over the leading-abbreviation title (#789) | 2b | +| 2l | the bracket escape reads the derived clash (#810) | 2a, 2b | +| 2m | surfacing, `Ambiguity.alternative` | the rest | + +2a and 2b are independent of each other. The sub-phase that gives a reserved name its meaning updates nameparser/vocab/README.md's reserved-names table with it: 2h for `default`, 2i for `attaches`. Measured changes that belonged outside phase 2 shipped as 2.x PRs: bare `wm` (#873), the `;` between credentials (#871), two dotted single letters as a surname (#872) and `sa` out of the vocabulary (#879). ## Phase 3: packs that prefer or restrict @@ -120,9 +174,9 @@ Consequences that follow from the definitions: - **Non-Latin vocabulary stays on under any choice**, because it selects itself by script (`씨`, `さん`, Cyrillic titles). Almost every clash between languages is within one script: `do`, `e`, `van` are all Latin. - **Whether a word needs a pack depends on where its clash is.** `do` clashes between languages (a Portuguese particle, an English credential, a Vietnamese surname), so a pack settles it. `fils` clashes within French, a generation word and a French surname, so no pack settles it and the phase 2 gate does the work. This reverses the premise of #684, which proposes a French pack to read `fils` as a suffix: under decisions 6 and 15, a French pack keeps both of French's readings, and `Marie Fils` would still need the gate to keep her family name. -**A pack can outrank a shape rule, not only a vocabulary reading** (decision 20). `Ma. Cristina Santos Reyes` reads title `Ma.` today because the leading period-abbreviation rule claims the word before any vocabulary is consulted; in Filipino names `Ma.` abbreviates María. So shape rules (period abbreviation, initials, capitals) must emit readings into the same pool as vocabulary, where a pack's preference can act on them, rather than final answers. +**A pack can outrank a shape rule, not only a vocabulary reading** (decision 20). Shape rules (period abbreviation, initials, capitals) emit readings into the same pool as vocabulary from phase 2 on, rather than final answers, so a pack's preference can act on them. Phase 2 is to settle one such clash in the default, since every language loads there: Filipino `Ma.`, which abbreviates María, is to get a `given` reading in `fil.toml` that outranks the leading period-abbreviation rule (`Ma. Cristina Santos Reyes`, title `Ma.` today); a pack's part is to prefer such readings where the default leaves both standing. -**What moves in phase 3:** the locale packs' vocabulary, still in Python today, becomes language files; `default = false` marks a reading for pack-only words too risky even gated; and the reserved names `prefer`, `strict` and `default` gain their meaning. How a tag filters (does `pt` match `pt-BR`?) and how a third-party language file joins the pool and the snapshot are open under #753. +**What moves in phase 3:** the locale packs' vocabulary, still in Python today, becomes language files; `default = false`, given its meaning in phase 2, marks the readings a pack turns on, for words too risky even gated; and the reserved names `prefer` and `strict` gain their meaning. How a tag filters (does `pt` match `pt-BR`?) and how a third-party language file joins the pool and the snapshot are open under #753. ## Phase 4: order, shape and the output model @@ -148,11 +202,13 @@ A language file will hold a `[rules]` table that turns on rules implemented in t | Rule | Input | Today | |---|---|---| -| surname span (es, pt) | `Gabriel García Márquez` | middle `García`, family `Márquez` | +| surname span | `Gabriel García Márquez` | middle `García`, family `Márquez` by default; family `García Márquez` under the opt-in `Policy(two_family_names=True)` | | compound given name (pt) | `Maria do Carmo Silva` | given `Maria`, family `do Carmo Silva` | | married-name ending (hu) | `Kovács Jánosné` | given `Kovács`, family `Jánosné` | | digraph initials (cs) | `Petr Chmelař` | initials `P. C.` | +The surname span has shipped ahead of the packs, as an opt-in `Policy` field rather than a pack, since the convention is not one language's ([rules/O6](rules/O6-two-family-names.md), two family names, #875 and #878; decisions/O6 has why it is not `locales.ES`). A pack can later switch it on in its `[rules]`, as it will a default order. + The real-name corpora agree on the priority. In the GND and Wikidata corpora, Spanish and Latin American double surnames written given-first disagreed with the source on 315 of 390 names, while the same kind of names in index form, where the comma settles it, disagreed on 0 of 60 (counted 2026-10-16 on master `4375b01b`, from `gnd_agreement.json` and `wikidata_agreement.json`): the gap is order and span, not vocabulary. ### The output model diff --git a/tools/differential/corpus_rules.jsonl b/tools/differential/corpus_rules.jsonl index 17531874..d0f9a937 100644 --- a/tools/differential/corpus_rules.jsonl +++ b/tools/differential/corpus_rules.jsonl @@ -321,6 +321,7 @@ "John Smith Ma" "John Smith Mc V" "John Smith Msc.Ed." +"John Smith Ph. D. Jones" "John Smith PhD" "John Smith PhD (MA)" "John Smith PhD Ed Ma" diff --git a/tools/differential/snapshot.jsonl b/tools/differential/snapshot.jsonl index f8b34342..afe7ab01 100644 --- a/tools/differential/snapshot.jsonl +++ b/tools/differential/snapshot.jsonl @@ -1631,6 +1631,7 @@ {"name": "John Smith Msc.Ed.", "given": "John", "family": "Smith", "suffix": "Msc.Ed.", "initials": "J. S."} {"name": "John Smith Ph.", "given": "John", "middle": "Smith", "family": "Ph.", "initials": "J. S. P."} {"name": "John Smith Ph. D.", "given": "John", "family": "Smith", "suffix": "Ph. D.", "initials": "J. S."} +{"name": "John Smith Ph. D. Jones", "given": "John", "middle": "Smith", "family": "Jones", "suffix": "Ph. D.", "initials": "J. S. J."} {"name": "John Smith PhD", "given": "John", "family": "Smith", "suffix": "PhD", "initials": "J. S."} {"name": "John Smith PhD (MA)", "given": "John", "family": "Smith", "suffix": "PhD", "nickname": "MA", "ambiguities": ["SUFFIX_OR_NICKNAME"], "initials": "J. S.", "capitalized_force": "John \"Ma\" Smith PhD"} {"name": "John Smith PhD Ed Ma", "given": "John", "family": "Smith", "suffix": "PhD Ed Ma", "ambiguities": ["SUFFIX_OR_NAME"], "initials": "J. S.", "capitalized_force": "John Smith PhD ED MA"}