From b732e67d9688152a4417b0308e2a890ecc40a6ec Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Tue, 27 Oct 2026 20:32:03 -0700 Subject: [PATCH 01/39] Propose vocabulary phase 3's design: a choice of languages selects the pool (#753) The RFC's Phase 3 section rewritten and a dated decisions/vocabulary-model entry, every item marked proposed until Derek decides it. Co-Authored-By: Claude Opus 5.5 --- docs/design/decisions/vocabulary-model.md | 13 ++++ docs/design/vocabulary-model-rfc.md | 95 +++++++++++++++++++++-- 2 files changed, 102 insertions(+), 6 deletions(-) diff --git a/docs/design/decisions/vocabulary-model.md b/docs/design/decisions/vocabulary-model.md index 82c1182ba..ff7766b91 100644 --- a/docs/design/decisions/vocabulary-model.md +++ b/docs/design/decisions/vocabulary-model.md @@ -236,3 +236,16 @@ Open: #913 (the title words phase 2j left, read as titles though borne as names) Adding the field changes `Lexicon`'s pickle layout, as phases 2h and 2i did. - **2026-10-26 #752, phase 2l — THE BRACKET ESCAPE READS THE DERIVED CLASH**, carrying out item 16 of the 2026-10-23 #752 entry above (#810). A member of the ambiguous class, the class 2a derives from name readings, alone in brackets or quotes after the name is the suffix where its writing settles it (`John Smith (MA)`, `John Smith (M.A)` → suffix), read in one pass, and stays the nickname otherwise (`John Smith (Ed)`, `JOHN SMITH (MA)`, `Jeffrey (JD) Bricken`). It is recorded as rules/S13, a rule of its own rather than the amendment to S1 item 16 planned, the position it reads being evidence S1's signal does not read; the writing is S3's and S6's evidence, which S13 cites. The record, the measurements and what was declined are decisions/S13's 2026-10-26 #810 entry. - **2026-10-26 #752, phase 2m — THE READING NOT TAKEN SURFACES AS `Ambiguity.alternative`, WITHOUT ITS LANGUAGES**, carrying out item 13 of the 2026-10-23 #752 entry above and item 7's source with it: a word the credential's run took by position alone reports its name reading's source as `position`, a fourth source beside the three the item named, and a word with a declared name reading reports `declared`. The pickle guard was widened as item 13 asked, every skew the narrow guard refused still refused. The reading's languages were left out: the per-word index costs about 0.4 ms of a 19.5 ms import and describes the default vocabulary, not a pack's or a caller's lexicon, so they wait for 3.0 with `Token.reading`. The record, the measurements and what was declined are decisions/A1's 2026-10-26 #752 entry. + +- **2026-10-27 #753 — PHASE 3'S DESIGN IS PROPOSED, NOT AGREED: A CHOICE OF LANGUAGES SELECTS THE POOL, ONE MECHANISM FOR TURNING READINGS ON, PREFER AND STRICT; NAME READINGS IN A PACK KEEP THE GATE ON; A READING INDEX CARRIES THE LANGUAGES TO RUN TIME.** Every item below is a proposal awaiting Derek's decision, and this head is rewritten when he makes it. The design overview is docs/design/vocabulary-model-rfc.md's Phase 3 section. Decisions 6, 15 and 20 are kept as written; the items say how a pack carries them out. The inputs are #753's comments of 2026-10-27: the inventory of 411 kept-out candidate readings with their verdicts, and the probe of foreign titles and function-word collisions. Measurements were taken 2026-10-27 on master `cfe8bf92` (Python 3.11.17) with throwaway Lexicon edits standing in for a pack, so each is evidence for the proposal and each sub-phase re-measures on its own base. Local-corpus figures are counts. + 1. **Proposed: a pack is a choice of languages carried by `Locale`.** `Locale` gains `languages`, a set of BCP 47 tags; `parser_for` unions the sets of the packs it is given, so the choice is order-independent while policy patches still fold in order; strict is an argument of the whole choice (`parser_for(locales.NE, locales.HI, strict=True)`), since a strict pack beside a preferring one has no meaning. The shipped language packs are `Locale` constants with an empty lexicon fragment (`locales.NE`, `locales.PT_BR`). Declined: `Parser(languages=…)` beside `Locale`, two ways to choose one thing, and a `Policy` field, `Policy` being behavior and vocabulary being `Lexicon`'s. + 2. **Proposed: one mechanism.** The chosen languages select which readings the compiler compiles, the default being that compiler with every language and no choice. Turning on, prefer and strict are derived from the file each reading sits in, and nothing is written per pack beyond its language's readings: a chosen language's `default = false` readings join; prefer drops other languages' vocabulary readings of a word on which a chosen language holds one, never `mul.toml`'s (`Haddad, Abd`, suffix `Abd` today, reads given `Abd` with the en credential gone); strict drops other languages' name readings and the markers they derive. A clash a property records with a SHAPE reading is settled for the chosen language's own reading, decision 20 applied: `may_be_initial` does not hold on it, so a `pt` choice reads `ANA SILVA E SOUZA` family `SILVA E SOUZA` where the default reads middle `SILVA E` with a report. This answers decisions/3-0-reevaluations' 2026-09-13 bullet that a pack cannot remove `e`, and its 2026-08-17 bullet's "overrides rather than additions": a pack overrides by selection, never by a removal list. Left open rather than built: a property a language's own reading should take only under its pack, Swedish `fru` and Icelandic `frú` as after-given salutations (rules/H8), refused in the default for the Cameroonian surname, the two readings in the inventory needing it. + 3. **Proposed: `default = false` is allowed on `given` and `family` readings, and turned on they derive markers like any name reading, never settling (decision 15 unchanged).** Measured on nepal-candidates' 126,981 Devanagari and romanized pairs, comparing every word's role across the two spellings: 123,440 read alike in the default; `given` readings on `श्री`, `guru` and `गुरु`, which make each a title only inside a title phrase (rules/H9), give 123,636 (198 fixed, 2 broken, both pairs whose romanization opens with a listed title); a `family` reading on `dc` as well, making the credential ambiguous, gives 123,639, the count still leaving `Ram Bahadur DC` a suffix; dropping `dc`'s credential, which settling by a name reading would do, gives 123,658, 19 more. Settling is not proposed: the 19 pairs are the price of decision 15, whose `do` case it protects. `bk`, `kc`, `gc` and `bc` read as the family in every shape tried (spoken order, a family comma, `FAMILY_FIRST`, dotted and Title case), and decision 11 refuses a name reading on a word no file claims as vocabulary, so none is proposed. Recompute: the readings moved into or out of `Lexicon` fields by hand, the pairs compared by `tuple(t.role for t in parser.parse(s).tokens)`. + 4. **Proposed: strict needs no sweep first.** Check 11 makes every particle reading, `default = false` included, carry a `given` reading on its word or state `may_be_given = false`, so the PR turning on `nl` `in` and `to` declares the Korean and Vietnamese given readings they collide with, with counts; in the default those readings derive nothing, the word having no live vocabulary reading there. + 5. **Proposed: the languages reach run time as a reading index.** A second generated module, one row per reading (text, tag, kind, behavioral properties), loaded only when a language is chosen; the compiler moves from tools/vocab/build.py into the package so build and pack share one implementation, and a test holds that compiling the index with no choice reproduces `_compiled.py`. It is the per-word index item 13 of the 2026-10-23 #752 entry says `Token.reading` needs. A choice is applied as a delta against the default's compile, so it composes with a caller's `base=` lexicon and a `Locale`'s own fragment. Measured: 2,140 readings over 1,797 words, 106 in more than one file; the index about 93 KB of source and 11 ms to execute without a bytecode cache, the compile 9 ms, both once per choice and only for a caller who makes one; loading the TOML 39 ms. Declined: per-language deltas compiled at build time, which do not compose (under prefer `{pt, en}` `do` keeps its particle and its credential, while Portuguese's delta drops the credential and English's the particle); every combination precompiled, combinatorial; the TOML read at run time, slower and either re-running the build's checks or skipping them. + 6. **Proposed: a chosen tag selects its file and its ancestors', never its descendants, and no macrolanguage table is read.** `pt-BR` selects `pt-BR.toml` and `pt.toml`; `pt` only `pt.toml`, so a caller in Portugal does not turn on the Brazilian agnomens (#843). The two files share no word today. Malay and Indonesian, which share `puan`, `putra` and `putri`, are chosen together by a caller who has both. The TOML names `prefer` and `strict` stay refused, the message saying a strength is the caller's choice and never data. + 7. **Proposed: a pack is measured three ways.** The default does not move (the committed snapshot, and the index invariant of item 5); a vocabulary pack's deviation surface, the words whose compiled readings its choice changes, is derived from the index, and tests/v2/test_locales.py's non-interference gate requires the default's reading for every corpus name without such a word, the policy packs (RU, TR_AZ) keeping their written `DEVIATES`; and `snapshot.py`, `--local` with `--base` included, takes a choice and prints what it moves by field, beside each pack's own ground truth (nepal-candidates' script pairs for `ne`, the CLDR agreement per pack, PSC's nationality slice). This answers decisions/3-0-reevaluations' 2026-09-13 bullet that the harness cannot see a pack. + 8. **Proposed: third-party language files are deferred to 3.0.** A file joining the pool at run time would skip the checks that run across files (check 6's exclusions, check 11's particle statement), the ones that refuse a third-party `nl` `in` written with neither a `given` reading nor `may_be_given = false`. `Locale(lexicon=…)` stays the 2.x way to add vocabulary. + 9. **Proposed: the pack-shaped exclusions become `default = false` readings,** so an `excluded` key again means wrong everywhere: the seven whose reason opens "pack-shaped" (cs `ml`, hr, sk and sl `st` and `ml`, fr `père`), the shape cs `st` already has. Recompute: `grep -c 'pack-shaped' nameparser/vocab/*.toml`. + 10. **Proposed: what stays out of phase 3.** ZH's surname list and the RU and TR_AZ endings are rule data for phase 4's `[rules]`, and a Python pack gains `languages` only where a measurement shows what it moves; the candidates that wait on a position rule (Turkish `Bey` and `Hanım`, Hungarian `úr`, Thai `นาย`, French `M.`) are phase 4's; same-language clashes (`kumari`, `sayyid`) take phase 2j's route in the default, since no pack settles one. + 11. **Proposed: the work order,** shape first and one PR each, measured on its own base: 3a the index and shared compiler with name readings allowed `default = false` (neutral); 3b the choice, tag selection, prefer, strict, the shape clash, the derived deviation surface and `snapshot.py`'s choice (neutral); 3c the pack-shaped exclusions converted (neutral); then the packs, `ne` (3d), `hi` and `bn`'s Latin forms (3e), `cs`, `sk`, `sl` and `hr` (3f), `es` and `pt` with a connective that counts only inside a title run (3g), and `ga`/`gd`, `nl` and `pt-BR` (3h). 3g's connective is a rule with its own design: with `profesora` added and no rule, `Profesora de Historia Ana López` reads family `de Historia Ana López` (the rules/P1 fold) and `Ministro da Fazenda Carlos Souza` given `da`, while `Doctora de la Vega` must keep family `de la Vega`, the two separated by whether a name stands behind the phrase; it must reach only readings a pack turns on. diff --git a/docs/design/vocabulary-model-rfc.md b/docs/design/vocabulary-model-rfc.md index a13fbf095..dda7145df 100644 --- a/docs/design/vocabulary-model-rfc.md +++ b/docs/design/vocabulary-model-rfc.md @@ -26,7 +26,7 @@ Four phases, in a fixed order: |---|---|---|---| | 1 | The per-word TOML format, compiled to today's sets | no | shipped in 2.6 (#737, #744); #749–#751 gave the words in `und.toml` their languages, leaving two particles of unrecorded origin | | 2 | Ambiguity derived from clashing readings; the ambiguity gate extended to every reading kind | yes, the first phase that does | #752, design agreed 2026-10-23; 2a, the two name-clash markers derived, landed without moving a parse (decisions/vocabulary-model, 2026-10-25) | -| 3 | Language packs that prefer or restrict their language | yes, for callers who choose one | #753, needs 1 and 2 | +| 3 | Language packs that prefer or restrict their language | yes, for callers who choose one | #753, design proposed 2026-10-27, not agreed | | 4 | Order and shape; the rules a pack can choose; the output model | yes | #754, mostly independent of 2 and 3 | Two commitments hold through all four. **The default loads every language and keeps working for names of any language, as today**; choosing a language narrows it (decision 5 in decisions/vocabulary-model). And **languages own vocabulary and defaults, the core owns structure**: a pack configures rules the core implements, and never ships code (mechanisms.md#LOCALE-PACKS-PURE-DATA). @@ -164,7 +164,9 @@ Each sub-phase is one PR, measured on its own base: the committed snapshot read ## Phase 3: packs that prefer or restrict -A pack is a choice of languages. Two strengths (decision 6): +Status: **proposed, not agreed** (2026-10-27, #753). The record is decisions/vocabulary-model's 2026-10-27 #753 entry, whose items are each marked proposed until Derek decides them; this section is the proposal those items describe. Phase 3 ships in 2.x, opt-in: a caller who chooses no language parses exactly as before, and nothing in this phase moves the default's parses. + +The semantics were fixed in 2026-10-15/16, and phase 3 keeps them. A pack is a choice of languages, in two strengths (decision 6): - **prefer** settles a clash where the chosen language has a vocabulary reading: the word keeps that language's readings and loses other languages' vocabulary readings. Under Portuguese, `do` is a particle only. - **strict** also drops other languages' name readings. It is for single-origin data. @@ -176,11 +178,92 @@ Consequences that follow from the definitions: - **Order does not matter.** Prefer and strict are sets of languages, so combining packs is commutative. - **Two chosen languages that each hold a vocabulary reading both keep it**, and the word stays gated, as in the default. - **Non-Latin vocabulary stays on under any choice**, because it selects itself by script (`씨`, `さん`, Cyrillic titles). Almost every clash between languages is within one script: `do`, `e`, `van` are all Latin. -- **Whether a word needs a pack depends on where its clash is.** `do` clashes between languages (a Portuguese particle, an English credential, a Vietnamese surname), so a pack settles it. `fils` clashes within French, a generation word and a French surname, so no pack settles it and the phase 2 gate does the work. This reverses the premise of #684, which proposes a French pack to read `fils` as a suffix: under decisions 6 and 15, a French pack keeps both of French's readings, and `Marie Fils` would still need the gate to keep her family name. +- **Whether a word needs a pack depends on where its clash is.** `do` clashes between languages, so a pack settles it. `fils` clashes within French, a generation word and a French surname, so no pack settles it and the phase 2 gate does the work, reversing the premise of #684. +- **A pack can outrank a shape rule, not only a vocabulary reading** (decision 20). Shape rules emit readings into the pool rather than final answers, so a pack's preference can act on them; phase 2k applied this in the default, a declared `given` reading of `Ma.` outranking the leading-abbreviation title. + +What the inventory of the kept-out words found (#753's comments of 2026-10-27, 411 candidate readings) changes the emphasis rather than the semantics. Only 9 words clash between languages (`de`, `do`, `i`, `st`, `sr`, `ra`, `mc`, `vd`, `abd`), so prefer alone settles little; most pack candidates are readings the default leaves out entirely, which a pack turns ON. Phase 3's main work is therefore turning readings on, with prefer and strict settling what turning them on collides with. + +### One mechanism: a choice selects the pool + +Everything a pack does is one operation: **the chosen languages select which readings of the pool are compiled**, by the same compiler that builds the default. The default is that compiler run with every language and no choice. Nothing is written per pack beyond the readings in its language's file, and every effect below is derived from the file a reading sits in: + +| Effect | Which readings | Example | +|---|---|---| +| turn on | a chosen language's `default = false` readings join the pool | cs `ml` (*mladší*, once 3c makes it a reading): `Jan Novák ml.` reads family `ml.` today; under `cs`, suffix `ml.` | +| prefer | on a word where a chosen language holds a vocabulary reading, other languages' vocabulary readings leave; `mul.toml`'s readings never do, being every language's | `Haddad, Abd` reads suffix `Abd` and no given name today; under `ar` the bound given reading stays and the en credential leaves; cs `st`: `Jiří Dienstbier st.` reads title `st.` today (rules/H5), and under `cs` its generation reading joins and the Saint readings leave, suffix `st.` | +| prefer, against a shape | a clash a property records with a SHAPE reading is settled for the chosen language's own reading, by decision 20: `may_be_initial` does not hold on it | `ANA SILVA E SOUZA` reads middle `SILVA E` with a conjunction-or-initial report today; under `pt` the conjunction holds, family `SILVA E SOUZA` | +| strict | other languages' name readings leave, and with them the markers they derive | an `nl` `in` turned on, with Korean In declared as a `given` reading, is a particle that may be given under `nl` and never-given under strict `nl` | + +The third row answers decisions/3-0-reevaluations' "a pack cannot remove `e`": `may_be_initial` records a clash between a conjunction and the initial a letter spells, and the initial is a shape reading, so a pack that prefers Portuguese settles it as decision 20 says a pack may. No property is toggled per pack, and no reading is written twice. + +Properties ride on their readings, so a choice that keeps a reading keeps its properties. The one shape this cannot express is a property a language's own reading should take only under its pack: Swedish `fru` and Icelandic `frú` as after-given salutations (rules/H8), refused in the default because Fru is also a Cameroonian surname. Two readings need it, so the proposal builds nothing for them and leaves them open. + +### Name readings in a pack + +`default = false` is allowed on `given` and `family` readings. A name reading turned on by a pack derives the markers any name reading derives, so it turns the gate on and never settles anything: decision 15 holds unchanged inside a pack. Measured on nepal-candidates (2026-10-27, `cfe8bf92`), comparing each candidate's Devanagari and romanized spellings by the role of every word: + +| Reading turned on | Effect | Pairs reading alike, of 126,981 | +|---|---|---| +| none (the default) | | 123,440 | +| `given` on `श्री`, `guru` and `गुरु` | each becomes a title only inside a title phrase (rules/H9), so it reads given at the head: `श्री कृष्ण श्रेष्ठ` → given `श्री` | 123,636 (198 fixed, 2 broken) | +| and `family` on `dc` | the family reading only makes the credential ambiguous: `Ram Bahadur DC` still reads suffix `DC`, the count leaving it one | 123,639 (3 more fixed) | +| `dc`'s credential dropped, as settling would do | `Ram Bahadur DC` → family `DC` | 123,658 (19 more fixed) | + +The 2 broken are pairs whose romanization opens with a listed title, which reads the `श्री` as a salutation (#870 counted `Mr` written for it twice). So the gate route delivers the `श्री` and `guru` fix without amending decision 15, and the 19 `dc` pairs it leaves are the price of keeping it. `bk`, `kc`, `gc` and `bc` need nothing: they read as the family in every shape tried (`Ram Bahadur BK`, `Ram Bahadur Kc`, `BK, Ram Bahadur`, `KC Sita` under `FAMILY_FIRST`), and decision 11 refuses a name reading on a word no file claims as vocabulary. + +A strict pack drops only the name readings that are recorded. The build already makes the author record them: a particle reading, `default = false` included, needs a `given` reading on its word or states `may_be_given = false` (check 11). So the pack PR that turns on `nl` `in` and `to` declares the Korean and Vietnamese given readings they collide with, with their counts, and no separate sweep comes first. In the default those readings derive nothing, since the word has no live vocabulary reading there. + +### Where the provenance lives at runtime + +The compiled module holds sets, and a set has forgotten which file each word came from. The proposal compiles a second module beside it: a **reading index**, one row per reading (its text, its file's tag, its kind and its behavioral properties), loaded only when a caller chooses a language. The compiler moves from tools/vocab/build.py into the package so that the build and a pack share one implementation, and a test holds the invariant that compiling the index with no choice reproduces `nameparser/config/_compiled.py` exactly. The same index is what decision 13 says `Token.reading` needs in 3.0. + +Measured 2026-10-27 on `cfe8bf92`: the pool holds 2,140 readings over 1,797 words, 106 of them in more than one file. An index of their behavioral properties is about 93 KB of generated source, executing in about 11 ms with no bytecode cache, and compiling the pool takes about 9 ms, both paid once per choice and only by a caller who makes one. Reading the TOML instead costs about 39 ms. Recompute: the build's `read_dir`, `load` and `compile_pool`, timed. + +Two cheaper-looking forms do not work. Per-language deltas compiled at build time do not compose: under prefer `{pt, en}`, `do` keeps both its particle and its credential, but Portuguese's delta removes the credential and English's removes the particle. And precompiling every combination of languages is combinatorial. + +A choice is applied as a delta against the default's compile, so it composes with a caller's own `Lexicon`: `parser_for(locales.NE, base=Parser(lexicon=custom))` keeps a title the caller added, and a `Locale`'s own `lexicon` fragment is still unioned on after the choice. + +### The API and tags + +A choice is carried by `Locale`, which gains a `languages` field, a set of BCP 47 tags. `parser_for` unions the `languages` of the packs it is given, so the choice is order-independent while their policy patches still fold in order. Strict is a property of the whole choice, not of one pack, and is an argument: `parser_for(locales.NE, locales.HI, strict=True)`. The shipped language packs are `Locale` constants like the four today (`locales.NE`, `locales.get("ne")`), each with an empty lexicon fragment; a code keeps `Locale`'s `[a-z0-9_]+`, so `pt-BR` is `locales.PT_BR`. + +A chosen tag selects its own file and its ancestors', never its descendants': `pt-BR` selects `pt-BR.toml` and `pt.toml`, and `pt` selects only `pt.toml`, so a caller in Portugal does not turn on the Brazilian agnomens (#843 left the single-letter `F` and `N` to a `pt-BR` pack). No macrolanguage table is read: Malay and Indonesian share `puan`, `putra` and `putri`, and a caller with both chooses both. The Czech, Slovak, Slovenian and Croatian files each hold the words they share, so each language is a pack of its own. Script subtags need no matching rule yet, since no file carries one. + +The TOML names `prefer` and `strict`, reserved for phase 3 since phase 1, stay refused, with a message saying that a strength is the caller's choice and never data. + +### Measuring a pack + +The snapshot and `compare.py` parse under the default, so neither can see a pack (decisions/3-0-reevaluations). Three measures, the first two in CI: + +- **The default does not move.** The committed snapshot holds it, as for any change, and so does the index invariant above. +- **A pack moves only names holding a word it changes.** The words whose compiled readings a choice changes are computable from the index, so a vocabulary pack's deviation surface is derived rather than hand-copied, and the non-interference gate in tests/v2/test_locales.py runs every corpus name without such a word under the pack and requires the default's reading. The packs whose behavior is policy (RU, TR_AZ) keep their hand-written `DEVIATES`. +- **What a pack does, read by shape.** `snapshot.py` and `snapshot.py --local … --base ` take a choice of languages and print the names it moves, grouped by changed field, as they do for the default; each pack PR reads its own ground truth beside that: nepal-candidates' script pairs for `ne` (mechanisms.md#PAIRED-SPELLINGS-PER-CHANGE), the CLDR agreement per pack where CLDR has the language, the PSC register's nationality slice where filers of that language are many. + +### What stays out of phase 3 + +- **Third-party language files.** A file joining the pool at run time would skip the checks that run across files, an exclusion in one file refusing a reading in any other (check 6) and check 11's particle statement. A third-party `nl` file adding the preposition `in` without a `given` reading or `may_be_given = false` is refused by the build and would ship on the unsafe side at run time. `Locale(lexicon=…)` stays the 2.x way to add vocabulary; a public file format is 3.0's question. +- **The Python packs' rule data.** ZH's 556 surnames are a segmentation list, outside the per-word model, and the RU and TR_AZ markers are endings inside rules (`-ovich`, `oğlu`); both wait for phase 4's `[rules]`. A pack gets a `languages` field only where measuring it shows what it moves. +- **Candidates that wait on a rule, not a word:** Turkish `Bey` and `Hanım` after the given name, Hungarian `úr` after the name, Thai `นาย` glued at the head, French `M.` against the initial it spells. Each is a position rule, phase 4's. +- **Same-language clashes.** No pack settles one (the `fils` argument): the Indic and Arabic titles that are names in their own language (`kumari`, `sayyid`) take phase 2j's route, a given reading and the title phrase, as default work. + +### The order of work + +Shape first, then the packs, each one PR measured on its own base, as phase 2's were. The first three move no default parse and no pack exists until 3c. + +| Sub-phase | What | Needs | +|---|---|---| +| 3a | the reading index and the shared compiler; `default = false` allowed on name readings; the no-choice invariant | — (neutral) | +| 3b | the choice: `Locale.languages`, `parser_for(strict=)`, tag selection, prefer and strict, decision 20 against the initial shape; the derived deviation surface; `snapshot.py` taking a choice | 3a (neutral) | +| 3c | the pack-shaped exclusions become `default = false` readings: cs `ml`, hr/sk/sl `st` and `ml`, fr `père` (seven exclusions whose reason opens "pack-shaped"), so an exclusion again means wrong everywhere | 3a (neutral) | +| 3d | `ne`: `given` on `श्री`, `guru` and `गुरु`, measured on nepal-candidates' script pairs | 3b | +| 3e | `hi` and `bn`, the Latin forms (`smt`, `shrimati`, `sushri`, `shriyut`, `shri`, `adhyapak`), with the names they collide with declared for strict (#345) | 3b | +| 3f | `cs`, `sk`, `sl`, `hr`: the generations of 3c, `pan`, `roz` and the academic abbreviations #848 left | 3b, 3c | +| 3g | `es` and `pt`: the occupational titles and `Don`/`Doña`/`Dona`, with a connective that counts only inside a title run, its own rule; `pt` also settles the one-case `e` | 3b | +| 3h | `ga` and `gd` (`nic`, `mhic`, `mic`, `uí`), `nl` (`an`, `in`, `ten`, `to`, `v`, with Korean and Vietnamese name readings declared), `pt-BR` (`F`, `N`, `sb`) | 3b | -**A pack can outrank a shape rule, not only a vocabulary reading** (decision 20). Shape rules (period abbreviation, initials, capitals) emit readings into the same pool as vocabulary from phase 2 on, rather than final answers, so a pack's preference can act on them. Phase 2 is to settle one such clash in the default, since every language loads there: Filipino `Ma.`, which abbreviates María, is to get a `given` reading in `fil.toml` that outranks the leading period-abbreviation rule (`Ma. Cristina Santos Reyes`, title `Ma.` until phase 2k gave it that reading on 2026-10-26); a pack's part is to prefer such readings where the default leaves both standing. +3g's connective needs a design of its own, because the pair that separates it is a count: `Profesora de Historia Ana López` should read title `Profesora de Historia`, while `Doctora de la Vega`, the same title, preposition and following word, has no name behind it and must keep family `de la Vega`. With `profesora` added and no rule, the first reads family `de Historia Ana López` (the rules/P1 fold), and `Ministro da Fazenda Carlos Souza` reads given `da`. The rule must reach only readings a pack turns on, so the default's parses stay where they are. -**What moves in phase 3:** the locale packs' vocabulary, still in Python today, becomes language files; `default = false`, given its meaning in phase 2, marks the readings a pack turns on, for words too risky even gated; and the reserved names `prefer` and `strict` gain their meaning. How a tag filters (does `pt` match `pt-BR`?) and how a third-party language file joins the pool and the snapshot are open under #753. +Languages with one to three candidates (`tr` `bay`, `it` `sig`, `lt` `jaun`, `ms` `bt`, `cy` `ab`/`ap`, `ber` `aït`) are a file each once 3b exists and need no sub-phase; `nb` and `sv`'s farm-name words wait for a measurement nobody has taken. ## Phase 4: order, shape and the output model @@ -227,7 +310,7 @@ Phase 4 also decides what a parse returns, aligned with CLDR's person-name field ## The first packs -Which packs come first once phase 3 exists is tracked in #747, not here. The surname-span measurement above is one input, but it was taken over corpora built to hold second surnames, so it shows that the gap exists, not how it ranks against the others. +The proposed order is phase 3's work order above (3d onward), led by `ne`, where the evidence is strongest; #747 tracks it. The surname-span measurement above is one input, but it was taken over corpora built to hold second surnames, so it shows that the gap exists, not how it ranks against the others. ## Where each fact lives From 73c99921962cf48cbc7fd068dbf57ac3542402df Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Tue, 27 Oct 2026 20:46:55 -0700 Subject: [PATCH 02/39] Phase 3 proposal: defer strict, name the amended contract, fix review findings (#753) Strict cannot be built on today's files (name readings sit where their count came from), so it waits; the add-only pack contract and the 3-0 Standing line are named as amended; counts, pointers and scope fixed. Co-Authored-By: Claude Opus 5.5 --- docs/design/decisions/vocabulary-model.md | 19 ++++++------ docs/design/vocabulary-model-rfc.md | 36 +++++++++++------------ 2 files changed, 28 insertions(+), 27 deletions(-) diff --git a/docs/design/decisions/vocabulary-model.md b/docs/design/decisions/vocabulary-model.md index ff7766b91..e7162d597 100644 --- a/docs/design/decisions/vocabulary-model.md +++ b/docs/design/decisions/vocabulary-model.md @@ -237,15 +237,16 @@ Open: #913 (the title words phase 2j left, read as titles though borne as names) - **2026-10-26 #752, phase 2l — THE BRACKET ESCAPE READS THE DERIVED CLASH**, carrying out item 16 of the 2026-10-23 #752 entry above (#810). A member of the ambiguous class, the class 2a derives from name readings, alone in brackets or quotes after the name is the suffix where its writing settles it (`John Smith (MA)`, `John Smith (M.A)` → suffix), read in one pass, and stays the nickname otherwise (`John Smith (Ed)`, `JOHN SMITH (MA)`, `Jeffrey (JD) Bricken`). It is recorded as rules/S13, a rule of its own rather than the amendment to S1 item 16 planned, the position it reads being evidence S1's signal does not read; the writing is S3's and S6's evidence, which S13 cites. The record, the measurements and what was declined are decisions/S13's 2026-10-26 #810 entry. - **2026-10-26 #752, phase 2m — THE READING NOT TAKEN SURFACES AS `Ambiguity.alternative`, WITHOUT ITS LANGUAGES**, carrying out item 13 of the 2026-10-23 #752 entry above and item 7's source with it: a word the credential's run took by position alone reports its name reading's source as `position`, a fourth source beside the three the item named, and a word with a declared name reading reports `declared`. The pickle guard was widened as item 13 asked, every skew the narrow guard refused still refused. The reading's languages were left out: the per-word index costs about 0.4 ms of a 19.5 ms import and describes the default vocabulary, not a pack's or a caller's lexicon, so they wait for 3.0 with `Token.reading`. The record, the measurements and what was declined are decisions/A1's 2026-10-26 #752 entry. -- **2026-10-27 #753 — PHASE 3'S DESIGN IS PROPOSED, NOT AGREED: A CHOICE OF LANGUAGES SELECTS THE POOL, ONE MECHANISM FOR TURNING READINGS ON, PREFER AND STRICT; NAME READINGS IN A PACK KEEP THE GATE ON; A READING INDEX CARRIES THE LANGUAGES TO RUN TIME.** Every item below is a proposal awaiting Derek's decision, and this head is rewritten when he makes it. The design overview is docs/design/vocabulary-model-rfc.md's Phase 3 section. Decisions 6, 15 and 20 are kept as written; the items say how a pack carries them out. The inputs are #753's comments of 2026-10-27: the inventory of 411 kept-out candidate readings with their verdicts, and the probe of foreign titles and function-word collisions. Measurements were taken 2026-10-27 on master `cfe8bf92` (Python 3.11.17) with throwaway Lexicon edits standing in for a pack, so each is evidence for the proposal and each sub-phase re-measures on its own base. Local-corpus figures are counts. - 1. **Proposed: a pack is a choice of languages carried by `Locale`.** `Locale` gains `languages`, a set of BCP 47 tags; `parser_for` unions the sets of the packs it is given, so the choice is order-independent while policy patches still fold in order; strict is an argument of the whole choice (`parser_for(locales.NE, locales.HI, strict=True)`), since a strict pack beside a preferring one has no meaning. The shipped language packs are `Locale` constants with an empty lexicon fragment (`locales.NE`, `locales.PT_BR`). Declined: `Parser(languages=…)` beside `Locale`, two ways to choose one thing, and a `Policy` field, `Policy` being behavior and vocabulary being `Lexicon`'s. - 2. **Proposed: one mechanism.** The chosen languages select which readings the compiler compiles, the default being that compiler with every language and no choice. Turning on, prefer and strict are derived from the file each reading sits in, and nothing is written per pack beyond its language's readings: a chosen language's `default = false` readings join; prefer drops other languages' vocabulary readings of a word on which a chosen language holds one, never `mul.toml`'s (`Haddad, Abd`, suffix `Abd` today, reads given `Abd` with the en credential gone); strict drops other languages' name readings and the markers they derive. A clash a property records with a SHAPE reading is settled for the chosen language's own reading, decision 20 applied: `may_be_initial` does not hold on it, so a `pt` choice reads `ANA SILVA E SOUZA` family `SILVA E SOUZA` where the default reads middle `SILVA E` with a report. This answers decisions/3-0-reevaluations' 2026-09-13 bullet that a pack cannot remove `e`, and its 2026-08-17 bullet's "overrides rather than additions": a pack overrides by selection, never by a removal list. Left open rather than built: a property a language's own reading should take only under its pack, Swedish `fru` and Icelandic `frú` as after-given salutations (rules/H8), refused in the default for the Cameroonian surname, the two readings in the inventory needing it. - 3. **Proposed: `default = false` is allowed on `given` and `family` readings, and turned on they derive markers like any name reading, never settling (decision 15 unchanged).** Measured on nepal-candidates' 126,981 Devanagari and romanized pairs, comparing every word's role across the two spellings: 123,440 read alike in the default; `given` readings on `श्री`, `guru` and `गुरु`, which make each a title only inside a title phrase (rules/H9), give 123,636 (198 fixed, 2 broken, both pairs whose romanization opens with a listed title); a `family` reading on `dc` as well, making the credential ambiguous, gives 123,639, the count still leaving `Ram Bahadur DC` a suffix; dropping `dc`'s credential, which settling by a name reading would do, gives 123,658, 19 more. Settling is not proposed: the 19 pairs are the price of decision 15, whose `do` case it protects. `bk`, `kc`, `gc` and `bc` read as the family in every shape tried (spoken order, a family comma, `FAMILY_FIRST`, dotted and Title case), and decision 11 refuses a name reading on a word no file claims as vocabulary, so none is proposed. Recompute: the readings moved into or out of `Lexicon` fields by hand, the pairs compared by `tuple(t.role for t in parser.parse(s).tokens)`. - 4. **Proposed: strict needs no sweep first.** Check 11 makes every particle reading, `default = false` included, carry a `given` reading on its word or state `may_be_given = false`, so the PR turning on `nl` `in` and `to` declares the Korean and Vietnamese given readings they collide with, with counts; in the default those readings derive nothing, the word having no live vocabulary reading there. - 5. **Proposed: the languages reach run time as a reading index.** A second generated module, one row per reading (text, tag, kind, behavioral properties), loaded only when a language is chosen; the compiler moves from tools/vocab/build.py into the package so build and pack share one implementation, and a test holds that compiling the index with no choice reproduces `_compiled.py`. It is the per-word index item 13 of the 2026-10-23 #752 entry says `Token.reading` needs. A choice is applied as a delta against the default's compile, so it composes with a caller's `base=` lexicon and a `Locale`'s own fragment. Measured: 2,140 readings over 1,797 words, 106 in more than one file; the index about 93 KB of source and 11 ms to execute without a bytecode cache, the compile 9 ms, both once per choice and only for a caller who makes one; loading the TOML 39 ms. Declined: per-language deltas compiled at build time, which do not compose (under prefer `{pt, en}` `do` keeps its particle and its credential, while Portuguese's delta drops the credential and English's the particle); every combination precompiled, combinatorial; the TOML read at run time, slower and either re-running the build's checks or skipping them. +- **2026-10-27 #753 — PHASE 3'S DESIGN IS PROPOSED, NOT AGREED: A CHOICE OF LANGUAGES SELECTS THE POOL, ONE MECHANISM FOR TURNING READINGS ON AND PREFER; NAME READINGS IN A PACK KEEP THE GATE ON; A READING INDEX CARRIES THE LANGUAGES TO RUN TIME; STRICT WAITS.** Every item below is a proposal awaiting Derek's decision, and this head is rewritten when he makes it. The design overview is docs/design/vocabulary-model-rfc.md's Phase 3 section. Decisions 15 and 20 are kept as written; decision 6's strict variant is reshaped to a property of the whole choice and deferred (items 1 and 4). The inputs are #753's comments of 2026-10-27: the inventory of 411 kept-out candidate readings with their verdicts, and the probe of foreign titles and function-word collisions. Measurements were taken 2026-10-27 on master `cfe8bf92` (Python 3.11.17) with throwaway Lexicon edits standing in for a pack, so each is evidence for the proposal and each sub-phase re-measures on its own base. Local-corpus figures are counts. + 1. **Proposed: a pack is a choice of languages carried by `Locale`.** `Locale` gains `languages`, a set of BCP 47 tags; `parser_for` unions the sets of the packs it is given, so the choice is order-independent while policy patches still fold in order; strict is an argument of the whole choice (`parser_for(locales.NE, locales.HI, strict=True)`), since a strict pack beside a preferring one has no meaning, reshaping decision 6's "strict variant". The shipped language packs are `Locale` constants with an empty lexicon fragment (`locales.NE`, `locales.PT_BR`). Weighed against: `Parser(languages=…)` beside `Locale`, two ways to choose one thing, and a `Policy` field, `Policy` being behavior and vocabulary being `Lexicon`'s. + 2. **Proposed: one mechanism.** The chosen languages select which readings the compiler compiles, the default being that compiler with every language and no choice. Turning on, prefer and strict are derived from the file each reading sits in, and nothing is written per pack beyond its language's readings: a chosen language's `default = false` readings join; prefer drops other languages' vocabulary readings of a word on which a chosen language holds one, never `mul.toml`'s (`Haddad, Abd`, suffix `Abd` today, reads given `Abd` with the en credential gone); strict drops other languages' name readings and the markers they derive. A clash a property records with a SHAPE reading is settled for the chosen language's own reading, decision 20 applied: `may_be_initial` does not hold on it, so a `pt` choice reads `ANA SILVA E SOUZA` family `SILVA E SOUZA` where the default reads middle `SILVA E` with a report. This answers decisions/3-0-reevaluations' 2026-09-13 bullet that a pack cannot remove `e`, and its 2026-08-17 bullet's "overrides rather than additions": a pack overrides by selection, never by a removal list. That removes base entries all the same, so mechanisms.md#LOCALE-PACKS-PURE-DATA's add-only contract is amended with it (item 12). The shape-clash rule follows from decision 7, which decision 20 permits; it reaches every reading `may_be_initial` stands on (pt and it `e`, ca and pl `i`, the Italian one unjudged), and it forecloses an `nl` pack marking `y` (the 2026-08-18 bullet there), which waits with `fru` below. New beside decision 6: `mul.toml`'s exemption from prefer. Left open rather than built: a property a language's own reading should take only under its pack, Swedish `fru` and Icelandic `frú` as after-given salutations (rules/H8), refused in the default for the Cameroonian surname, the two readings in the inventory needing it. + 3. **Proposed: `default = false` is allowed on `given` and `family` readings, and turned on they derive markers like any name reading, never settling (decision 15 unchanged).** Measured on nepal-candidates' 126,981 Devanagari and romanized pairs, comparing every word's role across the two spellings: 123,440 read alike in the default; `given` readings on `श्री`, `guru` and `गुरु`, which make each a title only inside a title phrase (rules/H9), give 123,636 (198 fixed, 2 broken, both pairs whose romanization opens with a listed title); a `family` reading on `dc` as well, making the credential ambiguous, gives 123,639, the count still leaving `Ram Bahadur DC` a suffix; dropping `dc`'s credential, which settling by a name reading would do, gives 123,658, 19 more. Settling is not proposed: the 19 pairs are the price of decision 15, whose `do` case it protects. `bk`, `kc`, `gc` and `bc` read as the family in every shape tried (spoken order, a family comma, `FAMILY_FIRST`, dotted and Title case), so a reading would move nothing and none is proposed. Recompute: the readings moved into or out of `Lexicon` fields by hand, the pairs compared by `tuple(t.role for t in parser.parse(s).tokens)`. + 4. **Proposed: strict waits, out of 2.8, until name readings say whose names they are.** A name reading sits in the file whose source counted it: 113 of the 211 name readings are in en.toml, `do`'s Vietnamese Đỗ readings among them, vi.toml holding no `do`, so strict built on the files drops them under a `vi` choice, and simulated, `Minh Do` reads suffix `Do` and `Do Van Minh` family `Do Van Minh`, decision 15's harm. A census count belongs to no one language either. The candidates needing strict wait with it, `nl`'s prepositions among them, which need more than strict: turned on as possibly-given particles, `in` and `to` read `Park In Hwan` family `In Hwan` and `Nguyen To Nga` family `To Nga` (the marker acts at the head only), and check 11 accepts `may_be_given = false` in place of the Korean and Vietnamese readings. Recompute: count `given` and `family` readings per file with tools/vocab/build.py's `load`; simulate by removing `do` from `suffix_acronyms_ambiguous` and `particles_ambiguous`. + 5. **Proposed: the languages reach run time as a reading index.** A second generated module, one row per reading (text, tag, kind, behavioral properties), loaded only when a language is chosen; the compiler moves from tools/vocab/build.py into the package so build and pack share one implementation, and a test holds that compiling the index with no choice reproduces `_compiled.py`. It is the per-word index item 13 of the 2026-10-23 #752 entry says `Token.reading` needs. A choice is applied as a delta against the default's compile, so it composes with a caller's `base=` lexicon and a `Locale`'s own fragment. Measured: 2,140 readings over 1,797 words, 106 in more than one file; the index about 93 KB of source and 11 ms to execute without a bytecode cache, the compile 9 ms, both once per choice and only for a caller who makes one; loading the TOML 39 ms. Weighed against: per-language deltas compiled at build time, which do not compose (under prefer `{pt, en}` `do` keeps its particle and its credential, while Portuguese's delta drops the credential and English's the particle); every combination precompiled, combinatorial; the TOML read at run time, slower and either re-running the build's checks or skipping them. 6. **Proposed: a chosen tag selects its file and its ancestors', never its descendants, and no macrolanguage table is read.** `pt-BR` selects `pt-BR.toml` and `pt.toml`; `pt` only `pt.toml`, so a caller in Portugal does not turn on the Brazilian agnomens (#843). The two files share no word today. Malay and Indonesian, which share `puan`, `putra` and `putri`, are chosen together by a caller who has both. The TOML names `prefer` and `strict` stay refused, the message saying a strength is the caller's choice and never data. - 7. **Proposed: a pack is measured three ways.** The default does not move (the committed snapshot, and the index invariant of item 5); a vocabulary pack's deviation surface, the words whose compiled readings its choice changes, is derived from the index, and tests/v2/test_locales.py's non-interference gate requires the default's reading for every corpus name without such a word, the policy packs (RU, TR_AZ) keeping their written `DEVIATES`; and `snapshot.py`, `--local` with `--base` included, takes a choice and prints what it moves by field, beside each pack's own ground truth (nepal-candidates' script pairs for `ne`, the CLDR agreement per pack, PSC's nationality slice). This answers decisions/3-0-reevaluations' 2026-09-13 bullet that the harness cannot see a pack. + 7. **Proposed: a pack is measured three ways.** The default does not move (the committed snapshot, and the index invariant of item 5); a vocabulary pack's deviation surface, the words whose compiled readings its choice changes, is derived from the index, and tests/v2/test_locales.py's non-interference gate requires the default's reading for every corpus name without such a word, the policy and segmentation packs (JA, RU, TR_AZ, ZH) keeping their written `DEVIATES`; and `snapshot.py`, `--local` with `--base` included, takes a choice and prints what it moves by field, beside each pack's own ground truth (nepal-candidates' script pairs for `ne`, the CLDR agreement per pack, PSC's nationality slice). This answers decisions/3-0-reevaluations' 2026-09-13 bullet that the harness cannot see a pack. 8. **Proposed: third-party language files are deferred to 3.0.** A file joining the pool at run time would skip the checks that run across files (check 6's exclusions, check 11's particle statement), the ones that refuse a third-party `nl` `in` written with neither a `given` reading nor `may_be_given = false`. `Locale(lexicon=…)` stays the 2.x way to add vocabulary. - 9. **Proposed: the pack-shaped exclusions become `default = false` readings,** so an `excluded` key again means wrong everywhere: the seven whose reason opens "pack-shaped" (cs `ml`, hr, sk and sl `st` and `ml`, fr `père`), the shape cs `st` already has. Recompute: `grep -c 'pack-shaped' nameparser/vocab/*.toml`. + 9. **Proposed: every exclusion a pack turns on becomes a `default = false` reading,** the shape cs `st` already has, so that once the packs land an `excluded` key means wrong everywhere: the eight whose reason opens "pack-shaped" in 3c (cs `ml`, hr, sk and sl `st` and `ml`, fr `père`), and the rest in the PR of the pack that turns each on (`pan`, `roz`, `nic`, `shri`, `sig`, `jaun`, `bt`, `bay`, `sb`). Recompute: `grep -h 'pack-shaped' nameparser/vocab/*.toml | wc -l`. 10. **Proposed: what stays out of phase 3.** ZH's surname list and the RU and TR_AZ endings are rule data for phase 4's `[rules]`, and a Python pack gains `languages` only where a measurement shows what it moves; the candidates that wait on a position rule (Turkish `Bey` and `Hanım`, Hungarian `úr`, Thai `นาย`, French `M.`) are phase 4's; same-language clashes (`kumari`, `sayyid`) take phase 2j's route in the default, since no pack settles one. - 11. **Proposed: the work order,** shape first and one PR each, measured on its own base: 3a the index and shared compiler with name readings allowed `default = false` (neutral); 3b the choice, tag selection, prefer, strict, the shape clash, the derived deviation surface and `snapshot.py`'s choice (neutral); 3c the pack-shaped exclusions converted (neutral); then the packs, `ne` (3d), `hi` and `bn`'s Latin forms (3e), `cs`, `sk`, `sl` and `hr` (3f), `es` and `pt` with a connective that counts only inside a title run (3g), and `ga`/`gd`, `nl` and `pt-BR` (3h). 3g's connective is a rule with its own design: with `profesora` added and no rule, `Profesora de Historia Ana López` reads family `de Historia Ana López` (the rules/P1 fold) and `Ministro da Fazenda Carlos Souza` given `da`, while `Doctora de la Vega` must keep family `de la Vega`, the two separated by whether a name stands behind the phrase; it must reach only readings a pack turns on. + 11. **Proposed: the work order,** shape first and one PR each, measured on its own base: 3a the index and shared compiler with name readings allowed `default = false` (neutral); 3b the choice, tag selection, turning on, prefer, the shape clash, the derived deviation surface, `snapshot.py`'s choice and the amended contract (neutral); 3c the pack-shaped exclusions converted (neutral); then the packs, `ne` (3d), `hi` and `bn`'s Latin forms (3e), `cs`, `sk`, `sl` and `hr` (3f), `es` and `pt` with a connective that counts only inside a title run (3g), and `ga`/`gd` and `pt-BR` (3h). 3g's connective is a rule with its own design: with `profesora` added and no rule, `Profesora de Historia Ana López` reads family `de Historia Ana López` (the rules/P1 fold) and `Ministro da Fazenda Carlos Souza` given `da`, while `Doctora de la Vega` must keep family `de la Vega`, the two separated by whether a name stands behind the phrase; it must reach only readings a pack turns on. + 12. **Proposed: phase 3 ships in 2.x, opt-in, the default's parses unmoved, and two standing texts are amended with 3b:** mechanisms.md#LOCALE-PACKS-PURE-DATA's contract that a pack is add-only and declares its deviations in its docstring (a pack adds, and selects among the default's readings by language, a vocabulary pack's deviations derived), and decisions/3-0-reevaluations' Standing line that removing base vocabulary through a pack is accepted for 2.x and part of 3.0's design. diff --git a/docs/design/vocabulary-model-rfc.md b/docs/design/vocabulary-model-rfc.md index dda7145df..1e50e0b92 100644 --- a/docs/design/vocabulary-model-rfc.md +++ b/docs/design/vocabulary-model-rfc.md @@ -164,7 +164,7 @@ Each sub-phase is one PR, measured on its own base: the committed snapshot read ## Phase 3: packs that prefer or restrict -Status: **proposed, not agreed** (2026-10-27, #753). The record is decisions/vocabulary-model's 2026-10-27 #753 entry, whose items are each marked proposed until Derek decides them; this section is the proposal those items describe. Phase 3 ships in 2.x, opt-in: a caller who chooses no language parses exactly as before, and nothing in this phase moves the default's parses. +Status: **proposed, not agreed** (2026-10-27, #753). The record is decisions/vocabulary-model's 2026-10-27 #753 entry, whose items are each marked proposed until Derek decides them; this section is the proposal those items describe. Phase 3 ships in 2.x, opt-in: a caller who chooses no language parses exactly as before, and nothing in this phase moves the default's parses. Two standing texts would be amended with it: mechanisms.md#LOCALE-PACKS-PURE-DATA's contract that a pack is add-only and declares its deviations in its docstring (a pack would add, and select among the default's readings by language, its deviations derived for a vocabulary pack), and decisions/3-0-reevaluations' line that removing base vocabulary through a pack waits for 3.0. The semantics were fixed in 2026-10-15/16, and phase 3 keeps them. A pack is a choice of languages, in two strengths (decision 6): @@ -175,7 +175,7 @@ The asymmetry is deliberate (decision 15): **vocabulary readings settle a clash; Consequences that follow from the definitions: -- **Order does not matter.** Prefer and strict are sets of languages, so combining packs is commutative. +- **Order does not matter.** A choice is a set of languages, so combining packs is commutative. Proposed below: strictness belongs to the whole choice rather than to one pack, reshaping decision 6's "strict variant", and strict waits until name readings say whose names they are. - **Two chosen languages that each hold a vocabulary reading both keep it**, and the word stays gated, as in the default. - **Non-Latin vocabulary stays on under any choice**, because it selects itself by script (`씨`, `さん`, Cyrillic titles). Almost every clash between languages is within one script: `do`, `e`, `van` are all Latin. - **Whether a word needs a pack depends on where its clash is.** `do` clashes between languages, so a pack settles it. `fils` clashes within French, a generation word and a French surname, so no pack settles it and the phase 2 gate does the work, reversing the premise of #684. @@ -190,13 +190,13 @@ Everything a pack does is one operation: **the chosen languages select which rea | Effect | Which readings | Example | |---|---|---| | turn on | a chosen language's `default = false` readings join the pool | cs `ml` (*mladší*, once 3c makes it a reading): `Jan Novák ml.` reads family `ml.` today; under `cs`, suffix `ml.` | -| prefer | on a word where a chosen language holds a vocabulary reading, other languages' vocabulary readings leave; `mul.toml`'s readings never do, being every language's | `Haddad, Abd` reads suffix `Abd` and no given name today; under `ar` the bound given reading stays and the en credential leaves; cs `st`: `Jiří Dienstbier st.` reads title `st.` today (rules/H5), and under `cs` its generation reading joins and the Saint readings leave, suffix `st.` | +| prefer | on a word where a chosen language holds a vocabulary reading, other languages' vocabulary readings leave; `mul.toml`'s readings never do, being every language's | `Haddad, Abd` reads suffix `Abd` and no given name today; under `ar` the bound given reading stays and the en credential leaves; cs `st`: `Jiří Dienstbier st.` reads title `st.` today (rules/H5), and under `cs` its generation reading joins and the Saint readings leave, suffix `st.`, reported as suffix-or-name since en's `given` reading of `st` still gates it | | prefer, against a shape | a clash a property records with a SHAPE reading is settled for the chosen language's own reading, by decision 20: `may_be_initial` does not hold on it | `ANA SILVA E SOUZA` reads middle `SILVA E` with a conjunction-or-initial report today; under `pt` the conjunction holds, family `SILVA E SOUZA` | -| strict | other languages' name readings leave, and with them the markers they derive | an `nl` `in` turned on, with Korean In declared as a `given` reading, is a particle that may be given under `nl` and never-given under strict `nl` | +| strict, deferred (below) | other languages' name readings leave, and with them the markers they derive | an `nl` `in` turned on, with Korean In declared as a `given` reading, is a particle that may be given under `nl` and never-given under strict `nl` | -The third row answers decisions/3-0-reevaluations' "a pack cannot remove `e`": `may_be_initial` records a clash between a conjunction and the initial a letter spells, and the initial is a shape reading, so a pack that prefers Portuguese settles it as decision 20 says a pack may. No property is toggled per pack, and no reading is written twice. +The third row answers decisions/3-0-reevaluations' "a pack cannot remove `e`": `may_be_initial` records a clash between a conjunction and the initial a letter spells, and the initial is a shape reading, so a pack that prefers Portuguese settles it, as decision 7 says choosing a language does and decision 20 permits. It reaches every reading the property stands on, pt and it `e` and ca and pl `i`, the Italian one unjudged by the inventory; and it forecloses the other direction of the 2026-08-18 bullet there, an `nl` pack marking `y` as an initial, which waits with the `fru` question below. Two parts of the mechanism are new beside decision 6: the exemption of `mul.toml` from prefer (under `ca`, `i` keeps its mul generation), and strictness as a property of the whole choice. No property is toggled per pack, and no reading is written twice. -Properties ride on their readings, so a choice that keeps a reading keeps its properties. The one shape this cannot express is a property a language's own reading should take only under its pack: Swedish `fru` and Icelandic `frú` as after-given salutations (rules/H8), refused in the default because Fru is also a Cameroonian surname. Two readings need it, so the proposal builds nothing for them and leaves them open. +Properties ride on their readings, so a choice that keeps a reading keeps its properties. The one shape this cannot express is a property a language's own reading should take only under its pack: Swedish `fru` and Icelandic `frú` as after-given salutations (rules/H8), refused in the default because Fru is also a Cameroonian surname. The inventory found two readings needing it (2026-10-27), so the proposal builds nothing for them and leaves them open. ### Name readings in a pack @@ -209,13 +209,13 @@ Properties ride on their readings, so a choice that keeps a reading keeps its pr | and `family` on `dc` | the family reading only makes the credential ambiguous: `Ram Bahadur DC` still reads suffix `DC`, the count leaving it one | 123,639 (3 more fixed) | | `dc`'s credential dropped, as settling would do | `Ram Bahadur DC` → family `DC` | 123,658 (19 more fixed) | -The 2 broken are pairs whose romanization opens with a listed title, which reads the `श्री` as a salutation (#870 counted `Mr` written for it twice). So the gate route delivers the `श्री` and `guru` fix without amending decision 15, and the 19 `dc` pairs it leaves are the price of keeping it. `bk`, `kc`, `gc` and `bc` need nothing: they read as the family in every shape tried (`Ram Bahadur BK`, `Ram Bahadur Kc`, `BK, Ram Bahadur`, `KC Sita` under `FAMILY_FIRST`), and decision 11 refuses a name reading on a word no file claims as vocabulary. +The 2 broken are pairs whose romanization opens with a listed title, which reads the `श्री` as a salutation (#870 counted `Mr` written for it twice). So the gate route delivers the `श्री` and `guru` fix without amending decision 15, and the 19 `dc` pairs it leaves are the price of keeping it. `bk`, `kc`, `gc` and `bc` need nothing: they read as the family in every shape tried (`Ram Bahadur BK`, `Ram Bahadur Kc`, `BK, Ram Bahadur`, `KC Sita` under `FAMILY_FIRST`), so a reading would move nothing. -A strict pack drops only the name readings that are recorded. The build already makes the author record them: a particle reading, `default = false` included, needs a `given` reading on its word or states `may_be_given = false` (check 11). So the pack PR that turns on `nl` `in` and `to` declares the Korean and Vietnamese given readings they collide with, with their counts, and no separate sweep comes first. In the default those readings derive nothing, since the word has no live vocabulary reading there. +**Strict cannot be built on today's files.** A name reading sits in the file whose source counted it, not the file of the language whose names carry it: 113 of the 211 name readings are in en.toml (2026-10-27), `do`'s Vietnamese Đỗ readings among them, and vi.toml holds no `do`. A strict `vi` choice built on the files would drop exactly those readings, and with them the gate: simulated, `Minh Do` reads suffix `Do` and `Do Van Minh` family `Do Van Minh`, the harm decision 15 is there to prevent. Nor does a census count belong to one language. So strict is proposed to wait, out of 2.8, until name readings say whose names they are, and the candidates that need it wait with it. The `nl` prepositions show they would need more than strict: turned on as possibly-given particles, `in` and `to` already read `Park In Hwan` family `In Hwan` and `Nguyen To Nga` family `To Nga`, the marker acting at the head only, and check 11 lets their author write `may_be_given = false` instead of declaring the Korean and Vietnamese readings. ### Where the provenance lives at runtime -The compiled module holds sets, and a set has forgotten which file each word came from. The proposal compiles a second module beside it: a **reading index**, one row per reading (its text, its file's tag, its kind and its behavioral properties), loaded only when a caller chooses a language. The compiler moves from tools/vocab/build.py into the package so that the build and a pack share one implementation, and a test holds the invariant that compiling the index with no choice reproduces `nameparser/config/_compiled.py` exactly. The same index is what decision 13 says `Token.reading` needs in 3.0. +The compiled module holds sets, and a set has forgotten which file each word came from. The proposal compiles a second module beside it: a **reading index**, one row per reading (its text, its file's tag, its kind and its behavioral properties), loaded only when a caller chooses a language. The compiler moves from tools/vocab/build.py into the package so that the build and a pack share one implementation, and a test holds the invariant that compiling the index with no choice reproduces `nameparser/config/_compiled.py` exactly. The same index is what item 13 of decisions/vocabulary-model's 2026-10-23 #752 entry says `Token.reading` needs in 3.0. Measured 2026-10-27 on `cfe8bf92`: the pool holds 2,140 readings over 1,797 words, 106 of them in more than one file. An index of their behavioral properties is about 93 KB of generated source, executing in about 11 ms with no bytecode cache, and compiling the pool takes about 9 ms, both paid once per choice and only by a caller who makes one. Reading the TOML instead costs about 39 ms. Recompute: the build's `read_dir`, `load` and `compile_pool`, timed. @@ -225,9 +225,9 @@ A choice is applied as a delta against the default's compile, so it composes wit ### The API and tags -A choice is carried by `Locale`, which gains a `languages` field, a set of BCP 47 tags. `parser_for` unions the `languages` of the packs it is given, so the choice is order-independent while their policy patches still fold in order. Strict is a property of the whole choice, not of one pack, and is an argument: `parser_for(locales.NE, locales.HI, strict=True)`. The shipped language packs are `Locale` constants like the four today (`locales.NE`, `locales.get("ne")`), each with an empty lexicon fragment; a code keeps `Locale`'s `[a-z0-9_]+`, so `pt-BR` is `locales.PT_BR`. +A choice is carried by `Locale`, which gains a `languages` field, a set of BCP 47 tags. `parser_for` unions the `languages` of the packs it is given, so the choice is order-independent while their policy patches still fold in order. Strict, once it can be built, is a property of the whole choice, not of one pack, and an argument: `parser_for(locales.NE, locales.HI, strict=True)`. The shipped language packs are `Locale` constants like the four today (`locales.NE`, `locales.get("ne")`), each with an empty lexicon fragment; a code keeps `Locale`'s `[a-z0-9_]+`, so `pt-BR` is `locales.PT_BR`. -A chosen tag selects its own file and its ancestors', never its descendants': `pt-BR` selects `pt-BR.toml` and `pt.toml`, and `pt` selects only `pt.toml`, so a caller in Portugal does not turn on the Brazilian agnomens (#843 left the single-letter `F` and `N` to a `pt-BR` pack). No macrolanguage table is read: Malay and Indonesian share `puan`, `putra` and `putri`, and a caller with both chooses both. The Czech, Slovak, Slovenian and Croatian files each hold the words they share, so each language is a pack of its own. Script subtags need no matching rule yet, since no file carries one. +A chosen tag selects its own file and its ancestors', never its descendants': `pt-BR` selects `pt-BR.toml` and `pt.toml`, and `pt` selects only `pt.toml`, so a caller in Portugal does not turn on the Brazilian agnomens (#843 left the single-letter `F` and `N` to a `pt-BR` pack). No macrolanguage table is read: Malay and Indonesian are separate tags, though BCP 47 counts Indonesian within the Malay macrolanguage, and a caller with names in both chooses both. The Czech, Slovak, Slovenian and Croatian files each hold the words they share, so each language is a pack of its own. Script subtags need no matching rule yet, since no file carries one. The TOML names `prefer` and `strict`, reserved for phase 3 since phase 1, stay refused, with a message saying that a strength is the caller's choice and never data. @@ -236,30 +236,30 @@ The TOML names `prefer` and `strict`, reserved for phase 3 since phase 1, stay r The snapshot and `compare.py` parse under the default, so neither can see a pack (decisions/3-0-reevaluations). Three measures, the first two in CI: - **The default does not move.** The committed snapshot holds it, as for any change, and so does the index invariant above. -- **A pack moves only names holding a word it changes.** The words whose compiled readings a choice changes are computable from the index, so a vocabulary pack's deviation surface is derived rather than hand-copied, and the non-interference gate in tests/v2/test_locales.py runs every corpus name without such a word under the pack and requires the default's reading. The packs whose behavior is policy (RU, TR_AZ) keep their hand-written `DEVIATES`. +- **A pack moves only names holding a word it changes.** The words whose compiled readings a choice changes are computable from the index, so a vocabulary pack's deviation surface is derived rather than hand-copied, and the non-interference gate in tests/v2/test_locales.py runs every corpus name without such a word under the pack and requires the default's reading. The packs whose behavior is policy or segmentation (JA, RU, TR_AZ, ZH) keep their hand-written `DEVIATES`. - **What a pack does, read by shape.** `snapshot.py` and `snapshot.py --local … --base ` take a choice of languages and print the names it moves, grouped by changed field, as they do for the default; each pack PR reads its own ground truth beside that: nepal-candidates' script pairs for `ne` (mechanisms.md#PAIRED-SPELLINGS-PER-CHANGE), the CLDR agreement per pack where CLDR has the language, the PSC register's nationality slice where filers of that language are many. ### What stays out of phase 3 - **Third-party language files.** A file joining the pool at run time would skip the checks that run across files, an exclusion in one file refusing a reading in any other (check 6) and check 11's particle statement. A third-party `nl` file adding the preposition `in` without a `given` reading or `may_be_given = false` is refused by the build and would ship on the unsafe side at run time. `Locale(lexicon=…)` stays the 2.x way to add vocabulary; a public file format is 3.0's question. -- **The Python packs' rule data.** ZH's 556 surnames are a segmentation list, outside the per-word model, and the RU and TR_AZ markers are endings inside rules (`-ovich`, `oğlu`); both wait for phase 4's `[rules]`. A pack gets a `languages` field only where measuring it shows what it moves. +- **The Python packs' rule data.** ZH's surnames (556 on 2026-10-27) are a segmentation list, outside the per-word model, and the RU and TR_AZ markers are endings inside rules (`-ovich`, `oğlu`); both wait for phase 4's `[rules]`. A pack gets a `languages` field only where measuring it shows what it moves. - **Candidates that wait on a rule, not a word:** Turkish `Bey` and `Hanım` after the given name, Hungarian `úr` after the name, Thai `นาย` glued at the head, French `M.` against the initial it spells. Each is a position rule, phase 4's. - **Same-language clashes.** No pack settles one (the `fils` argument): the Indic and Arabic titles that are names in their own language (`kumari`, `sayyid`) take phase 2j's route, a given reading and the title phrase, as default work. ### The order of work -Shape first, then the packs, each one PR measured on its own base, as phase 2's were. The first three move no default parse and no pack exists until 3c. +Shape first, then the packs, each one PR measured on its own base, as phase 2's were. The first three move no default parse, and no pack exists until 3d. Strict is not in 2.8's order (above). | Sub-phase | What | Needs | |---|---|---| | 3a | the reading index and the shared compiler; `default = false` allowed on name readings; the no-choice invariant | — (neutral) | -| 3b | the choice: `Locale.languages`, `parser_for(strict=)`, tag selection, prefer and strict, decision 20 against the initial shape; the derived deviation surface; `snapshot.py` taking a choice | 3a (neutral) | -| 3c | the pack-shaped exclusions become `default = false` readings: cs `ml`, hr/sk/sl `st` and `ml`, fr `père` (seven exclusions whose reason opens "pack-shaped"), so an exclusion again means wrong everywhere | 3a (neutral) | +| 3b | the choice: `Locale.languages`, tag selection, turning on and prefer, the shape clash of `may_be_initial`; the derived deviation surface; `snapshot.py` taking a choice; mechanisms.md#LOCALE-PACKS-PURE-DATA amended | 3a (neutral) | +| 3c | the eight exclusions whose reason opens "pack-shaped" become `default = false` readings: cs `ml`, hr, sk and sl `st` and `ml`, fr `père`; every other exclusion a pack turns on (`pan`, `roz`, `nic`, `shri`, `sig`, `jaun`, `bt`, `bay`, `sb`) converts in the pack's own PR, so that after the packs an exclusion means wrong everywhere | — (neutral) | | 3d | `ne`: `given` on `श्री`, `guru` and `गुरु`, measured on nepal-candidates' script pairs | 3b | -| 3e | `hi` and `bn`, the Latin forms (`smt`, `shrimati`, `sushri`, `shriyut`, `shri`, `adhyapak`), with the names they collide with declared for strict (#345) | 3b | +| 3e | `hi` and `bn`, the Latin forms (`smt`, `shrimati`, `sushri`, `shriyut`, `shri`, `adhyapak`), the ones whose collisions prefer cannot settle left for strict (#345) | 3b | | 3f | `cs`, `sk`, `sl`, `hr`: the generations of 3c, `pan`, `roz` and the academic abbreviations #848 left | 3b, 3c | | 3g | `es` and `pt`: the occupational titles and `Don`/`Doña`/`Dona`, with a connective that counts only inside a title run, its own rule; `pt` also settles the one-case `e` | 3b | -| 3h | `ga` and `gd` (`nic`, `mhic`, `mic`, `uí`), `nl` (`an`, `in`, `ten`, `to`, `v`, with Korean and Vietnamese name readings declared), `pt-BR` (`F`, `N`, `sb`) | 3b | +| 3h | `ga` and `gd` (`nic`, `mhic`, `mic`, `uí`), `pt-BR` (`F`, `N`, `sb`); `nl`'s prepositions wait with strict | 3b | 3g's connective needs a design of its own, because the pair that separates it is a count: `Profesora de Historia Ana López` should read title `Profesora de Historia`, while `Doctora de la Vega`, the same title, preposition and following word, has no name behind it and must keep family `de la Vega`. With `profesora` added and no rule, the first reads family `de Historia Ana López` (the rules/P1 fold), and `Ministro da Fazenda Carlos Souza` reads given `da`. The rule must reach only readings a pack turns on, so the default's parses stay where they are. From 77f15aba7b3b0f830e6e5a5b0c699be1a1d6c4ba Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Tue, 27 Oct 2026 21:12:37 -0700 Subject: [PATCH 03/39] Phase 3 proposal re-labelled as 3.0 design, behind phase 4's shape questions (#753) Co-Authored-By: Claude Opus 5.5 --- docs/design/decisions/vocabulary-model.md | 10 +++++----- docs/design/vocabulary-model-rfc.md | 14 ++++++++------ 2 files changed, 13 insertions(+), 11 deletions(-) diff --git a/docs/design/decisions/vocabulary-model.md b/docs/design/decisions/vocabulary-model.md index e7162d597..377ede506 100644 --- a/docs/design/decisions/vocabulary-model.md +++ b/docs/design/decisions/vocabulary-model.md @@ -237,16 +237,16 @@ Open: #913 (the title words phase 2j left, read as titles though borne as names) - **2026-10-26 #752, phase 2l — THE BRACKET ESCAPE READS THE DERIVED CLASH**, carrying out item 16 of the 2026-10-23 #752 entry above (#810). A member of the ambiguous class, the class 2a derives from name readings, alone in brackets or quotes after the name is the suffix where its writing settles it (`John Smith (MA)`, `John Smith (M.A)` → suffix), read in one pass, and stays the nickname otherwise (`John Smith (Ed)`, `JOHN SMITH (MA)`, `Jeffrey (JD) Bricken`). It is recorded as rules/S13, a rule of its own rather than the amendment to S1 item 16 planned, the position it reads being evidence S1's signal does not read; the writing is S3's and S6's evidence, which S13 cites. The record, the measurements and what was declined are decisions/S13's 2026-10-26 #810 entry. - **2026-10-26 #752, phase 2m — THE READING NOT TAKEN SURFACES AS `Ambiguity.alternative`, WITHOUT ITS LANGUAGES**, carrying out item 13 of the 2026-10-23 #752 entry above and item 7's source with it: a word the credential's run took by position alone reports its name reading's source as `position`, a fourth source beside the three the item named, and a word with a declared name reading reports `declared`. The pickle guard was widened as item 13 asked, every skew the narrow guard refused still refused. The reading's languages were left out: the per-word index costs about 0.4 ms of a 19.5 ms import and describes the default vocabulary, not a pack's or a caller's lexicon, so they wait for 3.0 with `Token.reading`. The record, the measurements and what was declined are decisions/A1's 2026-10-26 #752 entry. -- **2026-10-27 #753 — PHASE 3'S DESIGN IS PROPOSED, NOT AGREED: A CHOICE OF LANGUAGES SELECTS THE POOL, ONE MECHANISM FOR TURNING READINGS ON AND PREFER; NAME READINGS IN A PACK KEEP THE GATE ON; A READING INDEX CARRIES THE LANGUAGES TO RUN TIME; STRICT WAITS.** Every item below is a proposal awaiting Derek's decision, and this head is rewritten when he makes it. The design overview is docs/design/vocabulary-model-rfc.md's Phase 3 section. Decisions 15 and 20 are kept as written; decision 6's strict variant is reshaped to a property of the whole choice and deferred (items 1 and 4). The inputs are #753's comments of 2026-10-27: the inventory of 411 kept-out candidate readings with their verdicts, and the probe of foreign titles and function-word collisions. Measurements were taken 2026-10-27 on master `cfe8bf92` (Python 3.11.17) with throwaway Lexicon edits standing in for a pack, so each is evidence for the proposal and each sub-phase re-measures on its own base. Local-corpus figures are counts. - 1. **Proposed: a pack is a choice of languages carried by `Locale`.** `Locale` gains `languages`, a set of BCP 47 tags; `parser_for` unions the sets of the packs it is given, so the choice is order-independent while policy patches still fold in order; strict is an argument of the whole choice (`parser_for(locales.NE, locales.HI, strict=True)`), since a strict pack beside a preferring one has no meaning, reshaping decision 6's "strict variant". The shipped language packs are `Locale` constants with an empty lexicon fragment (`locales.NE`, `locales.PT_BR`). Weighed against: `Parser(languages=…)` beside `Locale`, two ways to choose one thing, and a `Policy` field, `Policy` being behavior and vocabulary being `Lexicon`'s. +- **2026-10-27 #753 — PHASE 3'S DESIGN IS PROPOSED, NOT AGREED: A CHOICE OF LANGUAGES SELECTS THE POOL, ONE MECHANISM FOR TURNING READINGS ON AND PREFER; NAME READINGS IN A PACK KEEP THE GATE ON; A READING INDEX CARRIES THE LANGUAGES TO RUN TIME; STRICT WAITS.** Every item below is a proposal awaiting Derek's decision, and this head is rewritten when he makes it. The design overview is docs/design/vocabulary-model-rfc.md's Phase 3 section. Decisions 15 and 20 are kept as written; decision 6's strict variant is reshaped to a property of the whole choice and deferred (items 1 and 4). Phase 3 is 3.0 work since the 2026-10-28 re-plan (item 12). The inputs are #753's comments of 2026-10-27: the inventory of 411 kept-out candidate readings with their verdicts, and the probe of foreign titles and function-word collisions. Measurements were taken 2026-10-27 on master `cfe8bf92` (Python 3.11.17) with throwaway Lexicon edits standing in for a pack, so each is evidence for the proposal and each sub-phase re-measures on its own base. Local-corpus figures are counts. + 1. **Proposed: a pack is a choice of languages carried by `Locale`.** `Locale` gains `languages`, a set of BCP 47 tags; `parser_for` unions the sets of the packs it is given, so the choice is order-independent while policy patches still fold in order; strict is an argument of the whole choice (`parser_for(locales.NE, locales.HI, strict=True)`), since a strict pack beside a preferring one has no meaning, reshaping decision 6's "strict variant". The shipped language packs are `Locale` constants with an empty lexicon fragment (`locales.NE`, `locales.PT_BR`). Open: this assumes `Locale` and `Policy` keep their 2.x shape once the facade is gone, which 3.0 has not decided. Weighed against: `Parser(languages=…)` beside `Locale`, two ways to choose one thing, and a `Policy` field, `Policy` being behavior and vocabulary being `Lexicon`'s. 2. **Proposed: one mechanism.** The chosen languages select which readings the compiler compiles, the default being that compiler with every language and no choice. Turning on, prefer and strict are derived from the file each reading sits in, and nothing is written per pack beyond its language's readings: a chosen language's `default = false` readings join; prefer drops other languages' vocabulary readings of a word on which a chosen language holds one, never `mul.toml`'s (`Haddad, Abd`, suffix `Abd` today, reads given `Abd` with the en credential gone); strict drops other languages' name readings and the markers they derive. A clash a property records with a SHAPE reading is settled for the chosen language's own reading, decision 20 applied: `may_be_initial` does not hold on it, so a `pt` choice reads `ANA SILVA E SOUZA` family `SILVA E SOUZA` where the default reads middle `SILVA E` with a report. This answers decisions/3-0-reevaluations' 2026-09-13 bullet that a pack cannot remove `e`, and its 2026-08-17 bullet's "overrides rather than additions": a pack overrides by selection, never by a removal list. That removes base entries all the same, so mechanisms.md#LOCALE-PACKS-PURE-DATA's add-only contract is amended with it (item 12). The shape-clash rule follows from decision 7, which decision 20 permits; it reaches every reading `may_be_initial` stands on (pt and it `e`, ca and pl `i`, the Italian one unjudged), and it forecloses an `nl` pack marking `y` (the 2026-08-18 bullet there), which waits with `fru` below. New beside decision 6: `mul.toml`'s exemption from prefer. Left open rather than built: a property a language's own reading should take only under its pack, Swedish `fru` and Icelandic `frú` as after-given salutations (rules/H8), refused in the default for the Cameroonian surname, the two readings in the inventory needing it. 3. **Proposed: `default = false` is allowed on `given` and `family` readings, and turned on they derive markers like any name reading, never settling (decision 15 unchanged).** Measured on nepal-candidates' 126,981 Devanagari and romanized pairs, comparing every word's role across the two spellings: 123,440 read alike in the default; `given` readings on `श्री`, `guru` and `गुरु`, which make each a title only inside a title phrase (rules/H9), give 123,636 (198 fixed, 2 broken, both pairs whose romanization opens with a listed title); a `family` reading on `dc` as well, making the credential ambiguous, gives 123,639, the count still leaving `Ram Bahadur DC` a suffix; dropping `dc`'s credential, which settling by a name reading would do, gives 123,658, 19 more. Settling is not proposed: the 19 pairs are the price of decision 15, whose `do` case it protects. `bk`, `kc`, `gc` and `bc` read as the family in every shape tried (spoken order, a family comma, `FAMILY_FIRST`, dotted and Title case), so a reading would move nothing and none is proposed. Recompute: the readings moved into or out of `Lexicon` fields by hand, the pairs compared by `tuple(t.role for t in parser.parse(s).tokens)`. - 4. **Proposed: strict waits, out of 2.8, until name readings say whose names they are.** A name reading sits in the file whose source counted it: 113 of the 211 name readings are in en.toml, `do`'s Vietnamese Đỗ readings among them, vi.toml holding no `do`, so strict built on the files drops them under a `vi` choice, and simulated, `Minh Do` reads suffix `Do` and `Do Van Minh` family `Do Van Minh`, decision 15's harm. A census count belongs to no one language either. The candidates needing strict wait with it, `nl`'s prepositions among them, which need more than strict: turned on as possibly-given particles, `in` and `to` read `Park In Hwan` family `In Hwan` and `Nguyen To Nga` family `To Nga` (the marker acts at the head only), and check 11 accepts `may_be_given = false` in place of the Korean and Vietnamese readings. Recompute: count `given` and `family` readings per file with tools/vocab/build.py's `load`; simulate by removing `do` from `suffix_acronyms_ambiguous` and `particles_ambiguous`. + 4. **Proposed: strict waits, out of the first packs, until name readings say whose names they are.** A name reading sits in the file whose source counted it: 113 of the 211 name readings are in en.toml, `do`'s Vietnamese Đỗ readings among them, vi.toml holding no `do`, so strict built on the files drops them under a `vi` choice, and simulated, `Minh Do` reads suffix `Do` and `Do Van Minh` family `Do Van Minh`, decision 15's harm. A census count belongs to no one language either. The candidates needing strict wait with it, `nl`'s prepositions among them, which need more than strict: turned on as possibly-given particles, `in` and `to` read `Park In Hwan` family `In Hwan` and `Nguyen To Nga` family `To Nga` (the marker acts at the head only), and check 11 accepts `may_be_given = false` in place of the Korean and Vietnamese readings. Recompute: count `given` and `family` readings per file with tools/vocab/build.py's `load`; simulate by removing `do` from `suffix_acronyms_ambiguous` and `particles_ambiguous`. 5. **Proposed: the languages reach run time as a reading index.** A second generated module, one row per reading (text, tag, kind, behavioral properties), loaded only when a language is chosen; the compiler moves from tools/vocab/build.py into the package so build and pack share one implementation, and a test holds that compiling the index with no choice reproduces `_compiled.py`. It is the per-word index item 13 of the 2026-10-23 #752 entry says `Token.reading` needs. A choice is applied as a delta against the default's compile, so it composes with a caller's `base=` lexicon and a `Locale`'s own fragment. Measured: 2,140 readings over 1,797 words, 106 in more than one file; the index about 93 KB of source and 11 ms to execute without a bytecode cache, the compile 9 ms, both once per choice and only for a caller who makes one; loading the TOML 39 ms. Weighed against: per-language deltas compiled at build time, which do not compose (under prefer `{pt, en}` `do` keeps its particle and its credential, while Portuguese's delta drops the credential and English's the particle); every combination precompiled, combinatorial; the TOML read at run time, slower and either re-running the build's checks or skipping them. 6. **Proposed: a chosen tag selects its file and its ancestors', never its descendants, and no macrolanguage table is read.** `pt-BR` selects `pt-BR.toml` and `pt.toml`; `pt` only `pt.toml`, so a caller in Portugal does not turn on the Brazilian agnomens (#843). The two files share no word today. Malay and Indonesian, which share `puan`, `putra` and `putri`, are chosen together by a caller who has both. The TOML names `prefer` and `strict` stay refused, the message saying a strength is the caller's choice and never data. 7. **Proposed: a pack is measured three ways.** The default does not move (the committed snapshot, and the index invariant of item 5); a vocabulary pack's deviation surface, the words whose compiled readings its choice changes, is derived from the index, and tests/v2/test_locales.py's non-interference gate requires the default's reading for every corpus name without such a word, the policy and segmentation packs (JA, RU, TR_AZ, ZH) keeping their written `DEVIATES`; and `snapshot.py`, `--local` with `--base` included, takes a choice and prints what it moves by field, beside each pack's own ground truth (nepal-candidates' script pairs for `ne`, the CLDR agreement per pack, PSC's nationality slice). This answers decisions/3-0-reevaluations' 2026-09-13 bullet that the harness cannot see a pack. - 8. **Proposed: third-party language files are deferred to 3.0.** A file joining the pool at run time would skip the checks that run across files (check 6's exclusions, check 11's particle statement), the ones that refuse a third-party `nl` `in` written with neither a `given` reading nor `may_be_given = false`. `Locale(lexicon=…)` stays the 2.x way to add vocabulary. + 8. **Proposed: third-party language files are deferred past phase 3.** A file joining the pool at run time would skip the checks that run across files (check 6's exclusions, check 11's particle statement), the ones that refuse a third-party `nl` `in` written with neither a `given` reading nor `may_be_given = false`. `Locale(lexicon=…)`, or its successor, stays the way to add vocabulary. 9. **Proposed: every exclusion a pack turns on becomes a `default = false` reading,** the shape cs `st` already has, so that once the packs land an `excluded` key means wrong everywhere: the eight whose reason opens "pack-shaped" in 3c (cs `ml`, hr, sk and sl `st` and `ml`, fr `père`), and the rest in the PR of the pack that turns each on (`pan`, `roz`, `nic`, `shri`, `sig`, `jaun`, `bt`, `bay`, `sb`). Recompute: `grep -h 'pack-shaped' nameparser/vocab/*.toml | wc -l`. 10. **Proposed: what stays out of phase 3.** ZH's surname list and the RU and TR_AZ endings are rule data for phase 4's `[rules]`, and a Python pack gains `languages` only where a measurement shows what it moves; the candidates that wait on a position rule (Turkish `Bey` and `Hanım`, Hungarian `úr`, Thai `นาย`, French `M.`) are phase 4's; same-language clashes (`kumari`, `sayyid`) take phase 2j's route in the default, since no pack settles one. 11. **Proposed: the work order,** shape first and one PR each, measured on its own base: 3a the index and shared compiler with name readings allowed `default = false` (neutral); 3b the choice, tag selection, turning on, prefer, the shape clash, the derived deviation surface, `snapshot.py`'s choice and the amended contract (neutral); 3c the pack-shaped exclusions converted (neutral); then the packs, `ne` (3d), `hi` and `bn`'s Latin forms (3e), `cs`, `sk`, `sl` and `hr` (3f), `es` and `pt` with a connective that counts only inside a title run (3g), and `ga`/`gd` and `pt-BR` (3h). 3g's connective is a rule with its own design: with `profesora` added and no rule, `Profesora de Historia Ana López` reads family `de Historia Ana López` (the rules/P1 fold) and `Ministro da Fazenda Carlos Souza` given `da`, while `Doctora de la Vega` must keep family `de la Vega`, the two separated by whether a name stands behind the phrase; it must reach only readings a pack turns on. - 12. **Proposed: phase 3 ships in 2.x, opt-in, the default's parses unmoved, and two standing texts are amended with 3b:** mechanisms.md#LOCALE-PACKS-PURE-DATA's contract that a pack is add-only and declares its deviations in its docstring (a pack adds, and selects among the default's readings by language, a vocabulary pack's deviations derived), and decisions/3-0-reevaluations' Standing line that removing base vocabulary through a pack is accepted for 2.x and part of 3.0's design. + 12. **Proposed: phase 3 is 3.0 work, opt-in, the default's parses unmoved, with phase 4's shape questions settled before or alongside its mechanism** (the 2026-10-28 re-plan: 2.8 is the last 2.x feature release, and 3.0 ships phases 3 and 4 with the facade's removal). The output model decides what a pack can express (a patronymic field, a second surname, #326's credentials apart from generations, #704's `middle`). 3a and 3c are independent of it, and so are 3b's selection semantics; 3b's API waits on item 1's open dependency; `cs` and `pt-BR` wait on #326 and `es`/`pt` on a second surname, while `ne` does not. One standing text is amended with 3b, mechanisms.md#LOCALE-PACKS-PURE-DATA's contract that a pack is add-only and declares its deviations in its docstring (a pack adds, and selects among the default's readings by language, a vocabulary pack's deviations derived); decisions/3-0-reevaluations' Standing line that removing base vocabulary through a pack is part of 3.0's pack design is carried out. diff --git a/docs/design/vocabulary-model-rfc.md b/docs/design/vocabulary-model-rfc.md index 1e50e0b92..f7179a75d 100644 --- a/docs/design/vocabulary-model-rfc.md +++ b/docs/design/vocabulary-model-rfc.md @@ -26,7 +26,7 @@ Four phases, in a fixed order: |---|---|---|---| | 1 | The per-word TOML format, compiled to today's sets | no | shipped in 2.6 (#737, #744); #749–#751 gave the words in `und.toml` their languages, leaving two particles of unrecorded origin | | 2 | Ambiguity derived from clashing readings; the ambiguity gate extended to every reading kind | yes, the first phase that does | #752, design agreed 2026-10-23; 2a, the two name-clash markers derived, landed without moving a parse (decisions/vocabulary-model, 2026-10-25) | -| 3 | Language packs that prefer or restrict their language | yes, for callers who choose one | #753, design proposed 2026-10-27, not agreed | +| 3 | Language packs that prefer or restrict their language | yes, for callers who choose one | #753, for 3.0; design proposed 2026-10-27, not agreed | | 4 | Order and shape; the rules a pack can choose; the output model | yes | #754, mostly independent of 2 and 3 | Two commitments hold through all four. **The default loads every language and keeps working for names of any language, as today**; choosing a language narrows it (decision 5 in decisions/vocabulary-model). And **languages own vocabulary and defaults, the core owns structure**: a pack configures rules the core implements, and never ships code (mechanisms.md#LOCALE-PACKS-PURE-DATA). @@ -164,7 +164,9 @@ Each sub-phase is one PR, measured on its own base: the committed snapshot read ## Phase 3: packs that prefer or restrict -Status: **proposed, not agreed** (2026-10-27, #753). The record is decisions/vocabulary-model's 2026-10-27 #753 entry, whose items are each marked proposed until Derek decides them; this section is the proposal those items describe. Phase 3 ships in 2.x, opt-in: a caller who chooses no language parses exactly as before, and nothing in this phase moves the default's parses. Two standing texts would be amended with it: mechanisms.md#LOCALE-PACKS-PURE-DATA's contract that a pack is add-only and declares its deviations in its docstring (a pack would add, and select among the default's readings by language, its deviations derived for a vocabulary pack), and decisions/3-0-reevaluations' line that removing base vocabulary through a pack waits for 3.0. +Status: **proposed, not agreed** (2026-10-27, #753). The record is decisions/vocabulary-model's 2026-10-27 #753 entry, whose items are each marked proposed until Derek decides them; this section is the proposal those items describe. Phase 3 is 3.0 work (re-planned 2026-10-28): 2.8 is the last 2.x feature release, and 3.0 ships phases 3 and 4 with the facade's removal. Packs stay opt-in, so a caller who chooses no language parses as the default does, and nothing in this phase moves the default's parses. One standing text would be amended with it: mechanisms.md#LOCALE-PACKS-PURE-DATA's contract that a pack is add-only and declares its deviations in its docstring (a pack would add, and select among the default's readings by language, its deviations derived for a vocabulary pack). decisions/3-0-reevaluations' line that removing base vocabulary through a pack is part of 3.0's pack design is carried out rather than moved. + +**Phase 4's shape questions come first, or alongside.** The output model decides what a pack can express: a patronymic field, a second surname, credentials apart from generation suffixes (#326), and what `middle` means (#704). A pack reading of a kind with no field to land in is a reading nothing can surface, so those questions are to be settled before or with phase 3's mechanism, by the same shape-before-value rule. Of the work order below, 3a (the reading index and the shared compiler) and 3c (the pack-shaped exclusions converted) are independent of the output model: the index carries whatever kinds the files hold, and 3c moves data between two encodings that compile alike. 3b's selection semantics are independent too, but its API is not: whether a choice rides on `Locale.languages` depends on what `Locale` and `Policy` look like once the facade is gone, so Q1 below is an open dependency. The packs depend on the output model in part: `cs`'s generations and `pt-BR`'s agnomens on #326, `es` and `pt` on a second surname; `ne`'s readings do not. The semantics were fixed in 2026-10-15/16, and phase 3 keeps them. A pack is a choice of languages, in two strengths (decision 6): @@ -211,7 +213,7 @@ Properties ride on their readings, so a choice that keeps a reading keeps its pr The 2 broken are pairs whose romanization opens with a listed title, which reads the `श्री` as a salutation (#870 counted `Mr` written for it twice). So the gate route delivers the `श्री` and `guru` fix without amending decision 15, and the 19 `dc` pairs it leaves are the price of keeping it. `bk`, `kc`, `gc` and `bc` need nothing: they read as the family in every shape tried (`Ram Bahadur BK`, `Ram Bahadur Kc`, `BK, Ram Bahadur`, `KC Sita` under `FAMILY_FIRST`), so a reading would move nothing. -**Strict cannot be built on today's files.** A name reading sits in the file whose source counted it, not the file of the language whose names carry it: 113 of the 211 name readings are in en.toml (2026-10-27), `do`'s Vietnamese Đỗ readings among them, and vi.toml holds no `do`. A strict `vi` choice built on the files would drop exactly those readings, and with them the gate: simulated, `Minh Do` reads suffix `Do` and `Do Van Minh` family `Do Van Minh`, the harm decision 15 is there to prevent. Nor does a census count belong to one language. So strict is proposed to wait, out of 2.8, until name readings say whose names they are, and the candidates that need it wait with it. The `nl` prepositions show they would need more than strict: turned on as possibly-given particles, `in` and `to` already read `Park In Hwan` family `In Hwan` and `Nguyen To Nga` family `To Nga`, the marker acting at the head only, and check 11 lets their author write `may_be_given = false` instead of declaring the Korean and Vietnamese readings. +**Strict cannot be built on today's files.** A name reading sits in the file whose source counted it, not the file of the language whose names carry it: 113 of the 211 name readings are in en.toml (2026-10-27), `do`'s Vietnamese Đỗ readings among them, and vi.toml holds no `do`. A strict `vi` choice built on the files would drop exactly those readings, and with them the gate: simulated, `Minh Do` reads suffix `Do` and `Do Van Minh` family `Do Van Minh`, the harm decision 15 is there to prevent. Nor does a census count belong to one language. So strict is proposed to wait, out of the first packs, until name readings say whose names they are, and the candidates that need it wait with it. The `nl` prepositions show they would need more than strict: turned on as possibly-given particles, `in` and `to` already read `Park In Hwan` family `In Hwan` and `Nguyen To Nga` family `To Nga`, the marker acting at the head only, and check 11 lets their author write `may_be_given = false` instead of declaring the Korean and Vietnamese readings. ### Where the provenance lives at runtime @@ -225,7 +227,7 @@ A choice is applied as a delta against the default's compile, so it composes wit ### The API and tags -A choice is carried by `Locale`, which gains a `languages` field, a set of BCP 47 tags. `parser_for` unions the `languages` of the packs it is given, so the choice is order-independent while their policy patches still fold in order. Strict, once it can be built, is a property of the whole choice, not of one pack, and an argument: `parser_for(locales.NE, locales.HI, strict=True)`. The shipped language packs are `Locale` constants like the four today (`locales.NE`, `locales.get("ne")`), each with an empty lexicon fragment; a code keeps `Locale`'s `[a-z0-9_]+`, so `pt-BR` is `locales.PT_BR`. +Open dependency: this assumes `Locale` and `Policy` keep their 2.x shape after the facade's removal, which is undecided. A choice is carried by `Locale`, which gains a `languages` field, a set of BCP 47 tags. `parser_for` unions the `languages` of the packs it is given, so the choice is order-independent while their policy patches still fold in order. Strict, once it can be built, is a property of the whole choice, not of one pack, and an argument: `parser_for(locales.NE, locales.HI, strict=True)`. The shipped language packs are `Locale` constants like the four today (`locales.NE`, `locales.get("ne")`), each with an empty lexicon fragment; a code keeps `Locale`'s `[a-z0-9_]+`, so `pt-BR` is `locales.PT_BR`. A chosen tag selects its own file and its ancestors', never its descendants': `pt-BR` selects `pt-BR.toml` and `pt.toml`, and `pt` selects only `pt.toml`, so a caller in Portugal does not turn on the Brazilian agnomens (#843 left the single-letter `F` and `N` to a `pt-BR` pack). No macrolanguage table is read: Malay and Indonesian are separate tags, though BCP 47 counts Indonesian within the Malay macrolanguage, and a caller with names in both chooses both. The Czech, Slovak, Slovenian and Croatian files each hold the words they share, so each language is a pack of its own. Script subtags need no matching rule yet, since no file carries one. @@ -241,14 +243,14 @@ The snapshot and `compare.py` parse under the default, so neither can see a pack ### What stays out of phase 3 -- **Third-party language files.** A file joining the pool at run time would skip the checks that run across files, an exclusion in one file refusing a reading in any other (check 6) and check 11's particle statement. A third-party `nl` file adding the preposition `in` without a `given` reading or `may_be_given = false` is refused by the build and would ship on the unsafe side at run time. `Locale(lexicon=…)` stays the 2.x way to add vocabulary; a public file format is 3.0's question. +- **Third-party language files.** A file joining the pool at run time would skip the checks that run across files, an exclusion in one file refusing a reading in any other (check 6) and check 11's particle statement. A third-party `nl` file adding the preposition `in` without a `given` reading or `may_be_given = false` is refused by the build and would ship on the unsafe side at run time. `Locale(lexicon=…)`, or its successor once the facade is gone, stays the way to add vocabulary; a public file format is a later question. - **The Python packs' rule data.** ZH's surnames (556 on 2026-10-27) are a segmentation list, outside the per-word model, and the RU and TR_AZ markers are endings inside rules (`-ovich`, `oğlu`); both wait for phase 4's `[rules]`. A pack gets a `languages` field only where measuring it shows what it moves. - **Candidates that wait on a rule, not a word:** Turkish `Bey` and `Hanım` after the given name, Hungarian `úr` after the name, Thai `นาย` glued at the head, French `M.` against the initial it spells. Each is a position rule, phase 4's. - **Same-language clashes.** No pack settles one (the `fils` argument): the Indic and Arabic titles that are names in their own language (`kumari`, `sayyid`) take phase 2j's route, a given reading and the title phrase, as default work. ### The order of work -Shape first, then the packs, each one PR measured on its own base, as phase 2's were. The first three move no default parse, and no pack exists until 3d. Strict is not in 2.8's order (above). +Shape first, then the packs, each one PR measured on its own base, as phase 2's were. The first three move no default parse, and no pack exists until 3d. Strict is not in this order (above). | Sub-phase | What | Needs | |---|---|---| From 8bbc50d189f1a1fb5e003976e8f4ef9d67ca7e8f Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Wed, 28 Oct 2026 02:28:07 -0700 Subject: [PATCH 04/39] Propose vocabulary phase 4's output model: the roles stay, new fields are views (#754) A spec draft for review, every item marked proposed. The RFC's Phase 4 section gains a status line, the [rules]-into-policy-patch proposal, the output model (Token.reading; generation/credentials, patronymic and surname2 as views; middle unchanged; render defaults following the read order), an API-change table against the 3.0 freeze, phase 3's Q1 confirmed, and a 4a-4h work order interleaved with 3a-3h. decisions/vocabulary-model gains the dated, proposed 2026-10-28 #754 entry carrying the measurements from the fielded corpora. Co-Authored-By: Claude Opus 5.5 --- docs/design/decisions/vocabulary-model.md | 12 ++++ docs/design/vocabulary-model-rfc.md | 82 ++++++++++++++++++++--- 2 files changed, 86 insertions(+), 8 deletions(-) diff --git a/docs/design/decisions/vocabulary-model.md b/docs/design/decisions/vocabulary-model.md index 70a56b730..2288a75a5 100644 --- a/docs/design/decisions/vocabulary-model.md +++ b/docs/design/decisions/vocabulary-model.md @@ -241,3 +241,15 @@ Open: #913 (the title words phase 2j left, read as titles though borne as names) Adding the field changes `Lexicon`'s pickle layout, as phases 2h and 2i did. - **2026-10-26 #752, phase 2l — THE BRACKET ESCAPE READS THE DERIVED CLASH**, carrying out item 16 of the 2026-10-23 #752 entry above (#810). A member of the ambiguous class, the class 2a derives from name readings, alone in brackets or quotes after the name is the suffix where its writing settles it (`John Smith (MA)`, `John Smith (M.A)` → suffix), read in one pass, and stays the nickname otherwise (`John Smith (Ed)`, `JOHN SMITH (MA)`, `Jeffrey (JD) Bricken`). It is recorded as rules/S13, a rule of its own rather than the amendment to S1 item 16 planned, the position it reads being evidence S1's signal does not read; the writing is S3's and S6's evidence, which S13 cites. The record, the measurements and what was declined are decisions/S13's 2026-10-26 #810 entry. - **2026-10-26 #752, phase 2m — THE READING NOT TAKEN SURFACES AS `Ambiguity.alternative`, WITHOUT ITS LANGUAGES**, carrying out item 13 of the 2026-10-23 #752 entry above and item 7's source with it: a word the credential's run took by position alone reports its name reading's source as `position`, a fourth source beside the three the item named, and a word with a declared name reading reports `declared`. The pickle guard was widened as item 13 asked, every skew the narrow guard refused still refused. The reading's languages were left out: the per-word index costs about 0.4 ms of a 19.5 ms import and describes the default vocabulary, not a pack's or a caller's lexicon, so they wait for 3.0 with `Token.reading`. The record, the measurements and what was declined are decisions/A1's 2026-10-26 #752 entry. + +- **2026-10-28 #754 — PHASE 4'S OUTPUT MODEL IS PROPOSED, NOT AGREED: THE SEVEN ROLES STAY AND EVERY NEW FIELD IS A VIEW READ FROM `Token.reading`; A PATRONYMIC NAMES ITS WORDS WITHOUT EMPTYING `family`; THE RENDER DEFAULTS FOLLOW THE ORDER A NAME WAS READ IN.** Every item below is a proposal awaiting Derek's decision, and this head is rewritten when he makes it. The design overview is docs/design/vocabulary-model-rfc.md's Phase 4 section. Decision 8 (a patronymic field in 3.0) is kept and its stated value revised (item 3); decisions 9 and 14 are kept as written. The core API is frozen for 3.0 except for additions (decisions/3-0-reevaluations, 2026-10-28 #925, THE CORE API), and each item says whether it is one. The inputs are #754 and its comment of 2026-10-23, #326, #704 and #694. Measurements were taken 2026-10-28 on master `c8357e74` (Python 3.11.17) over the fielded local corpora (`orcid-researchers`, `psc-individuals`, `patentsview-inventors`) and the committed fields files, with throwaway scripts that are not kept, so each figure is evidence for the proposal and each sub-phase re-measures on its own base. Local-corpus figures are counts. Recompute the box counts by reading each source's records with `local_corpora.read_records`, counting its boxes by word count and word ending per country or nationality, and the role the parse gives the counted word with `parse()`. + 1. **Proposed: `Role` keeps its seven members, and every field asked for is a view.** A role partitions the tokens, and each of the five fields asked for (a second surname, a patronymic, generation and credentials, a second given name, the reading taken) subdivides a role that exists, so each is computed from the tokens, a property and a render key, as `family_base` and `given_names` are (rules/R1, rules/R2). `as_dict()`, `comparison_key()` and the seven role properties are unchanged, and every view is an addition. Weighed against: a `Role` member for any of them, a break of the frozen `ParsedName` (`as_dict()` gains a key, `comparison_key()` a component, so a stored key stops matching, and the role its words leave returns less text than in 2.x), with no 2.x feature release left to warn in. + 2. **Proposed: `Token.reading`, the reading each token took**, a `Reading` (decisions/A1's type), `None` for a token `replace()` builds, with `ADDED_IN` metadata so a 2.x pickle loads, and left out of comparison so `==` keeps its 2.x meaning. `Reading` gains `languages`, the files holding the reading, filled from phase 3's reading index (3a) and empty where no file holds it; `ReadingKind` gains `PATRONYMIC`. The views read it. It carries out the 2026-10-23 #752 entry's item 13 and the 2026-10-26 phase 2m entry's deferral of the languages. + 3. **Proposed: a `patronymic` view, filled by a word or a rule that says patronymic, with `family` and `middle` unchanged.** The default fills it from the vocabulary's `patronymic` particles (`bin`, `binti`, `ibn`, `bint`, `bar`, `bat` and the native-script forms) with the name they introduce (`Mahathir bin Mohamad` → family `bin Mohamad`, as today, and patronymic `bin Mohamad`); RU's and TR_AZ's rules fill it where a caller turns them on (`Ivan Ivanovich Ivanov` under RU → middle `Ivanovich` and patronymic `Ivanovich`); an Icelandic ending rule fills it under an `is` pack only, since the default cannot tell `-dóttir` and `-son` from Swedish surnames (ORCID's Swedish records end their family box in `-son` or `-dóttir` in 2,184 of 20,000). A matronymic, written the same way, fills it too. SUPERSEDES, if agreed: the RFC's earlier statement of decision 8's value, that `family` can be empty for a name with a patronymic. Measured: filers put an Icelandic patronymic in the family box (ORCID 1,647 of 2,583 Icelandic records, PatentsView 535 of 722, PSC 101 of 125), and CLDR's samples put it in `surname` (9 of 9), while Wikidata's `patronymic-is` and `patronymic-my` strata record 108 names with a patronymic and no family name, of which the parse puts the patronymic in `family` 107 times. An East Slavic patronymic stands in the given part: CLDR's `ru` samples put it in `given2` (6 of 6); PatentsView's Russian given boxes end in one 8,423 times of 20,000, and the parse reads such a word as the middle 10,902 times of 10,962; 90 of the 112 Russian nationals in PSC who filled the middle-name box wrote one there. A `family` that keeps it and a `patronymic` that names it agree with both kinds of source. Weighed against: emptying `family` where the patronymic is recognized, which moves the field the fielded sources fill and changes `family` for every Icelandic and nasab name, a behavior change of a frozen property; and a neutral parent-names name for the view, which buys nothing the view can detect, since a matronymic and a patronymic are written alike. + 4. **Proposed: `generation` and `credentials` partition `suffix` (#326)**, `generation` the tokens read as a generation or an agnomen, `credentials` the rest, the post-nominals that are neither (`Esq.`, the honorifics after a name) falling in `credentials`, CLDR's catch-all, with `Token.reading.kind` telling them apart for a caller who needs it. Measured: of the 1,170 committed corpus names with a suffix, 88 hold a generation and a credential together and 87 only a listed post-nominal of another kind; of the 615 CLDR renderings with either field, the 75 whose suffix holds exactly CLDR's words split into CLDR's `generation` and `credentials` by the vocabulary's reading kind in all 75. Weighed against: two new `Role` members (item 1), and a third view for the honorifics, left open since CLDR has none. + 5. **Proposed: a `surname2` view, the family's second unit where rules/O6 read the family as two**, marked by a new stable tag where O6 takes its word, empty otherwise; `family` keeps both units as O6 makes it, and a family comma alone marks nothing (`Lloyd Webber, Andrew`). Measured: a two-word family box in ORCID's records from Spain 48.0%, Mexico 64.7%, Colombia 72.8% and Chile 50.5%, against 3.9% from the United States. Left open: a view for the first unit and its name, `surname` standing close to the `surnames` view that exists. + 6. **Proposed: the render defaults follow the order the name was read in.** `ParsedName` gains `order` (an addition, `ADDED_IN`, a 2.x pickle rendering as today); the default `str()` and `render()` spec put the family first where `order` does, and join two words that stand side by side in the original and the rendering with what separated them there (nothing, a space, an interpunct, a nakaguro); `initials()` gives nothing for a word in a script with no initials. A name with a comma or written given-first renders as today. Measured in the committed corpora: 264 names without a comma read family-first, `str()` renders all 264 differently from their text (`김민준` → `'민준 김'`), 159 of them written unspaced, and it drops the U+00B7 interpunct from 38 of the 43 names holding one. This carries out decisions/3-0-reevaluations' render-default clean slate. **Derek's decision needed:** no signature changes, but `str()`'s output changes for every name read family-first, and no 2.x release can warn about a `str()` result, so whether that counts as a break of the frozen `ParsedName` is his call. + 7. **Proposed: `middle` keeps its meaning (#704)**, every word of the given part after the given name, the given name one unit unless a rule joins words into it, as rules/P5 already does (`Abdul Aziz Tariq` → given `Abdul Aziz`); a compound given name is a `[rules]` rule a caller or a pack turns on, CLDR's `given2` is `middle`, and `Kaur` and an English name beside a Chinese one stay in `middle`, an accepted cost. Measured: in PSC, whose form has a middle-name box, 2,499 of 1,040,708 forename boxes hold two or more words, and of 6,101 Spanish nationals 25 do while 2,621 fill the middle box; ORCID's and PatentsView's given box holds middles too, so it cannot separate the two. The `Maria do Carmo` shape, a particle inside a given box of three or more words with a family box filled, reads the given box's words into the family in 1,812 of 1,923 ORCID records and 1,165 of 1,219 PatentsView records from Brazil, Portugal, Spain, Mexico, Argentina, Colombia, Chile and Italy, which is that rule's case. Weighed against: #704's option (b), one given unit by default, which changes `middle` for every English name of three words. + 8. **Proposed: a language file's `[rules]` compiles into its shipped `Locale`'s policy patch**, each rule kind a `Policy` field (an addition), so RU and TR_AZ are rebuilt from their files and `Locale` gains no field for rules. With it, **phase 3's Q1 is confirmed**: `Locale.languages` (the 2026-10-27 #753 entry's item 1) stands, its open dependency answered by the 2026-10-28 freeze, which keeps `Locale` and `Policy` in their 2.x shape. + 9. **Proposed: the work order**, one PR each, measured on its own base, the snapshot recording each new view where it is non-empty: 4a `Token.reading` and `ReadingKind.PATRONYMIC` (neutral; `languages` after 3a); 4b `generation` and `credentials` (neutral); 4c `patronymic` (neutral); 4d `surname2` (neutral); 4e the render defaults and `order`, once item 6 is decided; 4f `[rules]` into the policy patch, with or after 3b; 4g the compound given name, after 4f; 4h the shape inventory per order (#469). Interleaved with phase 3: 4a beside 3a; 4b to 4d before the packs that need them, `cs` (3f) and `pt-BR` (3h) on 4b, `es` and `pt` (3g) on 4d and 4g; an `is` pack on 3b, 4c and 4f. + 10. **Proposed: what stays out of the output model.** Gender (decision 9) waits, `Token.reading` able to carry it later as one more addition. Names of given names only are a rule's question, not a field's: ORCID's empty family box holds 68,006 of 1,831,533 records, 15.9% to 18.7% of those from India, Pakistan, Bangladesh and Indonesia, and the parse fills `family` in 51,367 of the 51,415 with two or more given-box words. #694's two-word surname after a given name (`Tariq Abdul Aziz`) is a parse rule for the default, outside phase 4. diff --git a/docs/design/vocabulary-model-rfc.md b/docs/design/vocabulary-model-rfc.md index a13fbf095..314c4874f 100644 --- a/docs/design/vocabulary-model-rfc.md +++ b/docs/design/vocabulary-model-rfc.md @@ -27,7 +27,7 @@ Four phases, in a fixed order: | 1 | The per-word TOML format, compiled to today's sets | no | shipped in 2.6 (#737, #744); #749–#751 gave the words in `und.toml` their languages, leaving two particles of unrecorded origin | | 2 | Ambiguity derived from clashing readings; the ambiguity gate extended to every reading kind | yes, the first phase that does | #752, design agreed 2026-10-23; 2a, the two name-clash markers derived, landed without moving a parse (decisions/vocabulary-model, 2026-10-25) | | 3 | Language packs that prefer or restrict their language | yes, for callers who choose one | #753, needs 1 and 2 | -| 4 | Order and shape; the rules a pack can choose; the output model | yes | #754, mostly independent of 2 and 3 | +| 4 | Order and shape; the rules a pack can choose; the output model | yes | #754, 3.0 work; its output model proposed 2026-10-28, ahead of or alongside phase 3's mechanism | Two commitments hold through all four. **The default loads every language and keeps working for names of any language, as today**; choosing a language narrows it (decision 5 in decisions/vocabulary-model). And **languages own vocabulary and defaults, the core owns structure**: a pack configures rules the core implements, and never ships code (mechanisms.md#LOCALE-PACKS-PURE-DATA). @@ -184,6 +184,8 @@ Consequences that follow from the definitions: ## Phase 4: order, shape and the output model +Status: **the output model is proposed, not agreed** (2026-10-28, #754). The record is decisions/vocabulary-model's 2026-10-28 #754 entry, whose items are each marked proposed until Derek decides them; the output-model and work-order subsections below are the proposal those items describe. The first two subsections, order and shape and the rules a pack can choose, are as written before and are amended only where the proposal reaches them. Phase 4 is 3.0 work: 3.0 ships it with phase 3 and the facade's removal, with alphas along the way, and it ships when phase 4 is done. The core API (`parse`, `Parser`, `Lexicon`, `Policy`, `Locale`, `ParsedName`) is frozen for 3.0 except for additions (decisions/3-0-reevaluations, 2026-10-28 #925, THE CORE API), so every proposal below says whether it is an addition. + ### Order is not shape Decision 14 separates two things a "language" was going to carry: @@ -213,17 +215,81 @@ A language file will hold a `[rules]` table that turns on rules implemented in t The surname span has shipped ahead of the packs, as an opt-in `Policy` field rather than a pack, since the convention is not one language's ([rules/O6](rules/O6-two-family-names.md), two family names, #875 and #878; decisions/O6 has why it is not `locales.ES`). A pack can later switch it on in its `[rules]`, as it will a default order. -The real-name corpora agree on the priority. In the GND and Wikidata corpora, Spanish and Latin American double surnames written given-first disagreed with the source on 315 of 390 names, while the same kind of names in index form, where the comma settles it, disagreed on 0 of 60 (counted 2026-10-16 on master `4375b01b`, from `gnd_agreement.json` and `wikidata_agreement.json`): the gap is order and span, not vocabulary. +**Proposed: a language file's `[rules]` compiles into its shipped `Locale`'s `policy` patch**, at build time as the vocabulary does, so a rule a pack turns on is a `Policy` field (`two_family_names`, `patronymic_rules`, and the new rule kinds as new fields) and `Locale` gains no field for rules. RU and TR_AZ become that: their `PolicyPatch` written by the build from `ru.toml`'s and `tr.toml`'s `[rules]` instead of by hand. The compound given name is measured in the fielded corpora: in ORCID and PatentsView records from Brazil, Portugal, Spain, Mexico, Argentina, Colombia, Chile and Italy whose given-names box holds three or more words with a particle inside (the `Maria do Carmo` shape) and whose family box is filled, 1,812 of 1,923 ORCID records and 1,165 of 1,219 PatentsView records read with the given box's words pulled into the family (2026-10-28, master `c8357e74`). Like the surname span, it is a rule a caller or a pack turns on, since `Juan de la Vega` is the same string shape with the opposite answer. + +The real-name corpora agree on the priority. In the GND and Wikidata corpora, Spanish and Latin American double surnames written given-first disagreed with the source on 315 of 390 names, while the same kind of names in index form, where the comma settles it, disagreed on 0 of 60 (counted 2026-10-16 on master `4375b01b`, from `gnd_agreement.json` and `wikidata_agreement.json`): the gap is order and span, not vocabulary. The PSC register ranks it (#754's comment of 2026-10-23): 22,168 of its 1,040,708 filed names read a two-word surname's first word as a middle name, the largest disagreement shape in its score. ### The output model -Phase 4 also decides what a parse returns, aligned with CLDR's person-name fields, which gives ground truth for free: +Phase 4 also decides what a parse returns, aligned with CLDR's person-name fields, which gives ground truth for free. Five fields were asked for: a second surname (`surname2`), a patronymic (decision 8), credentials apart from generations (#326), a meaning for `middle` when a given name is several words (#704), and the reading each token took (`Token.reading`, phase 2's item 13). The proposal answers all five with one shape. + +**Proposed: the seven roles stay, and every new field is a view.** A `Role` partitions the tokens: each word is in exactly one of title, given, middle, family, suffix, nickname and maiden. Each of the five asks to subdivide a role that exists, not to add a part a name has outside them: a second surname is part of the family, generation and credentials are parts of the suffix, a second given name is the middle, and a patronymic is a word the parse already puts in the family (Icelandic, Arabic and Malay names) or in the middle (East Slavic names). So each is a view computed from the tokens, as `family_base`, `family_particles`, `surnames` and `given_names` are today (rules/R1, rules/R2): a property and a render key, with `as_dict()`, `comparison_key()` and the seven role properties unchanged. Every view is an addition under the 3.0 freeze. A new `Role` member would not be: `as_dict()` would gain a key, `comparison_key()` a component (a stored key would stop matching), `for role in Role` a member, and the role a token moves out of would return less text than it does in 2.x, a change in what `family` or `suffix` means that no 2.x warning has announced. + +What decides a view is the reading each token took, so `Token.reading` comes first: + +- **`Token.reading`** (proposed): a `Token` field holding the `Reading` the parse took, the same type `Ambiguity.alternative` holds the reading not taken in (decisions/A1), `None` for a token `replace()` builds. Its `kind` says what the word was read as, its `source` why, and a `languages` field, added to `Reading`, names the language files holding that reading, filled from phase 3's reading index (3a) and left empty for a reading no file holds (a caller's own vocabulary, a shape, position alone). The field takes `ADDED_IN` metadata, so a 2.x pickle loads with `None` (the pickle guard as widened in 2.7), and is left out of comparison (`compare=False`), so `==` between two `ParsedName`s means what it means in 2.x. `ReadingKind` gains `PATRONYMIC`, its docstring already allowing new kinds. +- **Generation and credentials** (#326, proposed): `generation`, the suffix tokens whose reading is a generation or an agnomen (`Jr.`, `III`, `Filho`), and `credentials`, the rest of the suffix (`PhD`, `Esq.`, the honorifics after a name), so the two partition `suffix` and `suffix` is unchanged. CLDR has only these two buckets; the post-nominals that are neither (`Esq.`, the CJK and Hebrew honorifics, `Rinpoche`) fall in `credentials`, CLDR's catch-all, and a caller who wants them apart reads `Token.reading.kind`, which says `post-nominal`. Measured 2026-10-28 on `c8357e74`: of the 1,170 committed corpus names with a suffix, 88 hold a generation and a credential together, and 87 hold only a listed post-nominal of another kind. Of the 615 CLDR renderings with a generation or credentials field, 75 parse with exactly CLDR's suffix words, and in all 75 a split by the vocabulary's reading kind gives CLDR's `generation` and `credentials` both; the other 540 differ in which words the suffix holds, a parse question the split cannot fix. Recompute: classify each suffix token by the kind of its reading in nameparser/vocab/ (a roman numeral the generation), over `cldr_fields.jsonl`. +- **A patronymic** (decision 8, proposed): `patronymic`, the tokens whose reading is `PATRONYMIC`, in whichever role the parse put them, with `family` and `middle` unchanged. What fills it is a word or a rule that says patronymic, never a guess from position: the vocabulary's `patronymic` particles (`bin`, `binti`, `ibn`, `bint`, Hebrew `bar` and `bat` and the native-script forms) with the name after them, in the default, so `Mahathir bin Mohamad` reads family `bin Mohamad` as today and patronymic `bin Mohamad` beside it; the East Slavic and Turkic rules where RU or TR_AZ turns them on, so `Ivan Ivanovich Ivanov` under RU keeps middle `Ivanovich` and adds patronymic `Ivanovich`; and an Icelandic ending rule an `is` pack would turn on, the default unable to tell `Karlsdóttir` from the Swedish surname `Johansson`. The name `patronymic` is kept for a matronymic too, which is written the same way and cannot be told from one. **`family` is not emptied** by recognizing a patronymic, against what this section said before: the people filling in the fielded corpora put the patronymic in the family box (below), as CLDR's samples put an Icelandic patronymic in `surname` (9 of 9) and a Russian one in `given2` (6 of 6), our `middle`, while Wikidata records an Icelandic name as a patronymic with no family name; a `family` that keeps it and a `patronymic` that names it satisfy both. A caller writing "Mr. {family}" reads `patronymic` to see that the family is one. +- **A second surname** (proposed): `surname2`, the family's second name unit where a rule has read the family as two (rules/O6, which a caller or an `es` pack turns on), and empty otherwise, `family` keeping both units as O6 makes it today. O6 already knows which unit it took, so it marks where the second one starts with a new stable tag; a family comma alone marks nothing, since `Lloyd Webber, Andrew` is not two surnames, and a hyphenated family is one token and stays whole. Whether the first unit gets a view of its own, and its name (`surname` sits close to the `surnames` that exists), is a value question below. +- **`middle`** (#704, proposed): keeps its meaning, every word of the given part after the given name, and the given name is one name unit unless a rule joins words into it, as rules/P5 already does (`Abdul Aziz Tariq` → given `Abdul Aziz`). A compound given name (`Maria do Carmo`, `María José`) is the `[rules]` rule above, turned on by a caller or a pack, moving its words into `given` only under that rule. CLDR's `given2` is `middle`, so no view is added. A Sikh `Kaur` and an English name beside a Chinese one (`Lim Mei Ling Jessica`) stay in `middle`, an accepted cost. + +**Measured in the fielded corpora** (2026-10-28, master `c8357e74`, counts only; recompute by counting each record's boxes by word count and word ending with `local_corpora.read_records`, grouped by the record's country or nationality, and the parse's role for the counted word with `parse()`): + +| Question | Where | Count | +|---|---|---| +| patronymic in the family box | Icelandic records whose family box ends in `-son` or `-dóttir` | ORCID 1,647 of 2,583 (63.8%), PatentsView 535 of 722, PSC 101 of 125; the parse reads every one checked as the family (ORCID 1,509 of 1,509 ending `-sson` or `-dóttir`) | +| patronymic with no family | Wikidata's `patronymic-is` and `patronymic-my` strata, a patronymic and no surname stated | 108 rows, of which the parse puts the patronymic in `family` 107 times | +| patronymic in the given part | PatentsView records from Russia whose given box ends in an East Slavic patronymic | 8,423 of 20,000 (42.1%); across PatentsView the parse reads such a word as the middle 10,902 times of 10,962; in PSC, 90 of the 112 Russian nationals who filled the middle-name box wrote a patronymic there | +| nasab, filer's box | Malaysian records holding `bin`, `binti` and their spellings | ORCID: 319 in the family box, 335 in the given box; PatentsView: 106 and 269 | +| a second surname | family box of two words | ORCID Spain 48.0%, Mexico 64.7%, Colombia 72.8%, Chile 50.5%, Argentina 16.7%, United States 3.9% | +| a second given name | PSC, which has a middle-name box | 2,499 of 1,040,708 forename boxes hold two or more words; of 6,101 Spanish nationals, 25 do and 2,621 (43.0%) fill the middle box | +| no family name | ORCID records with an empty family box | 68,006 of 1,831,533 (3.7%); India 17.5%, Pakistan 16.4%, Bangladesh 18.7%, Indonesia 15.9%; of those with two or more given-box words, the parse fills `family` in 51,367 of 51,415 | + +ORCID's and PatentsView's given box holds the middle names too, so a two-word given box there cannot tell a compound given name from a middle name; PSC's middle box can, though its form steers what goes in it. The last row is a separate question from the patronymic: a name of given names only, written by people who have no family name, which no output field fixes and only a rule could read (an open question below). + +**Render defaults** (decisions/3-0-reevaluations' clean slate, proposed). The default `str()` reads every name given-first and joins its words with spaces, so a name read family-first is turned round and a written separator is lost: `str(parse("김민준"))` is `'민준 김'`, `str(parse("王 小明"))` is `'小明 王'` and `str(parse("威廉·莎士比亚"))` is `'威廉 莎士比亚'`. In the committed corpora (2026-10-28, `c8357e74`), 264 names without a comma read family-first, `str()` renders every one differently from the text, and 159 of them are written unspaced; of 43 names holding the U+00B7 interpunct, `str()` drops it from 38. Proposed: + +- `ParsedName` gains `order`, the order the name was read under (its script's or the declared `name_order`, a comma aside), an addition with `ADDED_IN` metadata whose missing value, in a 2.x pickle, renders as today. +- `str()` and `render()`'s default spec put the family first where `order` does, and two words that stand side by side in the original and in the rendering are joined by what separated them there, nothing, a space, an interpunct or a nakaguro, so `str(parse("김민준"))` is `'김민준'` and `str(parse("威廉·莎士比亚"))` keeps the dot. A name with a comma or written given-first renders as today (`Smith, John` → `'John Smith'`). +- `initials()` gives nothing for a word in a script with no initials (Han, Hangul, kana, `_policy._NO_INITIALS`), where today it gives the first character with a period (`'王.'`). -- **A second surname** (`surname2`) for Spanish, Catalan and Galician names, which CLDR's samples carry and nameparser has no field for. -- **A patronymic field** (decision 8), holding only a living patronymic. Its main value to callers is that `family` can be empty for the cultures that have no family name: `Soffía Karlsdóttir` reads family `Karlsdóttir` and `Mahathir bin Mohamad` family `bin Mohamad` today, where a letter addressed to "Mr. {family}" is wrong. Whether the field is `patronymic` or a neutral list of parent names, which a matronymic also fits, is open under #754. -- **Credentials separate from generation suffixes** (#326), and a decision on whether honorifics after a name get a field or stay a reading kind inside `suffix`. -- **What `middle` means** when a given name is several words (#704). -- **Gender**, recorded only where a word gives a definitive clue (decision 9): reported with its evidence, never inferred from given names. +These change no signature but change `str()`'s output for every name read family-first, so whether they count as a break of the frozen `ParsedName` is Derek's to decide (item 6 of the entry): no 2.x release can warn about a `str()` result. + +**What changes in the API**, by the 3.0 freeze: + +| Change | Kind | +|---|---| +| `Token.reading`, `Reading.languages`, `ReadingKind.PATRONYMIC` | additions; old pickles load | +| `generation`, `credentials`, `patronymic`, `surname2` properties and render keys | additions | +| a stable tag marking the second surname's start, joining `STABLE_TAGS` | addition | +| `ParsedName.order` | addition; old pickles load | +| new `Policy` fields for rules, and `Locale.languages` (phase 3) | additions | +| `Role`, `as_dict()`, `comparison_key()`, the seven role properties | unchanged | +| the default `str()`, `render()` spec and `initials()` output | behavior change, decision needed | +| a `Role` member for any of the four | a break, not proposed | + +**Phase 3's Q1 confirmed** (proposed): `Locale.languages` stands as phase 3's draft proposes. Its open dependency was what `Locale` and `Policy` look like once the facade is gone; the 2026-10-28 freeze answers it, both keeping their 2.x shape with additions. With the `[rules]` proposal above, a shipped language pack is a `Locale` whose `languages` choose its vocabulary and whose `policy` carries its rules, so `Locale` needs no third field. + +**Gender**, recorded only where a word gives a definitive clue (decision 9), is not part of this proposal: the vocabulary records it on readings (`bint`'s `gender = "f"`), and `Token.reading` could carry it later as one more addition. + +### The order of work + +Each sub-phase is one PR, measured on its own base as phase 2's and phase 3's are. The views and the reading move no field of any parse, so the committed snapshot holds still through 4a to 4d; the snapshot records the new views where they are non-empty, as it records the case views, so a reviewer sees what each fills. + +| Sub-phase | What | Needs | +|---|---|---| +| 4a | `Token.reading` and `ReadingKind.PATRONYMIC`, kind and source filled at the sites that decide today; `Reading.languages` once 3a's index exists | — (neutral); 3a for `languages` | +| 4b | `generation` and `credentials` (#326) | 4a (neutral) | +| 4c | `patronymic`: the vocabulary's patronymic particles, and the RU and TR_AZ rules writing the reading | 4a (neutral) | +| 4d | `surname2`: O6's stable tag | — (neutral) | +| 4e | the render defaults and `ParsedName.order` | the decision on item 6 | +| 4f | `[rules]` compiled into a `Locale`'s policy patch; RU and TR_AZ rebuilt from their files | 3b | +| 4g | the compound given name, a rule kind (#704) | 4f | +| 4h | the shape inventory per order (#469), starting with the title between a family and a given name under `FAMILY_FIRST` | — | + +**Interleaved with phase 3**: 4a can start beside 3a, its `languages` waiting for 3a's index; 4b to 4d need nothing of phase 3 and unblock its packs, `cs`'s generations (3f) and `pt-BR`'s agnomens (3h) on 4b, `es` and `pt` (3g) on 4d and 4g; 4f lands with or after 3b, since both decide what a shipped `Locale` holds; an `is` pack, which no phase 3 sub-phase lists, needs 3b, 4c and 4f. 4e and 4h stand apart from both. So the order proposed is 3a and 4a, then 4b to 4d with 3b and 3c, then 4f, then the packs with 4g, with 4e as soon as item 6 is decided. + +**Left open**, value questions inside the shapes above: the first surname's view and its name; whether `credentials` should leave the honorifics out, CLDR having no third bucket; which ending rules beside Icelandic a pack may name (Hungarian `-né` is a married name, not a patronymic); and the names of given names only (the last table row), whose reading needs a rule that tells a mononym from a given name and a family name, not an output field. ## The first packs From f39223959d901ef555a99802a673bd2773c3e715 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Wed, 28 Oct 2026 02:43:10 -0700 Subject: [PATCH 05/39] Record the render-default decision in phase 4's draft (#754) Co-Authored-By: Claude Opus 5.5 --- docs/design/decisions/vocabulary-model.md | 6 +++--- docs/design/vocabulary-model-rfc.md | 10 +++++----- 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/docs/design/decisions/vocabulary-model.md b/docs/design/decisions/vocabulary-model.md index 2288a75a5..19995714b 100644 --- a/docs/design/decisions/vocabulary-model.md +++ b/docs/design/decisions/vocabulary-model.md @@ -242,14 +242,14 @@ Open: #913 (the title words phase 2j left, read as titles though borne as names) - **2026-10-26 #752, phase 2l — THE BRACKET ESCAPE READS THE DERIVED CLASH**, carrying out item 16 of the 2026-10-23 #752 entry above (#810). A member of the ambiguous class, the class 2a derives from name readings, alone in brackets or quotes after the name is the suffix where its writing settles it (`John Smith (MA)`, `John Smith (M.A)` → suffix), read in one pass, and stays the nickname otherwise (`John Smith (Ed)`, `JOHN SMITH (MA)`, `Jeffrey (JD) Bricken`). It is recorded as rules/S13, a rule of its own rather than the amendment to S1 item 16 planned, the position it reads being evidence S1's signal does not read; the writing is S3's and S6's evidence, which S13 cites. The record, the measurements and what was declined are decisions/S13's 2026-10-26 #810 entry. - **2026-10-26 #752, phase 2m — THE READING NOT TAKEN SURFACES AS `Ambiguity.alternative`, WITHOUT ITS LANGUAGES**, carrying out item 13 of the 2026-10-23 #752 entry above and item 7's source with it: a word the credential's run took by position alone reports its name reading's source as `position`, a fourth source beside the three the item named, and a word with a declared name reading reports `declared`. The pickle guard was widened as item 13 asked, every skew the narrow guard refused still refused. The reading's languages were left out: the per-word index costs about 0.4 ms of a 19.5 ms import and describes the default vocabulary, not a pack's or a caller's lexicon, so they wait for 3.0 with `Token.reading`. The record, the measurements and what was declined are decisions/A1's 2026-10-26 #752 entry. -- **2026-10-28 #754 — PHASE 4'S OUTPUT MODEL IS PROPOSED, NOT AGREED: THE SEVEN ROLES STAY AND EVERY NEW FIELD IS A VIEW READ FROM `Token.reading`; A PATRONYMIC NAMES ITS WORDS WITHOUT EMPTYING `family`; THE RENDER DEFAULTS FOLLOW THE ORDER A NAME WAS READ IN.** Every item below is a proposal awaiting Derek's decision, and this head is rewritten when he makes it. The design overview is docs/design/vocabulary-model-rfc.md's Phase 4 section. Decision 8 (a patronymic field in 3.0) is kept and its stated value revised (item 3); decisions 9 and 14 are kept as written. The core API is frozen for 3.0 except for additions (decisions/3-0-reevaluations, 2026-10-28 #925, THE CORE API), and each item says whether it is one. The inputs are #754 and its comment of 2026-10-23, #326, #704 and #694. Measurements were taken 2026-10-28 on master `c8357e74` (Python 3.11.17) over the fielded local corpora (`orcid-researchers`, `psc-individuals`, `patentsview-inventors`) and the committed fields files, with throwaway scripts that are not kept, so each figure is evidence for the proposal and each sub-phase re-measures on its own base. Local-corpus figures are counts. Recompute the box counts by reading each source's records with `local_corpora.read_records`, counting its boxes by word count and word ending per country or nationality, and the role the parse gives the counted word with `parse()`. +- **2026-10-28 #754 — PHASE 4'S OUTPUT MODEL IS PROPOSED, NOT AGREED: THE SEVEN ROLES STAY AND EVERY NEW FIELD IS A VIEW READ FROM `Token.reading`; A PATRONYMIC NAMES ITS WORDS WITHOUT EMPTYING `family`; THE RENDER DEFAULTS FOLLOW THE ORDER A NAME WAS READ IN.** Every item below but item 6, decided 2026-10-28, is a proposal awaiting Derek's decision, and this head is rewritten when he makes them. The design overview is docs/design/vocabulary-model-rfc.md's Phase 4 section. Decision 8 (a patronymic field in 3.0) is kept and its stated value revised (item 3); decisions 9 and 14 are kept as written. The core API is frozen for 3.0 except for additions (decisions/3-0-reevaluations, 2026-10-28 #925, THE CORE API), and each item says whether it is one. The inputs are #754 and its comment of 2026-10-23, #326, #704 and #694. Measurements were taken 2026-10-28 on master `c8357e74` (Python 3.11.17) over the fielded local corpora (`orcid-researchers`, `psc-individuals`, `patentsview-inventors`) and the committed fields files, with throwaway scripts that are not kept, so each figure is evidence for the proposal and each sub-phase re-measures on its own base. Local-corpus figures are counts. Recompute the box counts by reading each source's records with `local_corpora.read_records`, counting its boxes by word count and word ending per country or nationality, and the role the parse gives the counted word with `parse()`. 1. **Proposed: `Role` keeps its seven members, and every field asked for is a view.** A role partitions the tokens, and each of the five fields asked for (a second surname, a patronymic, generation and credentials, a second given name, the reading taken) subdivides a role that exists, so each is computed from the tokens, a property and a render key, as `family_base` and `given_names` are (rules/R1, rules/R2). `as_dict()`, `comparison_key()` and the seven role properties are unchanged, and every view is an addition. Weighed against: a `Role` member for any of them, a break of the frozen `ParsedName` (`as_dict()` gains a key, `comparison_key()` a component, so a stored key stops matching, and the role its words leave returns less text than in 2.x), with no 2.x feature release left to warn in. 2. **Proposed: `Token.reading`, the reading each token took**, a `Reading` (decisions/A1's type), `None` for a token `replace()` builds, with `ADDED_IN` metadata so a 2.x pickle loads, and left out of comparison so `==` keeps its 2.x meaning. `Reading` gains `languages`, the files holding the reading, filled from phase 3's reading index (3a) and empty where no file holds it; `ReadingKind` gains `PATRONYMIC`. The views read it. It carries out the 2026-10-23 #752 entry's item 13 and the 2026-10-26 phase 2m entry's deferral of the languages. 3. **Proposed: a `patronymic` view, filled by a word or a rule that says patronymic, with `family` and `middle` unchanged.** The default fills it from the vocabulary's `patronymic` particles (`bin`, `binti`, `ibn`, `bint`, `bar`, `bat` and the native-script forms) with the name they introduce (`Mahathir bin Mohamad` → family `bin Mohamad`, as today, and patronymic `bin Mohamad`); RU's and TR_AZ's rules fill it where a caller turns them on (`Ivan Ivanovich Ivanov` under RU → middle `Ivanovich` and patronymic `Ivanovich`); an Icelandic ending rule fills it under an `is` pack only, since the default cannot tell `-dóttir` and `-son` from Swedish surnames (ORCID's Swedish records end their family box in `-son` or `-dóttir` in 2,184 of 20,000). A matronymic, written the same way, fills it too. SUPERSEDES, if agreed: the RFC's earlier statement of decision 8's value, that `family` can be empty for a name with a patronymic. Measured: filers put an Icelandic patronymic in the family box (ORCID 1,647 of 2,583 Icelandic records, PatentsView 535 of 722, PSC 101 of 125), and CLDR's samples put it in `surname` (9 of 9), while Wikidata's `patronymic-is` and `patronymic-my` strata record 108 names with a patronymic and no family name, of which the parse puts the patronymic in `family` 107 times. An East Slavic patronymic stands in the given part: CLDR's `ru` samples put it in `given2` (6 of 6); PatentsView's Russian given boxes end in one 8,423 times of 20,000, and the parse reads such a word as the middle 10,902 times of 10,962; 90 of the 112 Russian nationals in PSC who filled the middle-name box wrote one there. A `family` that keeps it and a `patronymic` that names it agree with both kinds of source. Weighed against: emptying `family` where the patronymic is recognized, which moves the field the fielded sources fill and changes `family` for every Icelandic and nasab name, a behavior change of a frozen property; and a neutral parent-names name for the view, which buys nothing the view can detect, since a matronymic and a patronymic are written alike. 4. **Proposed: `generation` and `credentials` partition `suffix` (#326)**, `generation` the tokens read as a generation or an agnomen, `credentials` the rest, the post-nominals that are neither (`Esq.`, the honorifics after a name) falling in `credentials`, CLDR's catch-all, with `Token.reading.kind` telling them apart for a caller who needs it. Measured: of the 1,170 committed corpus names with a suffix, 88 hold a generation and a credential together and 87 only a listed post-nominal of another kind; of the 615 CLDR renderings with either field, the 75 whose suffix holds exactly CLDR's words split into CLDR's `generation` and `credentials` by the vocabulary's reading kind in all 75. Weighed against: two new `Role` members (item 1), and a third view for the honorifics, left open since CLDR has none. 5. **Proposed: a `surname2` view, the family's second unit where rules/O6 read the family as two**, marked by a new stable tag where O6 takes its word, empty otherwise; `family` keeps both units as O6 makes it, and a family comma alone marks nothing (`Lloyd Webber, Andrew`). Measured: a two-word family box in ORCID's records from Spain 48.0%, Mexico 64.7%, Colombia 72.8% and Chile 50.5%, against 3.9% from the United States. Left open: a view for the first unit and its name, `surname` standing close to the `surnames` view that exists. - 6. **Proposed: the render defaults follow the order the name was read in.** `ParsedName` gains `order` (an addition, `ADDED_IN`, a 2.x pickle rendering as today); the default `str()` and `render()` spec put the family first where `order` does, and join two words that stand side by side in the original and the rendering with what separated them there (nothing, a space, an interpunct, a nakaguro); `initials()` gives nothing for a word in a script with no initials. A name with a comma or written given-first renders as today. Measured in the committed corpora: 264 names without a comma read family-first, `str()` renders all 264 differently from their text (`김민준` → `'민준 김'`), 159 of them written unspaced, and it drops the U+00B7 interpunct from 38 of the 43 names holding one. This carries out decisions/3-0-reevaluations' render-default clean slate. **Derek's decision needed:** no signature changes, but `str()`'s output changes for every name read family-first, and no 2.x release can warn about a `str()` result, so whether that counts as a break of the frozen `ParsedName` is his call. + 6. **Decided (Derek, 2026-10-28): the render defaults follow the order the name was read in, a behavior change of the output rather than a break of the API.** `ParsedName` gains `order` (an addition, `ADDED_IN`, a 2.x pickle rendering as today); the default `str()` and `render()` spec put the family first where `order` does, and join two words that stand side by side in the original and the rendering with what separated them there (nothing, a space, an interpunct, a nakaguro); `initials()` gives nothing for a word in a script with no initials. A name with a comma or written given-first renders as today. Measured in the committed corpora: 264 names without a comma read family-first, `str()` renders all 264 differently from their text (`김민준` → `'민준 김'`), 159 of them written unspaced, and it drops the U+00B7 interpunct from 38 of the 43 names holding one. This carries out decisions/3-0-reevaluations' render-default clean slate. No signature changes, but `str()`'s output changes for every name read family-first, and no 2.x release can warn about a `str()` result; Derek took it as an improvement to ship in 3.0's release notes, not a break needing a warning. 7. **Proposed: `middle` keeps its meaning (#704)**, every word of the given part after the given name, the given name one unit unless a rule joins words into it, as rules/P5 already does (`Abdul Aziz Tariq` → given `Abdul Aziz`); a compound given name is a `[rules]` rule a caller or a pack turns on, CLDR's `given2` is `middle`, and `Kaur` and an English name beside a Chinese one stay in `middle`, an accepted cost. Measured: in PSC, whose form has a middle-name box, 2,499 of 1,040,708 forename boxes hold two or more words, and of 6,101 Spanish nationals 25 do while 2,621 fill the middle box; ORCID's and PatentsView's given box holds middles too, so it cannot separate the two. The `Maria do Carmo` shape, a particle inside a given box of three or more words with a family box filled, reads the given box's words into the family in 1,812 of 1,923 ORCID records and 1,165 of 1,219 PatentsView records from Brazil, Portugal, Spain, Mexico, Argentina, Colombia, Chile and Italy, which is that rule's case. Weighed against: #704's option (b), one given unit by default, which changes `middle` for every English name of three words. 8. **Proposed: a language file's `[rules]` compiles into its shipped `Locale`'s policy patch**, each rule kind a `Policy` field (an addition), so RU and TR_AZ are rebuilt from their files and `Locale` gains no field for rules. With it, **phase 3's Q1 is confirmed**: `Locale.languages` (the 2026-10-27 #753 entry's item 1) stands, its open dependency answered by the 2026-10-28 freeze, which keeps `Locale` and `Policy` in their 2.x shape. - 9. **Proposed: the work order**, one PR each, measured on its own base, the snapshot recording each new view where it is non-empty: 4a `Token.reading` and `ReadingKind.PATRONYMIC` (neutral; `languages` after 3a); 4b `generation` and `credentials` (neutral); 4c `patronymic` (neutral); 4d `surname2` (neutral); 4e the render defaults and `order`, once item 6 is decided; 4f `[rules]` into the policy patch, with or after 3b; 4g the compound given name, after 4f; 4h the shape inventory per order (#469). Interleaved with phase 3: 4a beside 3a; 4b to 4d before the packs that need them, `cs` (3f) and `pt-BR` (3h) on 4b, `es` and `pt` (3g) on 4d and 4g; an `is` pack on 3b, 4c and 4f. + 9. **Proposed: the work order**, one PR each, measured on its own base, the snapshot recording each new view where it is non-empty: 4a `Token.reading` and `ReadingKind.PATRONYMIC` (neutral; `languages` after 3a); 4b `generation` and `credentials` (neutral); 4c `patronymic` (neutral); 4d `surname2` (neutral); 4e the render defaults and `order`; 4f `[rules]` into the policy patch, with or after 3b; 4g the compound given name, after 4f; 4h the shape inventory per order (#469). Interleaved with phase 3: 4a beside 3a; 4b to 4d before the packs that need them, `cs` (3f) and `pt-BR` (3h) on 4b, `es` and `pt` (3g) on 4d and 4g; an `is` pack on 3b, 4c and 4f. 10. **Proposed: what stays out of the output model.** Gender (decision 9) waits, `Token.reading` able to carry it later as one more addition. Names of given names only are a rule's question, not a field's: ORCID's empty family box holds 68,006 of 1,831,533 records, 15.9% to 18.7% of those from India, Pakistan, Bangladesh and Indonesia, and the parse fills `family` in 51,367 of the 51,415 with two or more given-box words. #694's two-word surname after a given name (`Tariq Abdul Aziz`) is a parse rule for the default, outside phase 4. diff --git a/docs/design/vocabulary-model-rfc.md b/docs/design/vocabulary-model-rfc.md index 314c4874f..feb0d09a6 100644 --- a/docs/design/vocabulary-model-rfc.md +++ b/docs/design/vocabulary-model-rfc.md @@ -247,13 +247,13 @@ What decides a view is the reading each token took, so `Token.reading` comes fir ORCID's and PatentsView's given box holds the middle names too, so a two-word given box there cannot tell a compound given name from a middle name; PSC's middle box can, though its form steers what goes in it. The last row is a separate question from the patronymic: a name of given names only, written by people who have no family name, which no output field fixes and only a rule could read (an open question below). -**Render defaults** (decisions/3-0-reevaluations' clean slate, proposed). The default `str()` reads every name given-first and joins its words with spaces, so a name read family-first is turned round and a written separator is lost: `str(parse("김민준"))` is `'민준 김'`, `str(parse("王 小明"))` is `'小明 王'` and `str(parse("威廉·莎士比亚"))` is `'威廉 莎士比亚'`. In the committed corpora (2026-10-28, `c8357e74`), 264 names without a comma read family-first, `str()` renders every one differently from the text, and 159 of them are written unspaced; of 43 names holding the U+00B7 interpunct, `str()` drops it from 38. Proposed: +**Render defaults** (decisions/3-0-reevaluations' clean slate, decided 2026-10-28). The default `str()` reads every name given-first and joins its words with spaces, so a name read family-first is turned round and a written separator is lost: `str(parse("김민준"))` is `'민준 김'`, `str(parse("王 小明"))` is `'小明 王'` and `str(parse("威廉·莎士比亚"))` is `'威廉 莎士比亚'`. In the committed corpora (2026-10-28, `c8357e74`), 264 names without a comma read family-first, `str()` renders every one differently from the text, and 159 of them are written unspaced; of 43 names holding the U+00B7 interpunct, `str()` drops it from 38. Decided: - `ParsedName` gains `order`, the order the name was read under (its script's or the declared `name_order`, a comma aside), an addition with `ADDED_IN` metadata whose missing value, in a 2.x pickle, renders as today. - `str()` and `render()`'s default spec put the family first where `order` does, and two words that stand side by side in the original and in the rendering are joined by what separated them there, nothing, a space, an interpunct or a nakaguro, so `str(parse("김민준"))` is `'김민준'` and `str(parse("威廉·莎士比亚"))` keeps the dot. A name with a comma or written given-first renders as today (`Smith, John` → `'John Smith'`). - `initials()` gives nothing for a word in a script with no initials (Han, Hangul, kana, `_policy._NO_INITIALS`), where today it gives the first character with a period (`'王.'`). -These change no signature but change `str()`'s output for every name read family-first, so whether they count as a break of the frozen `ParsedName` is Derek's to decide (item 6 of the entry): no 2.x release can warn about a `str()` result. +These change no signature but change `str()`'s output for every name read family-first, which no 2.x release can warn about; Derek took them as an improvement for 3.0's release notes rather than a break (item 6 of the entry). **What changes in the API**, by the 3.0 freeze: @@ -265,7 +265,7 @@ These change no signature but change `str()`'s output for every name read family | `ParsedName.order` | addition; old pickles load | | new `Policy` fields for rules, and `Locale.languages` (phase 3) | additions | | `Role`, `as_dict()`, `comparison_key()`, the seven role properties | unchanged | -| the default `str()`, `render()` spec and `initials()` output | behavior change, decision needed | +| the default `str()`, `render()` spec and `initials()` output | behavior change, decided | | a `Role` member for any of the four | a break, not proposed | **Phase 3's Q1 confirmed** (proposed): `Locale.languages` stands as phase 3's draft proposes. Its open dependency was what `Locale` and `Policy` look like once the facade is gone; the 2026-10-28 freeze answers it, both keeping their 2.x shape with additions. With the `[rules]` proposal above, a shipped language pack is a `Locale` whose `languages` choose its vocabulary and whose `policy` carries its rules, so `Locale` needs no third field. @@ -282,12 +282,12 @@ Each sub-phase is one PR, measured on its own base as phase 2's and phase 3's ar | 4b | `generation` and `credentials` (#326) | 4a (neutral) | | 4c | `patronymic`: the vocabulary's patronymic particles, and the RU and TR_AZ rules writing the reading | 4a (neutral) | | 4d | `surname2`: O6's stable tag | — (neutral) | -| 4e | the render defaults and `ParsedName.order` | the decision on item 6 | +| 4e | the render defaults and `ParsedName.order` | — | | 4f | `[rules]` compiled into a `Locale`'s policy patch; RU and TR_AZ rebuilt from their files | 3b | | 4g | the compound given name, a rule kind (#704) | 4f | | 4h | the shape inventory per order (#469), starting with the title between a family and a given name under `FAMILY_FIRST` | — | -**Interleaved with phase 3**: 4a can start beside 3a, its `languages` waiting for 3a's index; 4b to 4d need nothing of phase 3 and unblock its packs, `cs`'s generations (3f) and `pt-BR`'s agnomens (3h) on 4b, `es` and `pt` (3g) on 4d and 4g; 4f lands with or after 3b, since both decide what a shipped `Locale` holds; an `is` pack, which no phase 3 sub-phase lists, needs 3b, 4c and 4f. 4e and 4h stand apart from both. So the order proposed is 3a and 4a, then 4b to 4d with 3b and 3c, then 4f, then the packs with 4g, with 4e as soon as item 6 is decided. +**Interleaved with phase 3**: 4a can start beside 3a, its `languages` waiting for 3a's index; 4b to 4d need nothing of phase 3 and unblock its packs, `cs`'s generations (3f) and `pt-BR`'s agnomens (3h) on 4b, `es` and `pt` (3g) on 4d and 4g; 4f lands with or after 3b, since both decide what a shipped `Locale` holds; an `is` pack, which no phase 3 sub-phase lists, needs 3b, 4c and 4f. 4e and 4h stand apart from both. So the order proposed is 3a and 4a, then 4b to 4d with 3b and 3c, then 4f, then the packs with 4g, with 4e at any point. **Left open**, value questions inside the shapes above: the first surname's view and its name; whether `credentials` should leave the honorifics out, CLDR having no third bucket; which ending rules beside Icelandic a pack may name (Hungarian `-né` is a married name, not a patronymic); and the names of given names only (the last table row), whose reading needs a rule that tells a mononym from a given name and a family name, not an output field. From 321c37deaf4738653fa76c98b46e72e34bc0ece1 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Wed, 28 Oct 2026 02:47:40 -0700 Subject: [PATCH 06/39] Record the generation/credentials decision in phase 4's draft (#754) Co-Authored-By: Claude Opus 5.5 --- docs/design/decisions/vocabulary-model.md | 4 ++-- docs/design/vocabulary-model-rfc.md | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/design/decisions/vocabulary-model.md b/docs/design/decisions/vocabulary-model.md index 19995714b..50c0179f7 100644 --- a/docs/design/decisions/vocabulary-model.md +++ b/docs/design/decisions/vocabulary-model.md @@ -242,11 +242,11 @@ Open: #913 (the title words phase 2j left, read as titles though borne as names) - **2026-10-26 #752, phase 2l — THE BRACKET ESCAPE READS THE DERIVED CLASH**, carrying out item 16 of the 2026-10-23 #752 entry above (#810). A member of the ambiguous class, the class 2a derives from name readings, alone in brackets or quotes after the name is the suffix where its writing settles it (`John Smith (MA)`, `John Smith (M.A)` → suffix), read in one pass, and stays the nickname otherwise (`John Smith (Ed)`, `JOHN SMITH (MA)`, `Jeffrey (JD) Bricken`). It is recorded as rules/S13, a rule of its own rather than the amendment to S1 item 16 planned, the position it reads being evidence S1's signal does not read; the writing is S3's and S6's evidence, which S13 cites. The record, the measurements and what was declined are decisions/S13's 2026-10-26 #810 entry. - **2026-10-26 #752, phase 2m — THE READING NOT TAKEN SURFACES AS `Ambiguity.alternative`, WITHOUT ITS LANGUAGES**, carrying out item 13 of the 2026-10-23 #752 entry above and item 7's source with it: a word the credential's run took by position alone reports its name reading's source as `position`, a fourth source beside the three the item named, and a word with a declared name reading reports `declared`. The pickle guard was widened as item 13 asked, every skew the narrow guard refused still refused. The reading's languages were left out: the per-word index costs about 0.4 ms of a 19.5 ms import and describes the default vocabulary, not a pack's or a caller's lexicon, so they wait for 3.0 with `Token.reading`. The record, the measurements and what was declined are decisions/A1's 2026-10-26 #752 entry. -- **2026-10-28 #754 — PHASE 4'S OUTPUT MODEL IS PROPOSED, NOT AGREED: THE SEVEN ROLES STAY AND EVERY NEW FIELD IS A VIEW READ FROM `Token.reading`; A PATRONYMIC NAMES ITS WORDS WITHOUT EMPTYING `family`; THE RENDER DEFAULTS FOLLOW THE ORDER A NAME WAS READ IN.** Every item below but item 6, decided 2026-10-28, is a proposal awaiting Derek's decision, and this head is rewritten when he makes them. The design overview is docs/design/vocabulary-model-rfc.md's Phase 4 section. Decision 8 (a patronymic field in 3.0) is kept and its stated value revised (item 3); decisions 9 and 14 are kept as written. The core API is frozen for 3.0 except for additions (decisions/3-0-reevaluations, 2026-10-28 #925, THE CORE API), and each item says whether it is one. The inputs are #754 and its comment of 2026-10-23, #326, #704 and #694. Measurements were taken 2026-10-28 on master `c8357e74` (Python 3.11.17) over the fielded local corpora (`orcid-researchers`, `psc-individuals`, `patentsview-inventors`) and the committed fields files, with throwaway scripts that are not kept, so each figure is evidence for the proposal and each sub-phase re-measures on its own base. Local-corpus figures are counts. Recompute the box counts by reading each source's records with `local_corpora.read_records`, counting its boxes by word count and word ending per country or nationality, and the role the parse gives the counted word with `parse()`. +- **2026-10-28 #754 — PHASE 4'S OUTPUT MODEL IS PROPOSED, NOT AGREED: THE SEVEN ROLES STAY AND EVERY NEW FIELD IS A VIEW READ FROM `Token.reading`; A PATRONYMIC NAMES ITS WORDS WITHOUT EMPTYING `family`; THE RENDER DEFAULTS FOLLOW THE ORDER A NAME WAS READ IN.** Every item below but items 4 and 6, decided 2026-10-28, is a proposal awaiting Derek's decision, and this head is rewritten when he makes them. The design overview is docs/design/vocabulary-model-rfc.md's Phase 4 section. Decision 8 (a patronymic field in 3.0) is kept and its stated value revised (item 3); decisions 9 and 14 are kept as written. The core API is frozen for 3.0 except for additions (decisions/3-0-reevaluations, 2026-10-28 #925, THE CORE API), and each item says whether it is one. The inputs are #754 and its comment of 2026-10-23, #326, #704 and #694. Measurements were taken 2026-10-28 on master `c8357e74` (Python 3.11.17) over the fielded local corpora (`orcid-researchers`, `psc-individuals`, `patentsview-inventors`) and the committed fields files, with throwaway scripts that are not kept, so each figure is evidence for the proposal and each sub-phase re-measures on its own base. Local-corpus figures are counts. Recompute the box counts by reading each source's records with `local_corpora.read_records`, counting its boxes by word count and word ending per country or nationality, and the role the parse gives the counted word with `parse()`. 1. **Proposed: `Role` keeps its seven members, and every field asked for is a view.** A role partitions the tokens, and each of the five fields asked for (a second surname, a patronymic, generation and credentials, a second given name, the reading taken) subdivides a role that exists, so each is computed from the tokens, a property and a render key, as `family_base` and `given_names` are (rules/R1, rules/R2). `as_dict()`, `comparison_key()` and the seven role properties are unchanged, and every view is an addition. Weighed against: a `Role` member for any of them, a break of the frozen `ParsedName` (`as_dict()` gains a key, `comparison_key()` a component, so a stored key stops matching, and the role its words leave returns less text than in 2.x), with no 2.x feature release left to warn in. 2. **Proposed: `Token.reading`, the reading each token took**, a `Reading` (decisions/A1's type), `None` for a token `replace()` builds, with `ADDED_IN` metadata so a 2.x pickle loads, and left out of comparison so `==` keeps its 2.x meaning. `Reading` gains `languages`, the files holding the reading, filled from phase 3's reading index (3a) and empty where no file holds it; `ReadingKind` gains `PATRONYMIC`. The views read it. It carries out the 2026-10-23 #752 entry's item 13 and the 2026-10-26 phase 2m entry's deferral of the languages. 3. **Proposed: a `patronymic` view, filled by a word or a rule that says patronymic, with `family` and `middle` unchanged.** The default fills it from the vocabulary's `patronymic` particles (`bin`, `binti`, `ibn`, `bint`, `bar`, `bat` and the native-script forms) with the name they introduce (`Mahathir bin Mohamad` → family `bin Mohamad`, as today, and patronymic `bin Mohamad`); RU's and TR_AZ's rules fill it where a caller turns them on (`Ivan Ivanovich Ivanov` under RU → middle `Ivanovich` and patronymic `Ivanovich`); an Icelandic ending rule fills it under an `is` pack only, since the default cannot tell `-dóttir` and `-son` from Swedish surnames (ORCID's Swedish records end their family box in `-son` or `-dóttir` in 2,184 of 20,000). A matronymic, written the same way, fills it too. SUPERSEDES, if agreed: the RFC's earlier statement of decision 8's value, that `family` can be empty for a name with a patronymic. Measured: filers put an Icelandic patronymic in the family box (ORCID 1,647 of 2,583 Icelandic records, PatentsView 535 of 722, PSC 101 of 125), and CLDR's samples put it in `surname` (9 of 9), while Wikidata's `patronymic-is` and `patronymic-my` strata record 108 names with a patronymic and no family name, of which the parse puts the patronymic in `family` 107 times. An East Slavic patronymic stands in the given part: CLDR's `ru` samples put it in `given2` (6 of 6); PatentsView's Russian given boxes end in one 8,423 times of 20,000, and the parse reads such a word as the middle 10,902 times of 10,962; 90 of the 112 Russian nationals in PSC who filled the middle-name box wrote one there. A `family` that keeps it and a `patronymic` that names it agree with both kinds of source. Weighed against: emptying `family` where the patronymic is recognized, which moves the field the fielded sources fill and changes `family` for every Icelandic and nasab name, a behavior change of a frozen property; and a neutral parent-names name for the view, which buys nothing the view can detect, since a matronymic and a patronymic are written alike. - 4. **Proposed: `generation` and `credentials` partition `suffix` (#326)**, `generation` the tokens read as a generation or an agnomen, `credentials` the rest, the post-nominals that are neither (`Esq.`, the honorifics after a name) falling in `credentials`, CLDR's catch-all, with `Token.reading.kind` telling them apart for a caller who needs it. Measured: of the 1,170 committed corpus names with a suffix, 88 hold a generation and a credential together and 87 only a listed post-nominal of another kind; of the 615 CLDR renderings with either field, the 75 whose suffix holds exactly CLDR's words split into CLDR's `generation` and `credentials` by the vocabulary's reading kind in all 75. Weighed against: two new `Role` members (item 1), and a third view for the honorifics, left open since CLDR has none. + 4. **Decided (Derek, 2026-10-28): `generation` and `credentials` are views that partition `suffix` (#326)**, not roles, `generation` the tokens read as a generation or an agnomen, `credentials` the rest, the post-nominals that are neither (`Esq.`, the honorifics after a name) falling in `credentials`, CLDR's catch-all, with `Token.reading.kind` telling them apart for a caller who needs it. Measured: of the 1,170 committed corpus names with a suffix, 88 hold a generation and a credential together and 87 only a listed post-nominal of another kind; of the 615 CLDR renderings with either field, the 75 whose suffix holds exactly CLDR's words split into CLDR's `generation` and `credentials` by the vocabulary's reading kind in all 75. Weighed against: two new `Role` members (item 1), and a third view for the honorifics, left open since CLDR has none. 5. **Proposed: a `surname2` view, the family's second unit where rules/O6 read the family as two**, marked by a new stable tag where O6 takes its word, empty otherwise; `family` keeps both units as O6 makes it, and a family comma alone marks nothing (`Lloyd Webber, Andrew`). Measured: a two-word family box in ORCID's records from Spain 48.0%, Mexico 64.7%, Colombia 72.8% and Chile 50.5%, against 3.9% from the United States. Left open: a view for the first unit and its name, `surname` standing close to the `surnames` view that exists. 6. **Decided (Derek, 2026-10-28): the render defaults follow the order the name was read in, a behavior change of the output rather than a break of the API.** `ParsedName` gains `order` (an addition, `ADDED_IN`, a 2.x pickle rendering as today); the default `str()` and `render()` spec put the family first where `order` does, and join two words that stand side by side in the original and the rendering with what separated them there (nothing, a space, an interpunct, a nakaguro); `initials()` gives nothing for a word in a script with no initials. A name with a comma or written given-first renders as today. Measured in the committed corpora: 264 names without a comma read family-first, `str()` renders all 264 differently from their text (`김민준` → `'민준 김'`), 159 of them written unspaced, and it drops the U+00B7 interpunct from 38 of the 43 names holding one. This carries out decisions/3-0-reevaluations' render-default clean slate. No signature changes, but `str()`'s output changes for every name read family-first, and no 2.x release can warn about a `str()` result; Derek took it as an improvement to ship in 3.0's release notes, not a break needing a warning. 7. **Proposed: `middle` keeps its meaning (#704)**, every word of the given part after the given name, the given name one unit unless a rule joins words into it, as rules/P5 already does (`Abdul Aziz Tariq` → given `Abdul Aziz`); a compound given name is a `[rules]` rule a caller or a pack turns on, CLDR's `given2` is `middle`, and `Kaur` and an English name beside a Chinese one stay in `middle`, an accepted cost. Measured: in PSC, whose form has a middle-name box, 2,499 of 1,040,708 forename boxes hold two or more words, and of 6,101 Spanish nationals 25 do while 2,621 fill the middle box; ORCID's and PatentsView's given box holds middles too, so it cannot separate the two. The `Maria do Carmo` shape, a particle inside a given box of three or more words with a family box filled, reads the given box's words into the family in 1,812 of 1,923 ORCID records and 1,165 of 1,219 PatentsView records from Brazil, Portugal, Spain, Mexico, Argentina, Colombia, Chile and Italy, which is that rule's case. Weighed against: #704's option (b), one given unit by default, which changes `middle` for every English name of three words. diff --git a/docs/design/vocabulary-model-rfc.md b/docs/design/vocabulary-model-rfc.md index feb0d09a6..9ee1a6d47 100644 --- a/docs/design/vocabulary-model-rfc.md +++ b/docs/design/vocabulary-model-rfc.md @@ -228,7 +228,7 @@ Phase 4 also decides what a parse returns, aligned with CLDR's person-name field What decides a view is the reading each token took, so `Token.reading` comes first: - **`Token.reading`** (proposed): a `Token` field holding the `Reading` the parse took, the same type `Ambiguity.alternative` holds the reading not taken in (decisions/A1), `None` for a token `replace()` builds. Its `kind` says what the word was read as, its `source` why, and a `languages` field, added to `Reading`, names the language files holding that reading, filled from phase 3's reading index (3a) and left empty for a reading no file holds (a caller's own vocabulary, a shape, position alone). The field takes `ADDED_IN` metadata, so a 2.x pickle loads with `None` (the pickle guard as widened in 2.7), and is left out of comparison (`compare=False`), so `==` between two `ParsedName`s means what it means in 2.x. `ReadingKind` gains `PATRONYMIC`, its docstring already allowing new kinds. -- **Generation and credentials** (#326, proposed): `generation`, the suffix tokens whose reading is a generation or an agnomen (`Jr.`, `III`, `Filho`), and `credentials`, the rest of the suffix (`PhD`, `Esq.`, the honorifics after a name), so the two partition `suffix` and `suffix` is unchanged. CLDR has only these two buckets; the post-nominals that are neither (`Esq.`, the CJK and Hebrew honorifics, `Rinpoche`) fall in `credentials`, CLDR's catch-all, and a caller who wants them apart reads `Token.reading.kind`, which says `post-nominal`. Measured 2026-10-28 on `c8357e74`: of the 1,170 committed corpus names with a suffix, 88 hold a generation and a credential together, and 87 hold only a listed post-nominal of another kind. Of the 615 CLDR renderings with a generation or credentials field, 75 parse with exactly CLDR's suffix words, and in all 75 a split by the vocabulary's reading kind gives CLDR's `generation` and `credentials` both; the other 540 differ in which words the suffix holds, a parse question the split cannot fix. Recompute: classify each suffix token by the kind of its reading in nameparser/vocab/ (a roman numeral the generation), over `cldr_fields.jsonl`. +- **Generation and credentials** (#326, decided 2026-10-28): `generation`, the suffix tokens whose reading is a generation or an agnomen (`Jr.`, `III`, `Filho`), and `credentials`, the rest of the suffix (`PhD`, `Esq.`, the honorifics after a name), so the two partition `suffix` and `suffix` is unchanged. CLDR has only these two buckets; the post-nominals that are neither (`Esq.`, the CJK and Hebrew honorifics, `Rinpoche`) fall in `credentials`, CLDR's catch-all, and a caller who wants them apart reads `Token.reading.kind`, which says `post-nominal`. Measured 2026-10-28 on `c8357e74`: of the 1,170 committed corpus names with a suffix, 88 hold a generation and a credential together, and 87 hold only a listed post-nominal of another kind. Of the 615 CLDR renderings with a generation or credentials field, 75 parse with exactly CLDR's suffix words, and in all 75 a split by the vocabulary's reading kind gives CLDR's `generation` and `credentials` both; the other 540 differ in which words the suffix holds, a parse question the split cannot fix. Recompute: classify each suffix token by the kind of its reading in nameparser/vocab/ (a roman numeral the generation), over `cldr_fields.jsonl`. - **A patronymic** (decision 8, proposed): `patronymic`, the tokens whose reading is `PATRONYMIC`, in whichever role the parse put them, with `family` and `middle` unchanged. What fills it is a word or a rule that says patronymic, never a guess from position: the vocabulary's `patronymic` particles (`bin`, `binti`, `ibn`, `bint`, Hebrew `bar` and `bat` and the native-script forms) with the name after them, in the default, so `Mahathir bin Mohamad` reads family `bin Mohamad` as today and patronymic `bin Mohamad` beside it; the East Slavic and Turkic rules where RU or TR_AZ turns them on, so `Ivan Ivanovich Ivanov` under RU keeps middle `Ivanovich` and adds patronymic `Ivanovich`; and an Icelandic ending rule an `is` pack would turn on, the default unable to tell `Karlsdóttir` from the Swedish surname `Johansson`. The name `patronymic` is kept for a matronymic too, which is written the same way and cannot be told from one. **`family` is not emptied** by recognizing a patronymic, against what this section said before: the people filling in the fielded corpora put the patronymic in the family box (below), as CLDR's samples put an Icelandic patronymic in `surname` (9 of 9) and a Russian one in `given2` (6 of 6), our `middle`, while Wikidata records an Icelandic name as a patronymic with no family name; a `family` that keeps it and a `patronymic` that names it satisfy both. A caller writing "Mr. {family}" reads `patronymic` to see that the family is one. - **A second surname** (proposed): `surname2`, the family's second name unit where a rule has read the family as two (rules/O6, which a caller or an `es` pack turns on), and empty otherwise, `family` keeping both units as O6 makes it today. O6 already knows which unit it took, so it marks where the second one starts with a new stable tag; a family comma alone marks nothing, since `Lloyd Webber, Andrew` is not two surnames, and a hyphenated family is one token and stays whole. Whether the first unit gets a view of its own, and its name (`surname` sits close to the `surnames` that exists), is a value question below. - **`middle`** (#704, proposed): keeps its meaning, every word of the given part after the given name, and the given name is one name unit unless a rule joins words into it, as rules/P5 already does (`Abdul Aziz Tariq` → given `Abdul Aziz`). A compound given name (`Maria do Carmo`, `María José`) is the `[rules]` rule above, turned on by a caller or a pack, moving its words into `given` only under that rule. CLDR's `given2` is `middle`, so no view is added. A Sikh `Kaur` and an English name beside a Chinese one (`Lim Mei Ling Jessica`) stay in `middle`, an accepted cost. From fbc21af5325eca698384130fd2de0b80ed2bb544 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Wed, 28 Oct 2026 02:49:00 -0700 Subject: [PATCH 07/39] Record that surnames keeps its meaning in phase 4's draft (#754) Co-Authored-By: Claude Opus 5.5 --- docs/design/decisions/vocabulary-model.md | 2 +- docs/design/vocabulary-model-rfc.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/design/decisions/vocabulary-model.md b/docs/design/decisions/vocabulary-model.md index 50c0179f7..69ce3a1b3 100644 --- a/docs/design/decisions/vocabulary-model.md +++ b/docs/design/decisions/vocabulary-model.md @@ -247,7 +247,7 @@ Open: #913 (the title words phase 2j left, read as titles though borne as names) 2. **Proposed: `Token.reading`, the reading each token took**, a `Reading` (decisions/A1's type), `None` for a token `replace()` builds, with `ADDED_IN` metadata so a 2.x pickle loads, and left out of comparison so `==` keeps its 2.x meaning. `Reading` gains `languages`, the files holding the reading, filled from phase 3's reading index (3a) and empty where no file holds it; `ReadingKind` gains `PATRONYMIC`. The views read it. It carries out the 2026-10-23 #752 entry's item 13 and the 2026-10-26 phase 2m entry's deferral of the languages. 3. **Proposed: a `patronymic` view, filled by a word or a rule that says patronymic, with `family` and `middle` unchanged.** The default fills it from the vocabulary's `patronymic` particles (`bin`, `binti`, `ibn`, `bint`, `bar`, `bat` and the native-script forms) with the name they introduce (`Mahathir bin Mohamad` → family `bin Mohamad`, as today, and patronymic `bin Mohamad`); RU's and TR_AZ's rules fill it where a caller turns them on (`Ivan Ivanovich Ivanov` under RU → middle `Ivanovich` and patronymic `Ivanovich`); an Icelandic ending rule fills it under an `is` pack only, since the default cannot tell `-dóttir` and `-son` from Swedish surnames (ORCID's Swedish records end their family box in `-son` or `-dóttir` in 2,184 of 20,000). A matronymic, written the same way, fills it too. SUPERSEDES, if agreed: the RFC's earlier statement of decision 8's value, that `family` can be empty for a name with a patronymic. Measured: filers put an Icelandic patronymic in the family box (ORCID 1,647 of 2,583 Icelandic records, PatentsView 535 of 722, PSC 101 of 125), and CLDR's samples put it in `surname` (9 of 9), while Wikidata's `patronymic-is` and `patronymic-my` strata record 108 names with a patronymic and no family name, of which the parse puts the patronymic in `family` 107 times. An East Slavic patronymic stands in the given part: CLDR's `ru` samples put it in `given2` (6 of 6); PatentsView's Russian given boxes end in one 8,423 times of 20,000, and the parse reads such a word as the middle 10,902 times of 10,962; 90 of the 112 Russian nationals in PSC who filled the middle-name box wrote one there. A `family` that keeps it and a `patronymic` that names it agree with both kinds of source. Weighed against: emptying `family` where the patronymic is recognized, which moves the field the fielded sources fill and changes `family` for every Icelandic and nasab name, a behavior change of a frozen property; and a neutral parent-names name for the view, which buys nothing the view can detect, since a matronymic and a patronymic are written alike. 4. **Decided (Derek, 2026-10-28): `generation` and `credentials` are views that partition `suffix` (#326)**, not roles, `generation` the tokens read as a generation or an agnomen, `credentials` the rest, the post-nominals that are neither (`Esq.`, the honorifics after a name) falling in `credentials`, CLDR's catch-all, with `Token.reading.kind` telling them apart for a caller who needs it. Measured: of the 1,170 committed corpus names with a suffix, 88 hold a generation and a credential together and 87 only a listed post-nominal of another kind; of the 615 CLDR renderings with either field, the 75 whose suffix holds exactly CLDR's words split into CLDR's `generation` and `credentials` by the vocabulary's reading kind in all 75. Weighed against: two new `Role` members (item 1), and a third view for the honorifics, left open since CLDR has none. - 5. **Proposed: a `surname2` view, the family's second unit where rules/O6 read the family as two**, marked by a new stable tag where O6 takes its word, empty otherwise; `family` keeps both units as O6 makes it, and a family comma alone marks nothing (`Lloyd Webber, Andrew`). Measured: a two-word family box in ORCID's records from Spain 48.0%, Mexico 64.7%, Colombia 72.8% and Chile 50.5%, against 3.9% from the United States. Left open: a view for the first unit and its name, `surname` standing close to the `surnames` view that exists. + 5. **Proposed: a `surname2` view, the family's second unit where rules/O6 read the family as two**, marked by a new stable tag where O6 takes its word, empty otherwise; `family` keeps both units as O6 makes it, and a family comma alone marks nothing (`Lloyd Webber, Andrew`). Measured: a two-word family box in ORCID's records from Spain 48.0%, Mexico 64.7%, Colombia 72.8% and Chile 50.5%, against 3.9% from the United States. Left open: a view for the first unit and its name, `surname` standing close to the `surnames` view that exists. Declined (Derek, 2026-10-28): redefining `surnames` as the two units, which would repeat `family`, the family holding both where O6 reads two, and lose what `surnames` does today, joining `middle` and `family` so that a double surname the default splits (`Gabriel García Márquez` → middle `García`) reads whole, the view rules/P2 names for that reading. 6. **Decided (Derek, 2026-10-28): the render defaults follow the order the name was read in, a behavior change of the output rather than a break of the API.** `ParsedName` gains `order` (an addition, `ADDED_IN`, a 2.x pickle rendering as today); the default `str()` and `render()` spec put the family first where `order` does, and join two words that stand side by side in the original and the rendering with what separated them there (nothing, a space, an interpunct, a nakaguro); `initials()` gives nothing for a word in a script with no initials. A name with a comma or written given-first renders as today. Measured in the committed corpora: 264 names without a comma read family-first, `str()` renders all 264 differently from their text (`김민준` → `'민준 김'`), 159 of them written unspaced, and it drops the U+00B7 interpunct from 38 of the 43 names holding one. This carries out decisions/3-0-reevaluations' render-default clean slate. No signature changes, but `str()`'s output changes for every name read family-first, and no 2.x release can warn about a `str()` result; Derek took it as an improvement to ship in 3.0's release notes, not a break needing a warning. 7. **Proposed: `middle` keeps its meaning (#704)**, every word of the given part after the given name, the given name one unit unless a rule joins words into it, as rules/P5 already does (`Abdul Aziz Tariq` → given `Abdul Aziz`); a compound given name is a `[rules]` rule a caller or a pack turns on, CLDR's `given2` is `middle`, and `Kaur` and an English name beside a Chinese one stay in `middle`, an accepted cost. Measured: in PSC, whose form has a middle-name box, 2,499 of 1,040,708 forename boxes hold two or more words, and of 6,101 Spanish nationals 25 do while 2,621 fill the middle box; ORCID's and PatentsView's given box holds middles too, so it cannot separate the two. The `Maria do Carmo` shape, a particle inside a given box of three or more words with a family box filled, reads the given box's words into the family in 1,812 of 1,923 ORCID records and 1,165 of 1,219 PatentsView records from Brazil, Portugal, Spain, Mexico, Argentina, Colombia, Chile and Italy, which is that rule's case. Weighed against: #704's option (b), one given unit by default, which changes `middle` for every English name of three words. 8. **Proposed: a language file's `[rules]` compiles into its shipped `Locale`'s policy patch**, each rule kind a `Policy` field (an addition), so RU and TR_AZ are rebuilt from their files and `Locale` gains no field for rules. With it, **phase 3's Q1 is confirmed**: `Locale.languages` (the 2026-10-27 #753 entry's item 1) stands, its open dependency answered by the 2026-10-28 freeze, which keeps `Locale` and `Policy` in their 2.x shape. diff --git a/docs/design/vocabulary-model-rfc.md b/docs/design/vocabulary-model-rfc.md index 9ee1a6d47..b480d7e6d 100644 --- a/docs/design/vocabulary-model-rfc.md +++ b/docs/design/vocabulary-model-rfc.md @@ -230,7 +230,7 @@ What decides a view is the reading each token took, so `Token.reading` comes fir - **`Token.reading`** (proposed): a `Token` field holding the `Reading` the parse took, the same type `Ambiguity.alternative` holds the reading not taken in (decisions/A1), `None` for a token `replace()` builds. Its `kind` says what the word was read as, its `source` why, and a `languages` field, added to `Reading`, names the language files holding that reading, filled from phase 3's reading index (3a) and left empty for a reading no file holds (a caller's own vocabulary, a shape, position alone). The field takes `ADDED_IN` metadata, so a 2.x pickle loads with `None` (the pickle guard as widened in 2.7), and is left out of comparison (`compare=False`), so `==` between two `ParsedName`s means what it means in 2.x. `ReadingKind` gains `PATRONYMIC`, its docstring already allowing new kinds. - **Generation and credentials** (#326, decided 2026-10-28): `generation`, the suffix tokens whose reading is a generation or an agnomen (`Jr.`, `III`, `Filho`), and `credentials`, the rest of the suffix (`PhD`, `Esq.`, the honorifics after a name), so the two partition `suffix` and `suffix` is unchanged. CLDR has only these two buckets; the post-nominals that are neither (`Esq.`, the CJK and Hebrew honorifics, `Rinpoche`) fall in `credentials`, CLDR's catch-all, and a caller who wants them apart reads `Token.reading.kind`, which says `post-nominal`. Measured 2026-10-28 on `c8357e74`: of the 1,170 committed corpus names with a suffix, 88 hold a generation and a credential together, and 87 hold only a listed post-nominal of another kind. Of the 615 CLDR renderings with a generation or credentials field, 75 parse with exactly CLDR's suffix words, and in all 75 a split by the vocabulary's reading kind gives CLDR's `generation` and `credentials` both; the other 540 differ in which words the suffix holds, a parse question the split cannot fix. Recompute: classify each suffix token by the kind of its reading in nameparser/vocab/ (a roman numeral the generation), over `cldr_fields.jsonl`. - **A patronymic** (decision 8, proposed): `patronymic`, the tokens whose reading is `PATRONYMIC`, in whichever role the parse put them, with `family` and `middle` unchanged. What fills it is a word or a rule that says patronymic, never a guess from position: the vocabulary's `patronymic` particles (`bin`, `binti`, `ibn`, `bint`, Hebrew `bar` and `bat` and the native-script forms) with the name after them, in the default, so `Mahathir bin Mohamad` reads family `bin Mohamad` as today and patronymic `bin Mohamad` beside it; the East Slavic and Turkic rules where RU or TR_AZ turns them on, so `Ivan Ivanovich Ivanov` under RU keeps middle `Ivanovich` and adds patronymic `Ivanovich`; and an Icelandic ending rule an `is` pack would turn on, the default unable to tell `Karlsdóttir` from the Swedish surname `Johansson`. The name `patronymic` is kept for a matronymic too, which is written the same way and cannot be told from one. **`family` is not emptied** by recognizing a patronymic, against what this section said before: the people filling in the fielded corpora put the patronymic in the family box (below), as CLDR's samples put an Icelandic patronymic in `surname` (9 of 9) and a Russian one in `given2` (6 of 6), our `middle`, while Wikidata records an Icelandic name as a patronymic with no family name; a `family` that keeps it and a `patronymic` that names it satisfy both. A caller writing "Mr. {family}" reads `patronymic` to see that the family is one. -- **A second surname** (proposed): `surname2`, the family's second name unit where a rule has read the family as two (rules/O6, which a caller or an `es` pack turns on), and empty otherwise, `family` keeping both units as O6 makes it today. O6 already knows which unit it took, so it marks where the second one starts with a new stable tag; a family comma alone marks nothing, since `Lloyd Webber, Andrew` is not two surnames, and a hyphenated family is one token and stays whole. Whether the first unit gets a view of its own, and its name (`surname` sits close to the `surnames` that exists), is a value question below. +- **A second surname** (proposed): `surname2`, the family's second name unit where a rule has read the family as two (rules/O6, which a caller or an `es` pack turns on), and empty otherwise, `family` keeping both units as O6 makes it today. O6 already knows which unit it took, so it marks where the second one starts with a new stable tag; a family comma alone marks nothing, since `Lloyd Webber, Andrew` is not two surnames, and a hyphenated family is one token and stays whole. Whether the first unit gets a view of its own, and its name (`surname` sits close to the `surnames` that exists), is a value question below. `surnames` keeps its meaning, `middle` and `family` joined, the reading of a double surname the default splits. - **`middle`** (#704, proposed): keeps its meaning, every word of the given part after the given name, and the given name is one name unit unless a rule joins words into it, as rules/P5 already does (`Abdul Aziz Tariq` → given `Abdul Aziz`). A compound given name (`Maria do Carmo`, `María José`) is the `[rules]` rule above, turned on by a caller or a pack, moving its words into `given` only under that rule. CLDR's `given2` is `middle`, so no view is added. A Sikh `Kaur` and an English name beside a Chinese one (`Lim Mei Ling Jessica`) stay in `middle`, an accepted cost. **Measured in the fielded corpora** (2026-10-28, master `c8357e74`, counts only; recompute by counting each record's boxes by word count and word ending with `local_corpora.read_records`, grouped by the record's country or nationality, and the parse's role for the counted word with `parse()`): From 612a2e111fe3ad76546c3fda37a65e0436fa788a Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Wed, 28 Oct 2026 02:50:59 -0700 Subject: [PATCH 08/39] Record phase 4's output model as agreed (#754) Co-Authored-By: Claude Opus 5.5 --- docs/design/decisions/vocabulary-model.md | 25 +++++++++--------- docs/design/vocabulary-model-rfc.md | 32 +++++++++++------------ 2 files changed, 29 insertions(+), 28 deletions(-) diff --git a/docs/design/decisions/vocabulary-model.md b/docs/design/decisions/vocabulary-model.md index 69ce3a1b3..3bc401c8b 100644 --- a/docs/design/decisions/vocabulary-model.md +++ b/docs/design/decisions/vocabulary-model.md @@ -9,7 +9,8 @@ The vocabulary was a set of base sets plus marker subsets (`particles` and `part - A language file holds the words the parser might meet in names written in that language, immigrant particles included, not the words of its etymology, and a word sits in the file of every language whose names it occurs in, a Latin transliteration in its source language's file; `mul.toml` holds the words shared on purpose instead (the generations `i` to `v` and `2`, `&`, and the European titles below) — 2026-10-15/16; 2026-10-18 #737; 2026-10-19 #749; 2026-10-19 #750, title kinds. - The default loads every language and keeps working for names of any language. Choosing a language is to narrow it in phase 3 (#753): a pack is to settle only the clashes where its language has a vocabulary reading, a strict pack to drop other languages' name-level readings too, and a pack's preferred reading to be able to outrank one a shape rule derives — 2026-10-15/16. - `e`, `y` and `i` stay in the default; choosing a language is to settle their one-case fork, which no shipped pack does yet — 2026-10-15/16. -- A patronymic field is to come in 3.0, and the particle kinds already draw the `clan`/`patronymic` line for it: a patronymic is followed by a parent's name, a clan prefix by a family name the lineage shares — 2026-10-15/16; 2026-10-19 #749. +- A patronymic field is to come in 3.0, as a view that names its words without emptying `family`, and the particle kinds already draw the `clan`/`patronymic` line for it: a patronymic is followed by a parent's name, a clan prefix by a family name the lineage shares — 2026-10-15/16; 2026-10-19 #749; 2026-10-28 #754. +- 3.0's output model keeps the seven roles and adds every new field as a view read from `Token.reading`, the reading each token took: `generation` and `credentials` partitioning `suffix`, `patronymic`, and `surname2` where rules/O6 read two family names; `middle` and `surnames` keep their meaning, a compound given name being a rule a caller or pack turns on; the default `str()` renders a name in the order it was read in with its written separators; a language file's `[rules]` compile into its `Locale`'s policy patch — 2026-10-28 #754. - Gender is recorded only where a word gives a definitive clue: a feminine form or one half of a fixed pair, never an unmarked noun — 2026-10-15/16; 2026-10-19 #750, title kinds. - Title-phrase modifiers count as titles only beside a head title, at the head of the name only and never in a comma part (`Wing Lau` → given 'Wing', `John Smith, Vice` → title 'Vice'), save a modifier that may be a given name, which the head's test reads in every part (`Lau, Wing Kai` → given 'Wing'), rules/H9 — 2026-10-15/16; 2026-10-19 #750, title kinds; 2026-10-23 #752; 2026-10-25 #752, phase 2i; 2026-10-26 #752, phase 2j, `wing`. - Name readings are hand-declared, and only on a word some language file claims as vocabulary or refuses, so the vocabulary carries no surname list (the segmentation lists, Korean and the ZH pack's, are outside the per-word model) — 2026-10-15/16; 2026-10-19. @@ -242,14 +243,14 @@ Open: #913 (the title words phase 2j left, read as titles though borne as names) - **2026-10-26 #752, phase 2l — THE BRACKET ESCAPE READS THE DERIVED CLASH**, carrying out item 16 of the 2026-10-23 #752 entry above (#810). A member of the ambiguous class, the class 2a derives from name readings, alone in brackets or quotes after the name is the suffix where its writing settles it (`John Smith (MA)`, `John Smith (M.A)` → suffix), read in one pass, and stays the nickname otherwise (`John Smith (Ed)`, `JOHN SMITH (MA)`, `Jeffrey (JD) Bricken`). It is recorded as rules/S13, a rule of its own rather than the amendment to S1 item 16 planned, the position it reads being evidence S1's signal does not read; the writing is S3's and S6's evidence, which S13 cites. The record, the measurements and what was declined are decisions/S13's 2026-10-26 #810 entry. - **2026-10-26 #752, phase 2m — THE READING NOT TAKEN SURFACES AS `Ambiguity.alternative`, WITHOUT ITS LANGUAGES**, carrying out item 13 of the 2026-10-23 #752 entry above and item 7's source with it: a word the credential's run took by position alone reports its name reading's source as `position`, a fourth source beside the three the item named, and a word with a declared name reading reports `declared`. The pickle guard was widened as item 13 asked, every skew the narrow guard refused still refused. The reading's languages were left out: the per-word index costs about 0.4 ms of a 19.5 ms import and describes the default vocabulary, not a pack's or a caller's lexicon, so they wait for 3.0 with `Token.reading`. The record, the measurements and what was declined are decisions/A1's 2026-10-26 #752 entry. -- **2026-10-28 #754 — PHASE 4'S OUTPUT MODEL IS PROPOSED, NOT AGREED: THE SEVEN ROLES STAY AND EVERY NEW FIELD IS A VIEW READ FROM `Token.reading`; A PATRONYMIC NAMES ITS WORDS WITHOUT EMPTYING `family`; THE RENDER DEFAULTS FOLLOW THE ORDER A NAME WAS READ IN.** Every item below but items 4 and 6, decided 2026-10-28, is a proposal awaiting Derek's decision, and this head is rewritten when he makes them. The design overview is docs/design/vocabulary-model-rfc.md's Phase 4 section. Decision 8 (a patronymic field in 3.0) is kept and its stated value revised (item 3); decisions 9 and 14 are kept as written. The core API is frozen for 3.0 except for additions (decisions/3-0-reevaluations, 2026-10-28 #925, THE CORE API), and each item says whether it is one. The inputs are #754 and its comment of 2026-10-23, #326, #704 and #694. Measurements were taken 2026-10-28 on master `c8357e74` (Python 3.11.17) over the fielded local corpora (`orcid-researchers`, `psc-individuals`, `patentsview-inventors`) and the committed fields files, with throwaway scripts that are not kept, so each figure is evidence for the proposal and each sub-phase re-measures on its own base. Local-corpus figures are counts. Recompute the box counts by reading each source's records with `local_corpora.read_records`, counting its boxes by word count and word ending per country or nationality, and the role the parse gives the counted word with `parse()`. - 1. **Proposed: `Role` keeps its seven members, and every field asked for is a view.** A role partitions the tokens, and each of the five fields asked for (a second surname, a patronymic, generation and credentials, a second given name, the reading taken) subdivides a role that exists, so each is computed from the tokens, a property and a render key, as `family_base` and `given_names` are (rules/R1, rules/R2). `as_dict()`, `comparison_key()` and the seven role properties are unchanged, and every view is an addition. Weighed against: a `Role` member for any of them, a break of the frozen `ParsedName` (`as_dict()` gains a key, `comparison_key()` a component, so a stored key stops matching, and the role its words leave returns less text than in 2.x), with no 2.x feature release left to warn in. - 2. **Proposed: `Token.reading`, the reading each token took**, a `Reading` (decisions/A1's type), `None` for a token `replace()` builds, with `ADDED_IN` metadata so a 2.x pickle loads, and left out of comparison so `==` keeps its 2.x meaning. `Reading` gains `languages`, the files holding the reading, filled from phase 3's reading index (3a) and empty where no file holds it; `ReadingKind` gains `PATRONYMIC`. The views read it. It carries out the 2026-10-23 #752 entry's item 13 and the 2026-10-26 phase 2m entry's deferral of the languages. - 3. **Proposed: a `patronymic` view, filled by a word or a rule that says patronymic, with `family` and `middle` unchanged.** The default fills it from the vocabulary's `patronymic` particles (`bin`, `binti`, `ibn`, `bint`, `bar`, `bat` and the native-script forms) with the name they introduce (`Mahathir bin Mohamad` → family `bin Mohamad`, as today, and patronymic `bin Mohamad`); RU's and TR_AZ's rules fill it where a caller turns them on (`Ivan Ivanovich Ivanov` under RU → middle `Ivanovich` and patronymic `Ivanovich`); an Icelandic ending rule fills it under an `is` pack only, since the default cannot tell `-dóttir` and `-son` from Swedish surnames (ORCID's Swedish records end their family box in `-son` or `-dóttir` in 2,184 of 20,000). A matronymic, written the same way, fills it too. SUPERSEDES, if agreed: the RFC's earlier statement of decision 8's value, that `family` can be empty for a name with a patronymic. Measured: filers put an Icelandic patronymic in the family box (ORCID 1,647 of 2,583 Icelandic records, PatentsView 535 of 722, PSC 101 of 125), and CLDR's samples put it in `surname` (9 of 9), while Wikidata's `patronymic-is` and `patronymic-my` strata record 108 names with a patronymic and no family name, of which the parse puts the patronymic in `family` 107 times. An East Slavic patronymic stands in the given part: CLDR's `ru` samples put it in `given2` (6 of 6); PatentsView's Russian given boxes end in one 8,423 times of 20,000, and the parse reads such a word as the middle 10,902 times of 10,962; 90 of the 112 Russian nationals in PSC who filled the middle-name box wrote one there. A `family` that keeps it and a `patronymic` that names it agree with both kinds of source. Weighed against: emptying `family` where the patronymic is recognized, which moves the field the fielded sources fill and changes `family` for every Icelandic and nasab name, a behavior change of a frozen property; and a neutral parent-names name for the view, which buys nothing the view can detect, since a matronymic and a patronymic are written alike. - 4. **Decided (Derek, 2026-10-28): `generation` and `credentials` are views that partition `suffix` (#326)**, not roles, `generation` the tokens read as a generation or an agnomen, `credentials` the rest, the post-nominals that are neither (`Esq.`, the honorifics after a name) falling in `credentials`, CLDR's catch-all, with `Token.reading.kind` telling them apart for a caller who needs it. Measured: of the 1,170 committed corpus names with a suffix, 88 hold a generation and a credential together and 87 only a listed post-nominal of another kind; of the 615 CLDR renderings with either field, the 75 whose suffix holds exactly CLDR's words split into CLDR's `generation` and `credentials` by the vocabulary's reading kind in all 75. Weighed against: two new `Role` members (item 1), and a third view for the honorifics, left open since CLDR has none. - 5. **Proposed: a `surname2` view, the family's second unit where rules/O6 read the family as two**, marked by a new stable tag where O6 takes its word, empty otherwise; `family` keeps both units as O6 makes it, and a family comma alone marks nothing (`Lloyd Webber, Andrew`). Measured: a two-word family box in ORCID's records from Spain 48.0%, Mexico 64.7%, Colombia 72.8% and Chile 50.5%, against 3.9% from the United States. Left open: a view for the first unit and its name, `surname` standing close to the `surnames` view that exists. Declined (Derek, 2026-10-28): redefining `surnames` as the two units, which would repeat `family`, the family holding both where O6 reads two, and lose what `surnames` does today, joining `middle` and `family` so that a double surname the default splits (`Gabriel García Márquez` → middle `García`) reads whole, the view rules/P2 names for that reading. - 6. **Decided (Derek, 2026-10-28): the render defaults follow the order the name was read in, a behavior change of the output rather than a break of the API.** `ParsedName` gains `order` (an addition, `ADDED_IN`, a 2.x pickle rendering as today); the default `str()` and `render()` spec put the family first where `order` does, and join two words that stand side by side in the original and the rendering with what separated them there (nothing, a space, an interpunct, a nakaguro); `initials()` gives nothing for a word in a script with no initials. A name with a comma or written given-first renders as today. Measured in the committed corpora: 264 names without a comma read family-first, `str()` renders all 264 differently from their text (`김민준` → `'민준 김'`), 159 of them written unspaced, and it drops the U+00B7 interpunct from 38 of the 43 names holding one. This carries out decisions/3-0-reevaluations' render-default clean slate. No signature changes, but `str()`'s output changes for every name read family-first, and no 2.x release can warn about a `str()` result; Derek took it as an improvement to ship in 3.0's release notes, not a break needing a warning. - 7. **Proposed: `middle` keeps its meaning (#704)**, every word of the given part after the given name, the given name one unit unless a rule joins words into it, as rules/P5 already does (`Abdul Aziz Tariq` → given `Abdul Aziz`); a compound given name is a `[rules]` rule a caller or a pack turns on, CLDR's `given2` is `middle`, and `Kaur` and an English name beside a Chinese one stay in `middle`, an accepted cost. Measured: in PSC, whose form has a middle-name box, 2,499 of 1,040,708 forename boxes hold two or more words, and of 6,101 Spanish nationals 25 do while 2,621 fill the middle box; ORCID's and PatentsView's given box holds middles too, so it cannot separate the two. The `Maria do Carmo` shape, a particle inside a given box of three or more words with a family box filled, reads the given box's words into the family in 1,812 of 1,923 ORCID records and 1,165 of 1,219 PatentsView records from Brazil, Portugal, Spain, Mexico, Argentina, Colombia, Chile and Italy, which is that rule's case. Weighed against: #704's option (b), one given unit by default, which changes `middle` for every English name of three words. - 8. **Proposed: a language file's `[rules]` compiles into its shipped `Locale`'s policy patch**, each rule kind a `Policy` field (an addition), so RU and TR_AZ are rebuilt from their files and `Locale` gains no field for rules. With it, **phase 3's Q1 is confirmed**: `Locale.languages` (the 2026-10-27 #753 entry's item 1) stands, its open dependency answered by the 2026-10-28 freeze, which keeps `Locale` and `Policy` in their 2.x shape. - 9. **Proposed: the work order**, one PR each, measured on its own base, the snapshot recording each new view where it is non-empty: 4a `Token.reading` and `ReadingKind.PATRONYMIC` (neutral; `languages` after 3a); 4b `generation` and `credentials` (neutral); 4c `patronymic` (neutral); 4d `surname2` (neutral); 4e the render defaults and `order`; 4f `[rules]` into the policy patch, with or after 3b; 4g the compound given name, after 4f; 4h the shape inventory per order (#469). Interleaved with phase 3: 4a beside 3a; 4b to 4d before the packs that need them, `cs` (3f) and `pt-BR` (3h) on 4b, `es` and `pt` (3g) on 4d and 4g; an `is` pack on 3b, 4c and 4f. - 10. **Proposed: what stays out of the output model.** Gender (decision 9) waits, `Token.reading` able to carry it later as one more addition. Names of given names only are a rule's question, not a field's: ORCID's empty family box holds 68,006 of 1,831,533 records, 15.9% to 18.7% of those from India, Pakistan, Bangladesh and Indonesia, and the parse fills `family` in 51,367 of the 51,415 with two or more given-box words. #694's two-word surname after a given name (`Tariq Abdul Aziz`) is a parse rule for the default, outside phase 4. +- **2026-10-28 #754 (Derek) — PHASE 4'S OUTPUT MODEL IS AGREED: THE SEVEN ROLES STAY AND EVERY NEW FIELD IS A VIEW READ FROM `Token.reading`; A PATRONYMIC NAMES ITS WORDS WITHOUT EMPTYING `family`; THE RENDER DEFAULTS FOLLOW THE ORDER A NAME WAS READ IN.** Derek agreed every item in chat on 2026-10-28, items 6 and 4 and the `surnames` line of item 5 first, then the rest. The design overview is docs/design/vocabulary-model-rfc.md's Phase 4 section. Decision 8 (a patronymic field in 3.0) is kept and its stated value revised (item 3); decisions 9 and 14 are kept as written. The core API is frozen for 3.0 except for additions (decisions/3-0-reevaluations, 2026-10-28 #925, THE CORE API), and each item says whether it is one. The inputs are #754 and its comment of 2026-10-23, #326, #704 and #694. Measurements were taken 2026-10-28 on master `c8357e74` (Python 3.11.17) over the fielded local corpora (`orcid-researchers`, `psc-individuals`, `patentsview-inventors`) and the committed fields files, with throwaway scripts that are not kept, so each figure is evidence for the decision and each sub-phase re-measures on its own base. Local-corpus figures are counts. Recompute the box counts by reading each source's records with `local_corpora.read_records`, counting its boxes by word count and word ending per country or nationality, and the role the parse gives the counted word with `parse()`. + 1. **`Role` keeps its seven members, and every field asked for is a view.** A role partitions the tokens, and each of the five fields asked for (a second surname, a patronymic, generation and credentials, a second given name, the reading taken) subdivides a role that exists, so each is computed from the tokens, a property and a render key, as `family_base` and `given_names` are (rules/R1, rules/R2). `as_dict()`, `comparison_key()` and the seven role properties are unchanged, and every view is an addition. Weighed against: a `Role` member for any of them, a break of the frozen `ParsedName` (`as_dict()` gains a key, `comparison_key()` a component, so a stored key stops matching, and the role its words leave returns less text than in 2.x), with no 2.x feature release left to warn in. + 2. **`Token.reading`, the reading each token took**, a `Reading` (decisions/A1's type), `None` for a token `replace()` builds, with `ADDED_IN` metadata so a 2.x pickle loads, and left out of comparison so `==` keeps its 2.x meaning. `Reading` gains `languages`, the files holding the reading, filled from phase 3's reading index (3a) and empty where no file holds it; `ReadingKind` gains `PATRONYMIC`. The views read it. It carries out the 2026-10-23 #752 entry's item 13 and the 2026-10-26 phase 2m entry's deferral of the languages. + 3. **a `patronymic` view, filled by a word or a rule that says patronymic, with `family` and `middle` unchanged.** The default fills it from the vocabulary's `patronymic` particles (`bin`, `binti`, `ibn`, `bint`, `bar`, `bat` and the native-script forms) with the name they introduce (`Mahathir bin Mohamad` → family `bin Mohamad`, as today, and patronymic `bin Mohamad`); RU's and TR_AZ's rules fill it where a caller turns them on (`Ivan Ivanovich Ivanov` under RU → middle `Ivanovich` and patronymic `Ivanovich`); an Icelandic ending rule fills it under an `is` pack only, since the default cannot tell `-dóttir` and `-son` from Swedish surnames (ORCID's Swedish records end their family box in `-son` or `-dóttir` in 2,184 of 20,000). A matronymic, written the same way, fills it too. SUPERSEDES: the RFC's earlier statement of decision 8's value, that `family` can be empty for a name with a patronymic. Measured: filers put an Icelandic patronymic in the family box (ORCID 1,647 of 2,583 Icelandic records, PatentsView 535 of 722, PSC 101 of 125), and CLDR's samples put it in `surname` (9 of 9), while Wikidata's `patronymic-is` and `patronymic-my` strata record 108 names with a patronymic and no family name, of which the parse puts the patronymic in `family` 107 times. An East Slavic patronymic stands in the given part: CLDR's `ru` samples put it in `given2` (6 of 6); PatentsView's Russian given boxes end in one 8,423 times of 20,000, and the parse reads such a word as the middle 10,902 times of 10,962; 90 of the 112 Russian nationals in PSC who filled the middle-name box wrote one there. A `family` that keeps it and a `patronymic` that names it agree with both kinds of source. Weighed against: emptying `family` where the patronymic is recognized, which moves the field the fielded sources fill and changes `family` for every Icelandic and nasab name, a behavior change of a frozen property; and a neutral parent-names name for the view, which buys nothing the view can detect, since a matronymic and a patronymic are written alike. + 4. **`generation` and `credentials` are views that partition `suffix` (#326)**, not roles, `generation` the tokens read as a generation or an agnomen, `credentials` the rest, the post-nominals that are neither (`Esq.`, the honorifics after a name) falling in `credentials`, CLDR's catch-all, with `Token.reading.kind` telling them apart for a caller who needs it. Measured: of the 1,170 committed corpus names with a suffix, 88 hold a generation and a credential together and 87 only a listed post-nominal of another kind; of the 615 CLDR renderings with either field, the 75 whose suffix holds exactly CLDR's words split into CLDR's `generation` and `credentials` by the vocabulary's reading kind in all 75. Weighed against: two new `Role` members (item 1), and a third view for the honorifics, left open since CLDR has none. + 5. **a `surname2` view, the family's second unit where rules/O6 read the family as two**, marked by a new stable tag where O6 takes its word, empty otherwise; `family` keeps both units as O6 makes it, and a family comma alone marks nothing (`Lloyd Webber, Andrew`). Measured: a two-word family box in ORCID's records from Spain 48.0%, Mexico 64.7%, Colombia 72.8% and Chile 50.5%, against 3.9% from the United States. Left open: a view for the first unit and its name, `surname` standing close to the `surnames` view that exists. Declined: redefining `surnames` as the two units, which would repeat `family`, the family holding both where O6 reads two, and lose what `surnames` does today, joining `middle` and `family` so that a double surname the default splits (`Gabriel García Márquez` → middle `García`) reads whole, the view rules/P2 names for that reading. + 6. **the render defaults follow the order the name was read in, a behavior change of the output rather than a break of the API.** `ParsedName` gains `order` (an addition, `ADDED_IN`, a 2.x pickle rendering as today); the default `str()` and `render()` spec put the family first where `order` does, and join two words that stand side by side in the original and the rendering with what separated them there (nothing, a space, an interpunct, a nakaguro); `initials()` gives nothing for a word in a script with no initials. A name with a comma or written given-first renders as today. Measured in the committed corpora: 264 names without a comma read family-first, `str()` renders all 264 differently from their text (`김민준` → `'민준 김'`), 159 of them written unspaced, and it drops the U+00B7 interpunct from 38 of the 43 names holding one. This carries out decisions/3-0-reevaluations' render-default clean slate. No signature changes, but `str()`'s output changes for every name read family-first, and no 2.x release can warn about a `str()` result, so it is taken as an improvement for 3.0's release notes, not a break needing a warning. + 7. **`middle` keeps its meaning (#704)**, every word of the given part after the given name, the given name one unit unless a rule joins words into it, as rules/P5 already does (`Abdul Aziz Tariq` → given `Abdul Aziz`); a compound given name is a `[rules]` rule a caller or a pack turns on, CLDR's `given2` is `middle`, and `Kaur` and an English name beside a Chinese one stay in `middle`, an accepted cost. Measured: in PSC, whose form has a middle-name box, 2,499 of 1,040,708 forename boxes hold two or more words, and of 6,101 Spanish nationals 25 do while 2,621 fill the middle box; ORCID's and PatentsView's given box holds middles too, so it cannot separate the two. The `Maria do Carmo` shape, a particle inside a given box of three or more words with a family box filled, reads the given box's words into the family in 1,812 of 1,923 ORCID records and 1,165 of 1,219 PatentsView records from Brazil, Portugal, Spain, Mexico, Argentina, Colombia, Chile and Italy, which is that rule's case. Weighed against: #704's option (b), one given unit by default, which changes `middle` for every English name of three words. + 8. **a language file's `[rules]` compiles into its shipped `Locale`'s policy patch**, each rule kind a `Policy` field (an addition), so RU and TR_AZ are rebuilt from their files and `Locale` gains no field for rules. With it, **phase 3's Q1 is confirmed**: `Locale.languages` (the 2026-10-27 #753 entry's item 1) stands, its open dependency answered by the 2026-10-28 freeze, which keeps `Locale` and `Policy` in their 2.x shape. + 9. **the work order**, one PR each, measured on its own base, the snapshot recording each new view where it is non-empty: 4a `Token.reading` and `ReadingKind.PATRONYMIC` (neutral; `languages` after 3a); 4b `generation` and `credentials` (neutral); 4c `patronymic` (neutral); 4d `surname2` (neutral); 4e the render defaults and `order`; 4f `[rules]` into the policy patch, with or after 3b; 4g the compound given name, after 4f; 4h the shape inventory per order (#469). Interleaved with phase 3: 4a beside 3a; 4b to 4d before the packs that need them, `cs` (3f) and `pt-BR` (3h) on 4b, `es` and `pt` (3g) on 4d and 4g; an `is` pack on 3b, 4c and 4f. + 10. **what stays out of the output model.** Gender (decision 9) waits, `Token.reading` able to carry it later as one more addition. Names of given names only are a rule's question, not a field's: ORCID's empty family box holds 68,006 of 1,831,533 records, 15.9% to 18.7% of those from India, Pakistan, Bangladesh and Indonesia, and the parse fills `family` in 51,367 of the 51,415 with two or more given-box words. #694's two-word surname after a given name (`Tariq Abdul Aziz`) is a parse rule for the default, outside phase 4. diff --git a/docs/design/vocabulary-model-rfc.md b/docs/design/vocabulary-model-rfc.md index b480d7e6d..9cb4ccca2 100644 --- a/docs/design/vocabulary-model-rfc.md +++ b/docs/design/vocabulary-model-rfc.md @@ -27,7 +27,7 @@ Four phases, in a fixed order: | 1 | The per-word TOML format, compiled to today's sets | no | shipped in 2.6 (#737, #744); #749–#751 gave the words in `und.toml` their languages, leaving two particles of unrecorded origin | | 2 | Ambiguity derived from clashing readings; the ambiguity gate extended to every reading kind | yes, the first phase that does | #752, design agreed 2026-10-23; 2a, the two name-clash markers derived, landed without moving a parse (decisions/vocabulary-model, 2026-10-25) | | 3 | Language packs that prefer or restrict their language | yes, for callers who choose one | #753, needs 1 and 2 | -| 4 | Order and shape; the rules a pack can choose; the output model | yes | #754, 3.0 work; its output model proposed 2026-10-28, ahead of or alongside phase 3's mechanism | +| 4 | Order and shape; the rules a pack can choose; the output model | yes | #754, 3.0 work; its output model agreed 2026-10-28, ahead of or alongside phase 3's mechanism | Two commitments hold through all four. **The default loads every language and keeps working for names of any language, as today**; choosing a language narrows it (decision 5 in decisions/vocabulary-model). And **languages own vocabulary and defaults, the core owns structure**: a pack configures rules the core implements, and never ships code (mechanisms.md#LOCALE-PACKS-PURE-DATA). @@ -184,7 +184,7 @@ Consequences that follow from the definitions: ## Phase 4: order, shape and the output model -Status: **the output model is proposed, not agreed** (2026-10-28, #754). The record is decisions/vocabulary-model's 2026-10-28 #754 entry, whose items are each marked proposed until Derek decides them; the output-model and work-order subsections below are the proposal those items describe. The first two subsections, order and shape and the rules a pack can choose, are as written before and are amended only where the proposal reaches them. Phase 4 is 3.0 work: 3.0 ships it with phase 3 and the facade's removal, with alphas along the way, and it ships when phase 4 is done. The core API (`parse`, `Parser`, `Lexicon`, `Policy`, `Locale`, `ParsedName`) is frozen for 3.0 except for additions (decisions/3-0-reevaluations, 2026-10-28 #925, THE CORE API), so every proposal below says whether it is an addition. +Status: **the output model is agreed** (2026-10-28, #754). The record is decisions/vocabulary-model's 2026-10-28 #754 entry; the output-model and work-order subsections below are the design its items describe. The first two subsections, order and shape and the rules a pack can choose, are as written before and are amended only where the design reaches them. Phase 4 is 3.0 work: 3.0 ships it with phase 3 and the facade's removal, with alphas along the way, and it ships when phase 4 is done. The core API (`parse`, `Parser`, `Lexicon`, `Policy`, `Locale`, `ParsedName`) is frozen for 3.0 except for additions (decisions/3-0-reevaluations, 2026-10-28 #925, THE CORE API), so every change below says whether it is an addition. ### Order is not shape @@ -215,23 +215,23 @@ A language file will hold a `[rules]` table that turns on rules implemented in t The surname span has shipped ahead of the packs, as an opt-in `Policy` field rather than a pack, since the convention is not one language's ([rules/O6](rules/O6-two-family-names.md), two family names, #875 and #878; decisions/O6 has why it is not `locales.ES`). A pack can later switch it on in its `[rules]`, as it will a default order. -**Proposed: a language file's `[rules]` compiles into its shipped `Locale`'s `policy` patch**, at build time as the vocabulary does, so a rule a pack turns on is a `Policy` field (`two_family_names`, `patronymic_rules`, and the new rule kinds as new fields) and `Locale` gains no field for rules. RU and TR_AZ become that: their `PolicyPatch` written by the build from `ru.toml`'s and `tr.toml`'s `[rules]` instead of by hand. The compound given name is measured in the fielded corpora: in ORCID and PatentsView records from Brazil, Portugal, Spain, Mexico, Argentina, Colombia, Chile and Italy whose given-names box holds three or more words with a particle inside (the `Maria do Carmo` shape) and whose family box is filled, 1,812 of 1,923 ORCID records and 1,165 of 1,219 PatentsView records read with the given box's words pulled into the family (2026-10-28, master `c8357e74`). Like the surname span, it is a rule a caller or a pack turns on, since `Juan de la Vega` is the same string shape with the opposite answer. +**A language file's `[rules]` compiles into its shipped `Locale`'s `policy` patch**, at build time as the vocabulary does, so a rule a pack turns on is a `Policy` field (`two_family_names`, `patronymic_rules`, and the new rule kinds as new fields) and `Locale` gains no field for rules. RU and TR_AZ become that: their `PolicyPatch` written by the build from `ru.toml`'s and `tr.toml`'s `[rules]` instead of by hand. The compound given name is measured in the fielded corpora: in ORCID and PatentsView records from Brazil, Portugal, Spain, Mexico, Argentina, Colombia, Chile and Italy whose given-names box holds three or more words with a particle inside (the `Maria do Carmo` shape) and whose family box is filled, 1,812 of 1,923 ORCID records and 1,165 of 1,219 PatentsView records read with the given box's words pulled into the family (2026-10-28, master `c8357e74`). Like the surname span, it is a rule a caller or a pack turns on, since `Juan de la Vega` is the same string shape with the opposite answer. The real-name corpora agree on the priority. In the GND and Wikidata corpora, Spanish and Latin American double surnames written given-first disagreed with the source on 315 of 390 names, while the same kind of names in index form, where the comma settles it, disagreed on 0 of 60 (counted 2026-10-16 on master `4375b01b`, from `gnd_agreement.json` and `wikidata_agreement.json`): the gap is order and span, not vocabulary. The PSC register ranks it (#754's comment of 2026-10-23): 22,168 of its 1,040,708 filed names read a two-word surname's first word as a middle name, the largest disagreement shape in its score. ### The output model -Phase 4 also decides what a parse returns, aligned with CLDR's person-name fields, which gives ground truth for free. Five fields were asked for: a second surname (`surname2`), a patronymic (decision 8), credentials apart from generations (#326), a meaning for `middle` when a given name is several words (#704), and the reading each token took (`Token.reading`, phase 2's item 13). The proposal answers all five with one shape. +Phase 4 also decides what a parse returns, aligned with CLDR's person-name fields, which gives ground truth for free. Five fields were asked for: a second surname (`surname2`), a patronymic (decision 8), credentials apart from generations (#326), a meaning for `middle` when a given name is several words (#704), and the reading each token took (`Token.reading`, phase 2's item 13). The design answers all five with one shape. -**Proposed: the seven roles stay, and every new field is a view.** A `Role` partitions the tokens: each word is in exactly one of title, given, middle, family, suffix, nickname and maiden. Each of the five asks to subdivide a role that exists, not to add a part a name has outside them: a second surname is part of the family, generation and credentials are parts of the suffix, a second given name is the middle, and a patronymic is a word the parse already puts in the family (Icelandic, Arabic and Malay names) or in the middle (East Slavic names). So each is a view computed from the tokens, as `family_base`, `family_particles`, `surnames` and `given_names` are today (rules/R1, rules/R2): a property and a render key, with `as_dict()`, `comparison_key()` and the seven role properties unchanged. Every view is an addition under the 3.0 freeze. A new `Role` member would not be: `as_dict()` would gain a key, `comparison_key()` a component (a stored key would stop matching), `for role in Role` a member, and the role a token moves out of would return less text than it does in 2.x, a change in what `family` or `suffix` means that no 2.x warning has announced. +**The seven roles stay, and every new field is a view.** A `Role` partitions the tokens: each word is in exactly one of title, given, middle, family, suffix, nickname and maiden. Each of the five asks to subdivide a role that exists, not to add a part a name has outside them: a second surname is part of the family, generation and credentials are parts of the suffix, a second given name is the middle, and a patronymic is a word the parse already puts in the family (Icelandic, Arabic and Malay names) or in the middle (East Slavic names). So each is a view computed from the tokens, as `family_base`, `family_particles`, `surnames` and `given_names` are today (rules/R1, rules/R2): a property and a render key, with `as_dict()`, `comparison_key()` and the seven role properties unchanged. Every view is an addition under the 3.0 freeze. A new `Role` member would not be: `as_dict()` would gain a key, `comparison_key()` a component (a stored key would stop matching), `for role in Role` a member, and the role a token moves out of would return less text than it does in 2.x, a change in what `family` or `suffix` means that no 2.x warning has announced. What decides a view is the reading each token took, so `Token.reading` comes first: -- **`Token.reading`** (proposed): a `Token` field holding the `Reading` the parse took, the same type `Ambiguity.alternative` holds the reading not taken in (decisions/A1), `None` for a token `replace()` builds. Its `kind` says what the word was read as, its `source` why, and a `languages` field, added to `Reading`, names the language files holding that reading, filled from phase 3's reading index (3a) and left empty for a reading no file holds (a caller's own vocabulary, a shape, position alone). The field takes `ADDED_IN` metadata, so a 2.x pickle loads with `None` (the pickle guard as widened in 2.7), and is left out of comparison (`compare=False`), so `==` between two `ParsedName`s means what it means in 2.x. `ReadingKind` gains `PATRONYMIC`, its docstring already allowing new kinds. -- **Generation and credentials** (#326, decided 2026-10-28): `generation`, the suffix tokens whose reading is a generation or an agnomen (`Jr.`, `III`, `Filho`), and `credentials`, the rest of the suffix (`PhD`, `Esq.`, the honorifics after a name), so the two partition `suffix` and `suffix` is unchanged. CLDR has only these two buckets; the post-nominals that are neither (`Esq.`, the CJK and Hebrew honorifics, `Rinpoche`) fall in `credentials`, CLDR's catch-all, and a caller who wants them apart reads `Token.reading.kind`, which says `post-nominal`. Measured 2026-10-28 on `c8357e74`: of the 1,170 committed corpus names with a suffix, 88 hold a generation and a credential together, and 87 hold only a listed post-nominal of another kind. Of the 615 CLDR renderings with a generation or credentials field, 75 parse with exactly CLDR's suffix words, and in all 75 a split by the vocabulary's reading kind gives CLDR's `generation` and `credentials` both; the other 540 differ in which words the suffix holds, a parse question the split cannot fix. Recompute: classify each suffix token by the kind of its reading in nameparser/vocab/ (a roman numeral the generation), over `cldr_fields.jsonl`. -- **A patronymic** (decision 8, proposed): `patronymic`, the tokens whose reading is `PATRONYMIC`, in whichever role the parse put them, with `family` and `middle` unchanged. What fills it is a word or a rule that says patronymic, never a guess from position: the vocabulary's `patronymic` particles (`bin`, `binti`, `ibn`, `bint`, Hebrew `bar` and `bat` and the native-script forms) with the name after them, in the default, so `Mahathir bin Mohamad` reads family `bin Mohamad` as today and patronymic `bin Mohamad` beside it; the East Slavic and Turkic rules where RU or TR_AZ turns them on, so `Ivan Ivanovich Ivanov` under RU keeps middle `Ivanovich` and adds patronymic `Ivanovich`; and an Icelandic ending rule an `is` pack would turn on, the default unable to tell `Karlsdóttir` from the Swedish surname `Johansson`. The name `patronymic` is kept for a matronymic too, which is written the same way and cannot be told from one. **`family` is not emptied** by recognizing a patronymic, against what this section said before: the people filling in the fielded corpora put the patronymic in the family box (below), as CLDR's samples put an Icelandic patronymic in `surname` (9 of 9) and a Russian one in `given2` (6 of 6), our `middle`, while Wikidata records an Icelandic name as a patronymic with no family name; a `family` that keeps it and a `patronymic` that names it satisfy both. A caller writing "Mr. {family}" reads `patronymic` to see that the family is one. -- **A second surname** (proposed): `surname2`, the family's second name unit where a rule has read the family as two (rules/O6, which a caller or an `es` pack turns on), and empty otherwise, `family` keeping both units as O6 makes it today. O6 already knows which unit it took, so it marks where the second one starts with a new stable tag; a family comma alone marks nothing, since `Lloyd Webber, Andrew` is not two surnames, and a hyphenated family is one token and stays whole. Whether the first unit gets a view of its own, and its name (`surname` sits close to the `surnames` that exists), is a value question below. `surnames` keeps its meaning, `middle` and `family` joined, the reading of a double surname the default splits. -- **`middle`** (#704, proposed): keeps its meaning, every word of the given part after the given name, and the given name is one name unit unless a rule joins words into it, as rules/P5 already does (`Abdul Aziz Tariq` → given `Abdul Aziz`). A compound given name (`Maria do Carmo`, `María José`) is the `[rules]` rule above, turned on by a caller or a pack, moving its words into `given` only under that rule. CLDR's `given2` is `middle`, so no view is added. A Sikh `Kaur` and an English name beside a Chinese one (`Lim Mei Ling Jessica`) stay in `middle`, an accepted cost. +- **`Token.reading`**: a `Token` field holding the `Reading` the parse took, the same type `Ambiguity.alternative` holds the reading not taken in (decisions/A1), `None` for a token `replace()` builds. Its `kind` says what the word was read as, its `source` why, and a `languages` field, added to `Reading`, names the language files holding that reading, filled from phase 3's reading index (3a) and left empty for a reading no file holds (a caller's own vocabulary, a shape, position alone). The field takes `ADDED_IN` metadata, so a 2.x pickle loads with `None` (the pickle guard as widened in 2.7), and is left out of comparison (`compare=False`), so `==` between two `ParsedName`s means what it means in 2.x. `ReadingKind` gains `PATRONYMIC`, its docstring already allowing new kinds. +- **Generation and credentials** (#326): `generation`, the suffix tokens whose reading is a generation or an agnomen (`Jr.`, `III`, `Filho`), and `credentials`, the rest of the suffix (`PhD`, `Esq.`, the honorifics after a name), so the two partition `suffix` and `suffix` is unchanged. CLDR has only these two buckets; the post-nominals that are neither (`Esq.`, the CJK and Hebrew honorifics, `Rinpoche`) fall in `credentials`, CLDR's catch-all, and a caller who wants them apart reads `Token.reading.kind`, which says `post-nominal`. Measured 2026-10-28 on `c8357e74`: of the 1,170 committed corpus names with a suffix, 88 hold a generation and a credential together, and 87 hold only a listed post-nominal of another kind. Of the 615 CLDR renderings with a generation or credentials field, 75 parse with exactly CLDR's suffix words, and in all 75 a split by the vocabulary's reading kind gives CLDR's `generation` and `credentials` both; the other 540 differ in which words the suffix holds, a parse question the split cannot fix. Recompute: classify each suffix token by the kind of its reading in nameparser/vocab/ (a roman numeral the generation), over `cldr_fields.jsonl`. +- **A patronymic** (decision 8): `patronymic`, the tokens whose reading is `PATRONYMIC`, in whichever role the parse put them, with `family` and `middle` unchanged. What fills it is a word or a rule that says patronymic, never a guess from position: the vocabulary's `patronymic` particles (`bin`, `binti`, `ibn`, `bint`, Hebrew `bar` and `bat` and the native-script forms) with the name after them, in the default, so `Mahathir bin Mohamad` reads family `bin Mohamad` as today and patronymic `bin Mohamad` beside it; the East Slavic and Turkic rules where RU or TR_AZ turns them on, so `Ivan Ivanovich Ivanov` under RU keeps middle `Ivanovich` and adds patronymic `Ivanovich`; and an Icelandic ending rule an `is` pack would turn on, the default unable to tell `Karlsdóttir` from the Swedish surname `Johansson`. The name `patronymic` is kept for a matronymic too, which is written the same way and cannot be told from one. **`family` is not emptied** by recognizing a patronymic, revising what this section said before: the people filling in the fielded corpora put the patronymic in the family box (below), as CLDR's samples put an Icelandic patronymic in `surname` (9 of 9) and a Russian one in `given2` (6 of 6), our `middle`, while Wikidata records an Icelandic name as a patronymic with no family name; a `family` that keeps it and a `patronymic` that names it satisfy both. A caller writing "Mr. {family}" reads `patronymic` to see that the family is one. +- **A second surname**: `surname2`, the family's second name unit where a rule has read the family as two (rules/O6, which a caller or an `es` pack turns on), and empty otherwise, `family` keeping both units as O6 makes it today. O6 already knows which unit it took, so it marks where the second one starts with a new stable tag; a family comma alone marks nothing, since `Lloyd Webber, Andrew` is not two surnames, and a hyphenated family is one token and stays whole. Whether the first unit gets a view of its own, and its name (`surname` sits close to the `surnames` that exists), is a value question below. `surnames` keeps its meaning, `middle` and `family` joined, the reading of a double surname the default splits. +- **`middle`** (#704): keeps its meaning, every word of the given part after the given name, and the given name is one name unit unless a rule joins words into it, as rules/P5 already does (`Abdul Aziz Tariq` → given `Abdul Aziz`). A compound given name (`Maria do Carmo`, `María José`) is the `[rules]` rule above, turned on by a caller or a pack, moving its words into `given` only under that rule. CLDR's `given2` is `middle`, so no view is added. A Sikh `Kaur` and an English name beside a Chinese one (`Lim Mei Ling Jessica`) stay in `middle`, an accepted cost. **Measured in the fielded corpora** (2026-10-28, master `c8357e74`, counts only; recompute by counting each record's boxes by word count and word ending with `local_corpora.read_records`, grouped by the record's country or nationality, and the parse's role for the counted word with `parse()`): @@ -247,7 +247,7 @@ What decides a view is the reading each token took, so `Token.reading` comes fir ORCID's and PatentsView's given box holds the middle names too, so a two-word given box there cannot tell a compound given name from a middle name; PSC's middle box can, though its form steers what goes in it. The last row is a separate question from the patronymic: a name of given names only, written by people who have no family name, which no output field fixes and only a rule could read (an open question below). -**Render defaults** (decisions/3-0-reevaluations' clean slate, decided 2026-10-28). The default `str()` reads every name given-first and joins its words with spaces, so a name read family-first is turned round and a written separator is lost: `str(parse("김민준"))` is `'민준 김'`, `str(parse("王 小明"))` is `'小明 王'` and `str(parse("威廉·莎士比亚"))` is `'威廉 莎士比亚'`. In the committed corpora (2026-10-28, `c8357e74`), 264 names without a comma read family-first, `str()` renders every one differently from the text, and 159 of them are written unspaced; of 43 names holding the U+00B7 interpunct, `str()` drops it from 38. Decided: +**Render defaults** (decisions/3-0-reevaluations' clean slate). The default `str()` reads every name given-first and joins its words with spaces, so a name read family-first is turned round and a written separator is lost: `str(parse("김민준"))` is `'민준 김'`, `str(parse("王 小明"))` is `'小明 王'` and `str(parse("威廉·莎士比亚"))` is `'威廉 莎士比亚'`. In the committed corpora (2026-10-28, `c8357e74`), 264 names without a comma read family-first, `str()` renders every one differently from the text, and 159 of them are written unspaced; of 43 names holding the U+00B7 interpunct, `str()` drops it from 38. So: - `ParsedName` gains `order`, the order the name was read under (its script's or the declared `name_order`, a comma aside), an addition with `ADDED_IN` metadata whose missing value, in a 2.x pickle, renders as today. - `str()` and `render()`'s default spec put the family first where `order` does, and two words that stand side by side in the original and in the rendering are joined by what separated them there, nothing, a space, an interpunct or a nakaguro, so `str(parse("김민준"))` is `'김민준'` and `str(parse("威廉·莎士比亚"))` keeps the dot. A name with a comma or written given-first renders as today (`Smith, John` → `'John Smith'`). @@ -265,12 +265,12 @@ These change no signature but change `str()`'s output for every name read family | `ParsedName.order` | addition; old pickles load | | new `Policy` fields for rules, and `Locale.languages` (phase 3) | additions | | `Role`, `as_dict()`, `comparison_key()`, the seven role properties | unchanged | -| the default `str()`, `render()` spec and `initials()` output | behavior change, decided | -| a `Role` member for any of the four | a break, not proposed | +| the default `str()`, `render()` spec and `initials()` output | behavior change, accepted for 3.0 | +| a `Role` member for any of the four | a break, declined | -**Phase 3's Q1 confirmed** (proposed): `Locale.languages` stands as phase 3's draft proposes. Its open dependency was what `Locale` and `Policy` look like once the facade is gone; the 2026-10-28 freeze answers it, both keeping their 2.x shape with additions. With the `[rules]` proposal above, a shipped language pack is a `Locale` whose `languages` choose its vocabulary and whose `policy` carries its rules, so `Locale` needs no third field. +**Phase 3's Q1 confirmed**: `Locale.languages` stands as phase 3's draft proposes. Its open dependency was what `Locale` and `Policy` look like once the facade is gone; the 2026-10-28 freeze answers it, both keeping their 2.x shape with additions. With the `[rules]` proposal above, a shipped language pack is a `Locale` whose `languages` choose its vocabulary and whose `policy` carries its rules, so `Locale` needs no third field. -**Gender**, recorded only where a word gives a definitive clue (decision 9), is not part of this proposal: the vocabulary records it on readings (`bint`'s `gender = "f"`), and `Token.reading` could carry it later as one more addition. +**Gender**, recorded only where a word gives a definitive clue (decision 9), is not part of this design: the vocabulary records it on readings (`bint`'s `gender = "f"`), and `Token.reading` could carry it later as one more addition. ### The order of work @@ -287,7 +287,7 @@ Each sub-phase is one PR, measured on its own base as phase 2's and phase 3's ar | 4g | the compound given name, a rule kind (#704) | 4f | | 4h | the shape inventory per order (#469), starting with the title between a family and a given name under `FAMILY_FIRST` | — | -**Interleaved with phase 3**: 4a can start beside 3a, its `languages` waiting for 3a's index; 4b to 4d need nothing of phase 3 and unblock its packs, `cs`'s generations (3f) and `pt-BR`'s agnomens (3h) on 4b, `es` and `pt` (3g) on 4d and 4g; 4f lands with or after 3b, since both decide what a shipped `Locale` holds; an `is` pack, which no phase 3 sub-phase lists, needs 3b, 4c and 4f. 4e and 4h stand apart from both. So the order proposed is 3a and 4a, then 4b to 4d with 3b and 3c, then 4f, then the packs with 4g, with 4e at any point. +**Interleaved with phase 3**: 4a can start beside 3a, its `languages` waiting for 3a's index; 4b to 4d need nothing of phase 3 and unblock its packs, `cs`'s generations (3f) and `pt-BR`'s agnomens (3h) on 4b, `es` and `pt` (3g) on 4d and 4g; 4f lands with or after 3b, since both decide what a shipped `Locale` holds; an `is` pack, which no phase 3 sub-phase lists, needs 3b, 4c and 4f. 4e and 4h stand apart from both. So the order is 3a and 4a, then 4b to 4d with 3b and 3c, then 4f, then the packs with 4g, with 4e at any point. **Left open**, value questions inside the shapes above: the first surname's view and its name; whether `credentials` should leave the honorifics out, CLDR having no third bucket; which ending rules beside Icelandic a pack may name (Hungarian `-né` is a married name, not a patronymic); and the names of given names only (the last table row), whose reading needs a rule that tells a mononym from a given name and a family name, not an output field. From 563af7fdf6249b191f7a858303e710c0d9604632 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Wed, 28 Oct 2026 03:08:06 -0700 Subject: [PATCH 09/39] Fix the docs review's findings on phase 4's design record (#754) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Scope the render exclusion to word order, so a given-first interpunct name keeps its dot; re-count the corpus figures over distinct entries under their declared orders and name the ending tests; point at phase 3's unmerged draft instead of an entry master lacks; say surname2 is read from O6's tag; leave order out of comparison; list the four reserved names as without a sub-phase; drop the Đỗ promise Token.reading cannot keep and the release plan no entry records. Co-Authored-By: Claude Opus 5.5 --- docs/design/decisions/vocabulary-model.md | 18 +++++++-------- docs/design/vocabulary-model-rfc.md | 28 +++++++++++------------ 2 files changed, 23 insertions(+), 23 deletions(-) diff --git a/docs/design/decisions/vocabulary-model.md b/docs/design/decisions/vocabulary-model.md index 3bc401c8b..76d7f2f52 100644 --- a/docs/design/decisions/vocabulary-model.md +++ b/docs/design/decisions/vocabulary-model.md @@ -10,7 +10,7 @@ The vocabulary was a set of base sets plus marker subsets (`particles` and `part - The default loads every language and keeps working for names of any language. Choosing a language is to narrow it in phase 3 (#753): a pack is to settle only the clashes where its language has a vocabulary reading, a strict pack to drop other languages' name-level readings too, and a pack's preferred reading to be able to outrank one a shape rule derives — 2026-10-15/16. - `e`, `y` and `i` stay in the default; choosing a language is to settle their one-case fork, which no shipped pack does yet — 2026-10-15/16. - A patronymic field is to come in 3.0, as a view that names its words without emptying `family`, and the particle kinds already draw the `clan`/`patronymic` line for it: a patronymic is followed by a parent's name, a clan prefix by a family name the lineage shares — 2026-10-15/16; 2026-10-19 #749; 2026-10-28 #754. -- 3.0's output model keeps the seven roles and adds every new field as a view read from `Token.reading`, the reading each token took: `generation` and `credentials` partitioning `suffix`, `patronymic`, and `surname2` where rules/O6 read two family names; `middle` and `surnames` keep their meaning, a compound given name being a rule a caller or pack turns on; the default `str()` renders a name in the order it was read in with its written separators; a language file's `[rules]` compile into its `Locale`'s policy patch — 2026-10-28 #754. +- 3.0's output model keeps the seven roles and adds every new field as a view, read from `Token.reading`, the reading each token took, or for `surname2` from rules/O6's tag: `generation` and `credentials` partitioning `suffix`, `patronymic`, and `surname2` where rules/O6 read two family names; `middle` and `surnames` keep their meaning, a compound given name being a rule a caller or pack turns on; the default `str()` renders a name in the order it was read in with its written separators; a language file's `[rules]` compile into its `Locale`'s policy patch — 2026-10-28 #754. - Gender is recorded only where a word gives a definitive clue: a feminine form or one half of a fixed pair, never an unmarked noun — 2026-10-15/16; 2026-10-19 #750, title kinds. - Title-phrase modifiers count as titles only beside a head title, at the head of the name only and never in a comma part (`Wing Lau` → given 'Wing', `John Smith, Vice` → title 'Vice'), save a modifier that may be a given name, which the head's test reads in every part (`Lau, Wing Kai` → given 'Wing'), rules/H9 — 2026-10-15/16; 2026-10-19 #750, title kinds; 2026-10-23 #752; 2026-10-25 #752, phase 2i; 2026-10-26 #752, phase 2j, `wing`. - Name readings are hand-declared, and only on a word some language file claims as vocabulary or refuses, so the vocabulary carries no surname list (the segmentation lists, Korean and the ZH pack's, are outside the per-word model) — 2026-10-15/16; 2026-10-19. @@ -243,14 +243,14 @@ Open: #913 (the title words phase 2j left, read as titles though borne as names) - **2026-10-26 #752, phase 2l — THE BRACKET ESCAPE READS THE DERIVED CLASH**, carrying out item 16 of the 2026-10-23 #752 entry above (#810). A member of the ambiguous class, the class 2a derives from name readings, alone in brackets or quotes after the name is the suffix where its writing settles it (`John Smith (MA)`, `John Smith (M.A)` → suffix), read in one pass, and stays the nickname otherwise (`John Smith (Ed)`, `JOHN SMITH (MA)`, `Jeffrey (JD) Bricken`). It is recorded as rules/S13, a rule of its own rather than the amendment to S1 item 16 planned, the position it reads being evidence S1's signal does not read; the writing is S3's and S6's evidence, which S13 cites. The record, the measurements and what was declined are decisions/S13's 2026-10-26 #810 entry. - **2026-10-26 #752, phase 2m — THE READING NOT TAKEN SURFACES AS `Ambiguity.alternative`, WITHOUT ITS LANGUAGES**, carrying out item 13 of the 2026-10-23 #752 entry above and item 7's source with it: a word the credential's run took by position alone reports its name reading's source as `position`, a fourth source beside the three the item named, and a word with a declared name reading reports `declared`. The pickle guard was widened as item 13 asked, every skew the narrow guard refused still refused. The reading's languages were left out: the per-word index costs about 0.4 ms of a 19.5 ms import and describes the default vocabulary, not a pack's or a caller's lexicon, so they wait for 3.0 with `Token.reading`. The record, the measurements and what was declined are decisions/A1's 2026-10-26 #752 entry. -- **2026-10-28 #754 (Derek) — PHASE 4'S OUTPUT MODEL IS AGREED: THE SEVEN ROLES STAY AND EVERY NEW FIELD IS A VIEW READ FROM `Token.reading`; A PATRONYMIC NAMES ITS WORDS WITHOUT EMPTYING `family`; THE RENDER DEFAULTS FOLLOW THE ORDER A NAME WAS READ IN.** Derek agreed every item in chat on 2026-10-28, items 6 and 4 and the `surnames` line of item 5 first, then the rest. The design overview is docs/design/vocabulary-model-rfc.md's Phase 4 section. Decision 8 (a patronymic field in 3.0) is kept and its stated value revised (item 3); decisions 9 and 14 are kept as written. The core API is frozen for 3.0 except for additions (decisions/3-0-reevaluations, 2026-10-28 #925, THE CORE API), and each item says whether it is one. The inputs are #754 and its comment of 2026-10-23, #326, #704 and #694. Measurements were taken 2026-10-28 on master `c8357e74` (Python 3.11.17) over the fielded local corpora (`orcid-researchers`, `psc-individuals`, `patentsview-inventors`) and the committed fields files, with throwaway scripts that are not kept, so each figure is evidence for the decision and each sub-phase re-measures on its own base. Local-corpus figures are counts. Recompute the box counts by reading each source's records with `local_corpora.read_records`, counting its boxes by word count and word ending per country or nationality, and the role the parse gives the counted word with `parse()`. - 1. **`Role` keeps its seven members, and every field asked for is a view.** A role partitions the tokens, and each of the five fields asked for (a second surname, a patronymic, generation and credentials, a second given name, the reading taken) subdivides a role that exists, so each is computed from the tokens, a property and a render key, as `family_base` and `given_names` are (rules/R1, rules/R2). `as_dict()`, `comparison_key()` and the seven role properties are unchanged, and every view is an addition. Weighed against: a `Role` member for any of them, a break of the frozen `ParsedName` (`as_dict()` gains a key, `comparison_key()` a component, so a stored key stops matching, and the role its words leave returns less text than in 2.x), with no 2.x feature release left to warn in. +- **2026-10-28 #754 (Derek) — PHASE 4'S OUTPUT MODEL IS AGREED: THE SEVEN ROLES STAY AND EVERY NEW FIELD IS A VIEW, MOST READ FROM `Token.reading`; A PATRONYMIC NAMES ITS WORDS WITHOUT EMPTYING `family`; THE RENDER DEFAULTS FOLLOW THE ORDER A NAME WAS READ IN.** Derek agreed every item in chat on 2026-10-28, items 6 and 4 and the `surnames` line of item 5 first, then the rest. The design overview is docs/design/vocabulary-model-rfc.md's Phase 4 section. Decision 8 (a patronymic field in 3.0) is kept and its stated value revised (item 3); decisions 9 and 14 are kept as written. The core API is frozen for 3.0 except for additions (decisions/3-0-reevaluations, 2026-10-28 #925, THE CORE API), and each item says whether it is one. The inputs are #754 and its comment of 2026-10-23, #326, #704 and #694. Measurements were taken 2026-10-28 on master `c8357e74` (Python 3.11.17) over the fielded local corpora (`orcid-researchers`, `psc-individuals`, `patentsview-inventors`), the committed fields files and the committed corpus's distinct entries (`snapshot.corpus_entries()`, each parsed under its declared order), with throwaway scripts that are not kept, so each figure is evidence for the decision and each sub-phase re-measures on its own base. Local-corpus figures are counts. Recompute the box counts by reading each source's records with `local_corpora.read_records`, counting its boxes by word count and word ending per country or nationality, and the role the parse gives the counted word with `parse()`; the ending tests are the RFC's Phase 4 table's, and other tests give counts a few percent apart. + 1. **`Role` keeps its seven members, and every field asked for is a view.** A role partitions the tokens, and each of the four fields asked for (a second surname, a patronymic, generation and credentials, a second given name) subdivides a role that exists, so each is computed from the tokens, a property and a render key, as `family_base` and `given_names` are (rules/R1, rules/R2). `as_dict()`, `comparison_key()` and the seven role properties are unchanged, and every view is an addition. Weighed against: a `Role` member for any of them, a break of the frozen `ParsedName` (`as_dict()` gains a key, `comparison_key()` a component, so a stored key stops matching, and the role its words leave returns less text than in 2.x), with no 2.x feature release left to warn in. 2. **`Token.reading`, the reading each token took**, a `Reading` (decisions/A1's type), `None` for a token `replace()` builds, with `ADDED_IN` metadata so a 2.x pickle loads, and left out of comparison so `==` keeps its 2.x meaning. `Reading` gains `languages`, the files holding the reading, filled from phase 3's reading index (3a) and empty where no file holds it; `ReadingKind` gains `PATRONYMIC`. The views read it. It carries out the 2026-10-23 #752 entry's item 13 and the 2026-10-26 phase 2m entry's deferral of the languages. - 3. **a `patronymic` view, filled by a word or a rule that says patronymic, with `family` and `middle` unchanged.** The default fills it from the vocabulary's `patronymic` particles (`bin`, `binti`, `ibn`, `bint`, `bar`, `bat` and the native-script forms) with the name they introduce (`Mahathir bin Mohamad` → family `bin Mohamad`, as today, and patronymic `bin Mohamad`); RU's and TR_AZ's rules fill it where a caller turns them on (`Ivan Ivanovich Ivanov` under RU → middle `Ivanovich` and patronymic `Ivanovich`); an Icelandic ending rule fills it under an `is` pack only, since the default cannot tell `-dóttir` and `-son` from Swedish surnames (ORCID's Swedish records end their family box in `-son` or `-dóttir` in 2,184 of 20,000). A matronymic, written the same way, fills it too. SUPERSEDES: the RFC's earlier statement of decision 8's value, that `family` can be empty for a name with a patronymic. Measured: filers put an Icelandic patronymic in the family box (ORCID 1,647 of 2,583 Icelandic records, PatentsView 535 of 722, PSC 101 of 125), and CLDR's samples put it in `surname` (9 of 9), while Wikidata's `patronymic-is` and `patronymic-my` strata record 108 names with a patronymic and no family name, of which the parse puts the patronymic in `family` 107 times. An East Slavic patronymic stands in the given part: CLDR's `ru` samples put it in `given2` (6 of 6); PatentsView's Russian given boxes end in one 8,423 times of 20,000, and the parse reads such a word as the middle 10,902 times of 10,962; 90 of the 112 Russian nationals in PSC who filled the middle-name box wrote one there. A `family` that keeps it and a `patronymic` that names it agree with both kinds of source. Weighed against: emptying `family` where the patronymic is recognized, which moves the field the fielded sources fill and changes `family` for every Icelandic and nasab name, a behavior change of a frozen property; and a neutral parent-names name for the view, which buys nothing the view can detect, since a matronymic and a patronymic are written alike. - 4. **`generation` and `credentials` are views that partition `suffix` (#326)**, not roles, `generation` the tokens read as a generation or an agnomen, `credentials` the rest, the post-nominals that are neither (`Esq.`, the honorifics after a name) falling in `credentials`, CLDR's catch-all, with `Token.reading.kind` telling them apart for a caller who needs it. Measured: of the 1,170 committed corpus names with a suffix, 88 hold a generation and a credential together and 87 only a listed post-nominal of another kind; of the 615 CLDR renderings with either field, the 75 whose suffix holds exactly CLDR's words split into CLDR's `generation` and `credentials` by the vocabulary's reading kind in all 75. Weighed against: two new `Role` members (item 1), and a third view for the honorifics, left open since CLDR has none. + 3. **A `patronymic` view, filled by a word or a rule that says patronymic, with `family` and `middle` unchanged.** The default fills it from the vocabulary's `patronymic` particles (`bin`, `binti`, `ibn`, `bint`, `bar`, `bat` and the native-script forms) with the name they introduce (`Mahathir bin Mohamad` → family `bin Mohamad`, as today, and patronymic `bin Mohamad`); RU's and TR_AZ's rules fill it where a caller turns them on (`Ivan Ivanovich Ivanov` under RU → middle `Ivanovich` and patronymic `Ivanovich`); an Icelandic ending rule fills it under an `is` pack only, since the default cannot tell `-dóttir` and `-son` from Swedish surnames (ORCID's Swedish records end their family box in `-son` or `-dóttir` in 2,184 of 20,000). A matronymic, written the same way, fills it too. SUPERSEDES: the RFC's earlier statement of decision 8's value, that `family` can be empty for a name with a patronymic. Measured: filers put an Icelandic patronymic in the family box (ORCID 1,647 of 2,583 Icelandic records, PatentsView 535 of 722, PSC 101 of 125), and CLDR's `is` samples put it in `surname` or `surname-core` (9 of the 9 fields of the `is` locale holding one), while Wikidata's `patronymic-is` and `patronymic-my` strata record 108 names with a patronymic and no family name, of which the parse puts the patronymic in `family` 107 times. An East Slavic patronymic stands in the given part: CLDR's `ru` samples put it in `given2` (6 of 6); PatentsView's Russian given boxes end in one 8,423 times of 20,000, and the parse reads such a word as the middle 10,902 times of 10,962; 90 of the 112 Russian nationals in PSC who filled the middle-name box wrote one there. A `family` that keeps it and a `patronymic` that names it agree with both kinds of source. Weighed against: emptying `family` where the patronymic is recognized, which moves the field the fielded sources fill and changes `family` for every Icelandic and nasab name, a behavior change of a frozen property; and a neutral parent-names name for the view, which buys nothing the view can detect, since a matronymic and a patronymic are written alike. + 4. **`generation` and `credentials` are views that partition `suffix` (#326)**, not roles, `generation` the tokens read as a generation or an agnomen, `credentials` the rest, the post-nominals that are neither (`Esq.`, the honorifics after a name) falling in `credentials`, CLDR's catch-all, with `Token.reading.kind` telling them apart for a caller who needs it. Measured: of the 1,054 committed corpus entries with a suffix, 87 hold a generation (or agnomen) and a credential together and 65 only a listed post-nominal of another kind; of the 615 CLDR renderings with either field, the 75 whose suffix holds exactly CLDR's words split into CLDR's `generation` and `credentials` by the vocabulary's reading kind in all 75. Weighed against: two new `Role` members (item 1), and a third view for the honorifics, left open since CLDR has none. 5. **a `surname2` view, the family's second unit where rules/O6 read the family as two**, marked by a new stable tag where O6 takes its word, empty otherwise; `family` keeps both units as O6 makes it, and a family comma alone marks nothing (`Lloyd Webber, Andrew`). Measured: a two-word family box in ORCID's records from Spain 48.0%, Mexico 64.7%, Colombia 72.8% and Chile 50.5%, against 3.9% from the United States. Left open: a view for the first unit and its name, `surname` standing close to the `surnames` view that exists. Declined: redefining `surnames` as the two units, which would repeat `family`, the family holding both where O6 reads two, and lose what `surnames` does today, joining `middle` and `family` so that a double surname the default splits (`Gabriel García Márquez` → middle `García`) reads whole, the view rules/P2 names for that reading. - 6. **the render defaults follow the order the name was read in, a behavior change of the output rather than a break of the API.** `ParsedName` gains `order` (an addition, `ADDED_IN`, a 2.x pickle rendering as today); the default `str()` and `render()` spec put the family first where `order` does, and join two words that stand side by side in the original and the rendering with what separated them there (nothing, a space, an interpunct, a nakaguro); `initials()` gives nothing for a word in a script with no initials. A name with a comma or written given-first renders as today. Measured in the committed corpora: 264 names without a comma read family-first, `str()` renders all 264 differently from their text (`김민준` → `'민준 김'`), 159 of them written unspaced, and it drops the U+00B7 interpunct from 38 of the 43 names holding one. This carries out decisions/3-0-reevaluations' render-default clean slate. No signature changes, but `str()`'s output changes for every name read family-first, and no 2.x release can warn about a `str()` result, so it is taken as an improvement for 3.0's release notes, not a break needing a warning. + 6. **The render defaults follow the order the name was read in, a behavior change of the output rather than a break of the API.** `ParsedName` gains `order` (an addition, `ADDED_IN`, a 2.x pickle rendering as today, left out of comparison); the default `str()` and `render()` spec put the family first where `order` does, and join two words that stand side by side in the original and the rendering with what separated them there (nothing, a space, an interpunct, a nakaguro); `initials()` gives nothing for a word in a script with no initials. A name with a comma or written given-first keeps today's word order, the written separator kept for every name (`威廉·莎士比亚` keeps its dot). Measured over the committed corpus's distinct entries: 728 without a comma read family-first, `str()` renders all 728 differently from their text (`김민준` → `'민준 김'`), 150 of them written unspaced, and it drops the U+00B7 interpunct from 36 of the 41 holding one. This carries out decisions/3-0-reevaluations' render-default clean slate. No signature changes, but `str()`'s output changes for every name read family-first, and no 2.x release can warn about a `str()` result, so it is taken as an improvement for 3.0's release notes, not a break needing a warning. 7. **`middle` keeps its meaning (#704)**, every word of the given part after the given name, the given name one unit unless a rule joins words into it, as rules/P5 already does (`Abdul Aziz Tariq` → given `Abdul Aziz`); a compound given name is a `[rules]` rule a caller or a pack turns on, CLDR's `given2` is `middle`, and `Kaur` and an English name beside a Chinese one stay in `middle`, an accepted cost. Measured: in PSC, whose form has a middle-name box, 2,499 of 1,040,708 forename boxes hold two or more words, and of 6,101 Spanish nationals 25 do while 2,621 fill the middle box; ORCID's and PatentsView's given box holds middles too, so it cannot separate the two. The `Maria do Carmo` shape, a particle inside a given box of three or more words with a family box filled, reads the given box's words into the family in 1,812 of 1,923 ORCID records and 1,165 of 1,219 PatentsView records from Brazil, Portugal, Spain, Mexico, Argentina, Colombia, Chile and Italy, which is that rule's case. Weighed against: #704's option (b), one given unit by default, which changes `middle` for every English name of three words. - 8. **a language file's `[rules]` compiles into its shipped `Locale`'s policy patch**, each rule kind a `Policy` field (an addition), so RU and TR_AZ are rebuilt from their files and `Locale` gains no field for rules. With it, **phase 3's Q1 is confirmed**: `Locale.languages` (the 2026-10-27 #753 entry's item 1) stands, its open dependency answered by the 2026-10-28 freeze, which keeps `Locale` and `Policy` in their 2.x shape. - 9. **the work order**, one PR each, measured on its own base, the snapshot recording each new view where it is non-empty: 4a `Token.reading` and `ReadingKind.PATRONYMIC` (neutral; `languages` after 3a); 4b `generation` and `credentials` (neutral); 4c `patronymic` (neutral); 4d `surname2` (neutral); 4e the render defaults and `order`; 4f `[rules]` into the policy patch, with or after 3b; 4g the compound given name, after 4f; 4h the shape inventory per order (#469). Interleaved with phase 3: 4a beside 3a; 4b to 4d before the packs that need them, `cs` (3f) and `pt-BR` (3h) on 4b, `es` and `pt` (3g) on 4d and 4g; an `is` pack on 3b, 4c and 4f. - 10. **what stays out of the output model.** Gender (decision 9) waits, `Token.reading` able to carry it later as one more addition. Names of given names only are a rule's question, not a field's: ORCID's empty family box holds 68,006 of 1,831,533 records, 15.9% to 18.7% of those from India, Pakistan, Bangladesh and Indonesia, and the parse fills `family` in 51,367 of the 51,415 with two or more given-box words. #694's two-word surname after a given name (`Tariq Abdul Aziz`) is a parse rule for the default, outside phase 4. + 8. **A language file's `[rules]` compiles into its shipped `Locale`'s policy patch**, each rule kind a `Policy` field (an addition), so RU and TR_AZ are rebuilt from their files and `Locale` gains no field for rules. With it, **phase 3's open API dependency is answered**: its draft (branch `feat/vocabulary-phase3-design`, unmerged) carries a choice of languages on `Locale.languages`, assuming `Locale` and `Policy` keep their 2.x shape after the facade's removal, and the 2026-10-28 freeze keeps them so. The rest of that draft, strict as an argument of the whole choice included, stays phase 3's to decide; its entry, when it lands, says the dependency is answered. + 9. **The work order**, one PR each, measured on its own base, the snapshot recording each new view where it is non-empty: 4a `Token.reading` and `ReadingKind.PATRONYMIC` (neutral; `languages` after 3a); 4b `generation` and `credentials` (neutral); 4c `patronymic` (neutral); 4d `surname2` (neutral); 4e the render defaults and `order`; 4f `[rules]` into the policy patch, with or after 3b; 4g the compound given name, after 4f; 4h the shape inventory per order (#469). Interleaved with phase 3 (its sub-phases the draft's): 4a beside 3a; 4b to 4d before the packs that need them, `cs` (3f) and `pt-BR` (3h) on 4b, `es` and `pt` (3g) on 4d and 4g; an `is` pack on 3b, 4c and 4f. + 10. **What stays out of the output model.** Gender (decision 9) waits, `Token.reading` able to carry it later as one more addition. Names of given names only are a rule's question, not a field's: ORCID's empty family box holds 68,006 of 1,831,533 records, 15.9% to 18.7% of those from India, Pakistan, Bangladesh and Indonesia, and the parse fills `family` in 51,367 of the 51,415 with two or more given-box words. #694's two-word surname after a given name (`Tariq Abdul Aziz`) is a parse rule for the default, outside phase 4. Phase 1's four reserved names said to be phase 4's (`separators`, `bare_only`, `leading_only`, `trailing_only`) have no sub-phase yet. diff --git a/docs/design/vocabulary-model-rfc.md b/docs/design/vocabulary-model-rfc.md index 9cb4ccca2..217d96c25 100644 --- a/docs/design/vocabulary-model-rfc.md +++ b/docs/design/vocabulary-model-rfc.md @@ -134,7 +134,7 @@ Phase 2l carried this out as [rules/S13](rules/S13-bracketed-member-settled-by-i ### Readings on the parse -A caller should be able to see the reading behind a parse. **An `Ambiguity` carries the reading the parse did not take** (`Ambiguity.alternative`: its kind, the field it would have put the token in, and whether it came from the vocabulary, a declared name or a shape), and **a token carries the reading it did** (`Token.reading`), so `Nguyen Van Do`, which forks nothing, can still expose `Đỗ` without a report nobody would act on. A word the credential's run takes by position alone is told from a word with a declared name reading by that source, so it needs no new ambiguity kind. `Ambiguity.alternative` shipped in 2.7 (decisions/A1, 2026-10-26 #752): a `Reading` of a kind, the field the word would land in with every other word read as the parse read it, and a source, which is `vocabulary`, `declared`, `shape` or `position`, the fourth being the reading of an ordinary word nothing lists, which is what a word the credential's run took by position alone has. The pickle guard was widened to accept a field added later. A `languages` field, naming the files that hold a reading, was weighed and left out: it needs the per-word reading index, which costs about 0.4 ms of a 19.5 ms import and could not describe a pack's or a caller's lexicon, so it waits with `Token.reading` for 3.0. +A caller should be able to see the reading behind a parse. **An `Ambiguity` carries the reading the parse did not take** (`Ambiguity.alternative`: its kind, the field it would have put the token in, and whether it came from the vocabulary, a declared name or a shape), and **a token carries the reading it did** (`Token.reading`), so a word whose reading forks nothing still says what it was read as, without a report nobody would act on (phase 4's 4a; a native spelling such as `do`'s Vietnamese `Đỗ` is not part of it). A word the credential's run takes by position alone is told from a word with a declared name reading by that source, so it needs no new ambiguity kind. `Ambiguity.alternative` shipped in 2.7 (decisions/A1, 2026-10-26 #752): a `Reading` of a kind, the field the word would land in with every other word read as the parse read it, and a source, which is `vocabulary`, `declared`, `shape` or `position`, the fourth being the reading of an ordinary word nothing lists, which is what a word the credential's run took by position alone has. The pickle guard was widened to accept a field added later. A `languages` field, naming the files that hold a reading, was weighed and left out: it needs the per-word reading index, which costs about 0.4 ms of a 19.5 ms import and could not describe a pack's or a caller's lexicon, so it waits with `Token.reading` for 3.0. ### The phase 1 reserved names @@ -184,7 +184,7 @@ Consequences that follow from the definitions: ## Phase 4: order, shape and the output model -Status: **the output model is agreed** (2026-10-28, #754). The record is decisions/vocabulary-model's 2026-10-28 #754 entry; the output-model and work-order subsections below are the design its items describe. The first two subsections, order and shape and the rules a pack can choose, are as written before and are amended only where the design reaches them. Phase 4 is 3.0 work: 3.0 ships it with phase 3 and the facade's removal, with alphas along the way, and it ships when phase 4 is done. The core API (`parse`, `Parser`, `Lexicon`, `Policy`, `Locale`, `ParsedName`) is frozen for 3.0 except for additions (decisions/3-0-reevaluations, 2026-10-28 #925, THE CORE API), so every change below says whether it is an addition. +Status: **the output model is agreed** (2026-10-28, #754). The record is decisions/vocabulary-model's 2026-10-28 #754 entry; the output-model and work-order subsections below are the design its items describe. The first two subsections, order and shape and the rules a pack can choose, are as written before and are amended only where the design reaches them. Phase 4 is 3.0 work. The core API (`parse`, `Parser`, `Lexicon`, `Policy`, `Locale`, `ParsedName`) is frozen for 3.0 except for additions (decisions/3-0-reevaluations, 2026-10-28 #925, THE CORE API), so every change below says whether it is an addition. ### Order is not shape @@ -221,19 +221,19 @@ The real-name corpora agree on the priority. In the GND and Wikidata corpora, Sp ### The output model -Phase 4 also decides what a parse returns, aligned with CLDR's person-name fields, which gives ground truth for free. Five fields were asked for: a second surname (`surname2`), a patronymic (decision 8), credentials apart from generations (#326), a meaning for `middle` when a given name is several words (#704), and the reading each token took (`Token.reading`, phase 2's item 13). The design answers all five with one shape. +Phase 4 also decides what a parse returns, aligned with CLDR's person-name fields, which gives ground truth for free. Five things were asked for: a second surname (`surname2`), a patronymic (decision 8), credentials apart from generations (#326), a meaning for `middle` when a given name is several words (#704), and the reading each token took (`Token.reading`, phase 2's item 13). The first four are fields, and the design answers them with one shape; the fifth is what most of them are read from. -**The seven roles stay, and every new field is a view.** A `Role` partitions the tokens: each word is in exactly one of title, given, middle, family, suffix, nickname and maiden. Each of the five asks to subdivide a role that exists, not to add a part a name has outside them: a second surname is part of the family, generation and credentials are parts of the suffix, a second given name is the middle, and a patronymic is a word the parse already puts in the family (Icelandic, Arabic and Malay names) or in the middle (East Slavic names). So each is a view computed from the tokens, as `family_base`, `family_particles`, `surnames` and `given_names` are today (rules/R1, rules/R2): a property and a render key, with `as_dict()`, `comparison_key()` and the seven role properties unchanged. Every view is an addition under the 3.0 freeze. A new `Role` member would not be: `as_dict()` would gain a key, `comparison_key()` a component (a stored key would stop matching), `for role in Role` a member, and the role a token moves out of would return less text than it does in 2.x, a change in what `family` or `suffix` means that no 2.x warning has announced. +**The seven roles stay, and every new field is a view.** A `Role` partitions the tokens: each word is in exactly one of title, given, middle, family, suffix, nickname and maiden. Each of the four fields subdivides a role that exists, not to add a part a name has outside them: a second surname is part of the family, generation and credentials are parts of the suffix, a second given name is the middle, and a patronymic is a word the parse already puts in the family (Icelandic, Arabic and Malay names) or in the middle (East Slavic names). So each is a view computed from the tokens, as `family_base`, `family_particles`, `surnames` and `given_names` are today (rules/R1, rules/R2): a property and a render key, with `as_dict()`, `comparison_key()` and the seven role properties unchanged. Every view is an addition under the 3.0 freeze. A new `Role` member would not be: `as_dict()` would gain a key, `comparison_key()` a component (a stored key would stop matching), `for role in Role` a member, and the role a token moves out of would return less text than it does in 2.x, a change in what `family` or `suffix` means that no 2.x warning has announced. -What decides a view is the reading each token took, so `Token.reading` comes first: +What decides a view is the reading each token took, so `Token.reading` comes first; `surname2` alone is read from a tag O6 writes, a rule rather than a word deciding it: - **`Token.reading`**: a `Token` field holding the `Reading` the parse took, the same type `Ambiguity.alternative` holds the reading not taken in (decisions/A1), `None` for a token `replace()` builds. Its `kind` says what the word was read as, its `source` why, and a `languages` field, added to `Reading`, names the language files holding that reading, filled from phase 3's reading index (3a) and left empty for a reading no file holds (a caller's own vocabulary, a shape, position alone). The field takes `ADDED_IN` metadata, so a 2.x pickle loads with `None` (the pickle guard as widened in 2.7), and is left out of comparison (`compare=False`), so `==` between two `ParsedName`s means what it means in 2.x. `ReadingKind` gains `PATRONYMIC`, its docstring already allowing new kinds. -- **Generation and credentials** (#326): `generation`, the suffix tokens whose reading is a generation or an agnomen (`Jr.`, `III`, `Filho`), and `credentials`, the rest of the suffix (`PhD`, `Esq.`, the honorifics after a name), so the two partition `suffix` and `suffix` is unchanged. CLDR has only these two buckets; the post-nominals that are neither (`Esq.`, the CJK and Hebrew honorifics, `Rinpoche`) fall in `credentials`, CLDR's catch-all, and a caller who wants them apart reads `Token.reading.kind`, which says `post-nominal`. Measured 2026-10-28 on `c8357e74`: of the 1,170 committed corpus names with a suffix, 88 hold a generation and a credential together, and 87 hold only a listed post-nominal of another kind. Of the 615 CLDR renderings with a generation or credentials field, 75 parse with exactly CLDR's suffix words, and in all 75 a split by the vocabulary's reading kind gives CLDR's `generation` and `credentials` both; the other 540 differ in which words the suffix holds, a parse question the split cannot fix. Recompute: classify each suffix token by the kind of its reading in nameparser/vocab/ (a roman numeral the generation), over `cldr_fields.jsonl`. -- **A patronymic** (decision 8): `patronymic`, the tokens whose reading is `PATRONYMIC`, in whichever role the parse put them, with `family` and `middle` unchanged. What fills it is a word or a rule that says patronymic, never a guess from position: the vocabulary's `patronymic` particles (`bin`, `binti`, `ibn`, `bint`, Hebrew `bar` and `bat` and the native-script forms) with the name after them, in the default, so `Mahathir bin Mohamad` reads family `bin Mohamad` as today and patronymic `bin Mohamad` beside it; the East Slavic and Turkic rules where RU or TR_AZ turns them on, so `Ivan Ivanovich Ivanov` under RU keeps middle `Ivanovich` and adds patronymic `Ivanovich`; and an Icelandic ending rule an `is` pack would turn on, the default unable to tell `Karlsdóttir` from the Swedish surname `Johansson`. The name `patronymic` is kept for a matronymic too, which is written the same way and cannot be told from one. **`family` is not emptied** by recognizing a patronymic, revising what this section said before: the people filling in the fielded corpora put the patronymic in the family box (below), as CLDR's samples put an Icelandic patronymic in `surname` (9 of 9) and a Russian one in `given2` (6 of 6), our `middle`, while Wikidata records an Icelandic name as a patronymic with no family name; a `family` that keeps it and a `patronymic` that names it satisfy both. A caller writing "Mr. {family}" reads `patronymic` to see that the family is one. +- **Generation and credentials** (#326): `generation`, the suffix tokens whose reading is a generation or an agnomen (`Jr.`, `III`, `Filho`), and `credentials`, the rest of the suffix (`PhD`, `Esq.`, the honorifics after a name), so the two partition `suffix` and `suffix` is unchanged. CLDR has only these two buckets; the post-nominals that are neither (`Esq.`, the CJK and Hebrew honorifics, `Rinpoche`) fall in `credentials`, CLDR's catch-all, and a caller who wants them apart reads `Token.reading.kind`, which says `post-nominal`. Measured 2026-10-28 on `c8357e74`, over the committed corpus's 6,533 distinct entries (`snapshot.corpus_entries()`) each parsed under its declared order: of the 1,054 with a suffix, 87 hold a generation (or agnomen) and a credential together, and 65 hold only a listed post-nominal of another kind. Of the 615 CLDR renderings with a generation or credentials field, 75 parse with exactly CLDR's suffix words, and in all 75 a split by the vocabulary's reading kind gives CLDR's `generation` and `credentials` both; the other 540 differ in which words the suffix holds, a parse question the split cannot fix. Recompute: classify each suffix token by the kinds of its readings in nameparser/vocab/, skipping `default = false` ones, a generation or agnomen reading winning, an unlisted roman numeral `i` to `x` a generation; over the corpus entries above, and over `cldr_fields.jsonl` for CLDR. +- **A patronymic** (decision 8): `patronymic`, the tokens whose reading is `PATRONYMIC`, in whichever role the parse put them, with `family` and `middle` unchanged. What fills it is a word or a rule that says patronymic, never a guess from position: the vocabulary's `patronymic` particles (`bin`, `binti`, `ibn`, `bint`, Hebrew `bar` and `bat` and the native-script forms) with the name after them, in the default, so `Mahathir bin Mohamad` reads family `bin Mohamad` as today and patronymic `bin Mohamad` beside it; the East Slavic and Turkic rules where RU or TR_AZ turns them on, so `Ivan Ivanovich Ivanov` under RU keeps middle `Ivanovich` and adds patronymic `Ivanovich`; and an Icelandic ending rule an `is` pack would turn on, the default unable to tell `Karlsdóttir` from the Swedish surname `Johansson`. The name `patronymic` is kept for a matronymic too, which is written the same way and cannot be told from one. **`family` is not emptied** by recognizing a patronymic, revising what this section said before: the people filling in the fielded corpora put the patronymic in the family box (below), as CLDR's samples put an Icelandic patronymic in `surname` or `surname-core` (9 of the 9 fields of the `is` locale holding one) and a Russian one in `given2` (6 of 6), our `middle`, while Wikidata records an Icelandic name as a patronymic with no family name; a `family` that keeps it and a `patronymic` that names it satisfy both. A caller writing "Mr. {family}" reads `patronymic` to see that the family is one. - **A second surname**: `surname2`, the family's second name unit where a rule has read the family as two (rules/O6, which a caller or an `es` pack turns on), and empty otherwise, `family` keeping both units as O6 makes it today. O6 already knows which unit it took, so it marks where the second one starts with a new stable tag; a family comma alone marks nothing, since `Lloyd Webber, Andrew` is not two surnames, and a hyphenated family is one token and stays whole. Whether the first unit gets a view of its own, and its name (`surname` sits close to the `surnames` that exists), is a value question below. `surnames` keeps its meaning, `middle` and `family` joined, the reading of a double surname the default splits. - **`middle`** (#704): keeps its meaning, every word of the given part after the given name, and the given name is one name unit unless a rule joins words into it, as rules/P5 already does (`Abdul Aziz Tariq` → given `Abdul Aziz`). A compound given name (`Maria do Carmo`, `María José`) is the `[rules]` rule above, turned on by a caller or a pack, moving its words into `given` only under that rule. CLDR's `given2` is `middle`, so no view is added. A Sikh `Kaur` and an English name beside a Chinese one (`Lim Mei Ling Jessica`) stay in `middle`, an accepted cost. -**Measured in the fielded corpora** (2026-10-28, master `c8357e74`, counts only; recompute by counting each record's boxes by word count and word ending with `local_corpora.read_records`, grouped by the record's country or nationality, and the parse's role for the counted word with `parse()`): +**Measured in the fielded corpora** (2026-10-28, master `c8357e74`, counts only; recompute by counting each record's boxes by word count and word ending with `local_corpora.read_records`, grouped by the record's country or nationality, and the parse's role for the counted word with `parse()`. The ending tests, case-insensitive on the box's last word: an Icelandic patronymic a one-word family box ending `dóttir`, `dottir`, `sson` or `son` (the role count, `dóttir`, `dottir` or `sson`); an East Slavic patronymic `ovich`, `ovna`, `evich`, `evna`, `ichna` or their Cyrillic forms, in a given box of two or more words; nasab `bin`, `binti`, `bint`, `bte`, `bt`, `b.`, `ibn` or `ben` anywhere in the box. Other tests give other counts by a few percent): | Question | Where | Count | |---|---|---| @@ -247,10 +247,10 @@ What decides a view is the reading each token took, so `Token.reading` comes fir ORCID's and PatentsView's given box holds the middle names too, so a two-word given box there cannot tell a compound given name from a middle name; PSC's middle box can, though its form steers what goes in it. The last row is a separate question from the patronymic: a name of given names only, written by people who have no family name, which no output field fixes and only a rule could read (an open question below). -**Render defaults** (decisions/3-0-reevaluations' clean slate). The default `str()` reads every name given-first and joins its words with spaces, so a name read family-first is turned round and a written separator is lost: `str(parse("김민준"))` is `'민준 김'`, `str(parse("王 小明"))` is `'小明 王'` and `str(parse("威廉·莎士比亚"))` is `'威廉 莎士比亚'`. In the committed corpora (2026-10-28, `c8357e74`), 264 names without a comma read family-first, `str()` renders every one differently from the text, and 159 of them are written unspaced; of 43 names holding the U+00B7 interpunct, `str()` drops it from 38. So: +**Render defaults** (decisions/3-0-reevaluations' clean slate). The default `str()` reads every name given-first and joins its words with spaces, so a name read family-first is turned round and a written separator is lost: `str(parse("김민준"))` is `'민준 김'`, `str(parse("王 小明"))` is `'小明 王'` and `str(parse("威廉·莎士比亚"))` is `'威廉 莎士比亚'`. Over the committed corpus's distinct entries, each under its declared order (2026-10-28, `c8357e74`), 728 without a comma read family-first (the family's first word written before the given name's), `str()` renders every one differently from the text, and 150 of them are written unspaced; of 41 holding the U+00B7 interpunct, `str()` drops it from 36. So: -- `ParsedName` gains `order`, the order the name was read under (its script's or the declared `name_order`, a comma aside), an addition with `ADDED_IN` metadata whose missing value, in a 2.x pickle, renders as today. -- `str()` and `render()`'s default spec put the family first where `order` does, and two words that stand side by side in the original and in the rendering are joined by what separated them there, nothing, a space, an interpunct or a nakaguro, so `str(parse("김민준"))` is `'김민준'` and `str(parse("威廉·莎士比亚"))` keeps the dot. A name with a comma or written given-first renders as today (`Smith, John` → `'John Smith'`). +- `ParsedName` gains `order`, the order the name was read under (its script's or the declared `name_order`, a comma aside), an addition with `ADDED_IN` metadata whose missing value, in a 2.x pickle, renders as today, and left out of comparison as `Token.reading` is, so a loaded 2.x pickle still equals a fresh parse of its text. +- `str()` and `render()`'s default spec put the family first where `order` does, and two words that stand side by side in the original and in the rendering are joined by what separated them there, nothing, a space, an interpunct or a nakaguro, so `str(parse("김민준"))` is `'김민준'` and `str(parse("威廉·莎士比亚"))` keeps the dot. A name with a comma or written given-first keeps today's word order (`Smith, John` → `'John Smith'`), while the written separator is kept for every name, so the given-first transcription `威廉·莎士比亚`, most of the interpunct names measured, keeps its dot too. - `initials()` gives nothing for a word in a script with no initials (Han, Hangul, kana, `_policy._NO_INITIALS`), where today it gives the first character with a period (`'王.'`). These change no signature but change `str()`'s output for every name read family-first, which no 2.x release can warn about; Derek took them as an improvement for 3.0's release notes rather than a break (item 6 of the entry). @@ -268,7 +268,7 @@ These change no signature but change `str()`'s output for every name read family | the default `str()`, `render()` spec and `initials()` output | behavior change, accepted for 3.0 | | a `Role` member for any of the four | a break, declined | -**Phase 3's Q1 confirmed**: `Locale.languages` stands as phase 3's draft proposes. Its open dependency was what `Locale` and `Policy` look like once the facade is gone; the 2026-10-28 freeze answers it, both keeping their 2.x shape with additions. With the `[rules]` proposal above, a shipped language pack is a `Locale` whose `languages` choose its vocabulary and whose `policy` carries its rules, so `Locale` needs no third field. +**Phase 3's open API dependency answered**: phase 3's draft (branch `feat/vocabulary-phase3-design`, not yet merged) carries a choice of languages on `Locale.languages` and notes it assumes `Locale` and `Policy` keep their 2.x shape once the facade is gone; the 2026-10-28 freeze answers that, both keeping it with additions. The rest of that item, strict as an argument of the whole choice included, is phase 3's to decide, and its entry is to say the dependency is answered when it lands. With the `[rules]` proposal above, a shipped language pack is a `Locale` whose `languages` choose its vocabulary and whose `policy` carries its rules, so `Locale` needs no third field. **Gender**, recorded only where a word gives a definitive clue (decision 9), is not part of this design: the vocabulary records it on readings (`bint`'s `gender = "f"`), and `Token.reading` could carry it later as one more addition. @@ -287,9 +287,9 @@ Each sub-phase is one PR, measured on its own base as phase 2's and phase 3's ar | 4g | the compound given name, a rule kind (#704) | 4f | | 4h | the shape inventory per order (#469), starting with the title between a family and a given name under `FAMILY_FIRST` | — | -**Interleaved with phase 3**: 4a can start beside 3a, its `languages` waiting for 3a's index; 4b to 4d need nothing of phase 3 and unblock its packs, `cs`'s generations (3f) and `pt-BR`'s agnomens (3h) on 4b, `es` and `pt` (3g) on 4d and 4g; 4f lands with or after 3b, since both decide what a shipped `Locale` holds; an `is` pack, which no phase 3 sub-phase lists, needs 3b, 4c and 4f. 4e and 4h stand apart from both. So the order is 3a and 4a, then 4b to 4d with 3b and 3c, then 4f, then the packs with 4g, with 4e at any point. +**Interleaved with phase 3** (its sub-phases 3a to 3h are the draft's on `feat/vocabulary-phase3-design`): 4a can start beside 3a, its `languages` waiting for 3a's index; 4b to 4d need nothing of phase 3 and unblock its packs, `cs`'s generations (3f) and `pt-BR`'s agnomens (3h) on 4b, `es` and `pt` (3g) on 4d and 4g; 4f lands with or after 3b, since both decide what a shipped `Locale` holds; an `is` pack, which no phase 3 sub-phase lists, needs 3b, 4c and 4f. 4e and 4h stand apart from both. So the order is 3a and 4a, then 4b to 4d with 3b and 3c, then 4f, then the packs with 4g, with 4e at any point. -**Left open**, value questions inside the shapes above: the first surname's view and its name; whether `credentials` should leave the honorifics out, CLDR having no third bucket; which ending rules beside Icelandic a pack may name (Hungarian `-né` is a married name, not a patronymic); and the names of given names only (the last table row), whose reading needs a rule that tells a mononym from a given name and a family name, not an output field. +**Left open**, with no sub-phase yet: phase 1's four reserved names said to be phase 4's (`separators` with phase 4's glued words, and `bare_only`, `leading_only` and `trailing_only`, which this RFC's phase 2 section found not needed), whose build messages still name this phase. Value questions inside the shapes above: the first surname's view and its name; whether `credentials` should leave the honorifics out, CLDR having no third bucket; which ending rules beside Icelandic a pack may name (Hungarian `-né` is a married name, not a patronymic); and the names of given names only (the last table row), whose reading needs a rule that tells a mononym from a given name and a family name, not an output field. ## The first packs From aee660699ea60f7042a4493e216f3c41e23713de Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Wed, 28 Oct 2026 11:51:16 -0700 Subject: [PATCH 10/39] Propose which particles family_base removes (#754) rules/R2 says the base is the family without its leading particles; the code removes every working particle. Each interior one leads its own name unit, so the per-surname reading and the code agree. Proposed: amend R2 to the code. Measured over the committed corpus and tse-donors, in the default policy and with middle_as_family. Co-Authored-By: Claude Opus 5.5 --- .../decisions/R2-all-particles-family-base-divergence.md | 7 ++++++- docs/design/vocabulary-model-rfc.md | 2 +- 2 files changed, 7 insertions(+), 2 deletions(-) diff --git a/docs/design/decisions/R2-all-particles-family-base-divergence.md b/docs/design/decisions/R2-all-particles-family-base-divergence.md index 46813b491..1e85beeb8 100644 --- a/docs/design/decisions/R2-all-particles-family-base-divergence.md +++ b/docs/design/decisions/R2-all-particles-family-base-divergence.md @@ -11,7 +11,7 @@ Elsewhere: where a connective initials, a part holding nothing else for it to join (`Jon Dough and` → 'J. D.', `Juan de y` → 'J. y.') — decisions/R3's 2026-09-20 #461 entry; case repair keeping that connective lowercase, its carve-out not gated on the mark (under `Lexicon.default().add(particles={'y'})`, `Juan de y` forced → family 'De y') — decisions/R4's 2026-08-29, the CONJUNCTION CARVE-OUT entry; case repair reading the unjoined mark for the whole part (`ANH DO` → 'Anh Do') — decisions/R4's 2026-08-29 #407 entry; the snapshot recording `initials()`, so a change to the base is visible there — decisions/differential-ledger's 2026-10-13 #664, a committed snapshot replaces the classified gate entry. -Open: none in this file. +Open: #754 (which particles `family_base` removes, the leading run or every working particle; proposed in the 2026-10-28 #754 entry). ## Log @@ -34,6 +34,11 @@ Open: none in this file. - 2026-09-20 #461 — AMENDS the 2026-08-29 bullet above, which said of four readings that "none of those moved or may move". Two of the four moved and two did not, and the half that survived is the half the criterion was always about. WHAT MOVED: `parse("Juan y Garcia")` initials "J. y. G." where that bullet recorded "J. G.", and R3's own `"Juan de y"` example line reads "J. y." where it read "J.". WHAT DID NOT: `parse("Juan Velasquez y Garcia")` is still "J. V. G." over base "Velasquez y Garcia", and `parse("Jon Dough and")` is still "J. D." over base "Dough and" (all four measured 2026-09-20 on this branch). The criterion that separates them is the one the bullet above named, and it is still not a word count: the question is asked of the WHOLE PART, and a part holding another name word for the connective to join — the part's working particles set aside — is a part where the connective is doing a connective's work and contributes nothing. "Velasquez y Garcia" and "Dough and" each hold such a word, so the skip stands there; the family "y", the family "de y" and the middle "y" of "Juan y Garcia" hold none, so the connective is its part's own name word and initials with it. So that bullet was RIGHT ABOUT THE MECHANISM and wrong only about the clause it preserved. Its finding — that a mark honored for some of a part's words and not for one of them is honoring nothing — is exactly what rules/R3 now states, for all three groups at once, and it is why the fact is computed ONCE over the whole part rather than re-derived per word by each view. What changed is the clause it left standing: R3's "A CONJUNCTION never initials, so a base that is one contributes nothing even then" is conditional now, and "none of those may move" was the reading of the day rather than a decision. decisions/R3's 2026-09-20 bullet carries the rule as taken. The pin that bullet describes did its job: `test_initials_readmits_a_conjunction_in_a_particle_part` was written to fail when #461 was re-decided, and it failed. +- **2026-10-28 #754 — PROPOSED, NOT DECIDED: `family_base` REMOVES EVERY WORKING PARTICLE, EACH THE LEADING PARTICLE OF THE NAME UNIT IT JOINS, SO RULES/R2 IS AMENDED TO THE CODE RATHER THAN THE CODE TO R2.** Rules/R2's Then says the base is "the family without its leading particles", but the view drops every family word tagged `particle` and not marked unjoined, wherever it stands: `da Silva do Amaral de Souza, Miguel` reads family 'da Silva do Amaral de Souza', base 'Silva Amaral Souza', particles 'da do de', where the 2.x facade's `last_prefixes` gave 'da'; the P1 fold after a family comma makes a particle interior with none leading (`Smith, de Mesnil Jean` → base 'Smith Mesnil Jean', particles 'de'); and a leading run alone reads the same either way (`Juan de la Vega y Ortega` → base 'Vega y Ortega'). `test_particle_views_after_fold` (#937) pins the code's reading. Three answers were weighed: (1) amend R2 to the code; (2) remove only the leading run (base 'Silva do Amaral de Souza', particles 'da'); (3) derive the base from per-surname views, each surname keeping its own leading particle. + (1) and (3) are one answer. A particle joins forward to the word it modifies, so every working particle inside a family is the leading particle of the name unit it opens, and the unit bases joined are the code's base: under `two_family_names`, rules/O6 reads `Miguel da Silva do Amaral de Souza` as middle 'da Silva' and family 'do Amaral de Souza', whose units 'do Amaral' and 'de Souza' have bases 'Amaral' and 'Souza', and the family's base is 'Amaral Souza'. So `surname2` (the RFC's Phase 4, sub-phase 4d) reaches the question only where O6 runs and only for its two units; it needs nothing from `family_base`, and the first unit's view, left open there, would read the same way. One shape escapes the unit reading: a working particle ending a family of other words opens no unit and joins nothing (word-particle, 9 committed entries with `middle_as_family`), and the code drops it from the base as it does any other; whether a lone trailing particle is unjoined is R2's all-particles question asked of one word, left as the code reads it. The proposal is R2's Then as "the base (the family without its particles, each the leading particle of the name unit it joins) and the particles themselves", with `da Silva do Amaral de Souza, Miguel` and `Smith, de Mesnil Jean` as its examples, the code and #937's test unchanged; landing it re-quotes the R2 citation on `ParsedName.family_particles` in nameparser/_types.py, which tests/v2/test_doc_citations.py holds to R2's text. + Measured 2026-10-28 on master `c8357e74`, counts and token shapes only, over the committed corpus's 6,533 distinct entries (`snapshot.corpus_entries()`, each parsed under its declared order) and `tse-donors`' 854,531 records (`local_corpora.read_records`, the filed name and the registry name each parsed in the default order). A family is read in render order (FOLDED_TAG tokens first), a working particle is a token tagged `particle` and not UNJOINED_TAG, and a family has an interior particle where a working particle stands after its first word that is neither a particle nor a conjunction; that family is where (1) and (2) differ. In the default policy: committed, 24 against 684 with a leading run only (12 with no leading run, 8 whose interior particle stands after a family comma, 4 with a leading run as well); `tse-donors`, 12 filed names against 347,992 and 1 registry name against 392,134, since a Brazilian name's particles before its last word stand in `middle`. With `middle_as_family`: committed, 367 against 398 (262, 92 and 13); `tse-donors`' filed names, 268,591 against 79,926 (261,024, 5 and 7,562), the commonest family shape word-particle-word, 150,998 times. Recompute with those definitions; a change to the fold or to O6 moves them. + Weighed against (2): with `middle_as_family`, the population the per-surname reading serves, it gives the 261,024 `tse-donors` names with no leading run a base equal to the whole family and an empty particle view, so `family_particles` stops naming the particles it is for; and it changes a frozen property's output for 24 committed entries in the default policy, where (1) changes no output. The facade whose `last_prefixes` it matches is removed in 3.0. Its one gain, a base that is a written name, is what a sort by first surname wants, and that is the first unit's view left open in the RFC's Phase 4, not `family_base`. The cost of (1), accepted if agreed: the base is a sort key and not a name anyone writes ('Silva Amaral Souza'). + Declined: - 2026-08-18 — the GROUPING half of #404: a particle run that joins nothing does not chain, so "Jong van der" would split into middle 'van' plus family 'der'. Measured and rejected, though NOT for the reason first recorded here. The first draft said the split makes the family "no longer all-particle so the base fix stops firing" — false, and `der` and `la` are both shipped particles, so a family of either IS all-particle and the rule fires on it ("Juan Smith der" gives base 'der'). What the split actually costs is the SCOPE of the base and a stray particle relocated: grouping can decline to merge but cannot keep the words apart, because roles re-assemble them and two adjacent same-role pieces are one part at the field level. "Juan Smith van der" becomes middle 'Smith van', family 'der' — a base of 'der' rather than 'van der', and a middle name nobody wrote. Keeping the run whole in one part is what gives the base its full extent. The split reading needs the leftover distribution to know these are separate units, which is mechanisms.md#UNIT-PARTITION's problem. diff --git a/docs/design/vocabulary-model-rfc.md b/docs/design/vocabulary-model-rfc.md index 217d96c25..58ef1b1a0 100644 --- a/docs/design/vocabulary-model-rfc.md +++ b/docs/design/vocabulary-model-rfc.md @@ -289,7 +289,7 @@ Each sub-phase is one PR, measured on its own base as phase 2's and phase 3's ar **Interleaved with phase 3** (its sub-phases 3a to 3h are the draft's on `feat/vocabulary-phase3-design`): 4a can start beside 3a, its `languages` waiting for 3a's index; 4b to 4d need nothing of phase 3 and unblock its packs, `cs`'s generations (3f) and `pt-BR`'s agnomens (3h) on 4b, `es` and `pt` (3g) on 4d and 4g; 4f lands with or after 3b, since both decide what a shipped `Locale` holds; an `is` pack, which no phase 3 sub-phase lists, needs 3b, 4c and 4f. 4e and 4h stand apart from both. So the order is 3a and 4a, then 4b to 4d with 3b and 3c, then 4f, then the packs with 4g, with 4e at any point. -**Left open**, with no sub-phase yet: phase 1's four reserved names said to be phase 4's (`separators` with phase 4's glued words, and `bare_only`, `leading_only` and `trailing_only`, which this RFC's phase 2 section found not needed), whose build messages still name this phase. Value questions inside the shapes above: the first surname's view and its name; whether `credentials` should leave the honorifics out, CLDR having no third bucket; which ending rules beside Icelandic a pack may name (Hungarian `-né` is a married name, not a patronymic); and the names of given names only (the last table row), whose reading needs a rule that tells a mononym from a given name and a family name, not an output field. +**Left open**, with no sub-phase yet: phase 1's four reserved names said to be phase 4's (`separators` with phase 4's glued words, and `bare_only`, `leading_only` and `trailing_only`, which this RFC's phase 2 section found not needed), whose build messages still name this phase. Value questions inside the shapes above: the first surname's view and its name; which particles `family_base` removes, rules/R2 saying the leading ones and the code removing every working one, proposed in decisions/R2's 2026-10-28 #754 entry as R2 amended to the code, since each such particle leads its own name unit and the unit bases joined are the code's base; whether `credentials` should leave the honorifics out, CLDR having no third bucket; which ending rules beside Icelandic a pack may name (Hungarian `-né` is a married name, not a patronymic); and the names of given names only (the last table row), whose reading needs a rule that tells a mononym from a given name and a family name, not an output field. ## The first packs From 76ae6dbe3a698d32dcae6b10e675dd2c8b4a9d66 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Wed, 28 Oct 2026 12:04:16 -0700 Subject: [PATCH 11/39] Fix the second docs review's findings on phase 4 and the R2 proposal (#754) Comma names drop the comma as a separator; the str() change reaches given-first names with an interpunct; order's exclusion from == is left open; suffix counts re-measured per word (33/77/64) with the recipe stated; CLDR samples counted as distinct names; R2's proposal points at #937's unmerged branch, covers particle runs and trailing particles (39/505), and names migrate.rst among the sites landing it touches. Co-Authored-By: Claude Opus 5.5 --- .../R2-all-particles-family-base-divergence.md | 6 +++--- docs/design/decisions/vocabulary-model.md | 8 ++++---- docs/design/vocabulary-model-rfc.md | 16 ++++++++-------- 3 files changed, 15 insertions(+), 15 deletions(-) diff --git a/docs/design/decisions/R2-all-particles-family-base-divergence.md b/docs/design/decisions/R2-all-particles-family-base-divergence.md index 1e85beeb8..92c151730 100644 --- a/docs/design/decisions/R2-all-particles-family-base-divergence.md +++ b/docs/design/decisions/R2-all-particles-family-base-divergence.md @@ -34,9 +34,9 @@ Open: #754 (which particles `family_base` removes, the leading run or every work - 2026-09-20 #461 — AMENDS the 2026-08-29 bullet above, which said of four readings that "none of those moved or may move". Two of the four moved and two did not, and the half that survived is the half the criterion was always about. WHAT MOVED: `parse("Juan y Garcia")` initials "J. y. G." where that bullet recorded "J. G.", and R3's own `"Juan de y"` example line reads "J. y." where it read "J.". WHAT DID NOT: `parse("Juan Velasquez y Garcia")` is still "J. V. G." over base "Velasquez y Garcia", and `parse("Jon Dough and")` is still "J. D." over base "Dough and" (all four measured 2026-09-20 on this branch). The criterion that separates them is the one the bullet above named, and it is still not a word count: the question is asked of the WHOLE PART, and a part holding another name word for the connective to join — the part's working particles set aside — is a part where the connective is doing a connective's work and contributes nothing. "Velasquez y Garcia" and "Dough and" each hold such a word, so the skip stands there; the family "y", the family "de y" and the middle "y" of "Juan y Garcia" hold none, so the connective is its part's own name word and initials with it. So that bullet was RIGHT ABOUT THE MECHANISM and wrong only about the clause it preserved. Its finding — that a mark honored for some of a part's words and not for one of them is honoring nothing — is exactly what rules/R3 now states, for all three groups at once, and it is why the fact is computed ONCE over the whole part rather than re-derived per word by each view. What changed is the clause it left standing: R3's "A CONJUNCTION never initials, so a base that is one contributes nothing even then" is conditional now, and "none of those may move" was the reading of the day rather than a decision. decisions/R3's 2026-09-20 bullet carries the rule as taken. The pin that bullet describes did its job: `test_initials_readmits_a_conjunction_in_a_particle_part` was written to fail when #461 was re-decided, and it failed. -- **2026-10-28 #754 — PROPOSED, NOT DECIDED: `family_base` REMOVES EVERY WORKING PARTICLE, EACH THE LEADING PARTICLE OF THE NAME UNIT IT JOINS, SO RULES/R2 IS AMENDED TO THE CODE RATHER THAN THE CODE TO R2.** Rules/R2's Then says the base is "the family without its leading particles", but the view drops every family word tagged `particle` and not marked unjoined, wherever it stands: `da Silva do Amaral de Souza, Miguel` reads family 'da Silva do Amaral de Souza', base 'Silva Amaral Souza', particles 'da do de', where the 2.x facade's `last_prefixes` gave 'da'; the P1 fold after a family comma makes a particle interior with none leading (`Smith, de Mesnil Jean` → base 'Smith Mesnil Jean', particles 'de'); and a leading run alone reads the same either way (`Juan de la Vega y Ortega` → base 'Vega y Ortega'). `test_particle_views_after_fold` (#937) pins the code's reading. Three answers were weighed: (1) amend R2 to the code; (2) remove only the leading run (base 'Silva do Amaral de Souza', particles 'da'); (3) derive the base from per-surname views, each surname keeping its own leading particle. - (1) and (3) are one answer. A particle joins forward to the word it modifies, so every working particle inside a family is the leading particle of the name unit it opens, and the unit bases joined are the code's base: under `two_family_names`, rules/O6 reads `Miguel da Silva do Amaral de Souza` as middle 'da Silva' and family 'do Amaral de Souza', whose units 'do Amaral' and 'de Souza' have bases 'Amaral' and 'Souza', and the family's base is 'Amaral Souza'. So `surname2` (the RFC's Phase 4, sub-phase 4d) reaches the question only where O6 runs and only for its two units; it needs nothing from `family_base`, and the first unit's view, left open there, would read the same way. One shape escapes the unit reading: a working particle ending a family of other words opens no unit and joins nothing (word-particle, 9 committed entries with `middle_as_family`), and the code drops it from the base as it does any other; whether a lone trailing particle is unjoined is R2's all-particles question asked of one word, left as the code reads it. The proposal is R2's Then as "the base (the family without its particles, each the leading particle of the name unit it joins) and the particles themselves", with `da Silva do Amaral de Souza, Miguel` and `Smith, de Mesnil Jean` as its examples, the code and #937's test unchanged; landing it re-quotes the R2 citation on `ParsedName.family_particles` in nameparser/_types.py, which tests/v2/test_doc_citations.py holds to R2's text. - Measured 2026-10-28 on master `c8357e74`, counts and token shapes only, over the committed corpus's 6,533 distinct entries (`snapshot.corpus_entries()`, each parsed under its declared order) and `tse-donors`' 854,531 records (`local_corpora.read_records`, the filed name and the registry name each parsed in the default order). A family is read in render order (FOLDED_TAG tokens first), a working particle is a token tagged `particle` and not UNJOINED_TAG, and a family has an interior particle where a working particle stands after its first word that is neither a particle nor a conjunction; that family is where (1) and (2) differ. In the default policy: committed, 24 against 684 with a leading run only (12 with no leading run, 8 whose interior particle stands after a family comma, 4 with a leading run as well); `tse-donors`, 12 filed names against 347,992 and 1 registry name against 392,134, since a Brazilian name's particles before its last word stand in `middle`. With `middle_as_family`: committed, 367 against 398 (262, 92 and 13); `tse-donors`' filed names, 268,591 against 79,926 (261,024, 5 and 7,562), the commonest family shape word-particle-word, 150,998 times. Recompute with those definitions; a change to the fold or to O6 moves them. +- **2026-10-28 #754 — PROPOSED, NOT DECIDED: `family_base` REMOVES EVERY WORKING PARTICLE, EACH THE LEADING PARTICLE OF THE NAME UNIT IT JOINS, SO RULES/R2 IS AMENDED TO THE CODE RATHER THAN THE CODE TO R2.** Rules/R2's Then says the base is "the family without its leading particles", but the view drops every family word tagged `particle` and not marked unjoined, wherever it stands: `da Silva do Amaral de Souza, Miguel` reads family 'da Silva do Amaral de Souza', base 'Silva Amaral Souza', particles 'da do de', where the 2.x facade's `last_prefixes` gave 'da'; the P1 fold after a family comma makes a particle interior with none leading (`Smith, de Mesnil Jean` → base 'Smith Mesnil Jean', particles 'de'); and a leading run alone reads the same either way (`Juan de la Vega y Ortega` → base 'Vega y Ortega'). `test_particle_views_after_fold`, on #937's branch and not yet on master, pins the code's reading of `Miguel da Silva do Amaral de Souza` under `middle_as_family`. Three answers were weighed: (1) amend R2 to the code; (2) remove only the leading run (base 'Silva do Amaral de Souza', particles 'da'); (3) derive the base from per-surname views, each surname keeping its own leading particle. + (1) and (3) are one answer. A particle joins forward to the word it modifies, so every working particle inside a family stands in the particle run that opens a name unit (`la` in `de la Vega` is the run's second word), and the unit bases joined are the code's base: under `two_family_names`, rules/O6 reads `Miguel da Silva do Amaral de Souza` as middle 'da Silva' and family 'do Amaral de Souza', whose units 'do Amaral' and 'de Souza' have bases 'Amaral' and 'Souza', and the family's base is 'Amaral Souza'. So `surname2` (the RFC's Phase 4, sub-phase 4d) reaches the question only where O6 runs and only for its two units; it needs nothing from `family_base`, and the first unit's view, left open there, would read the same way. One shape escapes the unit reading: a working particle ending a family of other words opens no unit and joins nothing (a family of two or more words whose last is a working particle: 1 committed entry and 1 `tse-donors` filed name in the default policy, 39 and 505 with `middle_as_family`, the bare word-particle shape 9 of the 39), and the code drops it from the base as it does any other; whether a lone trailing particle is unjoined is R2's all-particles question asked of one word, left as the code reads it. The proposal is R2's Then as "the base (the family without its particles, each in the run that opens the name unit it joins, or ending the family) and the particles themselves", with `da Silva do Amaral de Souza, Miguel` and `Smith, de Mesnil Jean` as its examples, the code unchanged, and #937's test with it; landing it re-quotes the R2 citation on `ParsedName.family_particles` in nameparser/_types.py, which tests/v2/test_doc_citations.py holds to R2's text, and docs/migrate.rst's `family_base` line ("The surname with leading particles split off"), which nothing checks, beside its `last_prefixes` row, the mapping that reads 'da' where the core reads 'da do de'. + Measured 2026-10-28 on master `c8357e74`, counts and token shapes only, over the committed corpus's 6,533 distinct entries (`snapshot.corpus_entries()`, each parsed under its declared order) and `tse-donors`' 854,531 records (`local_corpora.read_records`, the filed name and the registry name each parsed in the default order). A family is read in render order (FOLDED_TAG tokens first), a working particle is a token tagged `particle` and not UNJOINED_TAG, and a family has an interior particle where a working particle stands after its first word that is neither a particle nor a conjunction; that family is where (1) and (2) differ. The three kinds of interior family are tested in this order, the first that fits counting: after a family comma, every interior particle standing after the first comma; with a leading run as well; with no leading run. In the default policy: committed, 24 against 684 with a leading run only (12 with no leading run, 8 whose interior particle stands after a family comma, 4 with a leading run as well); `tse-donors`, 12 filed names against 347,992 and 1 registry name against 392,134, since a Brazilian name's particles before its last word stand in `middle`. With `middle_as_family`: committed, 367 against 398 (262, 92 and 13); `tse-donors`' filed names, 268,591 against 79,926 (261,024, 5 and 7,562), the commonest family shape word-particle-word, 150,998 times. Recompute with those definitions; a change to the fold or to O6 moves them. Weighed against (2): with `middle_as_family`, the population the per-surname reading serves, it gives the 261,024 `tse-donors` names with no leading run a base equal to the whole family and an empty particle view, so `family_particles` stops naming the particles it is for; and it changes a frozen property's output for 24 committed entries in the default policy, where (1) changes no output. The facade whose `last_prefixes` it matches is removed in 3.0. Its one gain, a base that is a written name, is what a sort by first surname wants, and that is the first unit's view left open in the RFC's Phase 4, not `family_base`. The cost of (1), accepted if agreed: the base is a sort key and not a name anyone writes ('Silva Amaral Souza'). Declined: diff --git a/docs/design/decisions/vocabulary-model.md b/docs/design/decisions/vocabulary-model.md index 76d7f2f52..0cfbcaa2e 100644 --- a/docs/design/decisions/vocabulary-model.md +++ b/docs/design/decisions/vocabulary-model.md @@ -244,12 +244,12 @@ Open: #913 (the title words phase 2j left, read as titles though borne as names) - **2026-10-26 #752, phase 2m — THE READING NOT TAKEN SURFACES AS `Ambiguity.alternative`, WITHOUT ITS LANGUAGES**, carrying out item 13 of the 2026-10-23 #752 entry above and item 7's source with it: a word the credential's run took by position alone reports its name reading's source as `position`, a fourth source beside the three the item named, and a word with a declared name reading reports `declared`. The pickle guard was widened as item 13 asked, every skew the narrow guard refused still refused. The reading's languages were left out: the per-word index costs about 0.4 ms of a 19.5 ms import and describes the default vocabulary, not a pack's or a caller's lexicon, so they wait for 3.0 with `Token.reading`. The record, the measurements and what was declined are decisions/A1's 2026-10-26 #752 entry. - **2026-10-28 #754 (Derek) — PHASE 4'S OUTPUT MODEL IS AGREED: THE SEVEN ROLES STAY AND EVERY NEW FIELD IS A VIEW, MOST READ FROM `Token.reading`; A PATRONYMIC NAMES ITS WORDS WITHOUT EMPTYING `family`; THE RENDER DEFAULTS FOLLOW THE ORDER A NAME WAS READ IN.** Derek agreed every item in chat on 2026-10-28, items 6 and 4 and the `surnames` line of item 5 first, then the rest. The design overview is docs/design/vocabulary-model-rfc.md's Phase 4 section. Decision 8 (a patronymic field in 3.0) is kept and its stated value revised (item 3); decisions 9 and 14 are kept as written. The core API is frozen for 3.0 except for additions (decisions/3-0-reevaluations, 2026-10-28 #925, THE CORE API), and each item says whether it is one. The inputs are #754 and its comment of 2026-10-23, #326, #704 and #694. Measurements were taken 2026-10-28 on master `c8357e74` (Python 3.11.17) over the fielded local corpora (`orcid-researchers`, `psc-individuals`, `patentsview-inventors`), the committed fields files and the committed corpus's distinct entries (`snapshot.corpus_entries()`, each parsed under its declared order), with throwaway scripts that are not kept, so each figure is evidence for the decision and each sub-phase re-measures on its own base. Local-corpus figures are counts. Recompute the box counts by reading each source's records with `local_corpora.read_records`, counting its boxes by word count and word ending per country or nationality, and the role the parse gives the counted word with `parse()`; the ending tests are the RFC's Phase 4 table's, and other tests give counts a few percent apart. - 1. **`Role` keeps its seven members, and every field asked for is a view.** A role partitions the tokens, and each of the four fields asked for (a second surname, a patronymic, generation and credentials, a second given name) subdivides a role that exists, so each is computed from the tokens, a property and a render key, as `family_base` and `given_names` are (rules/R1, rules/R2). `as_dict()`, `comparison_key()` and the seven role properties are unchanged, and every view is an addition. Weighed against: a `Role` member for any of them, a break of the frozen `ParsedName` (`as_dict()` gains a key, `comparison_key()` a component, so a stored key stops matching, and the role its words leave returns less text than in 2.x), with no 2.x feature release left to warn in. + 1. **`Role` keeps its seven members, and every field asked for is a view.** A role partitions the tokens, and each field asked for (a second surname, a patronymic, generation and credentials, a second given name) subdivides a role that exists, so each is computed from the tokens, a property and a render key, except the second given name, which is the `middle` that exists (item 7), as `family_base` and `given_names` are (rules/R1, rules/R2). `as_dict()`, `comparison_key()` and the seven role properties are unchanged, and every view is an addition. Weighed against: a `Role` member for any of them, a break of the frozen `ParsedName` (`as_dict()` gains a key, `comparison_key()` a component, so a stored key stops matching, and the role its words leave returns less text than in 2.x), with no 2.x feature release left to warn in. 2. **`Token.reading`, the reading each token took**, a `Reading` (decisions/A1's type), `None` for a token `replace()` builds, with `ADDED_IN` metadata so a 2.x pickle loads, and left out of comparison so `==` keeps its 2.x meaning. `Reading` gains `languages`, the files holding the reading, filled from phase 3's reading index (3a) and empty where no file holds it; `ReadingKind` gains `PATRONYMIC`. The views read it. It carries out the 2026-10-23 #752 entry's item 13 and the 2026-10-26 phase 2m entry's deferral of the languages. - 3. **A `patronymic` view, filled by a word or a rule that says patronymic, with `family` and `middle` unchanged.** The default fills it from the vocabulary's `patronymic` particles (`bin`, `binti`, `ibn`, `bint`, `bar`, `bat` and the native-script forms) with the name they introduce (`Mahathir bin Mohamad` → family `bin Mohamad`, as today, and patronymic `bin Mohamad`); RU's and TR_AZ's rules fill it where a caller turns them on (`Ivan Ivanovich Ivanov` under RU → middle `Ivanovich` and patronymic `Ivanovich`); an Icelandic ending rule fills it under an `is` pack only, since the default cannot tell `-dóttir` and `-son` from Swedish surnames (ORCID's Swedish records end their family box in `-son` or `-dóttir` in 2,184 of 20,000). A matronymic, written the same way, fills it too. SUPERSEDES: the RFC's earlier statement of decision 8's value, that `family` can be empty for a name with a patronymic. Measured: filers put an Icelandic patronymic in the family box (ORCID 1,647 of 2,583 Icelandic records, PatentsView 535 of 722, PSC 101 of 125), and CLDR's `is` samples put it in `surname` or `surname-core` (9 of the 9 fields of the `is` locale holding one), while Wikidata's `patronymic-is` and `patronymic-my` strata record 108 names with a patronymic and no family name, of which the parse puts the patronymic in `family` 107 times. An East Slavic patronymic stands in the given part: CLDR's `ru` samples put it in `given2` (6 of 6); PatentsView's Russian given boxes end in one 8,423 times of 20,000, and the parse reads such a word as the middle 10,902 times of 10,962; 90 of the 112 Russian nationals in PSC who filled the middle-name box wrote one there. A `family` that keeps it and a `patronymic` that names it agree with both kinds of source. Weighed against: emptying `family` where the patronymic is recognized, which moves the field the fielded sources fill and changes `family` for every Icelandic and nasab name, a behavior change of a frozen property; and a neutral parent-names name for the view, which buys nothing the view can detect, since a matronymic and a patronymic are written alike. - 4. **`generation` and `credentials` are views that partition `suffix` (#326)**, not roles, `generation` the tokens read as a generation or an agnomen, `credentials` the rest, the post-nominals that are neither (`Esq.`, the honorifics after a name) falling in `credentials`, CLDR's catch-all, with `Token.reading.kind` telling them apart for a caller who needs it. Measured: of the 1,054 committed corpus entries with a suffix, 87 hold a generation (or agnomen) and a credential together and 65 only a listed post-nominal of another kind; of the 615 CLDR renderings with either field, the 75 whose suffix holds exactly CLDR's words split into CLDR's `generation` and `credentials` by the vocabulary's reading kind in all 75. Weighed against: two new `Role` members (item 1), and a third view for the honorifics, left open since CLDR has none. + 3. **A `patronymic` view, filled by a word or a rule that says patronymic, with `family` and `middle` unchanged.** The default fills it from the vocabulary's `patronymic` particles (`bin`, `binti`, `ibn`, `bint`, `bar`, `bat` and the native-script forms) with the name they introduce (`Mahathir bin Mohamad` → family `bin Mohamad`, as today, and patronymic `bin Mohamad`); RU's and TR_AZ's rules fill it where a caller turns them on (`Ivan Ivanovich Ivanov` under RU → middle `Ivanovich` and patronymic `Ivanovich`); an Icelandic ending rule fills it under an `is` pack only, since the default cannot tell `-dóttir` and `-son` from Swedish surnames (ORCID's Swedish records end their family box in `-son` or `-dóttir` in 2,184 of 20,000). A matronymic, written the same way, fills it too. SUPERSEDES: the RFC's earlier statement of decision 8's value, that `family` can be empty for a name with a patronymic. Measured: filers put an Icelandic patronymic in the family box (ORCID 1,647 of 2,583 Icelandic records, PatentsView 535 of 722, PSC 101 of 125), and CLDR's `is` file's 3 Icelandic samples put it in `surname` or `surname-core` in all 9 of their renderings (its one foreign sample puts a `-dóttir` word in `surname2`), while Wikidata's `patronymic-is` and `patronymic-my` strata record 108 names with a patronymic and no family name, of which the parse puts the patronymic in `family` 107 times. An East Slavic patronymic stands in the given part: CLDR's 2 `ru` samples put it in `given2` in all 6 renderings; PatentsView's Russian given boxes end in one 8,423 times of 20,000, and the parse reads such a word as the middle 10,902 times of 10,962; 90 of the 112 Russian nationals in PSC who filled the middle-name box wrote one there. A `family` that keeps it and a `patronymic` that names it agree with both kinds of source. Weighed against: emptying `family` where the patronymic is recognized, which moves the field the fielded sources fill and changes `family` for every Icelandic and nasab name, a behavior change of a frozen property; and a neutral parent-names name for the view, which buys nothing the view can detect, since a matronymic and a patronymic are written alike. + 4. **`generation` and `credentials` are views that partition `suffix` (#326)**, not roles, `generation` the tokens read as a generation or an agnomen, `credentials` the rest, the post-nominals that are neither (`Esq.`, the honorifics after a name) falling in `credentials`, CLDR's catch-all, with `Token.reading.kind` telling them apart for a caller who needs it. Measured: of the 1,054 committed corpus entries with a suffix, 33 hold a generation (or agnomen) and a listed credential together, 77 a generation beside any other suffix word, and 64 only listed post-nominals of other kinds (each suffix word classified by the RFC's Phase 4 recipe, joined tokens merged); of the 615 CLDR renderings with either field, the 75 whose suffix holds exactly CLDR's words split into CLDR's `generation` and `credentials` by the vocabulary's reading kind in all 75. Weighed against: two new `Role` members (item 1), and a third view for the honorifics, left open since CLDR has none. 5. **a `surname2` view, the family's second unit where rules/O6 read the family as two**, marked by a new stable tag where O6 takes its word, empty otherwise; `family` keeps both units as O6 makes it, and a family comma alone marks nothing (`Lloyd Webber, Andrew`). Measured: a two-word family box in ORCID's records from Spain 48.0%, Mexico 64.7%, Colombia 72.8% and Chile 50.5%, against 3.9% from the United States. Left open: a view for the first unit and its name, `surname` standing close to the `surnames` view that exists. Declined: redefining `surnames` as the two units, which would repeat `family`, the family holding both where O6 reads two, and lose what `surnames` does today, joining `middle` and `family` so that a double surname the default splits (`Gabriel García Márquez` → middle `García`) reads whole, the view rules/P2 names for that reading. - 6. **The render defaults follow the order the name was read in, a behavior change of the output rather than a break of the API.** `ParsedName` gains `order` (an addition, `ADDED_IN`, a 2.x pickle rendering as today, left out of comparison); the default `str()` and `render()` spec put the family first where `order` does, and join two words that stand side by side in the original and the rendering with what separated them there (nothing, a space, an interpunct, a nakaguro); `initials()` gives nothing for a word in a script with no initials. A name with a comma or written given-first keeps today's word order, the written separator kept for every name (`威廉·莎士比亚` keeps its dot). Measured over the committed corpus's distinct entries: 728 without a comma read family-first, `str()` renders all 728 differently from their text (`김민준` → `'민준 김'`), 150 of them written unspaced, and it drops the U+00B7 interpunct from 36 of the 41 holding one. This carries out decisions/3-0-reevaluations' render-default clean slate. No signature changes, but `str()`'s output changes for every name read family-first, and no 2.x release can warn about a `str()` result, so it is taken as an improvement for 3.0's release notes, not a break needing a warning. + 6. **The render defaults follow the order the name was read in, a behavior change of the output rather than a break of the API.** `ParsedName` gains `order` (an addition, `ADDED_IN`, a 2.x pickle rendering as today, left out of comparison, which lets two equal names render differently, left open in the RFC); the default `str()` and `render()` spec put the family first where `order` does, and join two words that stand side by side in the original and the rendering with what separated them there (nothing, a space, an interpunct, a nakaguro); `initials()` gives nothing for a word in a script with no initials. A name with a comma or written given-first keeps today's word order, the separator kept being any but a comma, which a space replaces (`Smith, John` → `'John Smith'`, while `威廉·莎士比亚` keeps its dot; 20 of the 36 names whose interpunct `str()` drops are read given-first). Measured over the committed corpus's distinct entries: 728 without a comma read family-first, `str()` renders all 728 differently from their text (`김민준` → `'민준 김'`), 150 of them with the family's last word touching the given name's first, and it drops the U+00B7 interpunct from 36 of the 41 holding one. This carries out decisions/3-0-reevaluations' render-default clean slate. No signature changes, but `str()`'s output changes for every name read family-first and every name whose written separator is neither a space nor a comma, and no 2.x release can warn about a `str()` result, so it is taken as an improvement for 3.0's release notes, not a break needing a warning. 7. **`middle` keeps its meaning (#704)**, every word of the given part after the given name, the given name one unit unless a rule joins words into it, as rules/P5 already does (`Abdul Aziz Tariq` → given `Abdul Aziz`); a compound given name is a `[rules]` rule a caller or a pack turns on, CLDR's `given2` is `middle`, and `Kaur` and an English name beside a Chinese one stay in `middle`, an accepted cost. Measured: in PSC, whose form has a middle-name box, 2,499 of 1,040,708 forename boxes hold two or more words, and of 6,101 Spanish nationals 25 do while 2,621 fill the middle box; ORCID's and PatentsView's given box holds middles too, so it cannot separate the two. The `Maria do Carmo` shape, a particle inside a given box of three or more words with a family box filled, reads the given box's words into the family in 1,812 of 1,923 ORCID records and 1,165 of 1,219 PatentsView records from Brazil, Portugal, Spain, Mexico, Argentina, Colombia, Chile and Italy, which is that rule's case. Weighed against: #704's option (b), one given unit by default, which changes `middle` for every English name of three words. 8. **A language file's `[rules]` compiles into its shipped `Locale`'s policy patch**, each rule kind a `Policy` field (an addition), so RU and TR_AZ are rebuilt from their files and `Locale` gains no field for rules. With it, **phase 3's open API dependency is answered**: its draft (branch `feat/vocabulary-phase3-design`, unmerged) carries a choice of languages on `Locale.languages`, assuming `Locale` and `Policy` keep their 2.x shape after the facade's removal, and the 2026-10-28 freeze keeps them so. The rest of that draft, strict as an argument of the whole choice included, stays phase 3's to decide; its entry, when it lands, says the dependency is answered. 9. **The work order**, one PR each, measured on its own base, the snapshot recording each new view where it is non-empty: 4a `Token.reading` and `ReadingKind.PATRONYMIC` (neutral; `languages` after 3a); 4b `generation` and `credentials` (neutral); 4c `patronymic` (neutral); 4d `surname2` (neutral); 4e the render defaults and `order`; 4f `[rules]` into the policy patch, with or after 3b; 4g the compound given name, after 4f; 4h the shape inventory per order (#469). Interleaved with phase 3 (its sub-phases the draft's): 4a beside 3a; 4b to 4d before the packs that need them, `cs` (3f) and `pt-BR` (3h) on 4b, `es` and `pt` (3g) on 4d and 4g; an `is` pack on 3b, 4c and 4f. diff --git a/docs/design/vocabulary-model-rfc.md b/docs/design/vocabulary-model-rfc.md index 58ef1b1a0..3028a1c2c 100644 --- a/docs/design/vocabulary-model-rfc.md +++ b/docs/design/vocabulary-model-rfc.md @@ -223,13 +223,13 @@ The real-name corpora agree on the priority. In the GND and Wikidata corpora, Sp Phase 4 also decides what a parse returns, aligned with CLDR's person-name fields, which gives ground truth for free. Five things were asked for: a second surname (`surname2`), a patronymic (decision 8), credentials apart from generations (#326), a meaning for `middle` when a given name is several words (#704), and the reading each token took (`Token.reading`, phase 2's item 13). The first four are fields, and the design answers them with one shape; the fifth is what most of them are read from. -**The seven roles stay, and every new field is a view.** A `Role` partitions the tokens: each word is in exactly one of title, given, middle, family, suffix, nickname and maiden. Each of the four fields subdivides a role that exists, not to add a part a name has outside them: a second surname is part of the family, generation and credentials are parts of the suffix, a second given name is the middle, and a patronymic is a word the parse already puts in the family (Icelandic, Arabic and Malay names) or in the middle (East Slavic names). So each is a view computed from the tokens, as `family_base`, `family_particles`, `surnames` and `given_names` are today (rules/R1, rules/R2): a property and a render key, with `as_dict()`, `comparison_key()` and the seven role properties unchanged. Every view is an addition under the 3.0 freeze. A new `Role` member would not be: `as_dict()` would gain a key, `comparison_key()` a component (a stored key would stop matching), `for role in Role` a member, and the role a token moves out of would return less text than it does in 2.x, a change in what `family` or `suffix` means that no 2.x warning has announced. +**The seven roles stay, and every new field is a view.** A `Role` partitions the tokens: each word is in exactly one of title, given, middle, family, suffix, nickname and maiden. Each field asked for subdivides a role that exists, not to add a part a name has outside them: a second surname is part of the family, generation and credentials are parts of the suffix, a second given name is the middle, and a patronymic is a word the parse already puts in the family (Icelandic, Arabic and Malay names) or in the middle (East Slavic names). So each is a view computed from the tokens, as `family_base`, `family_particles`, `surnames` and `given_names` are today (rules/R1, rules/R2), except the second given name, which is the `middle` that exists (below): a property and a render key, with `as_dict()`, `comparison_key()` and the seven role properties unchanged. Every view is an addition under the 3.0 freeze. A new `Role` member would not be: `as_dict()` would gain a key, `comparison_key()` a component (a stored key would stop matching), `for role in Role` a member, and the role a token moves out of would return less text than it does in 2.x, a change in what `family` or `suffix` means that no 2.x warning has announced. What decides a view is the reading each token took, so `Token.reading` comes first; `surname2` alone is read from a tag O6 writes, a rule rather than a word deciding it: - **`Token.reading`**: a `Token` field holding the `Reading` the parse took, the same type `Ambiguity.alternative` holds the reading not taken in (decisions/A1), `None` for a token `replace()` builds. Its `kind` says what the word was read as, its `source` why, and a `languages` field, added to `Reading`, names the language files holding that reading, filled from phase 3's reading index (3a) and left empty for a reading no file holds (a caller's own vocabulary, a shape, position alone). The field takes `ADDED_IN` metadata, so a 2.x pickle loads with `None` (the pickle guard as widened in 2.7), and is left out of comparison (`compare=False`), so `==` between two `ParsedName`s means what it means in 2.x. `ReadingKind` gains `PATRONYMIC`, its docstring already allowing new kinds. -- **Generation and credentials** (#326): `generation`, the suffix tokens whose reading is a generation or an agnomen (`Jr.`, `III`, `Filho`), and `credentials`, the rest of the suffix (`PhD`, `Esq.`, the honorifics after a name), so the two partition `suffix` and `suffix` is unchanged. CLDR has only these two buckets; the post-nominals that are neither (`Esq.`, the CJK and Hebrew honorifics, `Rinpoche`) fall in `credentials`, CLDR's catch-all, and a caller who wants them apart reads `Token.reading.kind`, which says `post-nominal`. Measured 2026-10-28 on `c8357e74`, over the committed corpus's 6,533 distinct entries (`snapshot.corpus_entries()`) each parsed under its declared order: of the 1,054 with a suffix, 87 hold a generation (or agnomen) and a credential together, and 65 hold only a listed post-nominal of another kind. Of the 615 CLDR renderings with a generation or credentials field, 75 parse with exactly CLDR's suffix words, and in all 75 a split by the vocabulary's reading kind gives CLDR's `generation` and `credentials` both; the other 540 differ in which words the suffix holds, a parse question the split cannot fix. Recompute: classify each suffix token by the kinds of its readings in nameparser/vocab/, skipping `default = false` ones, a generation or agnomen reading winning, an unlisted roman numeral `i` to `x` a generation; over the corpus entries above, and over `cldr_fields.jsonl` for CLDR. -- **A patronymic** (decision 8): `patronymic`, the tokens whose reading is `PATRONYMIC`, in whichever role the parse put them, with `family` and `middle` unchanged. What fills it is a word or a rule that says patronymic, never a guess from position: the vocabulary's `patronymic` particles (`bin`, `binti`, `ibn`, `bint`, Hebrew `bar` and `bat` and the native-script forms) with the name after them, in the default, so `Mahathir bin Mohamad` reads family `bin Mohamad` as today and patronymic `bin Mohamad` beside it; the East Slavic and Turkic rules where RU or TR_AZ turns them on, so `Ivan Ivanovich Ivanov` under RU keeps middle `Ivanovich` and adds patronymic `Ivanovich`; and an Icelandic ending rule an `is` pack would turn on, the default unable to tell `Karlsdóttir` from the Swedish surname `Johansson`. The name `patronymic` is kept for a matronymic too, which is written the same way and cannot be told from one. **`family` is not emptied** by recognizing a patronymic, revising what this section said before: the people filling in the fielded corpora put the patronymic in the family box (below), as CLDR's samples put an Icelandic patronymic in `surname` or `surname-core` (9 of the 9 fields of the `is` locale holding one) and a Russian one in `given2` (6 of 6), our `middle`, while Wikidata records an Icelandic name as a patronymic with no family name; a `family` that keeps it and a `patronymic` that names it satisfy both. A caller writing "Mr. {family}" reads `patronymic` to see that the family is one. +- **Generation and credentials** (#326): `generation`, the suffix tokens whose reading is a generation or an agnomen (`Jr.`, `III`, `Filho`), and `credentials`, the rest of the suffix (`PhD`, `Esq.`, the honorifics after a name), so the two partition `suffix` and `suffix` is unchanged. CLDR has only these two buckets; the post-nominals that are neither (`Esq.`, the CJK and Hebrew honorifics, `Rinpoche`) fall in `credentials`, CLDR's catch-all, and a caller who wants them apart reads `Token.reading.kind`, which says `post-nominal`. Measured 2026-10-28 on `c8357e74`, over the committed corpus's 6,533 distinct entries (`snapshot.corpus_entries()`) each parsed under its declared order: of the 1,054 with a suffix, 33 hold a generation (or agnomen) and a listed credential together, 77 a generation beside any other suffix word, and 64 only listed post-nominals of other kinds. Of the 615 CLDR renderings with a generation or credentials field, 75 parse with exactly CLDR's suffix words, and in all 75 a split by the vocabulary's reading kind gives CLDR's `generation` and `credentials` both; the other 540 differ in which words the suffix holds, a parse question the split cannot fix. Recompute: classify each suffix word (a token tagged `joined` merged into the one before it) by the kinds of its readings in nameparser/vocab/, matched lowercased with its periods and spaces removed, skipping `default = false` readings, a generation or agnomen reading winning over a credential and a credential over the other kinds, an unlisted roman numeral `i` to `x` a generation and any other unlisted word unlisted; over the corpus entries above, and over `cldr_fields.jsonl` for CLDR. +- **A patronymic** (decision 8): `patronymic`, the tokens whose reading is `PATRONYMIC`, in whichever role the parse put them, with `family` and `middle` unchanged. What fills it is a word or a rule that says patronymic, never a guess from position: the vocabulary's `patronymic` particles (`bin`, `binti`, `ibn`, `bint`, Hebrew `bar` and `bat` and the native-script forms) with the name after them, in the default, so `Mahathir bin Mohamad` reads family `bin Mohamad` as today and patronymic `bin Mohamad` beside it; the East Slavic and Turkic rules where RU or TR_AZ turns them on, so `Ivan Ivanovich Ivanov` under RU keeps middle `Ivanovich` and adds patronymic `Ivanovich`; and an Icelandic ending rule an `is` pack would turn on, the default unable to tell `Karlsdóttir` from the Swedish surname `Johansson`. The name `patronymic` is kept for a matronymic too, which is written the same way and cannot be told from one. **`family` is not emptied** by recognizing a patronymic, revising what this section said before: the people filling in the fielded corpora put the patronymic in the family box (below), as CLDR's samples put an Icelandic patronymic in `surname` or `surname-core` (CLDR's `is` file's 3 Icelandic samples, in all 9 of their renderings; its one foreign sample puts a `-dóttir` word in `surname2`) and a Russian one in `given2` (its 2 `ru` samples, in all 6 renderings), our `middle`, while Wikidata records an Icelandic name as a patronymic with no family name; a `family` that keeps it and a `patronymic` that names it satisfy both. A caller writing "Mr. {family}" reads `patronymic` to see that the family is one. - **A second surname**: `surname2`, the family's second name unit where a rule has read the family as two (rules/O6, which a caller or an `es` pack turns on), and empty otherwise, `family` keeping both units as O6 makes it today. O6 already knows which unit it took, so it marks where the second one starts with a new stable tag; a family comma alone marks nothing, since `Lloyd Webber, Andrew` is not two surnames, and a hyphenated family is one token and stays whole. Whether the first unit gets a view of its own, and its name (`surname` sits close to the `surnames` that exists), is a value question below. `surnames` keeps its meaning, `middle` and `family` joined, the reading of a double surname the default splits. - **`middle`** (#704): keeps its meaning, every word of the given part after the given name, and the given name is one name unit unless a rule joins words into it, as rules/P5 already does (`Abdul Aziz Tariq` → given `Abdul Aziz`). A compound given name (`Maria do Carmo`, `María José`) is the `[rules]` rule above, turned on by a caller or a pack, moving its words into `given` only under that rule. CLDR's `given2` is `middle`, so no view is added. A Sikh `Kaur` and an English name beside a Chinese one (`Lim Mei Ling Jessica`) stay in `middle`, an accepted cost. @@ -247,13 +247,13 @@ What decides a view is the reading each token took, so `Token.reading` comes fir ORCID's and PatentsView's given box holds the middle names too, so a two-word given box there cannot tell a compound given name from a middle name; PSC's middle box can, though its form steers what goes in it. The last row is a separate question from the patronymic: a name of given names only, written by people who have no family name, which no output field fixes and only a rule could read (an open question below). -**Render defaults** (decisions/3-0-reevaluations' clean slate). The default `str()` reads every name given-first and joins its words with spaces, so a name read family-first is turned round and a written separator is lost: `str(parse("김민준"))` is `'민준 김'`, `str(parse("王 小明"))` is `'小明 王'` and `str(parse("威廉·莎士比亚"))` is `'威廉 莎士比亚'`. Over the committed corpus's distinct entries, each under its declared order (2026-10-28, `c8357e74`), 728 without a comma read family-first (the family's first word written before the given name's), `str()` renders every one differently from the text, and 150 of them are written unspaced; of 41 holding the U+00B7 interpunct, `str()` drops it from 36. So: +**Render defaults** (decisions/3-0-reevaluations' clean slate). The default `str()` reads every name given-first and joins its words with spaces, so a name read family-first is turned round and a written separator is lost: `str(parse("김민준"))` is `'민준 김'`, `str(parse("王 小明"))` is `'小明 王'` and `str(parse("威廉·莎士比亚"))` is `'威廉 莎士比亚'`. Over the committed corpus's distinct entries, each under its declared order (2026-10-28, `c8357e74`), 728 without a comma read family-first (the family's first word written before the given name's), `str()` renders every one differently from the text, and in 150 of them the family's last word touches the given name's first, written unspaced; of 41 holding the U+00B7 interpunct, `str()` drops it from 36. So: - `ParsedName` gains `order`, the order the name was read under (its script's or the declared `name_order`, a comma aside), an addition with `ADDED_IN` metadata whose missing value, in a 2.x pickle, renders as today, and left out of comparison as `Token.reading` is, so a loaded 2.x pickle still equals a fresh parse of its text. -- `str()` and `render()`'s default spec put the family first where `order` does, and two words that stand side by side in the original and in the rendering are joined by what separated them there, nothing, a space, an interpunct or a nakaguro, so `str(parse("김민준"))` is `'김민준'` and `str(parse("威廉·莎士比亚"))` keeps the dot. A name with a comma or written given-first keeps today's word order (`Smith, John` → `'John Smith'`), while the written separator is kept for every name, so the given-first transcription `威廉·莎士比亚`, most of the interpunct names measured, keeps its dot too. +- `str()` and `render()`'s default spec put the family first where `order` does, and two words that stand side by side in the original and in the rendering are joined by what separated them there, nothing, a space, an interpunct or a nakaguro, so `str(parse("김민준"))` is `'김민준'` and `str(parse("威廉·莎士比亚"))` keeps the dot. A name with a comma or written given-first keeps today's word order, and the separator kept is any but a comma, which a space replaces (`Smith, John` → `'John Smith'`), so the given-first transcription `威廉·莎士比亚` keeps its dot too: 20 of the 36 names whose interpunct `str()` drops are read given-first, one of them with a comma. - `initials()` gives nothing for a word in a script with no initials (Han, Hangul, kana, `_policy._NO_INITIALS`), where today it gives the first character with a period (`'王.'`). -These change no signature but change `str()`'s output for every name read family-first, which no 2.x release can warn about; Derek took them as an improvement for 3.0's release notes rather than a break (item 6 of the entry). +These change no signature but change `str()`'s output for every name read family-first and every name whose written separator is neither a space nor a comma, which no 2.x release can warn about; Derek took them as an improvement for 3.0's release notes rather than a break (item 6 of the entry). **What changes in the API**, by the 3.0 freeze: @@ -266,7 +266,7 @@ These change no signature but change `str()`'s output for every name read family | new `Policy` fields for rules, and `Locale.languages` (phase 3) | additions | | `Role`, `as_dict()`, `comparison_key()`, the seven role properties | unchanged | | the default `str()`, `render()` spec and `initials()` output | behavior change, accepted for 3.0 | -| a `Role` member for any of the four | a break, declined | +| a `Role` member for any of these views | a break, declined | **Phase 3's open API dependency answered**: phase 3's draft (branch `feat/vocabulary-phase3-design`, not yet merged) carries a choice of languages on `Locale.languages` and notes it assumes `Locale` and `Policy` keep their 2.x shape once the facade is gone; the 2026-10-28 freeze answers that, both keeping it with additions. The rest of that item, strict as an argument of the whole choice included, is phase 3's to decide, and its entry is to say the dependency is answered when it lands. With the `[rules]` proposal above, a shipped language pack is a `Locale` whose `languages` choose its vocabulary and whose `policy` carries its rules, so `Locale` needs no third field. @@ -289,7 +289,7 @@ Each sub-phase is one PR, measured on its own base as phase 2's and phase 3's ar **Interleaved with phase 3** (its sub-phases 3a to 3h are the draft's on `feat/vocabulary-phase3-design`): 4a can start beside 3a, its `languages` waiting for 3a's index; 4b to 4d need nothing of phase 3 and unblock its packs, `cs`'s generations (3f) and `pt-BR`'s agnomens (3h) on 4b, `es` and `pt` (3g) on 4d and 4g; 4f lands with or after 3b, since both decide what a shipped `Locale` holds; an `is` pack, which no phase 3 sub-phase lists, needs 3b, 4c and 4f. 4e and 4h stand apart from both. So the order is 3a and 4a, then 4b to 4d with 3b and 3c, then 4f, then the packs with 4g, with 4e at any point. -**Left open**, with no sub-phase yet: phase 1's four reserved names said to be phase 4's (`separators` with phase 4's glued words, and `bare_only`, `leading_only` and `trailing_only`, which this RFC's phase 2 section found not needed), whose build messages still name this phase. Value questions inside the shapes above: the first surname's view and its name; which particles `family_base` removes, rules/R2 saying the leading ones and the code removing every working one, proposed in decisions/R2's 2026-10-28 #754 entry as R2 amended to the code, since each such particle leads its own name unit and the unit bases joined are the code's base; whether `credentials` should leave the honorifics out, CLDR having no third bucket; which ending rules beside Icelandic a pack may name (Hungarian `-né` is a married name, not a patronymic); and the names of given names only (the last table row), whose reading needs a rule that tells a mononym from a given name and a family name, not an output field. +**Left open**, with no sub-phase yet: phase 1's four reserved names said to be phase 4's (`separators` with phase 4's glued words, and `bare_only`, `leading_only` and `trailing_only`, which this RFC's phase 2 section found not needed), whose build messages still name this phase. Value questions inside the shapes above: the first surname's view and its name; whether `order`, and `Token.reading` beneath the new views, stay out of comparison, which keeps a loaded 2.x pickle equal to a fresh parse of its text but lets two equal names render differently (a 2.x pickle of `김민준` renders `'민준 김'` and a fresh parse `'김민준'`, and a set keeps one), where today every rendering is derived from what `==` compares; which particles `family_base` removes, rules/R2 saying the leading ones and the code removing every working one, proposed in decisions/R2's 2026-10-28 #754 entry as R2 amended to the code, since each such particle stands in the particle run that opens a name unit, or ends the family, and the unit bases joined are the code's base; whether `credentials` should leave the honorifics out, CLDR having no third bucket; which ending rules beside Icelandic a pack may name (Hungarian `-né` is a married name, not a patronymic); and the names of given names only (the last table row), whose reading needs a rule that tells a mononym from a given name and a family name, not an output field. ## The first packs From 72dad292b724d06f9a8ed24a0dc8f943f539fe44 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Wed, 28 Oct 2026 12:19:39 -0700 Subject: [PATCH 12/39] Amend rules/R2 to the code: the base drops every particle (#754) family_base removes every working particle wherever it stands, and R2 now says so, its Because calling the base a view the same under every naming convention rather than a sort key. decisions/R2 records the Portuguese last-surname and Spanish first-surname filing conventions with sources (RAE's rules a and e read from the page), and the RFC leaves open whether a pack supplies a convention's sort key. R2's size grows by three example lines and reads no new signal; budget re-recorded. Leaving ParsedName.order out of == is recorded as an accepted cost. Co-Authored-By: Claude Opus 5.5 --- .../R2-all-particles-family-base-divergence.md | 10 ++++++---- docs/design/decisions/vocabulary-model.md | 2 +- .../rules/R2-all-particles-family-base-divergence.md | 7 +++++-- docs/design/vocabulary-model-rfc.md | 4 ++-- nameparser/_types.py | 4 ++-- tests/v2/rules_budget.json | 2 +- 6 files changed, 17 insertions(+), 12 deletions(-) diff --git a/docs/design/decisions/R2-all-particles-family-base-divergence.md b/docs/design/decisions/R2-all-particles-family-base-divergence.md index 92c151730..fb378fc32 100644 --- a/docs/design/decisions/R2-all-particles-family-base-divergence.md +++ b/docs/design/decisions/R2-all-particles-family-base-divergence.md @@ -7,11 +7,12 @@ - The words keep their `particle` tag, which means "from the particle vocabulary wherever it lands", and the decision is recorded beside it as UNJOINED_TAG rather than by stripping the tag — 2026-08-18 #404/#385. - `initials()` reads the base, so the anchoring moves R3's output: `Anh Do` initials 'A. D.' and `Juan van der` 'J. v. d.' — 2026-08-16 (pre-merge coherence pass); 2026-08-18 #404/#385. - A particle run that joins nothing still chains whole into one part, so `Juan Smith van der` reads family 'van der' and middle 'Smith'; splitting it was declined, roles reassembling adjacent same-role pieces into one part and the base losing its full extent — 2026-08-18. +- The base is the family without its working particles, wherever in the family they stand (`da Silva do Amaral de Souza, Miguel` → family_base 'Silva Amaral Souza', family_particles 'da do de'), a view the same under every convention and no culture's sort key — 2026-10-28 #754. - The unjoined mark is a statement about a whole part, and `initials()` honors it for every word in it, a connective included: under `Lexicon.default().add(particles={'y'})`, `Juan de y` reads family_base 'de y' and initials 'J. d. y.'; narrowing the readmission to the `particle` tag alone was tried and backed out — 2026-08-29 #461; 2026-09-20 #461. Elsewhere: where a connective initials, a part holding nothing else for it to join (`Jon Dough and` → 'J. D.', `Juan de y` → 'J. y.') — decisions/R3's 2026-09-20 #461 entry; case repair keeping that connective lowercase, its carve-out not gated on the mark (under `Lexicon.default().add(particles={'y'})`, `Juan de y` forced → family 'De y') — decisions/R4's 2026-08-29, the CONJUNCTION CARVE-OUT entry; case repair reading the unjoined mark for the whole part (`ANH DO` → 'Anh Do') — decisions/R4's 2026-08-29 #407 entry; the snapshot recording `initials()`, so a change to the base is visible there — decisions/differential-ledger's 2026-10-13 #664, a committed snapshot replaces the classified gate entry. -Open: #754 (which particles `family_base` removes, the leading run or every working particle; proposed in the 2026-10-28 #754 entry). +Open: #754 (whether a language pack supplies a convention's sort key or entry element from Phase 4's per-surname units, the RFC's Phase 4 Left open). ## Log @@ -34,10 +35,11 @@ Open: #754 (which particles `family_base` removes, the leading run or every work - 2026-09-20 #461 — AMENDS the 2026-08-29 bullet above, which said of four readings that "none of those moved or may move". Two of the four moved and two did not, and the half that survived is the half the criterion was always about. WHAT MOVED: `parse("Juan y Garcia")` initials "J. y. G." where that bullet recorded "J. G.", and R3's own `"Juan de y"` example line reads "J. y." where it read "J.". WHAT DID NOT: `parse("Juan Velasquez y Garcia")` is still "J. V. G." over base "Velasquez y Garcia", and `parse("Jon Dough and")` is still "J. D." over base "Dough and" (all four measured 2026-09-20 on this branch). The criterion that separates them is the one the bullet above named, and it is still not a word count: the question is asked of the WHOLE PART, and a part holding another name word for the connective to join — the part's working particles set aside — is a part where the connective is doing a connective's work and contributes nothing. "Velasquez y Garcia" and "Dough and" each hold such a word, so the skip stands there; the family "y", the family "de y" and the middle "y" of "Juan y Garcia" hold none, so the connective is its part's own name word and initials with it. So that bullet was RIGHT ABOUT THE MECHANISM and wrong only about the clause it preserved. Its finding — that a mark honored for some of a part's words and not for one of them is honoring nothing — is exactly what rules/R3 now states, for all three groups at once, and it is why the fact is computed ONCE over the whole part rather than re-derived per word by each view. What changed is the clause it left standing: R3's "A CONJUNCTION never initials, so a base that is one contributes nothing even then" is conditional now, and "none of those may move" was the reading of the day rather than a decision. decisions/R3's 2026-09-20 bullet carries the rule as taken. The pin that bullet describes did its job: `test_initials_readmits_a_conjunction_in_a_particle_part` was written to fail when #461 was re-decided, and it failed. -- **2026-10-28 #754 — PROPOSED, NOT DECIDED: `family_base` REMOVES EVERY WORKING PARTICLE, EACH THE LEADING PARTICLE OF THE NAME UNIT IT JOINS, SO RULES/R2 IS AMENDED TO THE CODE RATHER THAN THE CODE TO R2.** Rules/R2's Then says the base is "the family without its leading particles", but the view drops every family word tagged `particle` and not marked unjoined, wherever it stands: `da Silva do Amaral de Souza, Miguel` reads family 'da Silva do Amaral de Souza', base 'Silva Amaral Souza', particles 'da do de', where the 2.x facade's `last_prefixes` gave 'da'; the P1 fold after a family comma makes a particle interior with none leading (`Smith, de Mesnil Jean` → base 'Smith Mesnil Jean', particles 'de'); and a leading run alone reads the same either way (`Juan de la Vega y Ortega` → base 'Vega y Ortega'). `test_particle_views_after_fold`, on #937's branch and not yet on master, pins the code's reading of `Miguel da Silva do Amaral de Souza` under `middle_as_family`. Three answers were weighed: (1) amend R2 to the code; (2) remove only the leading run (base 'Silva do Amaral de Souza', particles 'da'); (3) derive the base from per-surname views, each surname keeping its own leading particle. - (1) and (3) are one answer. A particle joins forward to the word it modifies, so every working particle inside a family stands in the particle run that opens a name unit (`la` in `de la Vega` is the run's second word), and the unit bases joined are the code's base: under `two_family_names`, rules/O6 reads `Miguel da Silva do Amaral de Souza` as middle 'da Silva' and family 'do Amaral de Souza', whose units 'do Amaral' and 'de Souza' have bases 'Amaral' and 'Souza', and the family's base is 'Amaral Souza'. So `surname2` (the RFC's Phase 4, sub-phase 4d) reaches the question only where O6 runs and only for its two units; it needs nothing from `family_base`, and the first unit's view, left open there, would read the same way. One shape escapes the unit reading: a working particle ending a family of other words opens no unit and joins nothing (a family of two or more words whose last is a working particle: 1 committed entry and 1 `tse-donors` filed name in the default policy, 39 and 505 with `middle_as_family`, the bare word-particle shape 9 of the 39), and the code drops it from the base as it does any other; whether a lone trailing particle is unjoined is R2's all-particles question asked of one word, left as the code reads it. The proposal is R2's Then as "the base (the family without its particles, each in the run that opens the name unit it joins, or ending the family) and the particles themselves", with `da Silva do Amaral de Souza, Miguel` and `Smith, de Mesnil Jean` as its examples, the code unchanged, and #937's test with it; landing it re-quotes the R2 citation on `ParsedName.family_particles` in nameparser/_types.py, which tests/v2/test_doc_citations.py holds to R2's text, and docs/migrate.rst's `family_base` line ("The surname with leading particles split off"), which nothing checks, beside its `last_prefixes` row, the mapping that reads 'da' where the core reads 'da do de'. +- **2026-10-28 #754 — `family_base` REMOVES EVERY WORKING PARTICLE, SO RULES/R2 IS AMENDED TO THE CODE RATHER THAN THE CODE TO R2, AND THE BASE IS A VIEW THE SAME UNDER EVERY CONVENTION, NOT ANY CULTURE'S SORT KEY.** Rules/R2's Then says the base is "the family without its leading particles", but the view drops every family word tagged `particle` and not marked unjoined, wherever it stands: `da Silva do Amaral de Souza, Miguel` reads family 'da Silva do Amaral de Souza', base 'Silva Amaral Souza', particles 'da do de', where the 2.x facade's `last_prefixes` gave 'da'; the P1 fold after a family comma makes a particle interior with none leading (`Smith, de Mesnil Jean` → base 'Smith Mesnil Jean', particles 'de'); and a leading run alone reads the same either way (`Juan de la Vega y Ortega` → base 'Vega y Ortega'). `test_particle_views_after_fold`, on #937's branch and not yet on master, pins the code's reading of `Miguel da Silva do Amaral de Souza` under `middle_as_family`. Three answers were weighed: (1) amend R2 to the code; (2) remove only the leading run (base 'Silva do Amaral de Souza', particles 'da'); (3) derive the base from per-surname views, each surname keeping its own leading particle. + (1) and (3) are one answer. A particle joins forward to the word it modifies, so every working particle inside a family stands in the particle run that opens a name unit (`la` in `de la Vega` is the run's second word), and the unit bases joined are the code's base: under `two_family_names`, rules/O6 reads `Miguel da Silva do Amaral de Souza` as middle 'da Silva' and family 'do Amaral de Souza', whose units 'do Amaral' and 'de Souza' have bases 'Amaral' and 'Souza', and the family's base is 'Amaral Souza'. So `surname2` (the RFC's Phase 4, sub-phase 4d) reaches the question only where O6 runs and only for its two units; it needs nothing from `family_base`, and the first unit's view, left open there, would read the same way. One shape escapes the unit reading: a working particle ending a family of other words opens no unit and joins nothing (a family of two or more words whose last is a working particle: 1 committed entry and 1 `tse-donors` filed name in the default policy, 39 and 505 with `middle_as_family`, the bare word-particle shape 9 of the 39), and the code drops it from the base as it does any other; whether a lone trailing particle is unjoined is R2's all-particles question asked of one word, left as the code reads it. R2's Then now reads "the base (the family without its particles) and the particles themselves", its Because saying the base is the same view under every convention rather than "sorting wants" it, with `da Silva do Amaral de Souza, Miguel` and `Smith, de Mesnil Jean` among its examples; the code is unchanged, and #937's test with it, and the R2 citation on `ParsedName.family_particles` in nameparser/_types.py is re-quoted. Still to move when #937 lands: docs/migrate.rst's `family_base` line ("The surname with leading particles split off"), which nothing checks, beside its `last_prefixes` row, the mapping that reads 'da' where the core reads 'da do de'; the facade docs are #937's to change. Measured 2026-10-28 on master `c8357e74`, counts and token shapes only, over the committed corpus's 6,533 distinct entries (`snapshot.corpus_entries()`, each parsed under its declared order) and `tse-donors`' 854,531 records (`local_corpora.read_records`, the filed name and the registry name each parsed in the default order). A family is read in render order (FOLDED_TAG tokens first), a working particle is a token tagged `particle` and not UNJOINED_TAG, and a family has an interior particle where a working particle stands after its first word that is neither a particle nor a conjunction; that family is where (1) and (2) differ. The three kinds of interior family are tested in this order, the first that fits counting: after a family comma, every interior particle standing after the first comma; with a leading run as well; with no leading run. In the default policy: committed, 24 against 684 with a leading run only (12 with no leading run, 8 whose interior particle stands after a family comma, 4 with a leading run as well); `tse-donors`, 12 filed names against 347,992 and 1 registry name against 392,134, since a Brazilian name's particles before its last word stand in `middle`. With `middle_as_family`: committed, 367 against 398 (262, 92 and 13); `tse-donors`' filed names, 268,591 against 79,926 (261,024, 5 and 7,562), the commonest family shape word-particle-word, 150,998 times. Recompute with those definitions; a change to the fold or to O6 moves them. - Weighed against (2): with `middle_as_family`, the population the per-surname reading serves, it gives the 261,024 `tse-donors` names with no leading run a base equal to the whole family and an empty particle view, so `family_particles` stops naming the particles it is for; and it changes a frozen property's output for 24 committed entries in the default policy, where (1) changes no output. The facade whose `last_prefixes` it matches is removed in 3.0. Its one gain, a base that is a written name, is what a sort by first surname wants, and that is the first unit's view left open in the RFC's Phase 4, not `family_base`. The cost of (1), accepted if agreed: the base is a sort key and not a name anyone writes ('Silva Amaral Souza'). + Weighed against (2): with `middle_as_family`, the population the per-surname reading serves, it gives the 261,024 `tse-donors` names with no leading run a base equal to the whole family and an empty particle view, so `family_particles` stops naming the particles it is for; and it changes a frozen property's output for 24 committed entries in the default policy, where (1) changes no output. The facade whose `last_prefixes` it matches is removed in 3.0. Its one gain is a base that is a written name, which no convention's key needs either (below). The cost of (1), accepted: the base is not a name anyone writes ('Silva Amaral Souza'), nor any convention's sort key. + Why the base is not a sort key, researched 2026-10-28. Portuguese and Brazilian practice files a name under its LAST surname, that surname's particle removed and set after the given names, the agnomens `Filho`, `Neto` and `Júnior` staying with the surname before them and an established compound surname kept whole: `SOUZA, Miguel da Silva do Amaral de`, `NOGUEIRA NETO, Wanderlino` (ABNT NBR 6023:2018, https://engenhariaeletrica.ufes.br/sites/engenhariaeletrica.ufes.br/files/field/anexo/abnt6023.2018.pdf; the ABNT Zotero style's comments, https://www.zotero.org/styles/universidade-federal-do-espirito-santo-abnt?source=1; USP's references manual, https://www.ip.usp.br/site/wp-content/uploads/2025/04/Manual-Referencias-ABNT-SabadiniSampaioNascimento.pdf; Cambridge University Library, https://www.lib.cam.ac.uk/node/3142; the UN cataloguing manual, https://www.un.org/Depts/dhl/unbisref_manual/nameguide/personal.htm; IFLA, Names of Persons: Portugal, 2009, https://ifla.org/files/assets/cataloguing/pubs/ifla_names_of_persons_portugal_2009.pdf). Spanish practice files under the FIRST surname, the second its secondary key (RAE, Ortografía de la lengua española, 2010, ch. VII, "Alfabetización de antropónimos", https://www.rae.es/ortografía/alfabetización-de-antropónimos, read 2026-10-28 in a browser, the page refusing a plain fetch): by its rule (a) a preposition, with or without an article, heading the FIRST surname is left out of the order and written after the given name (`Torre Ibarra, Ramón de la`); by rule (e) one heading the SECOND surname counts as part of the second key (`Gómez Casas`, `Gómez de la Iglesia`, `Gómez de Lara`, `Gómez del Castillo`, `Gómez Echagüe`, in that order); by rule (b) a leading article or `San`/`Santa` stays and is sorted on (`La Calle Ruiz, Carmen`); by rule (d) a `y` between the surnames counts (`Aguirre y Gómez-Pacheco`). The same page states the Portuguese convention from the other side, the paternal second surname being the axis. Neither convention's key is this base: Portuguese practice sorts `…da Silva de Souza do Amaral` before `…da Silva do Amaral de Souza` (Amaral before Souza) while both bases begin `Silva`, and Spanish practice keeps a second surname's particle (`Gómez de la Iglesia`) where the base drops it ('Gómez Iglesia'). The base dropping every working particle is nearest the Spanish first key alone. So the base is the convention-neutral view, and a convention's key is built from the surnames by whoever knows the convention, the RFC's Phase 4 Left open. In the default policy, `Miguel da Silva do Amaral de Souza` already reads family 'de Souza', base 'Souza', which is the Portuguese entry element by coincidence: only `middle_as_family` gives the long family. Declined: diff --git a/docs/design/decisions/vocabulary-model.md b/docs/design/decisions/vocabulary-model.md index 0cfbcaa2e..bfdc0bac9 100644 --- a/docs/design/decisions/vocabulary-model.md +++ b/docs/design/decisions/vocabulary-model.md @@ -249,7 +249,7 @@ Open: #913 (the title words phase 2j left, read as titles though borne as names) 3. **A `patronymic` view, filled by a word or a rule that says patronymic, with `family` and `middle` unchanged.** The default fills it from the vocabulary's `patronymic` particles (`bin`, `binti`, `ibn`, `bint`, `bar`, `bat` and the native-script forms) with the name they introduce (`Mahathir bin Mohamad` → family `bin Mohamad`, as today, and patronymic `bin Mohamad`); RU's and TR_AZ's rules fill it where a caller turns them on (`Ivan Ivanovich Ivanov` under RU → middle `Ivanovich` and patronymic `Ivanovich`); an Icelandic ending rule fills it under an `is` pack only, since the default cannot tell `-dóttir` and `-son` from Swedish surnames (ORCID's Swedish records end their family box in `-son` or `-dóttir` in 2,184 of 20,000). A matronymic, written the same way, fills it too. SUPERSEDES: the RFC's earlier statement of decision 8's value, that `family` can be empty for a name with a patronymic. Measured: filers put an Icelandic patronymic in the family box (ORCID 1,647 of 2,583 Icelandic records, PatentsView 535 of 722, PSC 101 of 125), and CLDR's `is` file's 3 Icelandic samples put it in `surname` or `surname-core` in all 9 of their renderings (its one foreign sample puts a `-dóttir` word in `surname2`), while Wikidata's `patronymic-is` and `patronymic-my` strata record 108 names with a patronymic and no family name, of which the parse puts the patronymic in `family` 107 times. An East Slavic patronymic stands in the given part: CLDR's 2 `ru` samples put it in `given2` in all 6 renderings; PatentsView's Russian given boxes end in one 8,423 times of 20,000, and the parse reads such a word as the middle 10,902 times of 10,962; 90 of the 112 Russian nationals in PSC who filled the middle-name box wrote one there. A `family` that keeps it and a `patronymic` that names it agree with both kinds of source. Weighed against: emptying `family` where the patronymic is recognized, which moves the field the fielded sources fill and changes `family` for every Icelandic and nasab name, a behavior change of a frozen property; and a neutral parent-names name for the view, which buys nothing the view can detect, since a matronymic and a patronymic are written alike. 4. **`generation` and `credentials` are views that partition `suffix` (#326)**, not roles, `generation` the tokens read as a generation or an agnomen, `credentials` the rest, the post-nominals that are neither (`Esq.`, the honorifics after a name) falling in `credentials`, CLDR's catch-all, with `Token.reading.kind` telling them apart for a caller who needs it. Measured: of the 1,054 committed corpus entries with a suffix, 33 hold a generation (or agnomen) and a listed credential together, 77 a generation beside any other suffix word, and 64 only listed post-nominals of other kinds (each suffix word classified by the RFC's Phase 4 recipe, joined tokens merged); of the 615 CLDR renderings with either field, the 75 whose suffix holds exactly CLDR's words split into CLDR's `generation` and `credentials` by the vocabulary's reading kind in all 75. Weighed against: two new `Role` members (item 1), and a third view for the honorifics, left open since CLDR has none. 5. **a `surname2` view, the family's second unit where rules/O6 read the family as two**, marked by a new stable tag where O6 takes its word, empty otherwise; `family` keeps both units as O6 makes it, and a family comma alone marks nothing (`Lloyd Webber, Andrew`). Measured: a two-word family box in ORCID's records from Spain 48.0%, Mexico 64.7%, Colombia 72.8% and Chile 50.5%, against 3.9% from the United States. Left open: a view for the first unit and its name, `surname` standing close to the `surnames` view that exists. Declined: redefining `surnames` as the two units, which would repeat `family`, the family holding both where O6 reads two, and lose what `surnames` does today, joining `middle` and `family` so that a double surname the default splits (`Gabriel García Márquez` → middle `García`) reads whole, the view rules/P2 names for that reading. - 6. **The render defaults follow the order the name was read in, a behavior change of the output rather than a break of the API.** `ParsedName` gains `order` (an addition, `ADDED_IN`, a 2.x pickle rendering as today, left out of comparison, which lets two equal names render differently, left open in the RFC); the default `str()` and `render()` spec put the family first where `order` does, and join two words that stand side by side in the original and the rendering with what separated them there (nothing, a space, an interpunct, a nakaguro); `initials()` gives nothing for a word in a script with no initials. A name with a comma or written given-first keeps today's word order, the separator kept being any but a comma, which a space replaces (`Smith, John` → `'John Smith'`, while `威廉·莎士比亚` keeps its dot; 20 of the 36 names whose interpunct `str()` drops are read given-first). Measured over the committed corpus's distinct entries: 728 without a comma read family-first, `str()` renders all 728 differently from their text (`김민준` → `'민준 김'`), 150 of them with the family's last word touching the given name's first, and it drops the U+00B7 interpunct from 36 of the 41 holding one. This carries out decisions/3-0-reevaluations' render-default clean slate. No signature changes, but `str()`'s output changes for every name read family-first and every name whose written separator is neither a space nor a comma, and no 2.x release can warn about a `str()` result, so it is taken as an improvement for 3.0's release notes, not a break needing a warning. + 6. **The render defaults follow the order the name was read in, a behavior change of the output rather than a break of the API.** `ParsedName` gains `order` (an addition, `ADDED_IN`, a 2.x pickle rendering as today, left out of comparison, which lets two equal names render differently, a cost accepted 2026-10-28: a 2.x pickle of `김민준` renders `'민준 김'` and a fresh parse `'김민준'`, and the two compare equal); the default `str()` and `render()` spec put the family first where `order` does, and join two words that stand side by side in the original and the rendering with what separated them there (nothing, a space, an interpunct, a nakaguro); `initials()` gives nothing for a word in a script with no initials. A name with a comma or written given-first keeps today's word order, the separator kept being any but a comma, which a space replaces (`Smith, John` → `'John Smith'`, while `威廉·莎士比亚` keeps its dot; 20 of the 36 names whose interpunct `str()` drops are read given-first). Measured over the committed corpus's distinct entries: 728 without a comma read family-first, `str()` renders all 728 differently from their text (`김민준` → `'민준 김'`), 150 of them with the family's last word touching the given name's first, and it drops the U+00B7 interpunct from 36 of the 41 holding one. This carries out decisions/3-0-reevaluations' render-default clean slate. No signature changes, but `str()`'s output changes for every name read family-first and every name whose written separator is neither a space nor a comma, and no 2.x release can warn about a `str()` result, so it is taken as an improvement for 3.0's release notes, not a break needing a warning. 7. **`middle` keeps its meaning (#704)**, every word of the given part after the given name, the given name one unit unless a rule joins words into it, as rules/P5 already does (`Abdul Aziz Tariq` → given `Abdul Aziz`); a compound given name is a `[rules]` rule a caller or a pack turns on, CLDR's `given2` is `middle`, and `Kaur` and an English name beside a Chinese one stay in `middle`, an accepted cost. Measured: in PSC, whose form has a middle-name box, 2,499 of 1,040,708 forename boxes hold two or more words, and of 6,101 Spanish nationals 25 do while 2,621 fill the middle box; ORCID's and PatentsView's given box holds middles too, so it cannot separate the two. The `Maria do Carmo` shape, a particle inside a given box of three or more words with a family box filled, reads the given box's words into the family in 1,812 of 1,923 ORCID records and 1,165 of 1,219 PatentsView records from Brazil, Portugal, Spain, Mexico, Argentina, Colombia, Chile and Italy, which is that rule's case. Weighed against: #704's option (b), one given unit by default, which changes `middle` for every English name of three words. 8. **A language file's `[rules]` compiles into its shipped `Locale`'s policy patch**, each rule kind a `Policy` field (an addition), so RU and TR_AZ are rebuilt from their files and `Locale` gains no field for rules. With it, **phase 3's open API dependency is answered**: its draft (branch `feat/vocabulary-phase3-design`, unmerged) carries a choice of languages on `Locale.languages`, assuming `Locale` and `Policy` keep their 2.x shape after the facade's removal, and the 2026-10-28 freeze keeps them so. The rest of that draft, strict as an argument of the whole choice included, stays phase 3's to decide; its entry, when it lands, says the dependency is answered. 9. **The work order**, one PR each, measured on its own base, the snapshot recording each new view where it is non-empty: 4a `Token.reading` and `ReadingKind.PATRONYMIC` (neutral; `languages` after 3a); 4b `generation` and `credentials` (neutral); 4c `patronymic` (neutral); 4d `surname2` (neutral); 4e the render defaults and `order`; 4f `[rules]` into the policy patch, with or after 3b; 4g the compound given name, after 4f; 4h the shape inventory per order (#469). Interleaved with phase 3 (its sub-phases the draft's): 4a beside 3a; 4b to 4d before the packs that need them, `cs` (3f) and `pt-BR` (3h) on 4b, `es` and `pt` (3g) on 4d and 4g; an `is` pack on 3b, 4c and 4f. diff --git a/docs/design/rules/R2-all-particles-family-base-divergence.md b/docs/design/rules/R2-all-particles-family-base-divergence.md index d794fca2f..69b307564 100644 --- a/docs/design/rules/R2-all-particles-family-base-divergence.md +++ b/docs/design/rules/R2-all-particles-family-base-divergence.md @@ -2,9 +2,9 @@ **When:** the family name of a parsed name is read with its particles set apart, as in "Dr. Juan Q. Xavier de la Vega III". -**Then:** the family name splits into further views: the base (the family without its leading particles) and the particles themselves. +**Then:** the family name splits into further views: the base (the family without its particles) and the particles themselves. -**Because:** callers need the surname with and without its particles — sorting wants "Vega", display wants "de la Vega". +**Because:** callers need the family name with and without its particles, and the base is that view under every naming convention, not one culture's collation or catalogue key. ``` "Dr. Juan Q. Xavier de la Vega III" → family_base="Vega" @@ -12,6 +12,9 @@ "Juan de la Vega" → family_base="Vega" · boundary "Juan de la Vega" → family_particles="de la" "Sean O'Connor" → family_base="O'Connor" +"da Silva do Amaral de Souza, Miguel" → family_base="Silva Amaral Souza" +"da Silva do Amaral de Souza, Miguel" → family_particles="da do de" +"Smith, de Mesnil Jean" → family_base="Smith Mesnil Jean" ``` **Except:** diff --git a/docs/design/vocabulary-model-rfc.md b/docs/design/vocabulary-model-rfc.md index 3028a1c2c..d39e96b51 100644 --- a/docs/design/vocabulary-model-rfc.md +++ b/docs/design/vocabulary-model-rfc.md @@ -249,7 +249,7 @@ ORCID's and PatentsView's given box holds the middle names too, so a two-word gi **Render defaults** (decisions/3-0-reevaluations' clean slate). The default `str()` reads every name given-first and joins its words with spaces, so a name read family-first is turned round and a written separator is lost: `str(parse("김민준"))` is `'민준 김'`, `str(parse("王 小明"))` is `'小明 王'` and `str(parse("威廉·莎士比亚"))` is `'威廉 莎士比亚'`. Over the committed corpus's distinct entries, each under its declared order (2026-10-28, `c8357e74`), 728 without a comma read family-first (the family's first word written before the given name's), `str()` renders every one differently from the text, and in 150 of them the family's last word touches the given name's first, written unspaced; of 41 holding the U+00B7 interpunct, `str()` drops it from 36. So: -- `ParsedName` gains `order`, the order the name was read under (its script's or the declared `name_order`, a comma aside), an addition with `ADDED_IN` metadata whose missing value, in a 2.x pickle, renders as today, and left out of comparison as `Token.reading` is, so a loaded 2.x pickle still equals a fresh parse of its text. +- `ParsedName` gains `order`, the order the name was read under (its script's or the declared `name_order`, a comma aside), an addition with `ADDED_IN` metadata whose missing value, in a 2.x pickle, renders as today, and left out of comparison as `Token.reading` is, so a loaded 2.x pickle still equals a fresh parse of its text. The cost, accepted 2026-10-28: two equal names can render differently, where today every rendering is derived from what `==` compares (a 2.x pickle of `김민준` renders `'민준 김'` and a fresh parse `'김민준'`, and a set keeps one of them); the views read from `Token.reading` share it. - `str()` and `render()`'s default spec put the family first where `order` does, and two words that stand side by side in the original and in the rendering are joined by what separated them there, nothing, a space, an interpunct or a nakaguro, so `str(parse("김민준"))` is `'김민준'` and `str(parse("威廉·莎士比亚"))` keeps the dot. A name with a comma or written given-first keeps today's word order, and the separator kept is any but a comma, which a space replaces (`Smith, John` → `'John Smith'`), so the given-first transcription `威廉·莎士比亚` keeps its dot too: 20 of the 36 names whose interpunct `str()` drops are read given-first, one of them with a comma. - `initials()` gives nothing for a word in a script with no initials (Han, Hangul, kana, `_policy._NO_INITIALS`), where today it gives the first character with a period (`'王.'`). @@ -289,7 +289,7 @@ Each sub-phase is one PR, measured on its own base as phase 2's and phase 3's ar **Interleaved with phase 3** (its sub-phases 3a to 3h are the draft's on `feat/vocabulary-phase3-design`): 4a can start beside 3a, its `languages` waiting for 3a's index; 4b to 4d need nothing of phase 3 and unblock its packs, `cs`'s generations (3f) and `pt-BR`'s agnomens (3h) on 4b, `es` and `pt` (3g) on 4d and 4g; 4f lands with or after 3b, since both decide what a shipped `Locale` holds; an `is` pack, which no phase 3 sub-phase lists, needs 3b, 4c and 4f. 4e and 4h stand apart from both. So the order is 3a and 4a, then 4b to 4d with 3b and 3c, then 4f, then the packs with 4g, with 4e at any point. -**Left open**, with no sub-phase yet: phase 1's four reserved names said to be phase 4's (`separators` with phase 4's glued words, and `bare_only`, `leading_only` and `trailing_only`, which this RFC's phase 2 section found not needed), whose build messages still name this phase. Value questions inside the shapes above: the first surname's view and its name; whether `order`, and `Token.reading` beneath the new views, stay out of comparison, which keeps a loaded 2.x pickle equal to a fresh parse of its text but lets two equal names render differently (a 2.x pickle of `김민준` renders `'민준 김'` and a fresh parse `'김민준'`, and a set keeps one), where today every rendering is derived from what `==` compares; which particles `family_base` removes, rules/R2 saying the leading ones and the code removing every working one, proposed in decisions/R2's 2026-10-28 #754 entry as R2 amended to the code, since each such particle stands in the particle run that opens a name unit, or ends the family, and the unit bases joined are the code's base; whether `credentials` should leave the honorifics out, CLDR having no third bucket; which ending rules beside Icelandic a pack may name (Hungarian `-né` is a married name, not a patronymic); and the names of given names only (the last table row), whose reading needs a rule that tells a mononym from a given name and a family name, not an output field. +**Left open**, with no sub-phase yet: phase 1's four reserved names said to be phase 4's (`separators` with phase 4's glued words, and `bare_only`, `leading_only` and `trailing_only`, which this RFC's phase 2 section found not needed), whose build messages still name this phase. Value questions inside the shapes above: the first surname's view and its name; whether a language pack supplies a convention's entry element or sort key, built from the per-surname units this phase reads, `family_base` being the family without every particle under every convention (decisions/R2's 2026-10-28 #754 entry): a `pt` or `pt-BR` pack would give the last surname's base first, keeping `Filho`, `Neto` and `Júnior` with it and a compound surname whole, and an `es` pack the first surname without its particle, then the second with its own (RAE's rules (a) and (e)); whether `credentials` should leave the honorifics out, CLDR having no third bucket; which ending rules beside Icelandic a pack may name (Hungarian `-né` is a married name, not a patronymic); and the names of given names only (the last table row), whose reading needs a rule that tells a mononym from a given name and a family name, not an output field. ## The first packs diff --git a/nameparser/_types.py b/nameparser/_types.py index 24df77e9c..4927c2dc8 100644 --- a/nameparser/_types.py +++ b/nameparser/_types.py @@ -1141,8 +1141,8 @@ def maiden(self) -> str: @property # rules/R2: "the family name splits into further views: the - # base (the family without its leading particles) and the - # particles themselves" + # base (the family without its particles) and the particles + # themselves" def family_particles(self) -> str: return self._text_for(Role.FAMILY, tag="particle", unless_tag=UNJOINED_TAG) diff --git a/tests/v2/rules_budget.json b/tests/v2/rules_budget.json index 6c69494e7..a9298b486 100644 --- a/tests/v2/rules_budget.json +++ b/tests/v2/rules_budget.json @@ -50,7 +50,7 @@ "P8": {"words": 515, "then_words": 450, "because_sentences": 2, "except": 0, "precedence": 0, "accepted": 2, "accepted_words": 309}, "P9": {"words": 405, "then_words": 265, "because_sentences": 1, "except": 2, "precedence": 0, "accepted": 0, "accepted_words": 0}, "R1": {"words": 176, "then_words": 145, "because_sentences": 1, "except": 0, "precedence": 0, "accepted": 1, "accepted_words": 146}, - "R2": {"words": 143, "then_words": 19, "because_sentences": 1, "except": 1, "precedence": 0, "accepted": 2, "accepted_words": 85}, + "R2": {"words": 151, "then_words": 18, "because_sentences": 1, "except": 1, "precedence": 0, "accepted": 2, "accepted_words": 85}, "R3": {"words": 483, "then_words": 407, "because_sentences": 2, "except": 1, "precedence": 0, "accepted": 3, "accepted_words": 409}, "R4": {"words": 991, "then_words": 729, "because_sentences": 1, "except": 4, "precedence": 0, "accepted": 4, "accepted_words": 600}, "R5": {"words": 250, "then_words": 154, "because_sentences": 2, "except": 0, "precedence": 0, "accepted": 0, "accepted_words": 0}, From 6e37ad9fd14f499b656be46b52d2c2f5af8a1f9e Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Wed, 28 Oct 2026 12:24:28 -0700 Subject: [PATCH 13/39] Keep a comma before a suffix in phase 4's render default (#754) Two words are joined by their written separator when they stand one after the other in the same order in the text and the rendering. A family comma never qualifies, since the rendering reverses its pair; a comma before a suffix does, so 'John Smith, Jr.' keeps it (456 of the 1,008 committed entries with a suffix after a family write one). Co-Authored-By: Claude Opus 5.5 --- docs/design/decisions/vocabulary-model.md | 2 +- docs/design/vocabulary-model-rfc.md | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/design/decisions/vocabulary-model.md b/docs/design/decisions/vocabulary-model.md index bfdc0bac9..036abb59a 100644 --- a/docs/design/decisions/vocabulary-model.md +++ b/docs/design/decisions/vocabulary-model.md @@ -249,7 +249,7 @@ Open: #913 (the title words phase 2j left, read as titles though borne as names) 3. **A `patronymic` view, filled by a word or a rule that says patronymic, with `family` and `middle` unchanged.** The default fills it from the vocabulary's `patronymic` particles (`bin`, `binti`, `ibn`, `bint`, `bar`, `bat` and the native-script forms) with the name they introduce (`Mahathir bin Mohamad` → family `bin Mohamad`, as today, and patronymic `bin Mohamad`); RU's and TR_AZ's rules fill it where a caller turns them on (`Ivan Ivanovich Ivanov` under RU → middle `Ivanovich` and patronymic `Ivanovich`); an Icelandic ending rule fills it under an `is` pack only, since the default cannot tell `-dóttir` and `-son` from Swedish surnames (ORCID's Swedish records end their family box in `-son` or `-dóttir` in 2,184 of 20,000). A matronymic, written the same way, fills it too. SUPERSEDES: the RFC's earlier statement of decision 8's value, that `family` can be empty for a name with a patronymic. Measured: filers put an Icelandic patronymic in the family box (ORCID 1,647 of 2,583 Icelandic records, PatentsView 535 of 722, PSC 101 of 125), and CLDR's `is` file's 3 Icelandic samples put it in `surname` or `surname-core` in all 9 of their renderings (its one foreign sample puts a `-dóttir` word in `surname2`), while Wikidata's `patronymic-is` and `patronymic-my` strata record 108 names with a patronymic and no family name, of which the parse puts the patronymic in `family` 107 times. An East Slavic patronymic stands in the given part: CLDR's 2 `ru` samples put it in `given2` in all 6 renderings; PatentsView's Russian given boxes end in one 8,423 times of 20,000, and the parse reads such a word as the middle 10,902 times of 10,962; 90 of the 112 Russian nationals in PSC who filled the middle-name box wrote one there. A `family` that keeps it and a `patronymic` that names it agree with both kinds of source. Weighed against: emptying `family` where the patronymic is recognized, which moves the field the fielded sources fill and changes `family` for every Icelandic and nasab name, a behavior change of a frozen property; and a neutral parent-names name for the view, which buys nothing the view can detect, since a matronymic and a patronymic are written alike. 4. **`generation` and `credentials` are views that partition `suffix` (#326)**, not roles, `generation` the tokens read as a generation or an agnomen, `credentials` the rest, the post-nominals that are neither (`Esq.`, the honorifics after a name) falling in `credentials`, CLDR's catch-all, with `Token.reading.kind` telling them apart for a caller who needs it. Measured: of the 1,054 committed corpus entries with a suffix, 33 hold a generation (or agnomen) and a listed credential together, 77 a generation beside any other suffix word, and 64 only listed post-nominals of other kinds (each suffix word classified by the RFC's Phase 4 recipe, joined tokens merged); of the 615 CLDR renderings with either field, the 75 whose suffix holds exactly CLDR's words split into CLDR's `generation` and `credentials` by the vocabulary's reading kind in all 75. Weighed against: two new `Role` members (item 1), and a third view for the honorifics, left open since CLDR has none. 5. **a `surname2` view, the family's second unit where rules/O6 read the family as two**, marked by a new stable tag where O6 takes its word, empty otherwise; `family` keeps both units as O6 makes it, and a family comma alone marks nothing (`Lloyd Webber, Andrew`). Measured: a two-word family box in ORCID's records from Spain 48.0%, Mexico 64.7%, Colombia 72.8% and Chile 50.5%, against 3.9% from the United States. Left open: a view for the first unit and its name, `surname` standing close to the `surnames` view that exists. Declined: redefining `surnames` as the two units, which would repeat `family`, the family holding both where O6 reads two, and lose what `surnames` does today, joining `middle` and `family` so that a double surname the default splits (`Gabriel García Márquez` → middle `García`) reads whole, the view rules/P2 names for that reading. - 6. **The render defaults follow the order the name was read in, a behavior change of the output rather than a break of the API.** `ParsedName` gains `order` (an addition, `ADDED_IN`, a 2.x pickle rendering as today, left out of comparison, which lets two equal names render differently, a cost accepted 2026-10-28: a 2.x pickle of `김민준` renders `'민준 김'` and a fresh parse `'김민준'`, and the two compare equal); the default `str()` and `render()` spec put the family first where `order` does, and join two words that stand side by side in the original and the rendering with what separated them there (nothing, a space, an interpunct, a nakaguro); `initials()` gives nothing for a word in a script with no initials. A name with a comma or written given-first keeps today's word order, the separator kept being any but a comma, which a space replaces (`Smith, John` → `'John Smith'`, while `威廉·莎士比亚` keeps its dot; 20 of the 36 names whose interpunct `str()` drops are read given-first). Measured over the committed corpus's distinct entries: 728 without a comma read family-first, `str()` renders all 728 differently from their text (`김민준` → `'민준 김'`), 150 of them with the family's last word touching the given name's first, and it drops the U+00B7 interpunct from 36 of the 41 holding one. This carries out decisions/3-0-reevaluations' render-default clean slate. No signature changes, but `str()`'s output changes for every name read family-first and every name whose written separator is neither a space nor a comma, and no 2.x release can warn about a `str()` result, so it is taken as an improvement for 3.0's release notes, not a break needing a warning. + 6. **The render defaults follow the order the name was read in, a behavior change of the output rather than a break of the API.** `ParsedName` gains `order` (an addition, `ADDED_IN`, a 2.x pickle rendering as today, left out of comparison, which lets two equal names render differently, a cost accepted 2026-10-28: a 2.x pickle of `김민준` renders `'민준 김'` and a fresh parse `'김민준'`, and the two compare equal); the default `str()` and `render()` spec put the family first where `order` does, and join two words that stand one directly after the other, in the same order, in the original and the rendering with what separated them there (nothing, a space, an interpunct, a nakaguro, a comma); `initials()` gives nothing for a word in a script with no initials. A name with a comma or written given-first keeps today's word order and its separators by the same rule (`威廉·莎士比亚` keeps its dot; 20 of the 36 names whose interpunct `str()` drops are read given-first). A family comma is never kept, the rendering reversing its pair (`Smith, John` → `'John Smith'`), and a comma before a suffix is, decided 2026-10-28 (`John Smith, Jr.` → `'John Smith, Jr.'`, today `'John Smith Jr.'`; 456 of the 1,008 committed corpus entries with a suffix after a family write one). Measured over the committed corpus's distinct entries: 728 without a comma read family-first, `str()` renders all 728 differently from their text (`김민준` → `'민준 김'`), 150 of them with the family's last word touching the given name's first, and it drops the U+00B7 interpunct from 36 of the 41 holding one. This carries out decisions/3-0-reevaluations' render-default clean slate. No signature changes, but `str()`'s output changes for every name read family-first, every name whose written separator is not a space, and every name with a comma before its suffix, and no 2.x release can warn about a `str()` result, so it is taken as an improvement for 3.0's release notes, not a break needing a warning. 7. **`middle` keeps its meaning (#704)**, every word of the given part after the given name, the given name one unit unless a rule joins words into it, as rules/P5 already does (`Abdul Aziz Tariq` → given `Abdul Aziz`); a compound given name is a `[rules]` rule a caller or a pack turns on, CLDR's `given2` is `middle`, and `Kaur` and an English name beside a Chinese one stay in `middle`, an accepted cost. Measured: in PSC, whose form has a middle-name box, 2,499 of 1,040,708 forename boxes hold two or more words, and of 6,101 Spanish nationals 25 do while 2,621 fill the middle box; ORCID's and PatentsView's given box holds middles too, so it cannot separate the two. The `Maria do Carmo` shape, a particle inside a given box of three or more words with a family box filled, reads the given box's words into the family in 1,812 of 1,923 ORCID records and 1,165 of 1,219 PatentsView records from Brazil, Portugal, Spain, Mexico, Argentina, Colombia, Chile and Italy, which is that rule's case. Weighed against: #704's option (b), one given unit by default, which changes `middle` for every English name of three words. 8. **A language file's `[rules]` compiles into its shipped `Locale`'s policy patch**, each rule kind a `Policy` field (an addition), so RU and TR_AZ are rebuilt from their files and `Locale` gains no field for rules. With it, **phase 3's open API dependency is answered**: its draft (branch `feat/vocabulary-phase3-design`, unmerged) carries a choice of languages on `Locale.languages`, assuming `Locale` and `Policy` keep their 2.x shape after the facade's removal, and the 2026-10-28 freeze keeps them so. The rest of that draft, strict as an argument of the whole choice included, stays phase 3's to decide; its entry, when it lands, says the dependency is answered. 9. **The work order**, one PR each, measured on its own base, the snapshot recording each new view where it is non-empty: 4a `Token.reading` and `ReadingKind.PATRONYMIC` (neutral; `languages` after 3a); 4b `generation` and `credentials` (neutral); 4c `patronymic` (neutral); 4d `surname2` (neutral); 4e the render defaults and `order`; 4f `[rules]` into the policy patch, with or after 3b; 4g the compound given name, after 4f; 4h the shape inventory per order (#469). Interleaved with phase 3 (its sub-phases the draft's): 4a beside 3a; 4b to 4d before the packs that need them, `cs` (3f) and `pt-BR` (3h) on 4b, `es` and `pt` (3g) on 4d and 4g; an `is` pack on 3b, 4c and 4f. diff --git a/docs/design/vocabulary-model-rfc.md b/docs/design/vocabulary-model-rfc.md index d39e96b51..f13452e4b 100644 --- a/docs/design/vocabulary-model-rfc.md +++ b/docs/design/vocabulary-model-rfc.md @@ -250,10 +250,10 @@ ORCID's and PatentsView's given box holds the middle names too, so a two-word gi **Render defaults** (decisions/3-0-reevaluations' clean slate). The default `str()` reads every name given-first and joins its words with spaces, so a name read family-first is turned round and a written separator is lost: `str(parse("김민준"))` is `'민준 김'`, `str(parse("王 小明"))` is `'小明 王'` and `str(parse("威廉·莎士比亚"))` is `'威廉 莎士比亚'`. Over the committed corpus's distinct entries, each under its declared order (2026-10-28, `c8357e74`), 728 without a comma read family-first (the family's first word written before the given name's), `str()` renders every one differently from the text, and in 150 of them the family's last word touches the given name's first, written unspaced; of 41 holding the U+00B7 interpunct, `str()` drops it from 36. So: - `ParsedName` gains `order`, the order the name was read under (its script's or the declared `name_order`, a comma aside), an addition with `ADDED_IN` metadata whose missing value, in a 2.x pickle, renders as today, and left out of comparison as `Token.reading` is, so a loaded 2.x pickle still equals a fresh parse of its text. The cost, accepted 2026-10-28: two equal names can render differently, where today every rendering is derived from what `==` compares (a 2.x pickle of `김민준` renders `'민준 김'` and a fresh parse `'김민준'`, and a set keeps one of them); the views read from `Token.reading` share it. -- `str()` and `render()`'s default spec put the family first where `order` does, and two words that stand side by side in the original and in the rendering are joined by what separated them there, nothing, a space, an interpunct or a nakaguro, so `str(parse("김민준"))` is `'김민준'` and `str(parse("威廉·莎士比亚"))` keeps the dot. A name with a comma or written given-first keeps today's word order, and the separator kept is any but a comma, which a space replaces (`Smith, John` → `'John Smith'`), so the given-first transcription `威廉·莎士比亚` keeps its dot too: 20 of the 36 names whose interpunct `str()` drops are read given-first, one of them with a comma. +- `str()` and `render()`'s default spec put the family first where `order` does, and two words that stand one directly after the other, in the same order, in the original and in the rendering are joined by what separated them there, nothing, a space, an interpunct, a nakaguro or a comma, so `str(parse("김민준"))` is `'김민준'` and `str(parse("威廉·莎士比亚"))` keeps the dot. A name with a comma or written given-first keeps today's word order, and its separators are kept by the same rule, so the given-first transcription `威廉·莎士比亚` keeps its dot too (20 of the 36 names whose interpunct `str()` drops are read given-first, one of them with a comma). A family comma is never kept, since the rendering reverses the pair it stands between (`Smith, John` → `'John Smith'`), while a comma before a suffix is (`John Smith, Jr.` → `'John Smith, Jr.'`, where `str()` gives `'John Smith Jr.'` today): 456 of the 1,008 committed corpus entries with a suffix after a family write a comma between the family's last word and the suffix's first (2026-10-28, `c8357e74`, each entry under its declared order). Pairs the rendering separates, as `John` and `Jr.` in `Smith, John, Jr.`, take the spec's own joiner, as today. - `initials()` gives nothing for a word in a script with no initials (Han, Hangul, kana, `_policy._NO_INITIALS`), where today it gives the first character with a period (`'王.'`). -These change no signature but change `str()`'s output for every name read family-first and every name whose written separator is neither a space nor a comma, which no 2.x release can warn about; Derek took them as an improvement for 3.0's release notes rather than a break (item 6 of the entry). +These change no signature but change `str()`'s output for every name read family-first, every name whose written separator is not a space, and every name written with a comma before its suffix, which no 2.x release can warn about; Derek took them as an improvement for 3.0's release notes rather than a break (item 6 of the entry). **What changes in the API**, by the 3.0 freeze: From de967a976dd2443d25fad441cf4efd44088b30d9 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Wed, 28 Oct 2026 12:30:24 -0700 Subject: [PATCH 14/39] Fix the third docs review's findings on phase 4 and R2 (#754) A kept separator is whitespace, nothing, an interpunct, a nakaguro or one comma; delimiters and consumed marker words take the spec's joiner. Only a family comma keeps a name given-first. The suffix-comma count is re-taken (214 of 1,003 keep the comma) and the differing render of 'Smith, John, Jr.' is recorded as accepted. R2's entry says where the base departs from the Spanish key (rule b's La and San), speaks of R2's old Then in the past, and the RFC keeps R2's all-particle exception. Co-Authored-By: Claude Opus 5.5 --- .../R2-all-particles-family-base-divergence.md | 4 ++-- docs/design/decisions/vocabulary-model.md | 2 +- docs/design/vocabulary-model-rfc.md | 10 +++++----- 3 files changed, 8 insertions(+), 8 deletions(-) diff --git a/docs/design/decisions/R2-all-particles-family-base-divergence.md b/docs/design/decisions/R2-all-particles-family-base-divergence.md index fb378fc32..15f73135c 100644 --- a/docs/design/decisions/R2-all-particles-family-base-divergence.md +++ b/docs/design/decisions/R2-all-particles-family-base-divergence.md @@ -35,11 +35,11 @@ Open: #754 (whether a language pack supplies a convention's sort key or entry el - 2026-09-20 #461 — AMENDS the 2026-08-29 bullet above, which said of four readings that "none of those moved or may move". Two of the four moved and two did not, and the half that survived is the half the criterion was always about. WHAT MOVED: `parse("Juan y Garcia")` initials "J. y. G." where that bullet recorded "J. G.", and R3's own `"Juan de y"` example line reads "J. y." where it read "J.". WHAT DID NOT: `parse("Juan Velasquez y Garcia")` is still "J. V. G." over base "Velasquez y Garcia", and `parse("Jon Dough and")` is still "J. D." over base "Dough and" (all four measured 2026-09-20 on this branch). The criterion that separates them is the one the bullet above named, and it is still not a word count: the question is asked of the WHOLE PART, and a part holding another name word for the connective to join — the part's working particles set aside — is a part where the connective is doing a connective's work and contributes nothing. "Velasquez y Garcia" and "Dough and" each hold such a word, so the skip stands there; the family "y", the family "de y" and the middle "y" of "Juan y Garcia" hold none, so the connective is its part's own name word and initials with it. So that bullet was RIGHT ABOUT THE MECHANISM and wrong only about the clause it preserved. Its finding — that a mark honored for some of a part's words and not for one of them is honoring nothing — is exactly what rules/R3 now states, for all three groups at once, and it is why the fact is computed ONCE over the whole part rather than re-derived per word by each view. What changed is the clause it left standing: R3's "A CONJUNCTION never initials, so a base that is one contributes nothing even then" is conditional now, and "none of those may move" was the reading of the day rather than a decision. decisions/R3's 2026-09-20 bullet carries the rule as taken. The pin that bullet describes did its job: `test_initials_readmits_a_conjunction_in_a_particle_part` was written to fail when #461 was re-decided, and it failed. -- **2026-10-28 #754 — `family_base` REMOVES EVERY WORKING PARTICLE, SO RULES/R2 IS AMENDED TO THE CODE RATHER THAN THE CODE TO R2, AND THE BASE IS A VIEW THE SAME UNDER EVERY CONVENTION, NOT ANY CULTURE'S SORT KEY.** Rules/R2's Then says the base is "the family without its leading particles", but the view drops every family word tagged `particle` and not marked unjoined, wherever it stands: `da Silva do Amaral de Souza, Miguel` reads family 'da Silva do Amaral de Souza', base 'Silva Amaral Souza', particles 'da do de', where the 2.x facade's `last_prefixes` gave 'da'; the P1 fold after a family comma makes a particle interior with none leading (`Smith, de Mesnil Jean` → base 'Smith Mesnil Jean', particles 'de'); and a leading run alone reads the same either way (`Juan de la Vega y Ortega` → base 'Vega y Ortega'). `test_particle_views_after_fold`, on #937's branch and not yet on master, pins the code's reading of `Miguel da Silva do Amaral de Souza` under `middle_as_family`. Three answers were weighed: (1) amend R2 to the code; (2) remove only the leading run (base 'Silva do Amaral de Souza', particles 'da'); (3) derive the base from per-surname views, each surname keeping its own leading particle. +- **2026-10-28 #754 — `family_base` REMOVES EVERY WORKING PARTICLE, SO RULES/R2 IS AMENDED TO THE CODE RATHER THAN THE CODE TO R2, AND THE BASE IS A VIEW THE SAME UNDER EVERY CONVENTION, NOT ANY CULTURE'S SORT KEY.** Rules/R2's Then said the base is "the family without its leading particles", but the view drops every family word tagged `particle` and not marked unjoined, wherever it stands: `da Silva do Amaral de Souza, Miguel` reads family 'da Silva do Amaral de Souza', base 'Silva Amaral Souza', particles 'da do de', where the 2.x facade's `last_prefixes` gave 'da'; the P1 fold after a family comma makes a particle interior with none leading (`Smith, de Mesnil Jean` → base 'Smith Mesnil Jean', particles 'de'); and a leading run alone reads the same either way (`Juan de la Vega y Ortega` → base 'Vega y Ortega'). `test_particle_views_after_fold`, on #937's branch and not yet on master, pins the code's reading of `Miguel da Silva do Amaral de Souza` under `middle_as_family`. Three answers were weighed: (1) amend R2 to the code; (2) remove only the leading run (base 'Silva do Amaral de Souza', particles 'da'); (3) derive the base from per-surname views, each surname keeping its own leading particle. (1) and (3) are one answer. A particle joins forward to the word it modifies, so every working particle inside a family stands in the particle run that opens a name unit (`la` in `de la Vega` is the run's second word), and the unit bases joined are the code's base: under `two_family_names`, rules/O6 reads `Miguel da Silva do Amaral de Souza` as middle 'da Silva' and family 'do Amaral de Souza', whose units 'do Amaral' and 'de Souza' have bases 'Amaral' and 'Souza', and the family's base is 'Amaral Souza'. So `surname2` (the RFC's Phase 4, sub-phase 4d) reaches the question only where O6 runs and only for its two units; it needs nothing from `family_base`, and the first unit's view, left open there, would read the same way. One shape escapes the unit reading: a working particle ending a family of other words opens no unit and joins nothing (a family of two or more words whose last is a working particle: 1 committed entry and 1 `tse-donors` filed name in the default policy, 39 and 505 with `middle_as_family`, the bare word-particle shape 9 of the 39), and the code drops it from the base as it does any other; whether a lone trailing particle is unjoined is R2's all-particles question asked of one word, left as the code reads it. R2's Then now reads "the base (the family without its particles) and the particles themselves", its Because saying the base is the same view under every convention rather than "sorting wants" it, with `da Silva do Amaral de Souza, Miguel` and `Smith, de Mesnil Jean` among its examples; the code is unchanged, and #937's test with it, and the R2 citation on `ParsedName.family_particles` in nameparser/_types.py is re-quoted. Still to move when #937 lands: docs/migrate.rst's `family_base` line ("The surname with leading particles split off"), which nothing checks, beside its `last_prefixes` row, the mapping that reads 'da' where the core reads 'da do de'; the facade docs are #937's to change. Measured 2026-10-28 on master `c8357e74`, counts and token shapes only, over the committed corpus's 6,533 distinct entries (`snapshot.corpus_entries()`, each parsed under its declared order) and `tse-donors`' 854,531 records (`local_corpora.read_records`, the filed name and the registry name each parsed in the default order). A family is read in render order (FOLDED_TAG tokens first), a working particle is a token tagged `particle` and not UNJOINED_TAG, and a family has an interior particle where a working particle stands after its first word that is neither a particle nor a conjunction; that family is where (1) and (2) differ. The three kinds of interior family are tested in this order, the first that fits counting: after a family comma, every interior particle standing after the first comma; with a leading run as well; with no leading run. In the default policy: committed, 24 against 684 with a leading run only (12 with no leading run, 8 whose interior particle stands after a family comma, 4 with a leading run as well); `tse-donors`, 12 filed names against 347,992 and 1 registry name against 392,134, since a Brazilian name's particles before its last word stand in `middle`. With `middle_as_family`: committed, 367 against 398 (262, 92 and 13); `tse-donors`' filed names, 268,591 against 79,926 (261,024, 5 and 7,562), the commonest family shape word-particle-word, 150,998 times. Recompute with those definitions; a change to the fold or to O6 moves them. Weighed against (2): with `middle_as_family`, the population the per-surname reading serves, it gives the 261,024 `tse-donors` names with no leading run a base equal to the whole family and an empty particle view, so `family_particles` stops naming the particles it is for; and it changes a frozen property's output for 24 committed entries in the default policy, where (1) changes no output. The facade whose `last_prefixes` it matches is removed in 3.0. Its one gain is a base that is a written name, which no convention's key needs either (below). The cost of (1), accepted: the base is not a name anyone writes ('Silva Amaral Souza'), nor any convention's sort key. - Why the base is not a sort key, researched 2026-10-28. Portuguese and Brazilian practice files a name under its LAST surname, that surname's particle removed and set after the given names, the agnomens `Filho`, `Neto` and `Júnior` staying with the surname before them and an established compound surname kept whole: `SOUZA, Miguel da Silva do Amaral de`, `NOGUEIRA NETO, Wanderlino` (ABNT NBR 6023:2018, https://engenhariaeletrica.ufes.br/sites/engenhariaeletrica.ufes.br/files/field/anexo/abnt6023.2018.pdf; the ABNT Zotero style's comments, https://www.zotero.org/styles/universidade-federal-do-espirito-santo-abnt?source=1; USP's references manual, https://www.ip.usp.br/site/wp-content/uploads/2025/04/Manual-Referencias-ABNT-SabadiniSampaioNascimento.pdf; Cambridge University Library, https://www.lib.cam.ac.uk/node/3142; the UN cataloguing manual, https://www.un.org/Depts/dhl/unbisref_manual/nameguide/personal.htm; IFLA, Names of Persons: Portugal, 2009, https://ifla.org/files/assets/cataloguing/pubs/ifla_names_of_persons_portugal_2009.pdf). Spanish practice files under the FIRST surname, the second its secondary key (RAE, Ortografía de la lengua española, 2010, ch. VII, "Alfabetización de antropónimos", https://www.rae.es/ortografía/alfabetización-de-antropónimos, read 2026-10-28 in a browser, the page refusing a plain fetch): by its rule (a) a preposition, with or without an article, heading the FIRST surname is left out of the order and written after the given name (`Torre Ibarra, Ramón de la`); by rule (e) one heading the SECOND surname counts as part of the second key (`Gómez Casas`, `Gómez de la Iglesia`, `Gómez de Lara`, `Gómez del Castillo`, `Gómez Echagüe`, in that order); by rule (b) a leading article or `San`/`Santa` stays and is sorted on (`La Calle Ruiz, Carmen`); by rule (d) a `y` between the surnames counts (`Aguirre y Gómez-Pacheco`). The same page states the Portuguese convention from the other side, the paternal second surname being the axis. Neither convention's key is this base: Portuguese practice sorts `…da Silva de Souza do Amaral` before `…da Silva do Amaral de Souza` (Amaral before Souza) while both bases begin `Silva`, and Spanish practice keeps a second surname's particle (`Gómez de la Iglesia`) where the base drops it ('Gómez Iglesia'). The base dropping every working particle is nearest the Spanish first key alone. So the base is the convention-neutral view, and a convention's key is built from the surnames by whoever knows the convention, the RFC's Phase 4 Left open. In the default policy, `Miguel da Silva do Amaral de Souza` already reads family 'de Souza', base 'Souza', which is the Portuguese entry element by coincidence: only `middle_as_family` gives the long family. + Why the base is not a sort key, researched 2026-10-28. Portuguese and Brazilian practice files a name under its LAST surname, that surname's particle removed and set after the given names, the agnomens `Filho`, `Neto` and `Júnior` staying with the surname before them and an established compound surname kept whole: `SOUZA, Miguel da Silva do Amaral de`, `NOGUEIRA NETO, Wanderlino` (ABNT NBR 6023:2018, https://engenhariaeletrica.ufes.br/sites/engenhariaeletrica.ufes.br/files/field/anexo/abnt6023.2018.pdf; the ABNT Zotero style's comments, https://www.zotero.org/styles/universidade-federal-do-espirito-santo-abnt?source=1; USP's references manual, https://www.ip.usp.br/site/wp-content/uploads/2025/04/Manual-Referencias-ABNT-SabadiniSampaioNascimento.pdf; Cambridge University Library, https://www.lib.cam.ac.uk/node/3142; the UN cataloguing manual, https://www.un.org/Depts/dhl/unbisref_manual/nameguide/personal.htm; IFLA, Names of Persons: Portugal, 2009, https://ifla.org/files/assets/cataloguing/pubs/ifla_names_of_persons_portugal_2009.pdf). Spanish practice files under the FIRST surname, the second its secondary key (RAE, Ortografía de la lengua española, 2010, ch. VII, "Alfabetización de antropónimos", https://www.rae.es/ortografía/alfabetización-de-antropónimos, read 2026-10-28 in a browser, the page refusing a plain fetch): by its rule (a) a preposition, with or without an article, heading the FIRST surname is left out of the order and written after the given name (`Torre Ibarra, Ramón de la`); by rule (e) one heading the SECOND surname counts as part of the second key (`Gómez Casas`, `Gómez de la Iglesia`, `Gómez de Lara`, `Gómez del Castillo`, `Gómez Echagüe`, in that order); by rule (b) a leading article or `San`/`Santa` stays and is sorted on (`La Calle Ruiz, Carmen`); by rule (d) a `y` between the surnames counts (`Aguirre y Gómez-Pacheco`). The same page states the Portuguese convention from the other side, the paternal second surname being the axis. Neither convention's key is this base: Portuguese practice sorts `…da Silva de Souza do Amaral` before `…da Silva do Amaral de Souza` (Amaral before Souza) while both bases begin `Silva`, and Spanish practice keeps a second surname's particle (`Gómez de la Iglesia`) where the base drops it ('Gómez Iglesia'). The base dropping every working particle is nearest the Spanish first key alone for a preposition, rule (a) (`Torre Ibarra, Ramón de la` → base 'Torre Ibarra'), and departs from it on what rule (b) keeps: a leading article or `San` is a particle to the parse, so `La Calle Ruiz, Carmen` reads base 'Calle Ruiz' and `San Martín López, Juan` base 'Martín López'. So the base is the convention-neutral view, and a convention's key is built from the surnames by whoever knows the convention, the RFC's Phase 4 Left open. In the default policy, `Miguel da Silva do Amaral de Souza` already reads family 'de Souza', base 'Souza', which is the Portuguese entry element by coincidence: only `middle_as_family` gives the long family. Declined: diff --git a/docs/design/decisions/vocabulary-model.md b/docs/design/decisions/vocabulary-model.md index 036abb59a..b1891145a 100644 --- a/docs/design/decisions/vocabulary-model.md +++ b/docs/design/decisions/vocabulary-model.md @@ -249,7 +249,7 @@ Open: #913 (the title words phase 2j left, read as titles though borne as names) 3. **A `patronymic` view, filled by a word or a rule that says patronymic, with `family` and `middle` unchanged.** The default fills it from the vocabulary's `patronymic` particles (`bin`, `binti`, `ibn`, `bint`, `bar`, `bat` and the native-script forms) with the name they introduce (`Mahathir bin Mohamad` → family `bin Mohamad`, as today, and patronymic `bin Mohamad`); RU's and TR_AZ's rules fill it where a caller turns them on (`Ivan Ivanovich Ivanov` under RU → middle `Ivanovich` and patronymic `Ivanovich`); an Icelandic ending rule fills it under an `is` pack only, since the default cannot tell `-dóttir` and `-son` from Swedish surnames (ORCID's Swedish records end their family box in `-son` or `-dóttir` in 2,184 of 20,000). A matronymic, written the same way, fills it too. SUPERSEDES: the RFC's earlier statement of decision 8's value, that `family` can be empty for a name with a patronymic. Measured: filers put an Icelandic patronymic in the family box (ORCID 1,647 of 2,583 Icelandic records, PatentsView 535 of 722, PSC 101 of 125), and CLDR's `is` file's 3 Icelandic samples put it in `surname` or `surname-core` in all 9 of their renderings (its one foreign sample puts a `-dóttir` word in `surname2`), while Wikidata's `patronymic-is` and `patronymic-my` strata record 108 names with a patronymic and no family name, of which the parse puts the patronymic in `family` 107 times. An East Slavic patronymic stands in the given part: CLDR's 2 `ru` samples put it in `given2` in all 6 renderings; PatentsView's Russian given boxes end in one 8,423 times of 20,000, and the parse reads such a word as the middle 10,902 times of 10,962; 90 of the 112 Russian nationals in PSC who filled the middle-name box wrote one there. A `family` that keeps it and a `patronymic` that names it agree with both kinds of source. Weighed against: emptying `family` where the patronymic is recognized, which moves the field the fielded sources fill and changes `family` for every Icelandic and nasab name, a behavior change of a frozen property; and a neutral parent-names name for the view, which buys nothing the view can detect, since a matronymic and a patronymic are written alike. 4. **`generation` and `credentials` are views that partition `suffix` (#326)**, not roles, `generation` the tokens read as a generation or an agnomen, `credentials` the rest, the post-nominals that are neither (`Esq.`, the honorifics after a name) falling in `credentials`, CLDR's catch-all, with `Token.reading.kind` telling them apart for a caller who needs it. Measured: of the 1,054 committed corpus entries with a suffix, 33 hold a generation (or agnomen) and a listed credential together, 77 a generation beside any other suffix word, and 64 only listed post-nominals of other kinds (each suffix word classified by the RFC's Phase 4 recipe, joined tokens merged); of the 615 CLDR renderings with either field, the 75 whose suffix holds exactly CLDR's words split into CLDR's `generation` and `credentials` by the vocabulary's reading kind in all 75. Weighed against: two new `Role` members (item 1), and a third view for the honorifics, left open since CLDR has none. 5. **a `surname2` view, the family's second unit where rules/O6 read the family as two**, marked by a new stable tag where O6 takes its word, empty otherwise; `family` keeps both units as O6 makes it, and a family comma alone marks nothing (`Lloyd Webber, Andrew`). Measured: a two-word family box in ORCID's records from Spain 48.0%, Mexico 64.7%, Colombia 72.8% and Chile 50.5%, against 3.9% from the United States. Left open: a view for the first unit and its name, `surname` standing close to the `surnames` view that exists. Declined: redefining `surnames` as the two units, which would repeat `family`, the family holding both where O6 reads two, and lose what `surnames` does today, joining `middle` and `family` so that a double surname the default splits (`Gabriel García Márquez` → middle `García`) reads whole, the view rules/P2 names for that reading. - 6. **The render defaults follow the order the name was read in, a behavior change of the output rather than a break of the API.** `ParsedName` gains `order` (an addition, `ADDED_IN`, a 2.x pickle rendering as today, left out of comparison, which lets two equal names render differently, a cost accepted 2026-10-28: a 2.x pickle of `김민준` renders `'민준 김'` and a fresh parse `'김민준'`, and the two compare equal); the default `str()` and `render()` spec put the family first where `order` does, and join two words that stand one directly after the other, in the same order, in the original and the rendering with what separated them there (nothing, a space, an interpunct, a nakaguro, a comma); `initials()` gives nothing for a word in a script with no initials. A name with a comma or written given-first keeps today's word order and its separators by the same rule (`威廉·莎士比亚` keeps its dot; 20 of the 36 names whose interpunct `str()` drops are read given-first). A family comma is never kept, the rendering reversing its pair (`Smith, John` → `'John Smith'`), and a comma before a suffix is, decided 2026-10-28 (`John Smith, Jr.` → `'John Smith, Jr.'`, today `'John Smith Jr.'`; 456 of the 1,008 committed corpus entries with a suffix after a family write one). Measured over the committed corpus's distinct entries: 728 without a comma read family-first, `str()` renders all 728 differently from their text (`김민준` → `'민준 김'`), 150 of them with the family's last word touching the given name's first, and it drops the U+00B7 interpunct from 36 of the 41 holding one. This carries out decisions/3-0-reevaluations' render-default clean slate. No signature changes, but `str()`'s output changes for every name read family-first, every name whose written separator is not a space, and every name with a comma before its suffix, and no 2.x release can warn about a `str()` result, so it is taken as an improvement for 3.0's release notes, not a break needing a warning. + 6. **The render defaults follow the order the name was read in, a behavior change of the output rather than a break of the API.** `ParsedName` gains `order` (an addition, `ADDED_IN`, a 2.x pickle rendering as today, left out of comparison, which lets two equal names render differently, a cost accepted 2026-10-28: a 2.x pickle of `김민준` renders `'민준 김'` and a fresh parse `'김민준'`, and the two compare equal); the default `str()` and `render()` spec put the family first where `order` does, and join two words that stand one directly after the other, in the same order, in the original and the rendering with what separated them there when that is nothing, whitespace, an interpunct, a nakaguro or one comma with whitespace, the spec's own joiner otherwise (a delimiter, a consumed marker word); `initials()` gives nothing for a word in a script with no initials. A name with a family comma or written given-first keeps today's word order and its separators by the same rule, a suffix comma alone leaving a family-first name family-first (`威廉·莎士比亚` keeps its dot; 20 of the 36 names whose interpunct `str()` drops are read given-first). A family comma is never kept, the rendering reversing its pair (`Smith, John` → `'John Smith'`), and one directly between the family and its suffix is, decided 2026-10-28 (`John Smith, Jr.` → `'John Smith, Jr.'`, today `'John Smith Jr.'`; 214 of the 1,003 committed corpus entries whose suffix follows the family write one there). Accepted with it: `Smith, John, Jr.`, whose suffix the rendering moves away from the given name before it, keeps `'John Smith Jr.'`, so two writings of one name render differently. Measured over the committed corpus's distinct entries: 728 without a comma read family-first, `str()` renders all 728 differently from their text (`김민준` → `'민준 김'`), 150 of them with the family's last word touching the given name's first, and it drops the U+00B7 interpunct from 36 of the 41 holding one. This carries out decisions/3-0-reevaluations' render-default clean slate. No signature changes, but `str()`'s output changes for every name read family-first, every name whose written separator is not a space, and every name with one comma directly between its family and its suffix, and no 2.x release can warn about a `str()` result, so it is taken as an improvement for 3.0's release notes, not a break needing a warning. 7. **`middle` keeps its meaning (#704)**, every word of the given part after the given name, the given name one unit unless a rule joins words into it, as rules/P5 already does (`Abdul Aziz Tariq` → given `Abdul Aziz`); a compound given name is a `[rules]` rule a caller or a pack turns on, CLDR's `given2` is `middle`, and `Kaur` and an English name beside a Chinese one stay in `middle`, an accepted cost. Measured: in PSC, whose form has a middle-name box, 2,499 of 1,040,708 forename boxes hold two or more words, and of 6,101 Spanish nationals 25 do while 2,621 fill the middle box; ORCID's and PatentsView's given box holds middles too, so it cannot separate the two. The `Maria do Carmo` shape, a particle inside a given box of three or more words with a family box filled, reads the given box's words into the family in 1,812 of 1,923 ORCID records and 1,165 of 1,219 PatentsView records from Brazil, Portugal, Spain, Mexico, Argentina, Colombia, Chile and Italy, which is that rule's case. Weighed against: #704's option (b), one given unit by default, which changes `middle` for every English name of three words. 8. **A language file's `[rules]` compiles into its shipped `Locale`'s policy patch**, each rule kind a `Policy` field (an addition), so RU and TR_AZ are rebuilt from their files and `Locale` gains no field for rules. With it, **phase 3's open API dependency is answered**: its draft (branch `feat/vocabulary-phase3-design`, unmerged) carries a choice of languages on `Locale.languages`, assuming `Locale` and `Policy` keep their 2.x shape after the facade's removal, and the 2026-10-28 freeze keeps them so. The rest of that draft, strict as an argument of the whole choice included, stays phase 3's to decide; its entry, when it lands, says the dependency is answered. 9. **The work order**, one PR each, measured on its own base, the snapshot recording each new view where it is non-empty: 4a `Token.reading` and `ReadingKind.PATRONYMIC` (neutral; `languages` after 3a); 4b `generation` and `credentials` (neutral); 4c `patronymic` (neutral); 4d `surname2` (neutral); 4e the render defaults and `order`; 4f `[rules]` into the policy patch, with or after 3b; 4g the compound given name, after 4f; 4h the shape inventory per order (#469). Interleaved with phase 3 (its sub-phases the draft's): 4a beside 3a; 4b to 4d before the packs that need them, `cs` (3f) and `pt-BR` (3h) on 4b, `es` and `pt` (3g) on 4d and 4g; an `is` pack on 3b, 4c and 4f. diff --git a/docs/design/vocabulary-model-rfc.md b/docs/design/vocabulary-model-rfc.md index f13452e4b..a1014cb80 100644 --- a/docs/design/vocabulary-model-rfc.md +++ b/docs/design/vocabulary-model-rfc.md @@ -228,7 +228,7 @@ Phase 4 also decides what a parse returns, aligned with CLDR's person-name field What decides a view is the reading each token took, so `Token.reading` comes first; `surname2` alone is read from a tag O6 writes, a rule rather than a word deciding it: - **`Token.reading`**: a `Token` field holding the `Reading` the parse took, the same type `Ambiguity.alternative` holds the reading not taken in (decisions/A1), `None` for a token `replace()` builds. Its `kind` says what the word was read as, its `source` why, and a `languages` field, added to `Reading`, names the language files holding that reading, filled from phase 3's reading index (3a) and left empty for a reading no file holds (a caller's own vocabulary, a shape, position alone). The field takes `ADDED_IN` metadata, so a 2.x pickle loads with `None` (the pickle guard as widened in 2.7), and is left out of comparison (`compare=False`), so `==` between two `ParsedName`s means what it means in 2.x. `ReadingKind` gains `PATRONYMIC`, its docstring already allowing new kinds. -- **Generation and credentials** (#326): `generation`, the suffix tokens whose reading is a generation or an agnomen (`Jr.`, `III`, `Filho`), and `credentials`, the rest of the suffix (`PhD`, `Esq.`, the honorifics after a name), so the two partition `suffix` and `suffix` is unchanged. CLDR has only these two buckets; the post-nominals that are neither (`Esq.`, the CJK and Hebrew honorifics, `Rinpoche`) fall in `credentials`, CLDR's catch-all, and a caller who wants them apart reads `Token.reading.kind`, which says `post-nominal`. Measured 2026-10-28 on `c8357e74`, over the committed corpus's 6,533 distinct entries (`snapshot.corpus_entries()`) each parsed under its declared order: of the 1,054 with a suffix, 33 hold a generation (or agnomen) and a listed credential together, 77 a generation beside any other suffix word, and 64 only listed post-nominals of other kinds. Of the 615 CLDR renderings with a generation or credentials field, 75 parse with exactly CLDR's suffix words, and in all 75 a split by the vocabulary's reading kind gives CLDR's `generation` and `credentials` both; the other 540 differ in which words the suffix holds, a parse question the split cannot fix. Recompute: classify each suffix word (a token tagged `joined` merged into the one before it) by the kinds of its readings in nameparser/vocab/, matched lowercased with its periods and spaces removed, skipping `default = false` readings, a generation or agnomen reading winning over a credential and a credential over the other kinds, an unlisted roman numeral `i` to `x` a generation and any other unlisted word unlisted; over the corpus entries above, and over `cldr_fields.jsonl` for CLDR. +- **Generation and credentials** (#326): `generation`, the suffix tokens whose reading is a generation or an agnomen (`Jr.`, `III`, `Filho`), and `credentials`, the rest of the suffix (`PhD`, `Esq.`, the honorifics after a name), so the two partition `suffix` and `suffix` is unchanged. CLDR has only these two buckets; the post-nominals that are neither (`Esq.`, the CJK and Hebrew honorifics, `Rinpoche`) fall in `credentials`, CLDR's catch-all, and a caller who wants them apart reads `Token.reading.kind`, which says `post-nominal`. Measured 2026-10-28 on `c8357e74`, over the committed corpus's 6,533 distinct entries (`snapshot.corpus_entries()`) each parsed under its declared order: of the 1,054 with a suffix, 33 hold a generation (or agnomen) and a listed credential together, 77 a generation beside any other suffix word, and 64 only listed post-nominals of other kinds. Of the 615 CLDR renderings with a generation or credentials field, 75 parse with exactly CLDR's suffix words, and in all 75 a split by the vocabulary's reading kind gives CLDR's `generation` and `credentials` both; the other 540 differ in which words the suffix holds, a parse question the split cannot fix. Recompute: classify each suffix word (a token tagged `joined` merged into the one before it) by the kinds of its suffix readings in nameparser/vocab/, matched lowercased with its periods and spaces removed, skipping `default = false` readings, a generation or agnomen reading winning over a credential and a credential over the other kinds, an unlisted roman numeral `i` to `x` a generation and any other unlisted word unlisted; over the corpus entries above, and over `cldr_fields.jsonl` for CLDR. - **A patronymic** (decision 8): `patronymic`, the tokens whose reading is `PATRONYMIC`, in whichever role the parse put them, with `family` and `middle` unchanged. What fills it is a word or a rule that says patronymic, never a guess from position: the vocabulary's `patronymic` particles (`bin`, `binti`, `ibn`, `bint`, Hebrew `bar` and `bat` and the native-script forms) with the name after them, in the default, so `Mahathir bin Mohamad` reads family `bin Mohamad` as today and patronymic `bin Mohamad` beside it; the East Slavic and Turkic rules where RU or TR_AZ turns them on, so `Ivan Ivanovich Ivanov` under RU keeps middle `Ivanovich` and adds patronymic `Ivanovich`; and an Icelandic ending rule an `is` pack would turn on, the default unable to tell `Karlsdóttir` from the Swedish surname `Johansson`. The name `patronymic` is kept for a matronymic too, which is written the same way and cannot be told from one. **`family` is not emptied** by recognizing a patronymic, revising what this section said before: the people filling in the fielded corpora put the patronymic in the family box (below), as CLDR's samples put an Icelandic patronymic in `surname` or `surname-core` (CLDR's `is` file's 3 Icelandic samples, in all 9 of their renderings; its one foreign sample puts a `-dóttir` word in `surname2`) and a Russian one in `given2` (its 2 `ru` samples, in all 6 renderings), our `middle`, while Wikidata records an Icelandic name as a patronymic with no family name; a `family` that keeps it and a `patronymic` that names it satisfy both. A caller writing "Mr. {family}" reads `patronymic` to see that the family is one. - **A second surname**: `surname2`, the family's second name unit where a rule has read the family as two (rules/O6, which a caller or an `es` pack turns on), and empty otherwise, `family` keeping both units as O6 makes it today. O6 already knows which unit it took, so it marks where the second one starts with a new stable tag; a family comma alone marks nothing, since `Lloyd Webber, Andrew` is not two surnames, and a hyphenated family is one token and stays whole. Whether the first unit gets a view of its own, and its name (`surname` sits close to the `surnames` that exists), is a value question below. `surnames` keeps its meaning, `middle` and `family` joined, the reading of a double surname the default splits. - **`middle`** (#704): keeps its meaning, every word of the given part after the given name, and the given name is one name unit unless a rule joins words into it, as rules/P5 already does (`Abdul Aziz Tariq` → given `Abdul Aziz`). A compound given name (`Maria do Carmo`, `María José`) is the `[rules]` rule above, turned on by a caller or a pack, moving its words into `given` only under that rule. CLDR's `given2` is `middle`, so no view is added. A Sikh `Kaur` and an English name beside a Chinese one (`Lim Mei Ling Jessica`) stay in `middle`, an accepted cost. @@ -249,11 +249,11 @@ ORCID's and PatentsView's given box holds the middle names too, so a two-word gi **Render defaults** (decisions/3-0-reevaluations' clean slate). The default `str()` reads every name given-first and joins its words with spaces, so a name read family-first is turned round and a written separator is lost: `str(parse("김민준"))` is `'민준 김'`, `str(parse("王 小明"))` is `'小明 王'` and `str(parse("威廉·莎士比亚"))` is `'威廉 莎士比亚'`. Over the committed corpus's distinct entries, each under its declared order (2026-10-28, `c8357e74`), 728 without a comma read family-first (the family's first word written before the given name's), `str()` renders every one differently from the text, and in 150 of them the family's last word touches the given name's first, written unspaced; of 41 holding the U+00B7 interpunct, `str()` drops it from 36. So: -- `ParsedName` gains `order`, the order the name was read under (its script's or the declared `name_order`, a comma aside), an addition with `ADDED_IN` metadata whose missing value, in a 2.x pickle, renders as today, and left out of comparison as `Token.reading` is, so a loaded 2.x pickle still equals a fresh parse of its text. The cost, accepted 2026-10-28: two equal names can render differently, where today every rendering is derived from what `==` compares (a 2.x pickle of `김민준` renders `'민준 김'` and a fresh parse `'김민준'`, and a set keeps one of them); the views read from `Token.reading` share it. -- `str()` and `render()`'s default spec put the family first where `order` does, and two words that stand one directly after the other, in the same order, in the original and in the rendering are joined by what separated them there, nothing, a space, an interpunct, a nakaguro or a comma, so `str(parse("김민준"))` is `'김민준'` and `str(parse("威廉·莎士比亚"))` keeps the dot. A name with a comma or written given-first keeps today's word order, and its separators are kept by the same rule, so the given-first transcription `威廉·莎士比亚` keeps its dot too (20 of the 36 names whose interpunct `str()` drops are read given-first, one of them with a comma). A family comma is never kept, since the rendering reverses the pair it stands between (`Smith, John` → `'John Smith'`), while a comma before a suffix is (`John Smith, Jr.` → `'John Smith, Jr.'`, where `str()` gives `'John Smith Jr.'` today): 456 of the 1,008 committed corpus entries with a suffix after a family write a comma between the family's last word and the suffix's first (2026-10-28, `c8357e74`, each entry under its declared order). Pairs the rendering separates, as `John` and `Jr.` in `Smith, John, Jr.`, take the spec's own joiner, as today. +- `ParsedName` gains `order`, the order the name was read under (its script's or the declared `name_order`, a family comma aside), an addition with `ADDED_IN` metadata whose missing value, in a 2.x pickle, renders as today, and left out of comparison as `Token.reading` is, so a loaded 2.x pickle still equals a fresh parse of its text. The cost, accepted 2026-10-28: two equal names can render differently, where today every rendering is derived from what `==` compares (a 2.x pickle of `김민준` renders `'민준 김'` and a fresh parse `'김민준'`, and a set keeps one of them); the views read from `Token.reading` share it. +- `str()` and `render()`'s default spec put the family first where `order` does, and two words that stand one directly after the other, in the same order, in the original and in the rendering are joined by what separated them there when that is nothing, whitespace, an interpunct, a nakaguro or one comma with whitespace, and by the spec's own joiner otherwise (a delimiter, a marker word the parse consumed, a doubled comma), so `John (Jack) Smith` keeps today's quoted nickname and `str(parse("김민준"))` is `'김민준'` and `str(parse("威廉·莎士比亚"))` keeps the dot. A name with a family comma or written given-first keeps today's word order, and its separators are kept by the same rule (a comma before a suffix alone does not make a family-first name given-first: `Wang Xiaoming, Jr.` under FAMILY_FIRST renders `'Wang Xiaoming, Jr.'`), so the given-first transcription `威廉·莎士比亚` keeps its dot too (20 of the 36 names whose interpunct `str()` drops are read given-first, one of them with a comma). A family comma is never kept, since the rendering reverses the pair it stands between (`Smith, John` → `'John Smith'`), while a comma directly between the family and its suffix is (`John Smith, Jr.` → `'John Smith, Jr.'`, where `str()` gives `'John Smith Jr.'` today): of the 1,003 committed corpus entries whose suffix starts after the family's last word ends, 214 have nothing between the two but one comma and whitespace (2026-10-28, `c8357e74`, each entry under its declared order); 456 hold a comma somewhere between, most with a given name or title standing there too. Pairs the rendering separates, as `John` and `Jr.` in `Smith, John, Jr.`, take the spec's own joiner, as today, so `Smith, John, Jr.` keeps rendering `'John Smith Jr.'` while `John Smith, Jr.` becomes `'John Smith, Jr.'`: two writings of one name rendering differently, a consequence of following the text accepted with the choice. - `initials()` gives nothing for a word in a script with no initials (Han, Hangul, kana, `_policy._NO_INITIALS`), where today it gives the first character with a period (`'王.'`). -These change no signature but change `str()`'s output for every name read family-first, every name whose written separator is not a space, and every name written with a comma before its suffix, which no 2.x release can warn about; Derek took them as an improvement for 3.0's release notes rather than a break (item 6 of the entry). +These change no signature but change `str()`'s output for every name read family-first, every name whose written separator is not a space, and every name written with one comma directly between its family and its suffix, which no 2.x release can warn about; Derek took them as an improvement for 3.0's release notes rather than a break (item 6 of the entry). **What changes in the API**, by the 3.0 freeze: @@ -289,7 +289,7 @@ Each sub-phase is one PR, measured on its own base as phase 2's and phase 3's ar **Interleaved with phase 3** (its sub-phases 3a to 3h are the draft's on `feat/vocabulary-phase3-design`): 4a can start beside 3a, its `languages` waiting for 3a's index; 4b to 4d need nothing of phase 3 and unblock its packs, `cs`'s generations (3f) and `pt-BR`'s agnomens (3h) on 4b, `es` and `pt` (3g) on 4d and 4g; 4f lands with or after 3b, since both decide what a shipped `Locale` holds; an `is` pack, which no phase 3 sub-phase lists, needs 3b, 4c and 4f. 4e and 4h stand apart from both. So the order is 3a and 4a, then 4b to 4d with 3b and 3c, then 4f, then the packs with 4g, with 4e at any point. -**Left open**, with no sub-phase yet: phase 1's four reserved names said to be phase 4's (`separators` with phase 4's glued words, and `bare_only`, `leading_only` and `trailing_only`, which this RFC's phase 2 section found not needed), whose build messages still name this phase. Value questions inside the shapes above: the first surname's view and its name; whether a language pack supplies a convention's entry element or sort key, built from the per-surname units this phase reads, `family_base` being the family without every particle under every convention (decisions/R2's 2026-10-28 #754 entry): a `pt` or `pt-BR` pack would give the last surname's base first, keeping `Filho`, `Neto` and `Júnior` with it and a compound surname whole, and an `es` pack the first surname without its particle, then the second with its own (RAE's rules (a) and (e)); whether `credentials` should leave the honorifics out, CLDR having no third bucket; which ending rules beside Icelandic a pack may name (Hungarian `-né` is a married name, not a patronymic); and the names of given names only (the last table row), whose reading needs a rule that tells a mononym from a given name and a family name, not an output field. +**Left open**, with no sub-phase yet: phase 1's four reserved names said to be phase 4's (`separators` with phase 4's glued words, and `bare_only`, `leading_only` and `trailing_only`, which this RFC's phase 2 section found not needed), whose build messages still name this phase. Value questions inside the shapes above: the first surname's view and its name; whether a language pack supplies a convention's entry element or sort key, built from the per-surname units this phase reads, `family_base` being the family without its working particles under every convention (an all-particle part's words staying, rules/R2's Except) (decisions/R2's 2026-10-28 #754 entry): a `pt` or `pt-BR` pack would give the last surname's base first, keeping `Filho`, `Neto` and `Júnior` with it and a compound surname whole, and an `es` pack the first surname without its particle, then the second with its own (RAE's rules (a) and (e)); whether `credentials` should leave the honorifics out, CLDR having no third bucket; which ending rules beside Icelandic a pack may name (Hungarian `-né` is a married name, not a patronymic); and the names of given names only (the last table row), whose reading needs a rule that tells a mononym from a given name and a family name, not an output field. ## The first packs From f88394030d929c891f40ca5a22ddd41562b069ae Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Wed, 28 Oct 2026 12:43:49 -0700 Subject: [PATCH 15/39] Record that a Locale may carry CLDR-shaped render patterns (#754) A shape for later: a pack may carry given-first, surname-first and sorting patterns. The RFC maps Phase 4's views to CLDR's person-name fields and names the gap (CLDR's prefix and core are the first surname's), proposes a casefolded key method, and leaves open whether a pattern is reached through the name or the parser. Two Standing pointers gain their entry's head words, now that two 2026-10-28 #754 entries exist. Co-Authored-By: Claude Opus 5.5 --- docs/design/decisions/vocabulary-model.md | 6 ++++-- docs/design/vocabulary-model-rfc.md | 18 +++++++++++++++++- 2 files changed, 21 insertions(+), 3 deletions(-) diff --git a/docs/design/decisions/vocabulary-model.md b/docs/design/decisions/vocabulary-model.md index b1891145a..77f6c9f0a 100644 --- a/docs/design/decisions/vocabulary-model.md +++ b/docs/design/decisions/vocabulary-model.md @@ -9,8 +9,9 @@ The vocabulary was a set of base sets plus marker subsets (`particles` and `part - A language file holds the words the parser might meet in names written in that language, immigrant particles included, not the words of its etymology, and a word sits in the file of every language whose names it occurs in, a Latin transliteration in its source language's file; `mul.toml` holds the words shared on purpose instead (the generations `i` to `v` and `2`, `&`, and the European titles below) — 2026-10-15/16; 2026-10-18 #737; 2026-10-19 #749; 2026-10-19 #750, title kinds. - The default loads every language and keeps working for names of any language. Choosing a language is to narrow it in phase 3 (#753): a pack is to settle only the clashes where its language has a vocabulary reading, a strict pack to drop other languages' name-level readings too, and a pack's preferred reading to be able to outrank one a shape rule derives — 2026-10-15/16. - `e`, `y` and `i` stay in the default; choosing a language is to settle their one-case fork, which no shipped pack does yet — 2026-10-15/16. -- A patronymic field is to come in 3.0, as a view that names its words without emptying `family`, and the particle kinds already draw the `clan`/`patronymic` line for it: a patronymic is followed by a parent's name, a clan prefix by a family name the lineage shares — 2026-10-15/16; 2026-10-19 #749; 2026-10-28 #754. -- 3.0's output model keeps the seven roles and adds every new field as a view, read from `Token.reading`, the reading each token took, or for `surname2` from rules/O6's tag: `generation` and `credentials` partitioning `suffix`, `patronymic`, and `surname2` where rules/O6 read two family names; `middle` and `surnames` keep their meaning, a compound given name being a rule a caller or pack turns on; the default `str()` renders a name in the order it was read in with its written separators; a language file's `[rules]` compile into its `Locale`'s policy patch — 2026-10-28 #754. +- A patronymic field is to come in 3.0, as a view that names its words without emptying `family`, and the particle kinds already draw the `clan`/`patronymic` line for it: a patronymic is followed by a parent's name, a clan prefix by a family name the lineage shares — 2026-10-15/16; 2026-10-19 #749; 2026-10-28 #754 (Derek). +- A `Locale` may carry CLDR-shaped render patterns, display and sorting, a later addition whose shape is agreed; how a call reaches one is open — 2026-10-28 #754, render patterns. +- 3.0's output model keeps the seven roles and adds every new field as a view, read from `Token.reading`, the reading each token took, or for `surname2` from rules/O6's tag: `generation` and `credentials` partitioning `suffix`, `patronymic`, and `surname2` where rules/O6 read two family names; `middle` and `surnames` keep their meaning, a compound given name being a rule a caller or pack turns on; the default `str()` renders a name in the order it was read in with its written separators; a language file's `[rules]` compile into its `Locale`'s policy patch — 2026-10-28 #754 (Derek). - Gender is recorded only where a word gives a definitive clue: a feminine form or one half of a fixed pair, never an unmarked noun — 2026-10-15/16; 2026-10-19 #750, title kinds. - Title-phrase modifiers count as titles only beside a head title, at the head of the name only and never in a comma part (`Wing Lau` → given 'Wing', `John Smith, Vice` → title 'Vice'), save a modifier that may be a given name, which the head's test reads in every part (`Lau, Wing Kai` → given 'Wing'), rules/H9 — 2026-10-15/16; 2026-10-19 #750, title kinds; 2026-10-23 #752; 2026-10-25 #752, phase 2i; 2026-10-26 #752, phase 2j, `wing`. - Name readings are hand-declared, and only on a word some language file claims as vocabulary or refuses, so the vocabulary carries no surname list (the segmentation lists, Korean and the ZH pack's, are outside the per-word model) — 2026-10-15/16; 2026-10-19. @@ -254,3 +255,4 @@ Open: #913 (the title words phase 2j left, read as titles though borne as names) 8. **A language file's `[rules]` compiles into its shipped `Locale`'s policy patch**, each rule kind a `Policy` field (an addition), so RU and TR_AZ are rebuilt from their files and `Locale` gains no field for rules. With it, **phase 3's open API dependency is answered**: its draft (branch `feat/vocabulary-phase3-design`, unmerged) carries a choice of languages on `Locale.languages`, assuming `Locale` and `Policy` keep their 2.x shape after the facade's removal, and the 2026-10-28 freeze keeps them so. The rest of that draft, strict as an argument of the whole choice included, stays phase 3's to decide; its entry, when it lands, says the dependency is answered. 9. **The work order**, one PR each, measured on its own base, the snapshot recording each new view where it is non-empty: 4a `Token.reading` and `ReadingKind.PATRONYMIC` (neutral; `languages` after 3a); 4b `generation` and `credentials` (neutral); 4c `patronymic` (neutral); 4d `surname2` (neutral); 4e the render defaults and `order`; 4f `[rules]` into the policy patch, with or after 3b; 4g the compound given name, after 4f; 4h the shape inventory per order (#469). Interleaved with phase 3 (its sub-phases the draft's): 4a beside 3a; 4b to 4d before the packs that need them, `cs` (3f) and `pt-BR` (3h) on 4b, `es` and `pt` (3g) on 4d and 4g; an `is` pack on 3b, 4c and 4f. 10. **What stays out of the output model.** Gender (decision 9) waits, `Token.reading` able to carry it later as one more addition. Names of given names only are a rule's question, not a field's: ORCID's empty family box holds 68,006 of 1,831,533 records, 15.9% to 18.7% of those from India, Pakistan, Bangladesh and Indonesia, and the parse fills `family` in 51,367 of the 51,415 with two or more given-box words. #694's two-word surname after a given name (`Tariq Abdul Aziz`) is a parse rule for the default, outside phase 4. Phase 1's four reserved names said to be phase 4's (`separators`, `bare_only`, `leading_only`, `trailing_only`) have no sub-phase yet. +- **2026-10-28 #754, render patterns — A `Locale` MAY CARRY CLDR-SHAPED RENDER PATTERNS (A GIVEN-FIRST AND A SURNAME-FIRST DISPLAY PATTERN AND A SORTING PATTERN), A SHAPE AGREED NOW FOR A FEATURE THAT COMES LATER AS AN ADDITION.** The RFC's Phase 4 section, "Render patterns a pack may carry", holds the shape: the pack field, a table of Phase 4's views against CLDR's person-name fields, and the gap it shows, CLDR's `surname-prefix` and `surname-core` belonging to the first surname where `family_particles` and `family_base` read the whole family, which the per-surname units close. CLDR is a starting point, not the authority: its `pt` sorting pattern files under the first surname and leaves `surname2` out, which is not ABNT's last-surname entry (decisions/R2). Proposed there, not decided: a casefolded key method beside `comparison_key()`, collation left to the caller; and, open for a decision, whether a pattern is reached through a convention recorded on the name or through the parser. diff --git a/docs/design/vocabulary-model-rfc.md b/docs/design/vocabulary-model-rfc.md index a1014cb80..59847cdde 100644 --- a/docs/design/vocabulary-model-rfc.md +++ b/docs/design/vocabulary-model-rfc.md @@ -270,6 +270,22 @@ These change no signature but change `str()`'s output for every name read family **Phase 3's open API dependency answered**: phase 3's draft (branch `feat/vocabulary-phase3-design`, not yet merged) carries a choice of languages on `Locale.languages` and notes it assumes `Locale` and `Policy` keep their 2.x shape once the facade is gone; the 2026-10-28 freeze answers that, both keeping it with additions. The rest of that item, strict as an argument of the whole choice included, is phase 3's to decide, and its entry is to say the dependency is answered when it lands. With the `[rules]` proposal above, a shipped language pack is a `Locale` whose `languages` choose its vocabulary and whose `policy` carries its rules, so `Locale` needs no third field. +**Render patterns a pack may carry** (a shape agreed 2026-10-28; the feature is later work and an addition, in 3.0 if time allows and in 3.x otherwise). A `Locale` is a code, a `Lexicon` and a `PolicyPatch`; it may also carry named render patterns in CLDR's person-name shape (Unicode TR35 part 8, https://www.unicode.org/reports/tr35/tr35-personNames.html), a given-first and a surname-first display pattern and a sorting pattern, so a caller gets a region's standard forms without knowing its convention. CLDR is where a pattern starts, not its authority. In `tools/differential/cldr_fields.jsonl` (CLDR release 48), `pt`'s sorting pattern files its native sample under the FIRST surname, its particle after the given names, and leaves `surname2` out (`Silva, José Pedro Eduardo da`, `surname2` 'Silveira'), which is CLDR's two-surname model and not ABNT's last-surname entry (decisions/R2's 2026-10-28 #754 entry); `es`'s sorting pattern keeps its foreign sample's `von` in front (`von Miller Jones, …`), where RAE's rule (a) sets a Spanish preposition after the given name, and its native sample holds no particle, so it cannot show rule (a) either way. + +| CLDR field | Phase 4 view | filled | +|---|---|---| +| `title`, `given`, `given2` | `title`, `given`, `middle` | today | +| `given-informal` | `nickname`, loosely | today | +| `surname-prefix`, `surname-core` | `family_particles`, `family_base` | today, but over the whole family | +| `surname`, `surname2` | the first unit's view (left open), `surname2` | 4d, where rules/O6 reads two units | +| `generation`, `credentials` | `generation`, `credentials` | 4b | + +What a pattern needs from this phase is the one gap the table shows: CLDR's prefix and core belong to the FIRST surname, while `family_particles` and `family_base` read the whole family, so filling them per surname needs the units told apart, each with its own particles and base. That is the first unit's view and the per-surname units the sort-key question under Left open reads, so they are one piece of work; CLDR has no prefix field for `surname2`. + +A sorting rendering is not a sort key: it is a string, and a string sorts by code point, so accents, `ñ` and case fall out of order. Proposed, not decided: a key method beside `comparison_key()` returning a convention's fields in its sort order, each casefolded, which a caller hands to real collation (ICU, `locale.strxfrm`) for anything past code-point order. + +How a call reaches a pack's pattern is open, and each answer costs something: `ParsedName` does not know the locale it was parsed under. (a) The parse records the convention on the name, as it records `order`, so `render("sorting")` finds it from the name alone: one more `ADDED_IN` field, missing on a 2.x pickle, and kept out of `==` as `order` is, with the cost accepted for `order` (two equal names rendering differently). (b) Pattern rendering goes through the parser, `Parser.render(name, "sorting")` or the like, as `Parser.capitalized` does for a convenience that depends on configuration: no field and no pickle question, but a name alone cannot give its convention's forms. The render default decided above forecloses neither: `str()` follows `order` and reads no pattern, and under both answers a pattern is something a caller asks for. + **Gender**, recorded only where a word gives a definitive clue (decision 9), is not part of this design: the vocabulary records it on readings (`bint`'s `gender = "f"`), and `Token.reading` could carry it later as one more addition. ### The order of work @@ -289,7 +305,7 @@ Each sub-phase is one PR, measured on its own base as phase 2's and phase 3's ar **Interleaved with phase 3** (its sub-phases 3a to 3h are the draft's on `feat/vocabulary-phase3-design`): 4a can start beside 3a, its `languages` waiting for 3a's index; 4b to 4d need nothing of phase 3 and unblock its packs, `cs`'s generations (3f) and `pt-BR`'s agnomens (3h) on 4b, `es` and `pt` (3g) on 4d and 4g; 4f lands with or after 3b, since both decide what a shipped `Locale` holds; an `is` pack, which no phase 3 sub-phase lists, needs 3b, 4c and 4f. 4e and 4h stand apart from both. So the order is 3a and 4a, then 4b to 4d with 3b and 3c, then 4f, then the packs with 4g, with 4e at any point. -**Left open**, with no sub-phase yet: phase 1's four reserved names said to be phase 4's (`separators` with phase 4's glued words, and `bare_only`, `leading_only` and `trailing_only`, which this RFC's phase 2 section found not needed), whose build messages still name this phase. Value questions inside the shapes above: the first surname's view and its name; whether a language pack supplies a convention's entry element or sort key, built from the per-surname units this phase reads, `family_base` being the family without its working particles under every convention (an all-particle part's words staying, rules/R2's Except) (decisions/R2's 2026-10-28 #754 entry): a `pt` or `pt-BR` pack would give the last surname's base first, keeping `Filho`, `Neto` and `Júnior` with it and a compound surname whole, and an `es` pack the first surname without its particle, then the second with its own (RAE's rules (a) and (e)); whether `credentials` should leave the honorifics out, CLDR having no third bucket; which ending rules beside Icelandic a pack may name (Hungarian `-né` is a married name, not a patronymic); and the names of given names only (the last table row), whose reading needs a rule that tells a mononym from a given name and a family name, not an output field. +**Left open**, with no sub-phase yet: phase 1's four reserved names said to be phase 4's (`separators` with phase 4's glued words, and `bare_only`, `leading_only` and `trailing_only`, which this RFC's phase 2 section found not needed), whose build messages still name this phase. Value questions inside the shapes above: the first surname's view and its name, with its own particles and base (the gap the render-pattern table shows); how a pack's render pattern is reached, and the key method (Render patterns, above); whether a language pack supplies a convention's entry element or sort key, built from the per-surname units this phase reads, `family_base` being the family without its working particles under every convention (an all-particle part's words staying, rules/R2's Except) (decisions/R2's 2026-10-28 #754 entry): a `pt` or `pt-BR` pack would give the last surname's base first, keeping `Filho`, `Neto` and `Júnior` with it and a compound surname whole, and an `es` pack the first surname without its particle, then the second with its own (RAE's rules (a) and (e)); whether `credentials` should leave the honorifics out, CLDR having no third bucket; which ending rules beside Icelandic a pack may name (Hungarian `-né` is a married name, not a patronymic); and the names of given names only (the last table row), whose reading needs a rule that tells a mononym from a given name and a family name, not an output field. ## The first packs From 4e2dbf1a34456f630bddf3444cc89ec6ce6aa457 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Wed, 28 Oct 2026 12:50:51 -0700 Subject: [PATCH 16/39] Fix the fourth docs review's findings on phase 4's render notes (#754) A spec joiner that decorates a field (the ') ' after a maiden name) stands over a kept separator; a comma may be ASCII or full-width, with the counts' ASCII scope stated. The CLDR table splits a one-unit surname (family, today) from a two-unit one, and option (a) names its cost of a new way to ask for a pattern, render() taking a format spec. Co-Authored-By: Claude Opus 5.5 --- docs/design/decisions/vocabulary-model.md | 2 +- docs/design/vocabulary-model-rfc.md | 7 ++++--- 2 files changed, 5 insertions(+), 4 deletions(-) diff --git a/docs/design/decisions/vocabulary-model.md b/docs/design/decisions/vocabulary-model.md index 77f6c9f0a..0ef9068c1 100644 --- a/docs/design/decisions/vocabulary-model.md +++ b/docs/design/decisions/vocabulary-model.md @@ -250,7 +250,7 @@ Open: #913 (the title words phase 2j left, read as titles though borne as names) 3. **A `patronymic` view, filled by a word or a rule that says patronymic, with `family` and `middle` unchanged.** The default fills it from the vocabulary's `patronymic` particles (`bin`, `binti`, `ibn`, `bint`, `bar`, `bat` and the native-script forms) with the name they introduce (`Mahathir bin Mohamad` → family `bin Mohamad`, as today, and patronymic `bin Mohamad`); RU's and TR_AZ's rules fill it where a caller turns them on (`Ivan Ivanovich Ivanov` under RU → middle `Ivanovich` and patronymic `Ivanovich`); an Icelandic ending rule fills it under an `is` pack only, since the default cannot tell `-dóttir` and `-son` from Swedish surnames (ORCID's Swedish records end their family box in `-son` or `-dóttir` in 2,184 of 20,000). A matronymic, written the same way, fills it too. SUPERSEDES: the RFC's earlier statement of decision 8's value, that `family` can be empty for a name with a patronymic. Measured: filers put an Icelandic patronymic in the family box (ORCID 1,647 of 2,583 Icelandic records, PatentsView 535 of 722, PSC 101 of 125), and CLDR's `is` file's 3 Icelandic samples put it in `surname` or `surname-core` in all 9 of their renderings (its one foreign sample puts a `-dóttir` word in `surname2`), while Wikidata's `patronymic-is` and `patronymic-my` strata record 108 names with a patronymic and no family name, of which the parse puts the patronymic in `family` 107 times. An East Slavic patronymic stands in the given part: CLDR's 2 `ru` samples put it in `given2` in all 6 renderings; PatentsView's Russian given boxes end in one 8,423 times of 20,000, and the parse reads such a word as the middle 10,902 times of 10,962; 90 of the 112 Russian nationals in PSC who filled the middle-name box wrote one there. A `family` that keeps it and a `patronymic` that names it agree with both kinds of source. Weighed against: emptying `family` where the patronymic is recognized, which moves the field the fielded sources fill and changes `family` for every Icelandic and nasab name, a behavior change of a frozen property; and a neutral parent-names name for the view, which buys nothing the view can detect, since a matronymic and a patronymic are written alike. 4. **`generation` and `credentials` are views that partition `suffix` (#326)**, not roles, `generation` the tokens read as a generation or an agnomen, `credentials` the rest, the post-nominals that are neither (`Esq.`, the honorifics after a name) falling in `credentials`, CLDR's catch-all, with `Token.reading.kind` telling them apart for a caller who needs it. Measured: of the 1,054 committed corpus entries with a suffix, 33 hold a generation (or agnomen) and a listed credential together, 77 a generation beside any other suffix word, and 64 only listed post-nominals of other kinds (each suffix word classified by the RFC's Phase 4 recipe, joined tokens merged); of the 615 CLDR renderings with either field, the 75 whose suffix holds exactly CLDR's words split into CLDR's `generation` and `credentials` by the vocabulary's reading kind in all 75. Weighed against: two new `Role` members (item 1), and a third view for the honorifics, left open since CLDR has none. 5. **a `surname2` view, the family's second unit where rules/O6 read the family as two**, marked by a new stable tag where O6 takes its word, empty otherwise; `family` keeps both units as O6 makes it, and a family comma alone marks nothing (`Lloyd Webber, Andrew`). Measured: a two-word family box in ORCID's records from Spain 48.0%, Mexico 64.7%, Colombia 72.8% and Chile 50.5%, against 3.9% from the United States. Left open: a view for the first unit and its name, `surname` standing close to the `surnames` view that exists. Declined: redefining `surnames` as the two units, which would repeat `family`, the family holding both where O6 reads two, and lose what `surnames` does today, joining `middle` and `family` so that a double surname the default splits (`Gabriel García Márquez` → middle `García`) reads whole, the view rules/P2 names for that reading. - 6. **The render defaults follow the order the name was read in, a behavior change of the output rather than a break of the API.** `ParsedName` gains `order` (an addition, `ADDED_IN`, a 2.x pickle rendering as today, left out of comparison, which lets two equal names render differently, a cost accepted 2026-10-28: a 2.x pickle of `김민준` renders `'민준 김'` and a fresh parse `'김민준'`, and the two compare equal); the default `str()` and `render()` spec put the family first where `order` does, and join two words that stand one directly after the other, in the same order, in the original and the rendering with what separated them there when that is nothing, whitespace, an interpunct, a nakaguro or one comma with whitespace, the spec's own joiner otherwise (a delimiter, a consumed marker word); `initials()` gives nothing for a word in a script with no initials. A name with a family comma or written given-first keeps today's word order and its separators by the same rule, a suffix comma alone leaving a family-first name family-first (`威廉·莎士比亚` keeps its dot; 20 of the 36 names whose interpunct `str()` drops are read given-first). A family comma is never kept, the rendering reversing its pair (`Smith, John` → `'John Smith'`), and one directly between the family and its suffix is, decided 2026-10-28 (`John Smith, Jr.` → `'John Smith, Jr.'`, today `'John Smith Jr.'`; 214 of the 1,003 committed corpus entries whose suffix follows the family write one there). Accepted with it: `Smith, John, Jr.`, whose suffix the rendering moves away from the given name before it, keeps `'John Smith Jr.'`, so two writings of one name render differently. Measured over the committed corpus's distinct entries: 728 without a comma read family-first, `str()` renders all 728 differently from their text (`김민준` → `'민준 김'`), 150 of them with the family's last word touching the given name's first, and it drops the U+00B7 interpunct from 36 of the 41 holding one. This carries out decisions/3-0-reevaluations' render-default clean slate. No signature changes, but `str()`'s output changes for every name read family-first, every name whose written separator is not a space, and every name with one comma directly between its family and its suffix, and no 2.x release can warn about a `str()` result, so it is taken as an improvement for 3.0's release notes, not a break needing a warning. + 6. **The render defaults follow the order the name was read in, a behavior change of the output rather than a break of the API.** `ParsedName` gains `order` (an addition, `ADDED_IN`, a 2.x pickle rendering as today, left out of comparison, which lets two equal names render differently, a cost accepted 2026-10-28: a 2.x pickle of `김민준` renders `'민준 김'` and a fresh parse `'김민준'`, and the two compare equal); the default `str()` and `render()` spec put the family first where `order` does, and join two words that stand one directly after the other, in the same order, in the original and the rendering with what separated them there when that is nothing, whitespace, an interpunct, a nakaguro or one comma with whitespace, the spec's own joiner otherwise (a delimiter, a consumed marker word, a joiner decorating a field, as the `) ` after a maiden name), a comma being ASCII or full-width; `initials()` gives nothing for a word in a script with no initials. A name with a family comma or written given-first keeps today's word order and its separators by the same rule, a suffix comma alone leaving a family-first name family-first (`威廉·莎士比亚` keeps its dot; 20 of the 36 names whose interpunct `str()` drops are read given-first). A family comma is never kept, the rendering reversing its pair (`Smith, John` → `'John Smith'`), and one directly between the family and its suffix is, decided 2026-10-28 (`John Smith, Jr.` → `'John Smith, Jr.'`, today `'John Smith Jr.'`; 214 of the 1,003 committed corpus entries whose suffix follows the family write one there). Accepted with it: `Smith, John, Jr.`, whose suffix the rendering moves away from the given name before it, keeps `'John Smith Jr.'`, so two writings of one name render differently. Measured over the committed corpus's distinct entries: 728 without a comma read family-first, `str()` renders all 728 differently from their text (`김민준` → `'민준 김'`), 150 of them with the family's last word touching the given name's first, and it drops the U+00B7 interpunct from 36 of the 41 holding one. This carries out decisions/3-0-reevaluations' render-default clean slate. No signature changes, but `str()`'s output changes for every name read family-first, every name whose written separator is not a space, and every name with one comma directly between its family and its suffix, and no 2.x release can warn about a `str()` result, so it is taken as an improvement for 3.0's release notes, not a break needing a warning. 7. **`middle` keeps its meaning (#704)**, every word of the given part after the given name, the given name one unit unless a rule joins words into it, as rules/P5 already does (`Abdul Aziz Tariq` → given `Abdul Aziz`); a compound given name is a `[rules]` rule a caller or a pack turns on, CLDR's `given2` is `middle`, and `Kaur` and an English name beside a Chinese one stay in `middle`, an accepted cost. Measured: in PSC, whose form has a middle-name box, 2,499 of 1,040,708 forename boxes hold two or more words, and of 6,101 Spanish nationals 25 do while 2,621 fill the middle box; ORCID's and PatentsView's given box holds middles too, so it cannot separate the two. The `Maria do Carmo` shape, a particle inside a given box of three or more words with a family box filled, reads the given box's words into the family in 1,812 of 1,923 ORCID records and 1,165 of 1,219 PatentsView records from Brazil, Portugal, Spain, Mexico, Argentina, Colombia, Chile and Italy, which is that rule's case. Weighed against: #704's option (b), one given unit by default, which changes `middle` for every English name of three words. 8. **A language file's `[rules]` compiles into its shipped `Locale`'s policy patch**, each rule kind a `Policy` field (an addition), so RU and TR_AZ are rebuilt from their files and `Locale` gains no field for rules. With it, **phase 3's open API dependency is answered**: its draft (branch `feat/vocabulary-phase3-design`, unmerged) carries a choice of languages on `Locale.languages`, assuming `Locale` and `Policy` keep their 2.x shape after the facade's removal, and the 2026-10-28 freeze keeps them so. The rest of that draft, strict as an argument of the whole choice included, stays phase 3's to decide; its entry, when it lands, says the dependency is answered. 9. **The work order**, one PR each, measured on its own base, the snapshot recording each new view where it is non-empty: 4a `Token.reading` and `ReadingKind.PATRONYMIC` (neutral; `languages` after 3a); 4b `generation` and `credentials` (neutral); 4c `patronymic` (neutral); 4d `surname2` (neutral); 4e the render defaults and `order`; 4f `[rules]` into the policy patch, with or after 3b; 4g the compound given name, after 4f; 4h the shape inventory per order (#469). Interleaved with phase 3 (its sub-phases the draft's): 4a beside 3a; 4b to 4d before the packs that need them, `cs` (3f) and `pt-BR` (3h) on 4b, `es` and `pt` (3g) on 4d and 4g; an `is` pack on 3b, 4c and 4f. diff --git a/docs/design/vocabulary-model-rfc.md b/docs/design/vocabulary-model-rfc.md index 59847cdde..55b92ce6a 100644 --- a/docs/design/vocabulary-model-rfc.md +++ b/docs/design/vocabulary-model-rfc.md @@ -250,7 +250,7 @@ ORCID's and PatentsView's given box holds the middle names too, so a two-word gi **Render defaults** (decisions/3-0-reevaluations' clean slate). The default `str()` reads every name given-first and joins its words with spaces, so a name read family-first is turned round and a written separator is lost: `str(parse("김민준"))` is `'민준 김'`, `str(parse("王 小明"))` is `'小明 王'` and `str(parse("威廉·莎士比亚"))` is `'威廉 莎士比亚'`. Over the committed corpus's distinct entries, each under its declared order (2026-10-28, `c8357e74`), 728 without a comma read family-first (the family's first word written before the given name's), `str()` renders every one differently from the text, and in 150 of them the family's last word touches the given name's first, written unspaced; of 41 holding the U+00B7 interpunct, `str()` drops it from 36. So: - `ParsedName` gains `order`, the order the name was read under (its script's or the declared `name_order`, a family comma aside), an addition with `ADDED_IN` metadata whose missing value, in a 2.x pickle, renders as today, and left out of comparison as `Token.reading` is, so a loaded 2.x pickle still equals a fresh parse of its text. The cost, accepted 2026-10-28: two equal names can render differently, where today every rendering is derived from what `==` compares (a 2.x pickle of `김민준` renders `'민준 김'` and a fresh parse `'김민준'`, and a set keeps one of them); the views read from `Token.reading` share it. -- `str()` and `render()`'s default spec put the family first where `order` does, and two words that stand one directly after the other, in the same order, in the original and in the rendering are joined by what separated them there when that is nothing, whitespace, an interpunct, a nakaguro or one comma with whitespace, and by the spec's own joiner otherwise (a delimiter, a marker word the parse consumed, a doubled comma), so `John (Jack) Smith` keeps today's quoted nickname and `str(parse("김민준"))` is `'김민준'` and `str(parse("威廉·莎士比亚"))` keeps the dot. A name with a family comma or written given-first keeps today's word order, and its separators are kept by the same rule (a comma before a suffix alone does not make a family-first name given-first: `Wang Xiaoming, Jr.` under FAMILY_FIRST renders `'Wang Xiaoming, Jr.'`), so the given-first transcription `威廉·莎士比亚` keeps its dot too (20 of the 36 names whose interpunct `str()` drops are read given-first, one of them with a comma). A family comma is never kept, since the rendering reverses the pair it stands between (`Smith, John` → `'John Smith'`), while a comma directly between the family and its suffix is (`John Smith, Jr.` → `'John Smith, Jr.'`, where `str()` gives `'John Smith Jr.'` today): of the 1,003 committed corpus entries whose suffix starts after the family's last word ends, 214 have nothing between the two but one comma and whitespace (2026-10-28, `c8357e74`, each entry under its declared order); 456 hold a comma somewhere between, most with a given name or title standing there too. Pairs the rendering separates, as `John` and `Jr.` in `Smith, John, Jr.`, take the spec's own joiner, as today, so `Smith, John, Jr.` keeps rendering `'John Smith Jr.'` while `John Smith, Jr.` becomes `'John Smith, Jr.'`: two writings of one name rendering differently, a consequence of following the text accepted with the choice. +- `str()` and `render()`'s default spec put the family first where `order` does, and two words that stand one directly after the other, in the same order, in the original and in the rendering are joined by what separated them there when that is nothing, whitespace, an interpunct, a nakaguro or one comma with whitespace, and by the spec's own joiner otherwise (a delimiter, a marker word the parse consumed, a doubled comma, or a joiner by which the spec decorates a field, as the `) ` after the maiden name, so `Jane Smith née Jones, PhD` keeps `'Jane Smith (Jones) PhD'`; 38 committed entries put a suffix directly after an undelimited maiden name), a comma being ASCII or full-width (U+FF0C), as the parse reads either, so `John (Jack) Smith` keeps today's quoted nickname and `str(parse("김민준"))` is `'김민준'` and `str(parse("威廉·莎士比亚"))` keeps the dot. A name with a family comma or written given-first keeps today's word order, and its separators are kept by the same rule (a comma before a suffix alone does not make a family-first name given-first: `Wang Xiaoming, Jr.` under FAMILY_FIRST renders `'Wang Xiaoming, Jr.'`), so the given-first transcription `威廉·莎士比亚` keeps its dot too (20 of the 36 names whose interpunct `str()` drops are read given-first, one of them with a comma). A family comma is never kept, since the rendering reverses the pair it stands between (`Smith, John` → `'John Smith'`), while a comma directly between the family and its suffix is (`John Smith, Jr.` → `'John Smith, Jr.'`, where `str()` gives `'John Smith Jr.'` today): of the 1,003 committed corpus entries whose suffix starts after the family's last word ends, 214 have nothing between the two but one comma and whitespace (2026-10-28, `c8357e74`, each entry under its declared order; these counts read the ASCII comma, and 7 more write a full-width one there, while 1 of the 728 family-first names above holds a full-width family comma); 456 hold a comma somewhere between, most with a given name or title standing there too. Pairs the rendering separates, as `John` and `Jr.` in `Smith, John, Jr.`, take the spec's own joiner, as today, so `Smith, John, Jr.` keeps rendering `'John Smith Jr.'` while `John Smith, Jr.` becomes `'John Smith, Jr.'`: two writings of one name rendering differently, a consequence of following the text accepted with the choice. - `initials()` gives nothing for a word in a script with no initials (Han, Hangul, kana, `_policy._NO_INITIALS`), where today it gives the first character with a period (`'王.'`). These change no signature but change `str()`'s output for every name read family-first, every name whose written separator is not a space, and every name written with one comma directly between its family and its suffix, which no 2.x release can warn about; Derek took them as an improvement for 3.0's release notes rather than a break (item 6 of the entry). @@ -277,14 +277,15 @@ These change no signature but change `str()`'s output for every name read family | `title`, `given`, `given2` | `title`, `given`, `middle` | today | | `given-informal` | `nickname`, loosely | today | | `surname-prefix`, `surname-core` | `family_particles`, `family_base` | today, but over the whole family | -| `surname`, `surname2` | the first unit's view (left open), `surname2` | 4d, where rules/O6 reads two units | +| `surname`, of a family read as one unit | `family` | today | +| `surname` and `surname2`, of a family read as two | the first unit's view, `surname2` | the first unit's view has no sub-phase (Left open); `surname2` is 4d, where rules/O6 reads two units | | `generation`, `credentials` | `generation`, `credentials` | 4b | What a pattern needs from this phase is the one gap the table shows: CLDR's prefix and core belong to the FIRST surname, while `family_particles` and `family_base` read the whole family, so filling them per surname needs the units told apart, each with its own particles and base. That is the first unit's view and the per-surname units the sort-key question under Left open reads, so they are one piece of work; CLDR has no prefix field for `surname2`. A sorting rendering is not a sort key: it is a string, and a string sorts by code point, so accents, `ñ` and case fall out of order. Proposed, not decided: a key method beside `comparison_key()` returning a convention's fields in its sort order, each casefolded, which a caller hands to real collation (ICU, `locale.strxfrm`) for anything past code-point order. -How a call reaches a pack's pattern is open, and each answer costs something: `ParsedName` does not know the locale it was parsed under. (a) The parse records the convention on the name, as it records `order`, so `render("sorting")` finds it from the name alone: one more `ADDED_IN` field, missing on a 2.x pickle, and kept out of `==` as `order` is, with the cost accepted for `order` (two equal names rendering differently). (b) Pattern rendering goes through the parser, `Parser.render(name, "sorting")` or the like, as `Parser.capitalized` does for a convenience that depends on configuration: no field and no pickle question, but a name alone cannot give its convention's forms. The render default decided above forecloses neither: `str()` follows `order` and reads no pattern, and under both answers a pattern is something a caller asks for. +How a call reaches a pack's pattern is open, and each answer costs something: `ParsedName` does not know the locale it was parsed under. (a) The parse records the convention on the name, as it records `order`, so a call on the name alone finds it: one more `ADDED_IN` field, and a new method or argument to ask for a pattern by, since `render()` takes a format spec today and `render("sorting")` returns `'sorting'`, so giving that argument a pattern's name would change behavior rather than add it; the field is missing on a 2.x pickle and kept out of `==` as `order` is, with the cost accepted for `order` (two equal names rendering differently). (b) Pattern rendering goes through the parser, `Parser.render(name, "sorting")` or the like, as `Parser.capitalized` does for a convenience that depends on configuration: no field and no pickle question, but a name alone cannot give its convention's forms. The render default decided above forecloses neither: `str()` follows `order` and reads no pattern, and under both answers a pattern is something a caller asks for. **Gender**, recorded only where a word gives a definitive clue (decision 9), is not part of this design: the vocabulary records it on readings (`bint`'s `gender = "f"`), and `Token.reading` could carry it later as one more addition. From acd9054846f7105060dd4fff06e2911071171a72 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Wed, 28 Oct 2026 13:34:58 -0700 Subject: [PATCH 17/39] Leave open whether a token's title kind reaches callers (#754) Title tokens show behaviors, not kinds, and ReadingKind reads every title as TITLE. The RFC's Phase 4 Left open asks whether Reading gains an optional vocabulary-kind field, and so whether phase 3's reading index carries kinds; no title word holds two kinds today (816 checked). Co-Authored-By: Claude Opus 5.5 --- docs/design/vocabulary-model-rfc.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/design/vocabulary-model-rfc.md b/docs/design/vocabulary-model-rfc.md index 55b92ce6a..5cb6859a0 100644 --- a/docs/design/vocabulary-model-rfc.md +++ b/docs/design/vocabulary-model-rfc.md @@ -306,7 +306,7 @@ Each sub-phase is one PR, measured on its own base as phase 2's and phase 3's ar **Interleaved with phase 3** (its sub-phases 3a to 3h are the draft's on `feat/vocabulary-phase3-design`): 4a can start beside 3a, its `languages` waiting for 3a's index; 4b to 4d need nothing of phase 3 and unblock its packs, `cs`'s generations (3f) and `pt-BR`'s agnomens (3h) on 4b, `es` and `pt` (3g) on 4d and 4g; 4f lands with or after 3b, since both decide what a shipped `Locale` holds; an `is` pack, which no phase 3 sub-phase lists, needs 3b, 4c and 4f. 4e and 4h stand apart from both. So the order is 3a and 4a, then 4b to 4d with 3b and 3c, then 4f, then the packs with 4g, with 4e at any point. -**Left open**, with no sub-phase yet: phase 1's four reserved names said to be phase 4's (`separators` with phase 4's glued words, and `bare_only`, `leading_only` and `trailing_only`, which this RFC's phase 2 section found not needed), whose build messages still name this phase. Value questions inside the shapes above: the first surname's view and its name, with its own particles and base (the gap the render-pattern table shows); how a pack's render pattern is reached, and the key method (Render patterns, above); whether a language pack supplies a convention's entry element or sort key, built from the per-surname units this phase reads, `family_base` being the family without its working particles under every convention (an all-particle part's words staying, rules/R2's Except) (decisions/R2's 2026-10-28 #754 entry): a `pt` or `pt-BR` pack would give the last surname's base first, keeping `Filho`, `Neto` and `Júnior` with it and a compound surname whole, and an `es` pack the first surname without its particle, then the second with its own (RAE's rules (a) and (e)); whether `credentials` should leave the honorifics out, CLDR having no third bucket; which ending rules beside Icelandic a pack may name (Hungarian `-né` is a married name, not a patronymic); and the names of given names only (the last table row), whose reading needs a rule that tells a mononym from a given name and a family name, not an output field. +**Left open**, with no sub-phase yet: phase 1's four reserved names said to be phase 4's (`separators` with phase 4's glued words, and `bare_only`, `leading_only` and `trailing_only`, which this RFC's phase 2 section found not needed), whose build messages still name this phase. Value questions inside the shapes above: the first surname's view and its name, with its own particles and base (the gap the render-pattern table shows); how a pack's render pattern is reached, and the key method (Render patterns, above); whether a language pack supplies a convention's entry element or sort key, built from the per-surname units this phase reads, `family_base` being the family without its working particles under every convention (an all-particle part's words staying, rules/R2's Except) (decisions/R2's 2026-10-28 #754 entry): a `pt` or `pt-BR` pack would give the last surname's base first, keeping `Filho`, `Neto` and `Júnior` with it and a compound surname whole, and an `es` pack the first surname without its particle, then the second with its own (RAE's rules (a) and (e)); whether a token's title kind should reach callers: today a title token's tags name only behaviors (`Dr. Mr. Sir John Smith` gives `Dr.` `vocab:title`, `Mr.` `vocab:after-given-title` and `vocab:title`, `Sir` `vocab:given-title` and `vocab:title`), `Lexicon` keeps the titles as one set beside `title_modifiers` and `title_styles`, the vocabulary's salutation, academic, religious, nobility, military, office, occupation and kinship kinds live only in the TOML, and `ReadingKind`, being "the parse's, not the vocabulary's", reads every one as `TITLE` where the suffix readings split four ways; so whether `Reading` gains an optional field holding the vocabulary's kind, filled only where the reading's source is the vocabulary (a title read by its shape, `Insp.`, or a word a caller added by `Lexicon.add` has none, so its absence means unknown, not that the word has none), and so whether phase 3's reading index (its "Where the provenance lives at runtime", on `feat/vocabulary-phase3-design`) carries the kinds as well as the languages and properties. It is an addition and blocks nothing in 3.0; it would let a view name the salutation alone (`Dear Mr. Smith`), which CLDR's one `title` field does not model. Whether it holds one kind or a set is part of the question: of the 816 title words in nameparser/vocab/ on 2026-10-28 none carries two title kinds, `father` being `religious` only, the README's "kinship word used as one"; whether `credentials` should leave the honorifics out, CLDR having no third bucket; which ending rules beside Icelandic a pack may name (Hungarian `-né` is a married name, not a patronymic); and the names of given names only (the last table row), whose reading needs a rule that tells a mononym from a given name and a family name, not an output field. ## The first packs From 3b290dd95f914dd3030e4772aeb31d3577f29adf Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Wed, 28 Oct 2026 13:37:12 -0700 Subject: [PATCH 18/39] Leave open whether 3.0's Lexicon exposes its compiled sets (#754) A shape question for before the freeze: keep a public field per behavior set (22 today, 11 added since 2.0.0), or answer word, kind and property queries from phase 3's reading index and keep the sets internal. The RFC states the costs on each side: frozen fields, pickles, reprs, how a caller adds a word, and per-token frames. Co-Authored-By: Claude Opus 5.5 --- docs/design/vocabulary-model-rfc.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/design/vocabulary-model-rfc.md b/docs/design/vocabulary-model-rfc.md index 5cb6859a0..8d0f1d933 100644 --- a/docs/design/vocabulary-model-rfc.md +++ b/docs/design/vocabulary-model-rfc.md @@ -306,7 +306,7 @@ Each sub-phase is one PR, measured on its own base as phase 2's and phase 3's ar **Interleaved with phase 3** (its sub-phases 3a to 3h are the draft's on `feat/vocabulary-phase3-design`): 4a can start beside 3a, its `languages` waiting for 3a's index; 4b to 4d need nothing of phase 3 and unblock its packs, `cs`'s generations (3f) and `pt-BR`'s agnomens (3h) on 4b, `es` and `pt` (3g) on 4d and 4g; 4f lands with or after 3b, since both decide what a shipped `Locale` holds; an `is` pack, which no phase 3 sub-phase lists, needs 3b, 4c and 4f. 4e and 4h stand apart from both. So the order is 3a and 4a, then 4b to 4d with 3b and 3c, then 4f, then the packs with 4g, with 4e at any point. -**Left open**, with no sub-phase yet: phase 1's four reserved names said to be phase 4's (`separators` with phase 4's glued words, and `bare_only`, `leading_only` and `trailing_only`, which this RFC's phase 2 section found not needed), whose build messages still name this phase. Value questions inside the shapes above: the first surname's view and its name, with its own particles and base (the gap the render-pattern table shows); how a pack's render pattern is reached, and the key method (Render patterns, above); whether a language pack supplies a convention's entry element or sort key, built from the per-surname units this phase reads, `family_base` being the family without its working particles under every convention (an all-particle part's words staying, rules/R2's Except) (decisions/R2's 2026-10-28 #754 entry): a `pt` or `pt-BR` pack would give the last surname's base first, keeping `Filho`, `Neto` and `Júnior` with it and a compound surname whole, and an `es` pack the first surname without its particle, then the second with its own (RAE's rules (a) and (e)); whether a token's title kind should reach callers: today a title token's tags name only behaviors (`Dr. Mr. Sir John Smith` gives `Dr.` `vocab:title`, `Mr.` `vocab:after-given-title` and `vocab:title`, `Sir` `vocab:given-title` and `vocab:title`), `Lexicon` keeps the titles as one set beside `title_modifiers` and `title_styles`, the vocabulary's salutation, academic, religious, nobility, military, office, occupation and kinship kinds live only in the TOML, and `ReadingKind`, being "the parse's, not the vocabulary's", reads every one as `TITLE` where the suffix readings split four ways; so whether `Reading` gains an optional field holding the vocabulary's kind, filled only where the reading's source is the vocabulary (a title read by its shape, `Insp.`, or a word a caller added by `Lexicon.add` has none, so its absence means unknown, not that the word has none), and so whether phase 3's reading index (its "Where the provenance lives at runtime", on `feat/vocabulary-phase3-design`) carries the kinds as well as the languages and properties. It is an addition and blocks nothing in 3.0; it would let a view name the salutation alone (`Dear Mr. Smith`), which CLDR's one `title` field does not model. Whether it holds one kind or a set is part of the question: of the 816 title words in nameparser/vocab/ on 2026-10-28 none carries two title kinds, `father` being `religious` only, the README's "kinship word used as one"; whether `credentials` should leave the honorifics out, CLDR having no third bucket; which ending rules beside Icelandic a pack may name (Hungarian `-né` is a married name, not a patronymic); and the names of given names only (the last table row), whose reading needs a rule that tells a mononym from a given name and a family name, not an output field. +**Left open**, with no sub-phase yet: phase 1's four reserved names said to be phase 4's (`separators` with phase 4's glued words, and `bare_only`, `leading_only` and `trailing_only`, which this RFC's phase 2 section found not needed), whose build messages still name this phase. Value questions inside the shapes above: the first surname's view and its name, with its own particles and base (the gap the render-pattern table shows); how a pack's render pattern is reached, and the key method (Render patterns, above); whether a language pack supplies a convention's entry element or sort key, built from the per-surname units this phase reads, `family_base` being the family without its working particles under every convention (an all-particle part's words staying, rules/R2's Except) (decisions/R2's 2026-10-28 #754 entry): a `pt` or `pt-BR` pack would give the last surname's base first, keeping `Filho`, `Neto` and `Júnior` with it and a compound surname whole, and an `es` pack the first surname without its particle, then the second with its own (RAE's rules (a) and (e)); whether a token's title kind should reach callers: today a title token's tags name only behaviors (`Dr. Mr. Sir John Smith` gives `Dr.` `vocab:title`, `Mr.` `vocab:after-given-title` and `vocab:title`, `Sir` `vocab:given-title` and `vocab:title`), `Lexicon` keeps the titles as one set beside `title_modifiers` and `title_styles`, the vocabulary's salutation, academic, religious, nobility, military, office, occupation and kinship kinds live only in the TOML, and `ReadingKind`, being "the parse's, not the vocabulary's", reads every one as `TITLE` where the suffix readings split four ways; so whether `Reading` gains an optional field holding the vocabulary's kind, filled only where the reading's source is the vocabulary (a title read by its shape, `Insp.`, or a word a caller added by `Lexicon.add` has none, so its absence means unknown, not that the word has none), and so whether phase 3's reading index (its "Where the provenance lives at runtime", on `feat/vocabulary-phase3-design`) carries the kinds as well as the languages and properties. It is an addition and blocks nothing in 3.0; it would let a view name the salutation alone (`Dear Mr. Smith`), which CLDR's one `title` field does not model. Whether it holds one kind or a set is part of the question: of the 816 title words in nameparser/vocab/ on 2026-10-28 none carries two title kinds, `father` being `religious` only, the README's "kinship word used as one"; whether `Lexicon` keeps exposing each compiled set as a public field or answers word, kind and property queries from phase 3's reading index, its sets kept internal, a SHAPE question to settle before the 3.0 freeze makes every field permanent: `dataclasses.fields(Lexicon)` gives 22 public fields on 2026-10-28, 11 of them added since 2.0.0's 11 (`surnames`, `honorific_tails`, `conjunctions_ambiguous`, `after_given_titles`, `suffix_words_ambiguous`, `agnomens`, `given_name_abbreviations`, `titles_ambiguous`, `title_modifiers`, `title_styles`, `title_phrases`), each a new parse behavior's set, and a rule keyed on a title kind or on one language's reading of a kind (phase 3's draft, "a property a language's own reading should take only under its pack") would add more. The costs on each side: every field is frozen API, pickled with the `Lexicon` (`_guarded_setstate` refuses a pickle whose fields differ unless a field carries `ADDED_IN`), and shown in its bounded repr (`Lexicon(default + titles: +1)`), while turning existing fields internal is a break that would have to land in 3.0, where adding a query beside the fields is an addition that can come later; a caller configures vocabulary today by the field (`Lexicon.add(titles=...)`, `dataclasses.replace`), and a query-based `Lexicon` needs a way in for a caller's word, which has no kind and no language; the parser need not pay either way, a frozenset test per word costing no frame and an internal cache keeping that path, while a query method called per word would add one per token, the cost the frame budget counts as scaling with the input; whether `credentials` should leave the honorifics out, CLDR having no third bucket; which ending rules beside Icelandic a pack may name (Hungarian `-né` is a married name, not a patronymic); and the names of given names only (the last table row), whose reading needs a rule that tells a mononym from a given name and a family name, not an output field. ## The first packs From b9fb7eb13e5266b15a32dd71997c6a371f4765f8 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Wed, 28 Oct 2026 13:40:57 -0700 Subject: [PATCH 19/39] Fix the fifth docs review's findings on phase 4's open items (#754) A spec joiner that decorates a field now precedes the kept-separator branch, and a kept comma carries the whitespace written beside it, so an unspaced full-width comma is kept too. Five of the 728 hold a full-width family comma; the decisions copy scopes its counts to the ASCII comma. Left open becomes lists, with the Lexicon fields question set apart as a shape question; its pickle cost is corrected (Lexicon's own guard admits no ADDED_IN field) and the index's load cost added. The title-kind item notes that phase 3's index already holds a kind per row, and that titles are split by name collision, not kind. Co-Authored-By: Claude Opus 5.5 --- docs/design/decisions/vocabulary-model.md | 2 +- docs/design/vocabulary-model-rfc.md | 18 ++++++++++++++++-- 2 files changed, 17 insertions(+), 3 deletions(-) diff --git a/docs/design/decisions/vocabulary-model.md b/docs/design/decisions/vocabulary-model.md index 0ef9068c1..f7bf8bb4f 100644 --- a/docs/design/decisions/vocabulary-model.md +++ b/docs/design/decisions/vocabulary-model.md @@ -250,7 +250,7 @@ Open: #913 (the title words phase 2j left, read as titles though borne as names) 3. **A `patronymic` view, filled by a word or a rule that says patronymic, with `family` and `middle` unchanged.** The default fills it from the vocabulary's `patronymic` particles (`bin`, `binti`, `ibn`, `bint`, `bar`, `bat` and the native-script forms) with the name they introduce (`Mahathir bin Mohamad` → family `bin Mohamad`, as today, and patronymic `bin Mohamad`); RU's and TR_AZ's rules fill it where a caller turns them on (`Ivan Ivanovich Ivanov` under RU → middle `Ivanovich` and patronymic `Ivanovich`); an Icelandic ending rule fills it under an `is` pack only, since the default cannot tell `-dóttir` and `-son` from Swedish surnames (ORCID's Swedish records end their family box in `-son` or `-dóttir` in 2,184 of 20,000). A matronymic, written the same way, fills it too. SUPERSEDES: the RFC's earlier statement of decision 8's value, that `family` can be empty for a name with a patronymic. Measured: filers put an Icelandic patronymic in the family box (ORCID 1,647 of 2,583 Icelandic records, PatentsView 535 of 722, PSC 101 of 125), and CLDR's `is` file's 3 Icelandic samples put it in `surname` or `surname-core` in all 9 of their renderings (its one foreign sample puts a `-dóttir` word in `surname2`), while Wikidata's `patronymic-is` and `patronymic-my` strata record 108 names with a patronymic and no family name, of which the parse puts the patronymic in `family` 107 times. An East Slavic patronymic stands in the given part: CLDR's 2 `ru` samples put it in `given2` in all 6 renderings; PatentsView's Russian given boxes end in one 8,423 times of 20,000, and the parse reads such a word as the middle 10,902 times of 10,962; 90 of the 112 Russian nationals in PSC who filled the middle-name box wrote one there. A `family` that keeps it and a `patronymic` that names it agree with both kinds of source. Weighed against: emptying `family` where the patronymic is recognized, which moves the field the fielded sources fill and changes `family` for every Icelandic and nasab name, a behavior change of a frozen property; and a neutral parent-names name for the view, which buys nothing the view can detect, since a matronymic and a patronymic are written alike. 4. **`generation` and `credentials` are views that partition `suffix` (#326)**, not roles, `generation` the tokens read as a generation or an agnomen, `credentials` the rest, the post-nominals that are neither (`Esq.`, the honorifics after a name) falling in `credentials`, CLDR's catch-all, with `Token.reading.kind` telling them apart for a caller who needs it. Measured: of the 1,054 committed corpus entries with a suffix, 33 hold a generation (or agnomen) and a listed credential together, 77 a generation beside any other suffix word, and 64 only listed post-nominals of other kinds (each suffix word classified by the RFC's Phase 4 recipe, joined tokens merged); of the 615 CLDR renderings with either field, the 75 whose suffix holds exactly CLDR's words split into CLDR's `generation` and `credentials` by the vocabulary's reading kind in all 75. Weighed against: two new `Role` members (item 1), and a third view for the honorifics, left open since CLDR has none. 5. **a `surname2` view, the family's second unit where rules/O6 read the family as two**, marked by a new stable tag where O6 takes its word, empty otherwise; `family` keeps both units as O6 makes it, and a family comma alone marks nothing (`Lloyd Webber, Andrew`). Measured: a two-word family box in ORCID's records from Spain 48.0%, Mexico 64.7%, Colombia 72.8% and Chile 50.5%, against 3.9% from the United States. Left open: a view for the first unit and its name, `surname` standing close to the `surnames` view that exists. Declined: redefining `surnames` as the two units, which would repeat `family`, the family holding both where O6 reads two, and lose what `surnames` does today, joining `middle` and `family` so that a double surname the default splits (`Gabriel García Márquez` → middle `García`) reads whole, the view rules/P2 names for that reading. - 6. **The render defaults follow the order the name was read in, a behavior change of the output rather than a break of the API.** `ParsedName` gains `order` (an addition, `ADDED_IN`, a 2.x pickle rendering as today, left out of comparison, which lets two equal names render differently, a cost accepted 2026-10-28: a 2.x pickle of `김민준` renders `'민준 김'` and a fresh parse `'김민준'`, and the two compare equal); the default `str()` and `render()` spec put the family first where `order` does, and join two words that stand one directly after the other, in the same order, in the original and the rendering with what separated them there when that is nothing, whitespace, an interpunct, a nakaguro or one comma with whitespace, the spec's own joiner otherwise (a delimiter, a consumed marker word, a joiner decorating a field, as the `) ` after a maiden name), a comma being ASCII or full-width; `initials()` gives nothing for a word in a script with no initials. A name with a family comma or written given-first keeps today's word order and its separators by the same rule, a suffix comma alone leaving a family-first name family-first (`威廉·莎士比亚` keeps its dot; 20 of the 36 names whose interpunct `str()` drops are read given-first). A family comma is never kept, the rendering reversing its pair (`Smith, John` → `'John Smith'`), and one directly between the family and its suffix is, decided 2026-10-28 (`John Smith, Jr.` → `'John Smith, Jr.'`, today `'John Smith Jr.'`; 214 of the 1,003 committed corpus entries whose suffix follows the family write one there). Accepted with it: `Smith, John, Jr.`, whose suffix the rendering moves away from the given name before it, keeps `'John Smith Jr.'`, so two writings of one name render differently. Measured over the committed corpus's distinct entries: 728 without a comma read family-first, `str()` renders all 728 differently from their text (`김민준` → `'민준 김'`), 150 of them with the family's last word touching the given name's first, and it drops the U+00B7 interpunct from 36 of the 41 holding one. This carries out decisions/3-0-reevaluations' render-default clean slate. No signature changes, but `str()`'s output changes for every name read family-first, every name whose written separator is not a space, and every name with one comma directly between its family and its suffix, and no 2.x release can warn about a `str()` result, so it is taken as an improvement for 3.0's release notes, not a break needing a warning. + 6. **The render defaults follow the order the name was read in, a behavior change of the output rather than a break of the API.** `ParsedName` gains `order` (an addition, `ADDED_IN`, a 2.x pickle rendering as today, left out of comparison, which lets two equal names render differently, a cost accepted 2026-10-28: a 2.x pickle of `김민준` renders `'민준 김'` and a fresh parse `'김민준'`, and the two compare equal); the default `str()` and `render()` spec put the family first where `order` does, and join two words that stand one directly after the other, in the same order, in the original and the rendering with the spec's own joiner where the spec decorates a field between them (the nickname's quotes, the `) ` after a maiden name) or the original separates them by a delimiter or a consumed marker word, and otherwise with what separated them there, nothing, whitespace, an interpunct, a nakaguro or one comma with the whitespace beside it, a comma being ASCII or full-width; `initials()` gives nothing for a word in a script with no initials. A name with a family comma or written given-first keeps today's word order and its separators by the same rule, a suffix comma alone leaving a family-first name family-first (`威廉·莎士比亚` keeps its dot; 20 of the 36 names whose interpunct `str()` drops are read given-first). A family comma is never kept, the rendering reversing its pair (`Smith, John` → `'John Smith'`), and one directly between the family and its suffix is, decided 2026-10-28 (`John Smith, Jr.` → `'John Smith, Jr.'`, today `'John Smith Jr.'`; 214 of the 1,003 committed corpus entries whose suffix follows the family write one there). Accepted with it: `Smith, John, Jr.`, whose suffix the rendering moves away from the given name before it, keeps `'John Smith Jr.'`, so two writings of one name render differently. Measured over the committed corpus's distinct entries, the counts reading the ASCII comma (the RFC gives the full-width ones): 728 without a comma read family-first, `str()` renders all 728 differently from their text (`김민준` → `'민준 김'`), 150 of them with the family's last word touching the given name's first, and it drops the U+00B7 interpunct from 36 of the 41 holding one. This carries out decisions/3-0-reevaluations' render-default clean slate. No signature changes, but `str()`'s output changes for every name read family-first, every name whose written separator is not a space, and every name with one comma directly between its family and its suffix, and no 2.x release can warn about a `str()` result, so it is taken as an improvement for 3.0's release notes, not a break needing a warning. 7. **`middle` keeps its meaning (#704)**, every word of the given part after the given name, the given name one unit unless a rule joins words into it, as rules/P5 already does (`Abdul Aziz Tariq` → given `Abdul Aziz`); a compound given name is a `[rules]` rule a caller or a pack turns on, CLDR's `given2` is `middle`, and `Kaur` and an English name beside a Chinese one stay in `middle`, an accepted cost. Measured: in PSC, whose form has a middle-name box, 2,499 of 1,040,708 forename boxes hold two or more words, and of 6,101 Spanish nationals 25 do while 2,621 fill the middle box; ORCID's and PatentsView's given box holds middles too, so it cannot separate the two. The `Maria do Carmo` shape, a particle inside a given box of three or more words with a family box filled, reads the given box's words into the family in 1,812 of 1,923 ORCID records and 1,165 of 1,219 PatentsView records from Brazil, Portugal, Spain, Mexico, Argentina, Colombia, Chile and Italy, which is that rule's case. Weighed against: #704's option (b), one given unit by default, which changes `middle` for every English name of three words. 8. **A language file's `[rules]` compiles into its shipped `Locale`'s policy patch**, each rule kind a `Policy` field (an addition), so RU and TR_AZ are rebuilt from their files and `Locale` gains no field for rules. With it, **phase 3's open API dependency is answered**: its draft (branch `feat/vocabulary-phase3-design`, unmerged) carries a choice of languages on `Locale.languages`, assuming `Locale` and `Policy` keep their 2.x shape after the facade's removal, and the 2026-10-28 freeze keeps them so. The rest of that draft, strict as an argument of the whole choice included, stays phase 3's to decide; its entry, when it lands, says the dependency is answered. 9. **The work order**, one PR each, measured on its own base, the snapshot recording each new view where it is non-empty: 4a `Token.reading` and `ReadingKind.PATRONYMIC` (neutral; `languages` after 3a); 4b `generation` and `credentials` (neutral); 4c `patronymic` (neutral); 4d `surname2` (neutral); 4e the render defaults and `order`; 4f `[rules]` into the policy patch, with or after 3b; 4g the compound given name, after 4f; 4h the shape inventory per order (#469). Interleaved with phase 3 (its sub-phases the draft's): 4a beside 3a; 4b to 4d before the packs that need them, `cs` (3f) and `pt-BR` (3h) on 4b, `es` and `pt` (3g) on 4d and 4g; an `is` pack on 3b, 4c and 4f. diff --git a/docs/design/vocabulary-model-rfc.md b/docs/design/vocabulary-model-rfc.md index 8d0f1d933..fa98d665a 100644 --- a/docs/design/vocabulary-model-rfc.md +++ b/docs/design/vocabulary-model-rfc.md @@ -250,7 +250,7 @@ ORCID's and PatentsView's given box holds the middle names too, so a two-word gi **Render defaults** (decisions/3-0-reevaluations' clean slate). The default `str()` reads every name given-first and joins its words with spaces, so a name read family-first is turned round and a written separator is lost: `str(parse("김민준"))` is `'민준 김'`, `str(parse("王 小明"))` is `'小明 王'` and `str(parse("威廉·莎士比亚"))` is `'威廉 莎士比亚'`. Over the committed corpus's distinct entries, each under its declared order (2026-10-28, `c8357e74`), 728 without a comma read family-first (the family's first word written before the given name's), `str()` renders every one differently from the text, and in 150 of them the family's last word touches the given name's first, written unspaced; of 41 holding the U+00B7 interpunct, `str()` drops it from 36. So: - `ParsedName` gains `order`, the order the name was read under (its script's or the declared `name_order`, a family comma aside), an addition with `ADDED_IN` metadata whose missing value, in a 2.x pickle, renders as today, and left out of comparison as `Token.reading` is, so a loaded 2.x pickle still equals a fresh parse of its text. The cost, accepted 2026-10-28: two equal names can render differently, where today every rendering is derived from what `==` compares (a 2.x pickle of `김민준` renders `'민준 김'` and a fresh parse `'김민준'`, and a set keeps one of them); the views read from `Token.reading` share it. -- `str()` and `render()`'s default spec put the family first where `order` does, and two words that stand one directly after the other, in the same order, in the original and in the rendering are joined by what separated them there when that is nothing, whitespace, an interpunct, a nakaguro or one comma with whitespace, and by the spec's own joiner otherwise (a delimiter, a marker word the parse consumed, a doubled comma, or a joiner by which the spec decorates a field, as the `) ` after the maiden name, so `Jane Smith née Jones, PhD` keeps `'Jane Smith (Jones) PhD'`; 38 committed entries put a suffix directly after an undelimited maiden name), a comma being ASCII or full-width (U+FF0C), as the parse reads either, so `John (Jack) Smith` keeps today's quoted nickname and `str(parse("김민준"))` is `'김민준'` and `str(parse("威廉·莎士比亚"))` keeps the dot. A name with a family comma or written given-first keeps today's word order, and its separators are kept by the same rule (a comma before a suffix alone does not make a family-first name given-first: `Wang Xiaoming, Jr.` under FAMILY_FIRST renders `'Wang Xiaoming, Jr.'`), so the given-first transcription `威廉·莎士比亚` keeps its dot too (20 of the 36 names whose interpunct `str()` drops are read given-first, one of them with a comma). A family comma is never kept, since the rendering reverses the pair it stands between (`Smith, John` → `'John Smith'`), while a comma directly between the family and its suffix is (`John Smith, Jr.` → `'John Smith, Jr.'`, where `str()` gives `'John Smith Jr.'` today): of the 1,003 committed corpus entries whose suffix starts after the family's last word ends, 214 have nothing between the two but one comma and whitespace (2026-10-28, `c8357e74`, each entry under its declared order; these counts read the ASCII comma, and 7 more write a full-width one there, while 1 of the 728 family-first names above holds a full-width family comma); 456 hold a comma somewhere between, most with a given name or title standing there too. Pairs the rendering separates, as `John` and `Jr.` in `Smith, John, Jr.`, take the spec's own joiner, as today, so `Smith, John, Jr.` keeps rendering `'John Smith Jr.'` while `John Smith, Jr.` becomes `'John Smith, Jr.'`: two writings of one name rendering differently, a consequence of following the text accepted with the choice. +- `str()` and `render()`'s default spec put the family first where `order` does, and two words that stand one directly after the other, in the same order, in the original and in the rendering take the spec's own joiner where the spec decorates a field between them (the nickname's quotes, the `) ` after the maiden name, so `Jane Smith née Jones, PhD` keeps `'Jane Smith (Jones) PhD'`; 38 committed entries put a suffix directly after an undelimited maiden name) or where the original separates them by a delimiter, a marker word the parse consumed or a doubled comma; otherwise they are joined by what separated them in the original, nothing, whitespace, an interpunct, a nakaguro or one comma with whatever whitespace stands beside it, a comma being ASCII or full-width (U+FF0C), as the parse reads either, so `John Smith，Jr.` keeps its comma unspaced as written, so `John (Jack) Smith` keeps today's quoted nickname and `str(parse("김민준"))` is `'김민준'` and `str(parse("威廉·莎士比亚"))` keeps the dot. A name with a family comma or written given-first keeps today's word order, and its separators are kept by the same rule (a comma before a suffix alone does not make a family-first name given-first: `Wang Xiaoming, Jr.` under FAMILY_FIRST renders `'Wang Xiaoming, Jr.'`), so the given-first transcription `威廉·莎士比亚` keeps its dot too (20 of the 36 names whose interpunct `str()` drops are read given-first, one of them with a comma). A family comma is never kept, since the rendering reverses the pair it stands between (`Smith, John` → `'John Smith'`), while a comma directly between the family and its suffix is (`John Smith, Jr.` → `'John Smith, Jr.'`, where `str()` gives `'John Smith Jr.'` today): of the 1,003 committed corpus entries whose suffix starts after the family's last word ends, 214 have nothing between the two but one comma and whitespace (2026-10-28, `c8357e74`, each entry under its declared order; these counts read the ASCII comma, and 7 more write a full-width one there, while 5 of the 728 family-first names above hold a full-width family comma, and so keep today's word order under the rule); 456 hold a comma somewhere between, most with a given name or title standing there too. Pairs the rendering separates, as `John` and `Jr.` in `Smith, John, Jr.`, take the spec's own joiner, as today, so `Smith, John, Jr.` keeps rendering `'John Smith Jr.'` while `John Smith, Jr.` becomes `'John Smith, Jr.'`: two writings of one name rendering differently, a consequence of following the text accepted with the choice. - `initials()` gives nothing for a word in a script with no initials (Han, Hangul, kana, `_policy._NO_INITIALS`), where today it gives the first character with a period (`'王.'`). These change no signature but change `str()`'s output for every name read family-first, every name whose written separator is not a space, and every name written with one comma directly between its family and its suffix, which no 2.x release can warn about; Derek took them as an improvement for 3.0's release notes rather than a break (item 6 of the entry). @@ -306,7 +306,21 @@ Each sub-phase is one PR, measured on its own base as phase 2's and phase 3's ar **Interleaved with phase 3** (its sub-phases 3a to 3h are the draft's on `feat/vocabulary-phase3-design`): 4a can start beside 3a, its `languages` waiting for 3a's index; 4b to 4d need nothing of phase 3 and unblock its packs, `cs`'s generations (3f) and `pt-BR`'s agnomens (3h) on 4b, `es` and `pt` (3g) on 4d and 4g; 4f lands with or after 3b, since both decide what a shipped `Locale` holds; an `is` pack, which no phase 3 sub-phase lists, needs 3b, 4c and 4f. 4e and 4h stand apart from both. So the order is 3a and 4a, then 4b to 4d with 3b and 3c, then 4f, then the packs with 4g, with 4e at any point. -**Left open**, with no sub-phase yet: phase 1's four reserved names said to be phase 4's (`separators` with phase 4's glued words, and `bare_only`, `leading_only` and `trailing_only`, which this RFC's phase 2 section found not needed), whose build messages still name this phase. Value questions inside the shapes above: the first surname's view and its name, with its own particles and base (the gap the render-pattern table shows); how a pack's render pattern is reached, and the key method (Render patterns, above); whether a language pack supplies a convention's entry element or sort key, built from the per-surname units this phase reads, `family_base` being the family without its working particles under every convention (an all-particle part's words staying, rules/R2's Except) (decisions/R2's 2026-10-28 #754 entry): a `pt` or `pt-BR` pack would give the last surname's base first, keeping `Filho`, `Neto` and `Júnior` with it and a compound surname whole, and an `es` pack the first surname without its particle, then the second with its own (RAE's rules (a) and (e)); whether a token's title kind should reach callers: today a title token's tags name only behaviors (`Dr. Mr. Sir John Smith` gives `Dr.` `vocab:title`, `Mr.` `vocab:after-given-title` and `vocab:title`, `Sir` `vocab:given-title` and `vocab:title`), `Lexicon` keeps the titles as one set beside `title_modifiers` and `title_styles`, the vocabulary's salutation, academic, religious, nobility, military, office, occupation and kinship kinds live only in the TOML, and `ReadingKind`, being "the parse's, not the vocabulary's", reads every one as `TITLE` where the suffix readings split four ways; so whether `Reading` gains an optional field holding the vocabulary's kind, filled only where the reading's source is the vocabulary (a title read by its shape, `Insp.`, or a word a caller added by `Lexicon.add` has none, so its absence means unknown, not that the word has none), and so whether phase 3's reading index (its "Where the provenance lives at runtime", on `feat/vocabulary-phase3-design`) carries the kinds as well as the languages and properties. It is an addition and blocks nothing in 3.0; it would let a view name the salutation alone (`Dear Mr. Smith`), which CLDR's one `title` field does not model. Whether it holds one kind or a set is part of the question: of the 816 title words in nameparser/vocab/ on 2026-10-28 none carries two title kinds, `father` being `religious` only, the README's "kinship word used as one"; whether `Lexicon` keeps exposing each compiled set as a public field or answers word, kind and property queries from phase 3's reading index, its sets kept internal, a SHAPE question to settle before the 3.0 freeze makes every field permanent: `dataclasses.fields(Lexicon)` gives 22 public fields on 2026-10-28, 11 of them added since 2.0.0's 11 (`surnames`, `honorific_tails`, `conjunctions_ambiguous`, `after_given_titles`, `suffix_words_ambiguous`, `agnomens`, `given_name_abbreviations`, `titles_ambiguous`, `title_modifiers`, `title_styles`, `title_phrases`), each a new parse behavior's set, and a rule keyed on a title kind or on one language's reading of a kind (phase 3's draft, "a property a language's own reading should take only under its pack") would add more. The costs on each side: every field is frozen API, pickled with the `Lexicon` (`_guarded_setstate` refuses a pickle whose fields differ unless a field carries `ADDED_IN`), and shown in its bounded repr (`Lexicon(default + titles: +1)`), while turning existing fields internal is a break that would have to land in 3.0, where adding a query beside the fields is an addition that can come later; a caller configures vocabulary today by the field (`Lexicon.add(titles=...)`, `dataclasses.replace`), and a query-based `Lexicon` needs a way in for a caller's word, which has no kind and no language; the parser need not pay either way, a frozenset test per word costing no frame and an internal cache keeping that path, while a query method called per word would add one per token, the cost the frame budget counts as scaling with the input; whether `credentials` should leave the honorifics out, CLDR having no third bucket; which ending rules beside Icelandic a pack may name (Hungarian `-né` is a married name, not a patronymic); and the names of given names only (the last table row), whose reading needs a rule that tells a mononym from a given name and a family name, not an output field. +**Left open.** With no sub-phase yet: phase 1's four reserved names said to be phase 4's (`separators` with phase 4's glued words, and `bare_only`, `leading_only` and `trailing_only`, which this RFC's phase 2 section found not needed), whose build messages still name this phase. + +A shape question, to settle before the 3.0 freeze makes every field permanent: + +- **Whether `Lexicon` keeps a public field for each compiled set**, or answers word, kind and property queries from phase 3's reading index and keeps its sets internal. `dataclasses.fields(Lexicon)` gives 22 public fields on 2026-10-28, 11 of them added since 2.0.0's 11 (`surnames`, `honorific_tails`, `conjunctions_ambiguous`, `after_given_titles`, `suffix_words_ambiguous`, `agnomens`, `given_name_abbreviations`, `titles_ambiguous`, `title_modifiers`, `title_styles`, `title_phrases`), each a new parse behavior's set, and a rule keyed on a title kind or on one language's reading of a kind (phase 3's draft, "a property a language's own reading should take only under its pack") would add more. Keeping the fields: each is frozen API, shown in the bounded repr (`Lexicon(default + titles: +1)`), and pickled, where `Lexicon`'s own `__setstate__`, unlike the `ADDED_IN` guard the other types share, refuses any pickle whose fields differ, so every new field breaks every older `Lexicon` pickle, and with it every pickled `Parser` and locale pack, until that guard learns the marker. Turning them internal: a break that has to land in 3.0, where a query added beside the fields is an addition that can come later; a caller who configures vocabulary by the field today (`Lexicon.add(titles=...)`, `dataclasses.replace`) needs another way in, for words that carry no kind and no language; and phase 3's draft loads the reading index only when a caller chooses a language (about 11 ms to execute), so a `Lexicon` answering from it loads it for every caller. The parser need not pay either way: a frozenset test per word costs no frame and an internal cache keeps that path, while a query method called per word would add one per token, the cost the frame budget counts as scaling with the input. + +Value questions inside the shapes above: + +- The first surname's view and its name, with its own particles and base (the gap the render-pattern table shows). +- How a pack's render pattern is reached, and the key method (Render patterns, above). +- Whether a language pack supplies a convention's entry element or sort key, built from the per-surname units this phase reads, `family_base` being the family without its working particles under every convention (an all-particle part's words staying, rules/R2's Except; decisions/R2's 2026-10-28 #754 entry). A `pt` or `pt-BR` pack would give the last surname's base first, keeping `Filho`, `Neto` and `Júnior` with it and a compound surname whole, and an `es` pack the first surname without its particle, then the second with its own (RAE's rules (a) and (e)). +- Whether a token's title kind reaches callers. A title token's tags name only behaviors (`Dr. Mr. Sir John Smith` gives `Dr.` `vocab:title`, `Mr.` `vocab:after-given-title` and `vocab:title`, `Sir` `vocab:given-title` and `vocab:title`); `Lexicon` splits the titles into `titles` and `titles_ambiguous` by whether a word is also a name, not by kind, and only `title_modifiers` and `title_styles` are sets of a kind; the salutation, academic, religious, nobility, military, office, occupation and kinship kinds live only in the TOML; and `ReadingKind`, being "the parse's, not the vocabulary's", would read every one as `TITLE` in the planned `Token.reading` (today these tokens carry no reading), where the suffix readings split four ways. Phase 3's draft already gives its reading index "its kind" per row ("Where the provenance lives at runtime", on `feat/vocabulary-phase3-design`), so what stays open is whether that is the vocabulary's kind and whether `Reading` gains an optional field exposing it, filled only where the reading's source is the vocabulary: a title read by its shape (`Insp.`) or a word a caller added by `Lexicon.add` has none, so its absence means unknown, not that the word has none. It is an addition and blocks nothing in 3.0, and it would let a view name the salutation alone (`Dear Mr. Smith`), which CLDR's one `title` field does not model. Whether it holds one kind or a set is part of the question: of the 816 title words in nameparser/vocab/ on 2026-10-28 none carries two title kinds, `father` being `religious` only, the README's "kinship word used as one". +- Whether `credentials` should leave the honorifics out, CLDR having no third bucket. +- Which ending rules beside Icelandic a pack may name (Hungarian `-né` is a married name, not a patronymic). +- The names of given names only (the last table row), whose reading needs a rule that tells a mononym from a given name and a family name, not an output field. ## The first packs From 8f2bbc54c87a01dfad745dbebd316a95b6172c49 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Wed, 28 Oct 2026 13:56:05 -0700 Subject: [PATCH 20/39] Fix the sixth docs review's findings on phase 4's render rule (#754) The rule now covers every separator: kept ones (nothing, whitespace, interpunct, nakaguro, one comma of any kind the parse reads, Arabic included), the spec's decoration between two fields standing instead, and the spec's joiner for everything else, in both copies. 36 of the 728 hold a family comma the parse reads (5 full-width, 31 Arabic). The index load cost carries its no-cache qualifier, and the title-kind item narrows to whether Reading exposes the vocabulary's kind. Co-Authored-By: Claude Opus 5.5 --- docs/design/decisions/vocabulary-model.md | 2 +- docs/design/vocabulary-model-rfc.md | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/design/decisions/vocabulary-model.md b/docs/design/decisions/vocabulary-model.md index f7bf8bb4f..48f29e0cb 100644 --- a/docs/design/decisions/vocabulary-model.md +++ b/docs/design/decisions/vocabulary-model.md @@ -250,7 +250,7 @@ Open: #913 (the title words phase 2j left, read as titles though borne as names) 3. **A `patronymic` view, filled by a word or a rule that says patronymic, with `family` and `middle` unchanged.** The default fills it from the vocabulary's `patronymic` particles (`bin`, `binti`, `ibn`, `bint`, `bar`, `bat` and the native-script forms) with the name they introduce (`Mahathir bin Mohamad` → family `bin Mohamad`, as today, and patronymic `bin Mohamad`); RU's and TR_AZ's rules fill it where a caller turns them on (`Ivan Ivanovich Ivanov` under RU → middle `Ivanovich` and patronymic `Ivanovich`); an Icelandic ending rule fills it under an `is` pack only, since the default cannot tell `-dóttir` and `-son` from Swedish surnames (ORCID's Swedish records end their family box in `-son` or `-dóttir` in 2,184 of 20,000). A matronymic, written the same way, fills it too. SUPERSEDES: the RFC's earlier statement of decision 8's value, that `family` can be empty for a name with a patronymic. Measured: filers put an Icelandic patronymic in the family box (ORCID 1,647 of 2,583 Icelandic records, PatentsView 535 of 722, PSC 101 of 125), and CLDR's `is` file's 3 Icelandic samples put it in `surname` or `surname-core` in all 9 of their renderings (its one foreign sample puts a `-dóttir` word in `surname2`), while Wikidata's `patronymic-is` and `patronymic-my` strata record 108 names with a patronymic and no family name, of which the parse puts the patronymic in `family` 107 times. An East Slavic patronymic stands in the given part: CLDR's 2 `ru` samples put it in `given2` in all 6 renderings; PatentsView's Russian given boxes end in one 8,423 times of 20,000, and the parse reads such a word as the middle 10,902 times of 10,962; 90 of the 112 Russian nationals in PSC who filled the middle-name box wrote one there. A `family` that keeps it and a `patronymic` that names it agree with both kinds of source. Weighed against: emptying `family` where the patronymic is recognized, which moves the field the fielded sources fill and changes `family` for every Icelandic and nasab name, a behavior change of a frozen property; and a neutral parent-names name for the view, which buys nothing the view can detect, since a matronymic and a patronymic are written alike. 4. **`generation` and `credentials` are views that partition `suffix` (#326)**, not roles, `generation` the tokens read as a generation or an agnomen, `credentials` the rest, the post-nominals that are neither (`Esq.`, the honorifics after a name) falling in `credentials`, CLDR's catch-all, with `Token.reading.kind` telling them apart for a caller who needs it. Measured: of the 1,054 committed corpus entries with a suffix, 33 hold a generation (or agnomen) and a listed credential together, 77 a generation beside any other suffix word, and 64 only listed post-nominals of other kinds (each suffix word classified by the RFC's Phase 4 recipe, joined tokens merged); of the 615 CLDR renderings with either field, the 75 whose suffix holds exactly CLDR's words split into CLDR's `generation` and `credentials` by the vocabulary's reading kind in all 75. Weighed against: two new `Role` members (item 1), and a third view for the honorifics, left open since CLDR has none. 5. **a `surname2` view, the family's second unit where rules/O6 read the family as two**, marked by a new stable tag where O6 takes its word, empty otherwise; `family` keeps both units as O6 makes it, and a family comma alone marks nothing (`Lloyd Webber, Andrew`). Measured: a two-word family box in ORCID's records from Spain 48.0%, Mexico 64.7%, Colombia 72.8% and Chile 50.5%, against 3.9% from the United States. Left open: a view for the first unit and its name, `surname` standing close to the `surnames` view that exists. Declined: redefining `surnames` as the two units, which would repeat `family`, the family holding both where O6 reads two, and lose what `surnames` does today, joining `middle` and `family` so that a double surname the default splits (`Gabriel García Márquez` → middle `García`) reads whole, the view rules/P2 names for that reading. - 6. **The render defaults follow the order the name was read in, a behavior change of the output rather than a break of the API.** `ParsedName` gains `order` (an addition, `ADDED_IN`, a 2.x pickle rendering as today, left out of comparison, which lets two equal names render differently, a cost accepted 2026-10-28: a 2.x pickle of `김민준` renders `'민준 김'` and a fresh parse `'김민준'`, and the two compare equal); the default `str()` and `render()` spec put the family first where `order` does, and join two words that stand one directly after the other, in the same order, in the original and the rendering with the spec's own joiner where the spec decorates a field between them (the nickname's quotes, the `) ` after a maiden name) or the original separates them by a delimiter or a consumed marker word, and otherwise with what separated them there, nothing, whitespace, an interpunct, a nakaguro or one comma with the whitespace beside it, a comma being ASCII or full-width; `initials()` gives nothing for a word in a script with no initials. A name with a family comma or written given-first keeps today's word order and its separators by the same rule, a suffix comma alone leaving a family-first name family-first (`威廉·莎士比亚` keeps its dot; 20 of the 36 names whose interpunct `str()` drops are read given-first). A family comma is never kept, the rendering reversing its pair (`Smith, John` → `'John Smith'`), and one directly between the family and its suffix is, decided 2026-10-28 (`John Smith, Jr.` → `'John Smith, Jr.'`, today `'John Smith Jr.'`; 214 of the 1,003 committed corpus entries whose suffix follows the family write one there). Accepted with it: `Smith, John, Jr.`, whose suffix the rendering moves away from the given name before it, keeps `'John Smith Jr.'`, so two writings of one name render differently. Measured over the committed corpus's distinct entries, the counts reading the ASCII comma (the RFC gives the full-width ones): 728 without a comma read family-first, `str()` renders all 728 differently from their text (`김민준` → `'민준 김'`), 150 of them with the family's last word touching the given name's first, and it drops the U+00B7 interpunct from 36 of the 41 holding one. This carries out decisions/3-0-reevaluations' render-default clean slate. No signature changes, but `str()`'s output changes for every name read family-first, every name whose written separator is not a space, and every name with one comma directly between its family and its suffix, and no 2.x release can warn about a `str()` result, so it is taken as an improvement for 3.0's release notes, not a break needing a warning. + 6. **The render defaults follow the order the name was read in, a behavior change of the output rather than a break of the API.** `ParsedName` gains `order` (an addition, `ADDED_IN`, a 2.x pickle rendering as today, left out of comparison, which lets two equal names render differently, a cost accepted 2026-10-28: a 2.x pickle of `김민준` renders `'민준 김'` and a fresh parse `'김민준'`, and the two compare equal); the default `str()` and `render()` spec put the family first where `order` does, and join two words that stand one directly after the other, in the same order, in the original and the rendering with what separated them there when it is nothing, whitespace, an interpunct, a nakaguro, or one comma with the whitespace beside it (any comma the parse reads: ASCII, full-width, Arabic), the spec's decoration between the two words' fields standing instead (the nickname's quotes, the `) ` after a maiden name), and any other pair or separator (a delimiter, a consumed marker word, two or more commas, a hyphen, a slash, a zero-width mark) taking the spec's own joiner; `initials()` gives nothing for a word in a script with no initials. A name with a family comma or written given-first keeps today's word order and its separators by the same rule, a suffix comma alone leaving a family-first name family-first (`威廉·莎士比亚` keeps its dot; 20 of the 36 names whose interpunct `str()` drops are read given-first). A family comma is never kept, the rendering reversing its pair (`Smith, John` → `'John Smith'`), and one directly between the family and its suffix is, decided 2026-10-28 (`John Smith, Jr.` → `'John Smith, Jr.'`, today `'John Smith Jr.'`; 214 of the 1,003 committed corpus entries whose suffix follows the family write one there). Accepted with it: `Smith, John, Jr.`, whose suffix the rendering moves away from the given name before it, keeps `'John Smith Jr.'`, so two writings of one name render differently. Measured over the committed corpus's distinct entries, the counts reading the ASCII comma (the RFC gives the full-width ones): 728 without a comma read family-first, `str()` renders all 728 differently from their text (`김민준` → `'민준 김'`), 150 of them with the family's last word touching the given name's first, and it drops the U+00B7 interpunct from 36 of the 41 holding one. This carries out decisions/3-0-reevaluations' render-default clean slate. No signature changes, but `str()`'s output changes for every name read family-first, every name whose written separator is not a space, and every name with one comma directly between its family and its suffix, and no 2.x release can warn about a `str()` result, so it is taken as an improvement for 3.0's release notes, not a break needing a warning. 7. **`middle` keeps its meaning (#704)**, every word of the given part after the given name, the given name one unit unless a rule joins words into it, as rules/P5 already does (`Abdul Aziz Tariq` → given `Abdul Aziz`); a compound given name is a `[rules]` rule a caller or a pack turns on, CLDR's `given2` is `middle`, and `Kaur` and an English name beside a Chinese one stay in `middle`, an accepted cost. Measured: in PSC, whose form has a middle-name box, 2,499 of 1,040,708 forename boxes hold two or more words, and of 6,101 Spanish nationals 25 do while 2,621 fill the middle box; ORCID's and PatentsView's given box holds middles too, so it cannot separate the two. The `Maria do Carmo` shape, a particle inside a given box of three or more words with a family box filled, reads the given box's words into the family in 1,812 of 1,923 ORCID records and 1,165 of 1,219 PatentsView records from Brazil, Portugal, Spain, Mexico, Argentina, Colombia, Chile and Italy, which is that rule's case. Weighed against: #704's option (b), one given unit by default, which changes `middle` for every English name of three words. 8. **A language file's `[rules]` compiles into its shipped `Locale`'s policy patch**, each rule kind a `Policy` field (an addition), so RU and TR_AZ are rebuilt from their files and `Locale` gains no field for rules. With it, **phase 3's open API dependency is answered**: its draft (branch `feat/vocabulary-phase3-design`, unmerged) carries a choice of languages on `Locale.languages`, assuming `Locale` and `Policy` keep their 2.x shape after the facade's removal, and the 2026-10-28 freeze keeps them so. The rest of that draft, strict as an argument of the whole choice included, stays phase 3's to decide; its entry, when it lands, says the dependency is answered. 9. **The work order**, one PR each, measured on its own base, the snapshot recording each new view where it is non-empty: 4a `Token.reading` and `ReadingKind.PATRONYMIC` (neutral; `languages` after 3a); 4b `generation` and `credentials` (neutral); 4c `patronymic` (neutral); 4d `surname2` (neutral); 4e the render defaults and `order`; 4f `[rules]` into the policy patch, with or after 3b; 4g the compound given name, after 4f; 4h the shape inventory per order (#469). Interleaved with phase 3 (its sub-phases the draft's): 4a beside 3a; 4b to 4d before the packs that need them, `cs` (3f) and `pt-BR` (3h) on 4b, `es` and `pt` (3g) on 4d and 4g; an `is` pack on 3b, 4c and 4f. diff --git a/docs/design/vocabulary-model-rfc.md b/docs/design/vocabulary-model-rfc.md index fa98d665a..1b5f2cb1c 100644 --- a/docs/design/vocabulary-model-rfc.md +++ b/docs/design/vocabulary-model-rfc.md @@ -250,7 +250,7 @@ ORCID's and PatentsView's given box holds the middle names too, so a two-word gi **Render defaults** (decisions/3-0-reevaluations' clean slate). The default `str()` reads every name given-first and joins its words with spaces, so a name read family-first is turned round and a written separator is lost: `str(parse("김민준"))` is `'민준 김'`, `str(parse("王 小明"))` is `'小明 王'` and `str(parse("威廉·莎士比亚"))` is `'威廉 莎士比亚'`. Over the committed corpus's distinct entries, each under its declared order (2026-10-28, `c8357e74`), 728 without a comma read family-first (the family's first word written before the given name's), `str()` renders every one differently from the text, and in 150 of them the family's last word touches the given name's first, written unspaced; of 41 holding the U+00B7 interpunct, `str()` drops it from 36. So: - `ParsedName` gains `order`, the order the name was read under (its script's or the declared `name_order`, a family comma aside), an addition with `ADDED_IN` metadata whose missing value, in a 2.x pickle, renders as today, and left out of comparison as `Token.reading` is, so a loaded 2.x pickle still equals a fresh parse of its text. The cost, accepted 2026-10-28: two equal names can render differently, where today every rendering is derived from what `==` compares (a 2.x pickle of `김민준` renders `'민준 김'` and a fresh parse `'김민준'`, and a set keeps one of them); the views read from `Token.reading` share it. -- `str()` and `render()`'s default spec put the family first where `order` does, and two words that stand one directly after the other, in the same order, in the original and in the rendering take the spec's own joiner where the spec decorates a field between them (the nickname's quotes, the `) ` after the maiden name, so `Jane Smith née Jones, PhD` keeps `'Jane Smith (Jones) PhD'`; 38 committed entries put a suffix directly after an undelimited maiden name) or where the original separates them by a delimiter, a marker word the parse consumed or a doubled comma; otherwise they are joined by what separated them in the original, nothing, whitespace, an interpunct, a nakaguro or one comma with whatever whitespace stands beside it, a comma being ASCII or full-width (U+FF0C), as the parse reads either, so `John Smith，Jr.` keeps its comma unspaced as written, so `John (Jack) Smith` keeps today's quoted nickname and `str(parse("김민준"))` is `'김민준'` and `str(parse("威廉·莎士比亚"))` keeps the dot. A name with a family comma or written given-first keeps today's word order, and its separators are kept by the same rule (a comma before a suffix alone does not make a family-first name given-first: `Wang Xiaoming, Jr.` under FAMILY_FIRST renders `'Wang Xiaoming, Jr.'`), so the given-first transcription `威廉·莎士比亚` keeps its dot too (20 of the 36 names whose interpunct `str()` drops are read given-first, one of them with a comma). A family comma is never kept, since the rendering reverses the pair it stands between (`Smith, John` → `'John Smith'`), while a comma directly between the family and its suffix is (`John Smith, Jr.` → `'John Smith, Jr.'`, where `str()` gives `'John Smith Jr.'` today): of the 1,003 committed corpus entries whose suffix starts after the family's last word ends, 214 have nothing between the two but one comma and whitespace (2026-10-28, `c8357e74`, each entry under its declared order; these counts read the ASCII comma, and 7 more write a full-width one there, while 5 of the 728 family-first names above hold a full-width family comma, and so keep today's word order under the rule); 456 hold a comma somewhere between, most with a given name or title standing there too. Pairs the rendering separates, as `John` and `Jr.` in `Smith, John, Jr.`, take the spec's own joiner, as today, so `Smith, John, Jr.` keeps rendering `'John Smith Jr.'` while `John Smith, Jr.` becomes `'John Smith, Jr.'`: two writings of one name rendering differently, a consequence of following the text accepted with the choice. +- `str()` and `render()`'s default spec put the family first where `order` does, and two words that stand one directly after the other, in the same order, in the original and in the rendering keep what separated them in the original when it is nothing, whitespace, an interpunct, a nakaguro, or one comma with whatever whitespace stands beside it, a comma being any the parse reads as one (ASCII, full-width U+FF0C, Arabic U+060C), so `John Smith，Jr.` keeps its comma unspaced as written; the spec's decoration between the two fields the words belong to stands instead (the nickname's quotes, the `) ` after the maiden name, so `Jane Smith née Jones, PhD` keeps `'Jane Smith (Jones) PhD'`; 38 committed entries put a suffix directly after an undelimited maiden name); and any other pair, or any other separator (a delimiter, a marker word the parse consumed, two or more commas, a hyphen, a slash, a zero-width or directional mark), takes the spec's own joiner, as today, so `John (Jack) Smith` keeps today's quoted nickname and `str(parse("김민준"))` is `'김민준'` and `str(parse("威廉·莎士比亚"))` keeps the dot. A name with a family comma or written given-first keeps today's word order, and its separators are kept by the same rule (a comma before a suffix alone does not make a family-first name given-first: `Wang Xiaoming, Jr.` under FAMILY_FIRST renders `'Wang Xiaoming, Jr.'`), so the given-first transcription `威廉·莎士比亚` keeps its dot too (20 of the 36 names whose interpunct `str()` drops are read given-first, one of them with a comma). A family comma is never kept, since the rendering reverses the pair it stands between (`Smith, John` → `'John Smith'`), while a comma directly between the family and its suffix is (`John Smith, Jr.` → `'John Smith, Jr.'`, where `str()` gives `'John Smith Jr.'` today): of the 1,003 committed corpus entries whose suffix starts after the family's last word ends, 214 have nothing between the two but one comma and whitespace (2026-10-28, `c8357e74`, each entry under its declared order; these counts read the ASCII comma, and 7 more write a full-width one there, while 36 of the 728 family-first names above hold a family comma the parse reads, 5 full-width and 31 Arabic (30 of the Arabic in `corpus_cldr.jsonl`), and so keep today's word order under the rule); 456 hold a comma somewhere between, most with a given name or title standing there too. Pairs the rendering separates, as `John` and `Jr.` in `Smith, John, Jr.`, take the spec's own joiner, as today, so `Smith, John, Jr.` keeps rendering `'John Smith Jr.'` while `John Smith, Jr.` becomes `'John Smith, Jr.'`: two writings of one name rendering differently, a consequence of following the text accepted with the choice. - `initials()` gives nothing for a word in a script with no initials (Han, Hangul, kana, `_policy._NO_INITIALS`), where today it gives the first character with a period (`'王.'`). These change no signature but change `str()`'s output for every name read family-first, every name whose written separator is not a space, and every name written with one comma directly between its family and its suffix, which no 2.x release can warn about; Derek took them as an improvement for 3.0's release notes rather than a break (item 6 of the entry). @@ -310,14 +310,14 @@ Each sub-phase is one PR, measured on its own base as phase 2's and phase 3's ar A shape question, to settle before the 3.0 freeze makes every field permanent: -- **Whether `Lexicon` keeps a public field for each compiled set**, or answers word, kind and property queries from phase 3's reading index and keeps its sets internal. `dataclasses.fields(Lexicon)` gives 22 public fields on 2026-10-28, 11 of them added since 2.0.0's 11 (`surnames`, `honorific_tails`, `conjunctions_ambiguous`, `after_given_titles`, `suffix_words_ambiguous`, `agnomens`, `given_name_abbreviations`, `titles_ambiguous`, `title_modifiers`, `title_styles`, `title_phrases`), each a new parse behavior's set, and a rule keyed on a title kind or on one language's reading of a kind (phase 3's draft, "a property a language's own reading should take only under its pack") would add more. Keeping the fields: each is frozen API, shown in the bounded repr (`Lexicon(default + titles: +1)`), and pickled, where `Lexicon`'s own `__setstate__`, unlike the `ADDED_IN` guard the other types share, refuses any pickle whose fields differ, so every new field breaks every older `Lexicon` pickle, and with it every pickled `Parser` and locale pack, until that guard learns the marker. Turning them internal: a break that has to land in 3.0, where a query added beside the fields is an addition that can come later; a caller who configures vocabulary by the field today (`Lexicon.add(titles=...)`, `dataclasses.replace`) needs another way in, for words that carry no kind and no language; and phase 3's draft loads the reading index only when a caller chooses a language (about 11 ms to execute), so a `Lexicon` answering from it loads it for every caller. The parser need not pay either way: a frozenset test per word costs no frame and an internal cache keeps that path, while a query method called per word would add one per token, the cost the frame budget counts as scaling with the input. +- **Whether `Lexicon` keeps a public field for each compiled set**, or answers word, kind and property queries from phase 3's reading index and keeps its sets internal. `dataclasses.fields(Lexicon)` gives 22 public fields on 2026-10-28, 11 of them added since 2.0.0's 11 (`surnames`, `honorific_tails`, `conjunctions_ambiguous`, `after_given_titles`, `suffix_words_ambiguous`, `agnomens`, `given_name_abbreviations`, `titles_ambiguous`, `title_modifiers`, `title_styles`, `title_phrases`), each a new parse behavior's set, and a rule keyed on a title kind or on one language's reading of a kind (phase 3's draft, "a property a language's own reading should take only under its pack") would add more. Keeping the fields: each is frozen API, shown in the bounded repr (`Lexicon(default + titles: +1)`), and pickled, where `Lexicon`'s own `__setstate__`, unlike the `ADDED_IN` guard the other types share, refuses any pickle whose fields differ, so every new field breaks every older `Lexicon` pickle, and with it every pickled `Parser` and locale pack, until that guard learns the marker. Turning them internal: a break that has to land in 3.0, where a query added beside the fields is an addition that can come later; a caller who configures vocabulary by the field today (`Lexicon.add(titles=...)`, `dataclasses.replace`) needs another way in, for words that carry no kind and no language; and phase 3's draft loads the reading index only when a caller chooses a language (about 11 ms to execute with no bytecode cache, by the draft's timing; the per-word index this RFC weighs for `Reading.languages` above costs about 0.4 ms of a 19.5 ms import), so a `Lexicon` answering from it loads it for every caller. The parser need not pay either way: a frozenset test per word costs no frame and an internal cache keeps that path, while a query method called per word would add one per token, the cost the frame budget counts as scaling with the input. Value questions inside the shapes above: - The first surname's view and its name, with its own particles and base (the gap the render-pattern table shows). - How a pack's render pattern is reached, and the key method (Render patterns, above). - Whether a language pack supplies a convention's entry element or sort key, built from the per-surname units this phase reads, `family_base` being the family without its working particles under every convention (an all-particle part's words staying, rules/R2's Except; decisions/R2's 2026-10-28 #754 entry). A `pt` or `pt-BR` pack would give the last surname's base first, keeping `Filho`, `Neto` and `Júnior` with it and a compound surname whole, and an `es` pack the first surname without its particle, then the second with its own (RAE's rules (a) and (e)). -- Whether a token's title kind reaches callers. A title token's tags name only behaviors (`Dr. Mr. Sir John Smith` gives `Dr.` `vocab:title`, `Mr.` `vocab:after-given-title` and `vocab:title`, `Sir` `vocab:given-title` and `vocab:title`); `Lexicon` splits the titles into `titles` and `titles_ambiguous` by whether a word is also a name, not by kind, and only `title_modifiers` and `title_styles` are sets of a kind; the salutation, academic, religious, nobility, military, office, occupation and kinship kinds live only in the TOML; and `ReadingKind`, being "the parse's, not the vocabulary's", would read every one as `TITLE` in the planned `Token.reading` (today these tokens carry no reading), where the suffix readings split four ways. Phase 3's draft already gives its reading index "its kind" per row ("Where the provenance lives at runtime", on `feat/vocabulary-phase3-design`), so what stays open is whether that is the vocabulary's kind and whether `Reading` gains an optional field exposing it, filled only where the reading's source is the vocabulary: a title read by its shape (`Insp.`) or a word a caller added by `Lexicon.add` has none, so its absence means unknown, not that the word has none. It is an addition and blocks nothing in 3.0, and it would let a view name the salutation alone (`Dear Mr. Smith`), which CLDR's one `title` field does not model. Whether it holds one kind or a set is part of the question: of the 816 title words in nameparser/vocab/ on 2026-10-28 none carries two title kinds, `father` being `religious` only, the README's "kinship word used as one". +- Whether a token's title kind reaches callers. A title token's tags name only behaviors (`Dr. Mr. Sir John Smith` gives `Dr.` `vocab:title`, `Mr.` `vocab:after-given-title` and `vocab:title`, `Sir` `vocab:given-title` and `vocab:title`); `Lexicon` splits the titles into `titles` and `titles_ambiguous` by whether a word is also a name, not by kind, and only `title_modifiers` and `title_styles` are sets of a kind; the salutation, academic, religious, nobility, military, office, occupation and kinship kinds live only in the TOML; and `ReadingKind`, being "the parse's, not the vocabulary's", would read every one as `TITLE` in the planned `Token.reading` (today these tokens carry no reading), where the suffix readings split four ways. Phase 3's draft already gives its reading index "its kind" per row ("Where the provenance lives at runtime", on `feat/vocabulary-phase3-design`), and a title reading is keyed by its kind in the TOML (`religious`, `academic`), so that row's kind is the vocabulary's by construction; what stays open is whether `Reading` gains an optional field exposing it, filled only where the reading's source is the vocabulary: a title read by its shape (`Insp.`) or a word a caller added by `Lexicon.add` has none, so its absence means unknown, not that the word has none. It is an addition and blocks nothing in 3.0, and it would let a view name the salutation alone (`Dear Mr. Smith`), which CLDR's one `title` field does not model. Whether it holds one kind or a set is part of the question: of the 816 title words in nameparser/vocab/ on 2026-10-28 none carries two title kinds, `father` being `religious` only, the README's "kinship word used as one". - Whether `credentials` should leave the honorifics out, CLDR having no third bucket. - Which ending rules beside Icelandic a pack may name (Hungarian `-né` is a married name, not a patronymic). - The names of given names only (the last table row), whose reading needs a rule that tells a mononym from a given name and a family name, not an output field. From a411862111258c430e2d123ec6eea4e1ca496995 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Wed, 28 Oct 2026 13:58:55 -0700 Subject: [PATCH 21/39] Normalize the separators phase 4's render default keeps (#754) A kept separator keeps its kind, not its written whitespace: any whitespace becomes one space, an interpunct or nakaguro stands alone, an ASCII or Arabic comma becomes comma-space, and a full-width comma stands alone. The written whitespace stays in ParsedName.original and the token spans. Co-Authored-By: Claude Opus 5.5 --- docs/design/decisions/vocabulary-model.md | 2 +- docs/design/vocabulary-model-rfc.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/design/decisions/vocabulary-model.md b/docs/design/decisions/vocabulary-model.md index 48f29e0cb..9ea47dfff 100644 --- a/docs/design/decisions/vocabulary-model.md +++ b/docs/design/decisions/vocabulary-model.md @@ -250,7 +250,7 @@ Open: #913 (the title words phase 2j left, read as titles though borne as names) 3. **A `patronymic` view, filled by a word or a rule that says patronymic, with `family` and `middle` unchanged.** The default fills it from the vocabulary's `patronymic` particles (`bin`, `binti`, `ibn`, `bint`, `bar`, `bat` and the native-script forms) with the name they introduce (`Mahathir bin Mohamad` → family `bin Mohamad`, as today, and patronymic `bin Mohamad`); RU's and TR_AZ's rules fill it where a caller turns them on (`Ivan Ivanovich Ivanov` under RU → middle `Ivanovich` and patronymic `Ivanovich`); an Icelandic ending rule fills it under an `is` pack only, since the default cannot tell `-dóttir` and `-son` from Swedish surnames (ORCID's Swedish records end their family box in `-son` or `-dóttir` in 2,184 of 20,000). A matronymic, written the same way, fills it too. SUPERSEDES: the RFC's earlier statement of decision 8's value, that `family` can be empty for a name with a patronymic. Measured: filers put an Icelandic patronymic in the family box (ORCID 1,647 of 2,583 Icelandic records, PatentsView 535 of 722, PSC 101 of 125), and CLDR's `is` file's 3 Icelandic samples put it in `surname` or `surname-core` in all 9 of their renderings (its one foreign sample puts a `-dóttir` word in `surname2`), while Wikidata's `patronymic-is` and `patronymic-my` strata record 108 names with a patronymic and no family name, of which the parse puts the patronymic in `family` 107 times. An East Slavic patronymic stands in the given part: CLDR's 2 `ru` samples put it in `given2` in all 6 renderings; PatentsView's Russian given boxes end in one 8,423 times of 20,000, and the parse reads such a word as the middle 10,902 times of 10,962; 90 of the 112 Russian nationals in PSC who filled the middle-name box wrote one there. A `family` that keeps it and a `patronymic` that names it agree with both kinds of source. Weighed against: emptying `family` where the patronymic is recognized, which moves the field the fielded sources fill and changes `family` for every Icelandic and nasab name, a behavior change of a frozen property; and a neutral parent-names name for the view, which buys nothing the view can detect, since a matronymic and a patronymic are written alike. 4. **`generation` and `credentials` are views that partition `suffix` (#326)**, not roles, `generation` the tokens read as a generation or an agnomen, `credentials` the rest, the post-nominals that are neither (`Esq.`, the honorifics after a name) falling in `credentials`, CLDR's catch-all, with `Token.reading.kind` telling them apart for a caller who needs it. Measured: of the 1,054 committed corpus entries with a suffix, 33 hold a generation (or agnomen) and a listed credential together, 77 a generation beside any other suffix word, and 64 only listed post-nominals of other kinds (each suffix word classified by the RFC's Phase 4 recipe, joined tokens merged); of the 615 CLDR renderings with either field, the 75 whose suffix holds exactly CLDR's words split into CLDR's `generation` and `credentials` by the vocabulary's reading kind in all 75. Weighed against: two new `Role` members (item 1), and a third view for the honorifics, left open since CLDR has none. 5. **a `surname2` view, the family's second unit where rules/O6 read the family as two**, marked by a new stable tag where O6 takes its word, empty otherwise; `family` keeps both units as O6 makes it, and a family comma alone marks nothing (`Lloyd Webber, Andrew`). Measured: a two-word family box in ORCID's records from Spain 48.0%, Mexico 64.7%, Colombia 72.8% and Chile 50.5%, against 3.9% from the United States. Left open: a view for the first unit and its name, `surname` standing close to the `surnames` view that exists. Declined: redefining `surnames` as the two units, which would repeat `family`, the family holding both where O6 reads two, and lose what `surnames` does today, joining `middle` and `family` so that a double surname the default splits (`Gabriel García Márquez` → middle `García`) reads whole, the view rules/P2 names for that reading. - 6. **The render defaults follow the order the name was read in, a behavior change of the output rather than a break of the API.** `ParsedName` gains `order` (an addition, `ADDED_IN`, a 2.x pickle rendering as today, left out of comparison, which lets two equal names render differently, a cost accepted 2026-10-28: a 2.x pickle of `김민준` renders `'민준 김'` and a fresh parse `'김민준'`, and the two compare equal); the default `str()` and `render()` spec put the family first where `order` does, and join two words that stand one directly after the other, in the same order, in the original and the rendering with what separated them there when it is nothing, whitespace, an interpunct, a nakaguro, or one comma with the whitespace beside it (any comma the parse reads: ASCII, full-width, Arabic), the spec's decoration between the two words' fields standing instead (the nickname's quotes, the `) ` after a maiden name), and any other pair or separator (a delimiter, a consumed marker word, two or more commas, a hyphen, a slash, a zero-width mark) taking the spec's own joiner; `initials()` gives nothing for a word in a script with no initials. A name with a family comma or written given-first keeps today's word order and its separators by the same rule, a suffix comma alone leaving a family-first name family-first (`威廉·莎士比亚` keeps its dot; 20 of the 36 names whose interpunct `str()` drops are read given-first). A family comma is never kept, the rendering reversing its pair (`Smith, John` → `'John Smith'`), and one directly between the family and its suffix is, decided 2026-10-28 (`John Smith, Jr.` → `'John Smith, Jr.'`, today `'John Smith Jr.'`; 214 of the 1,003 committed corpus entries whose suffix follows the family write one there). Accepted with it: `Smith, John, Jr.`, whose suffix the rendering moves away from the given name before it, keeps `'John Smith Jr.'`, so two writings of one name render differently. Measured over the committed corpus's distinct entries, the counts reading the ASCII comma (the RFC gives the full-width ones): 728 without a comma read family-first, `str()` renders all 728 differently from their text (`김민준` → `'민준 김'`), 150 of them with the family's last word touching the given name's first, and it drops the U+00B7 interpunct from 36 of the 41 holding one. This carries out decisions/3-0-reevaluations' render-default clean slate. No signature changes, but `str()`'s output changes for every name read family-first, every name whose written separator is not a space, and every name with one comma directly between its family and its suffix, and no 2.x release can warn about a `str()` result, so it is taken as an improvement for 3.0's release notes, not a break needing a warning. + 6. **The render defaults follow the order the name was read in, a behavior change of the output rather than a break of the API.** `ParsedName` gains `order` (an addition, `ADDED_IN`, a 2.x pickle rendering as today, left out of comparison, which lets two equal names render differently, a cost accepted 2026-10-28: a 2.x pickle of `김민준` renders `'민준 김'` and a fresh parse `'김민준'`, and the two compare equal); the default `str()` and `render()` spec put the family first where `order` does, and join two words that stand one directly after the other, in the same order, in the original and the rendering with the kind of separator that stood between them there, normalized, decided 2026-10-28 (nothing; one space for any whitespace; an interpunct or a nakaguro alone; one comma the parse reads as `, ` if ASCII, `، ` if Arabic and alone if full-width, so `John Smith ,Jr.` renders `'John Smith, Jr.'`; the written whitespace staying in `ParsedName.original` and the token spans), the spec's decoration between the two words' fields standing instead (the nickname's quotes, the `) ` after a maiden name), and any other pair or separator (a delimiter, a consumed marker word, two or more commas, a hyphen, a slash, a zero-width mark) taking the spec's own joiner; `initials()` gives nothing for a word in a script with no initials. A name with a family comma or written given-first keeps today's word order and its separators by the same rule, a suffix comma alone leaving a family-first name family-first (`威廉·莎士比亚` keeps its dot; 20 of the 36 names whose interpunct `str()` drops are read given-first). A family comma is never kept, the rendering reversing its pair (`Smith, John` → `'John Smith'`), and one directly between the family and its suffix is, decided 2026-10-28 (`John Smith, Jr.` → `'John Smith, Jr.'`, today `'John Smith Jr.'`; 214 of the 1,003 committed corpus entries whose suffix follows the family write one there). Accepted with it: `Smith, John, Jr.`, whose suffix the rendering moves away from the given name before it, keeps `'John Smith Jr.'`, so two writings of one name render differently. Measured over the committed corpus's distinct entries, the counts reading the ASCII comma (the RFC gives the full-width ones): 728 without a comma read family-first, `str()` renders all 728 differently from their text (`김민준` → `'민준 김'`), 150 of them with the family's last word touching the given name's first, and it drops the U+00B7 interpunct from 36 of the 41 holding one. This carries out decisions/3-0-reevaluations' render-default clean slate. No signature changes, but `str()`'s output changes for every name read family-first, every name whose written separator is not a space, and every name with one comma directly between its family and its suffix, and no 2.x release can warn about a `str()` result, so it is taken as an improvement for 3.0's release notes, not a break needing a warning. 7. **`middle` keeps its meaning (#704)**, every word of the given part after the given name, the given name one unit unless a rule joins words into it, as rules/P5 already does (`Abdul Aziz Tariq` → given `Abdul Aziz`); a compound given name is a `[rules]` rule a caller or a pack turns on, CLDR's `given2` is `middle`, and `Kaur` and an English name beside a Chinese one stay in `middle`, an accepted cost. Measured: in PSC, whose form has a middle-name box, 2,499 of 1,040,708 forename boxes hold two or more words, and of 6,101 Spanish nationals 25 do while 2,621 fill the middle box; ORCID's and PatentsView's given box holds middles too, so it cannot separate the two. The `Maria do Carmo` shape, a particle inside a given box of three or more words with a family box filled, reads the given box's words into the family in 1,812 of 1,923 ORCID records and 1,165 of 1,219 PatentsView records from Brazil, Portugal, Spain, Mexico, Argentina, Colombia, Chile and Italy, which is that rule's case. Weighed against: #704's option (b), one given unit by default, which changes `middle` for every English name of three words. 8. **A language file's `[rules]` compiles into its shipped `Locale`'s policy patch**, each rule kind a `Policy` field (an addition), so RU and TR_AZ are rebuilt from their files and `Locale` gains no field for rules. With it, **phase 3's open API dependency is answered**: its draft (branch `feat/vocabulary-phase3-design`, unmerged) carries a choice of languages on `Locale.languages`, assuming `Locale` and `Policy` keep their 2.x shape after the facade's removal, and the 2026-10-28 freeze keeps them so. The rest of that draft, strict as an argument of the whole choice included, stays phase 3's to decide; its entry, when it lands, says the dependency is answered. 9. **The work order**, one PR each, measured on its own base, the snapshot recording each new view where it is non-empty: 4a `Token.reading` and `ReadingKind.PATRONYMIC` (neutral; `languages` after 3a); 4b `generation` and `credentials` (neutral); 4c `patronymic` (neutral); 4d `surname2` (neutral); 4e the render defaults and `order`; 4f `[rules]` into the policy patch, with or after 3b; 4g the compound given name, after 4f; 4h the shape inventory per order (#469). Interleaved with phase 3 (its sub-phases the draft's): 4a beside 3a; 4b to 4d before the packs that need them, `cs` (3f) and `pt-BR` (3h) on 4b, `es` and `pt` (3g) on 4d and 4g; an `is` pack on 3b, 4c and 4f. diff --git a/docs/design/vocabulary-model-rfc.md b/docs/design/vocabulary-model-rfc.md index 1b5f2cb1c..6e9455082 100644 --- a/docs/design/vocabulary-model-rfc.md +++ b/docs/design/vocabulary-model-rfc.md @@ -250,7 +250,7 @@ ORCID's and PatentsView's given box holds the middle names too, so a two-word gi **Render defaults** (decisions/3-0-reevaluations' clean slate). The default `str()` reads every name given-first and joins its words with spaces, so a name read family-first is turned round and a written separator is lost: `str(parse("김민준"))` is `'민준 김'`, `str(parse("王 小明"))` is `'小明 王'` and `str(parse("威廉·莎士比亚"))` is `'威廉 莎士比亚'`. Over the committed corpus's distinct entries, each under its declared order (2026-10-28, `c8357e74`), 728 without a comma read family-first (the family's first word written before the given name's), `str()` renders every one differently from the text, and in 150 of them the family's last word touches the given name's first, written unspaced; of 41 holding the U+00B7 interpunct, `str()` drops it from 36. So: - `ParsedName` gains `order`, the order the name was read under (its script's or the declared `name_order`, a family comma aside), an addition with `ADDED_IN` metadata whose missing value, in a 2.x pickle, renders as today, and left out of comparison as `Token.reading` is, so a loaded 2.x pickle still equals a fresh parse of its text. The cost, accepted 2026-10-28: two equal names can render differently, where today every rendering is derived from what `==` compares (a 2.x pickle of `김민준` renders `'민준 김'` and a fresh parse `'김민준'`, and a set keeps one of them); the views read from `Token.reading` share it. -- `str()` and `render()`'s default spec put the family first where `order` does, and two words that stand one directly after the other, in the same order, in the original and in the rendering keep what separated them in the original when it is nothing, whitespace, an interpunct, a nakaguro, or one comma with whatever whitespace stands beside it, a comma being any the parse reads as one (ASCII, full-width U+FF0C, Arabic U+060C), so `John Smith，Jr.` keeps its comma unspaced as written; the spec's decoration between the two fields the words belong to stands instead (the nickname's quotes, the `) ` after the maiden name, so `Jane Smith née Jones, PhD` keeps `'Jane Smith (Jones) PhD'`; 38 committed entries put a suffix directly after an undelimited maiden name); and any other pair, or any other separator (a delimiter, a marker word the parse consumed, two or more commas, a hyphen, a slash, a zero-width or directional mark), takes the spec's own joiner, as today, so `John (Jack) Smith` keeps today's quoted nickname and `str(parse("김민준"))` is `'김민준'` and `str(parse("威廉·莎士比亚"))` keeps the dot. A name with a family comma or written given-first keeps today's word order, and its separators are kept by the same rule (a comma before a suffix alone does not make a family-first name given-first: `Wang Xiaoming, Jr.` under FAMILY_FIRST renders `'Wang Xiaoming, Jr.'`), so the given-first transcription `威廉·莎士比亚` keeps its dot too (20 of the 36 names whose interpunct `str()` drops are read given-first, one of them with a comma). A family comma is never kept, since the rendering reverses the pair it stands between (`Smith, John` → `'John Smith'`), while a comma directly between the family and its suffix is (`John Smith, Jr.` → `'John Smith, Jr.'`, where `str()` gives `'John Smith Jr.'` today): of the 1,003 committed corpus entries whose suffix starts after the family's last word ends, 214 have nothing between the two but one comma and whitespace (2026-10-28, `c8357e74`, each entry under its declared order; these counts read the ASCII comma, and 7 more write a full-width one there, while 36 of the 728 family-first names above hold a family comma the parse reads, 5 full-width and 31 Arabic (30 of the Arabic in `corpus_cldr.jsonl`), and so keep today's word order under the rule); 456 hold a comma somewhere between, most with a given name or title standing there too. Pairs the rendering separates, as `John` and `Jr.` in `Smith, John, Jr.`, take the spec's own joiner, as today, so `Smith, John, Jr.` keeps rendering `'John Smith Jr.'` while `John Smith, Jr.` becomes `'John Smith, Jr.'`: two writings of one name rendering differently, a consequence of following the text accepted with the choice. +- `str()` and `render()`'s default spec put the family first where `order` does, and two words that stand one directly after the other, in the same order, in the original and in the rendering keep the kind of separator that stood between them in the original, written in a normalized form (decided 2026-10-28): nothing stays nothing; whitespace of any width or length becomes one ASCII space; an interpunct or a nakaguro stands alone, with no space around it; and one comma the parse reads as one becomes `, ` if ASCII and `، ` if Arabic (U+060C), no space before it and one after, and stands alone if full-width (U+FF0C), which carries its own width in CJK text, so `John Smith ,Jr.` and `John Smith, Jr.` render `'John Smith, Jr.'` and `John Smith， Jr.` renders `'John Smith，Jr.'`. The written whitespace is not lost: `ParsedName.original` and the token spans keep it; the spec's decoration between the two fields the words belong to stands instead (the nickname's quotes, the `) ` after the maiden name, so `Jane Smith née Jones, PhD` keeps `'Jane Smith (Jones) PhD'`; 38 committed entries put a suffix directly after an undelimited maiden name); and any other pair, or any other separator (a delimiter, a marker word the parse consumed, two or more commas, a hyphen, a slash, a zero-width or directional mark), takes the spec's own joiner, as today, so `John (Jack) Smith` keeps today's quoted nickname and `str(parse("김민준"))` is `'김민준'` and `str(parse("威廉·莎士比亚"))` keeps the dot. A name with a family comma or written given-first keeps today's word order, and its separators are kept by the same rule (a comma before a suffix alone does not make a family-first name given-first: `Wang Xiaoming, Jr.` under FAMILY_FIRST renders `'Wang Xiaoming, Jr.'`), so the given-first transcription `威廉·莎士比亚` keeps its dot too (20 of the 36 names whose interpunct `str()` drops are read given-first, one of them with a comma). A family comma is never kept, since the rendering reverses the pair it stands between (`Smith, John` → `'John Smith'`), while a comma directly between the family and its suffix is (`John Smith, Jr.` → `'John Smith, Jr.'`, where `str()` gives `'John Smith Jr.'` today): of the 1,003 committed corpus entries whose suffix starts after the family's last word ends, 214 have nothing between the two but one comma and whitespace (2026-10-28, `c8357e74`, each entry under its declared order; these counts read the ASCII comma, and 7 more write a full-width one there, while 36 of the 728 family-first names above hold a family comma the parse reads, 5 full-width and 31 Arabic (30 of the Arabic in `corpus_cldr.jsonl`), and so keep today's word order under the rule); 456 hold a comma somewhere between, most with a given name or title standing there too. Pairs the rendering separates, as `John` and `Jr.` in `Smith, John, Jr.`, take the spec's own joiner, as today, so `Smith, John, Jr.` keeps rendering `'John Smith Jr.'` while `John Smith, Jr.` becomes `'John Smith, Jr.'`: two writings of one name rendering differently, a consequence of following the text accepted with the choice. - `initials()` gives nothing for a word in a script with no initials (Han, Hangul, kana, `_policy._NO_INITIALS`), where today it gives the first character with a period (`'王.'`). These change no signature but change `str()`'s output for every name read family-first, every name whose written separator is not a space, and every name written with one comma directly between its family and its suffix, which no 2.x release can warn about; Derek took them as an improvement for 3.0's release notes rather than a break (item 6 of the entry). From 4ab18e4c3b9b649f67e962439b9a42e91d11f049 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Wed, 28 Oct 2026 14:04:25 -0700 Subject: [PATCH 22/39] Fix the seventh docs review's findings on phase 4's render rule (#754) The change list no longer counts whitespace-separated names, which the normalized rule leaves as they are, and the Standing line says the separators are normalized. Only an interpunct the parse reads as a separator stands alone; a spaced one is a name word (rules/T3). The interpunct comma count reads every comma the parse reads, the decisions copy names directional marks and the Arabic counts, the whitespace note sits after the rule, and the index-cost comparison names the cache. Co-Authored-By: Claude Opus 5.5 --- docs/design/decisions/vocabulary-model.md | 4 ++-- docs/design/vocabulary-model-rfc.md | 6 +++--- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/design/decisions/vocabulary-model.md b/docs/design/decisions/vocabulary-model.md index 9ea47dfff..d4dd3cfbd 100644 --- a/docs/design/decisions/vocabulary-model.md +++ b/docs/design/decisions/vocabulary-model.md @@ -11,7 +11,7 @@ The vocabulary was a set of base sets plus marker subsets (`particles` and `part - `e`, `y` and `i` stay in the default; choosing a language is to settle their one-case fork, which no shipped pack does yet — 2026-10-15/16. - A patronymic field is to come in 3.0, as a view that names its words without emptying `family`, and the particle kinds already draw the `clan`/`patronymic` line for it: a patronymic is followed by a parent's name, a clan prefix by a family name the lineage shares — 2026-10-15/16; 2026-10-19 #749; 2026-10-28 #754 (Derek). - A `Locale` may carry CLDR-shaped render patterns, display and sorting, a later addition whose shape is agreed; how a call reaches one is open — 2026-10-28 #754, render patterns. -- 3.0's output model keeps the seven roles and adds every new field as a view, read from `Token.reading`, the reading each token took, or for `surname2` from rules/O6's tag: `generation` and `credentials` partitioning `suffix`, `patronymic`, and `surname2` where rules/O6 read two family names; `middle` and `surnames` keep their meaning, a compound given name being a rule a caller or pack turns on; the default `str()` renders a name in the order it was read in with its written separators; a language file's `[rules]` compile into its `Locale`'s policy patch — 2026-10-28 #754 (Derek). +- 3.0's output model keeps the seven roles and adds every new field as a view, read from `Token.reading`, the reading each token took, or for `surname2` from rules/O6's tag: `generation` and `credentials` partitioning `suffix`, `patronymic`, and `surname2` where rules/O6 read two family names; `middle` and `surnames` keep their meaning, a compound given name being a rule a caller or pack turns on; the default `str()` renders a name in the order it was read in with its written separators in a normalized form; a language file's `[rules]` compile into its `Locale`'s policy patch — 2026-10-28 #754 (Derek). - Gender is recorded only where a word gives a definitive clue: a feminine form or one half of a fixed pair, never an unmarked noun — 2026-10-15/16; 2026-10-19 #750, title kinds. - Title-phrase modifiers count as titles only beside a head title, at the head of the name only and never in a comma part (`Wing Lau` → given 'Wing', `John Smith, Vice` → title 'Vice'), save a modifier that may be a given name, which the head's test reads in every part (`Lau, Wing Kai` → given 'Wing'), rules/H9 — 2026-10-15/16; 2026-10-19 #750, title kinds; 2026-10-23 #752; 2026-10-25 #752, phase 2i; 2026-10-26 #752, phase 2j, `wing`. - Name readings are hand-declared, and only on a word some language file claims as vocabulary or refuses, so the vocabulary carries no surname list (the segmentation lists, Korean and the ZH pack's, are outside the per-word model) — 2026-10-15/16; 2026-10-19. @@ -250,7 +250,7 @@ Open: #913 (the title words phase 2j left, read as titles though borne as names) 3. **A `patronymic` view, filled by a word or a rule that says patronymic, with `family` and `middle` unchanged.** The default fills it from the vocabulary's `patronymic` particles (`bin`, `binti`, `ibn`, `bint`, `bar`, `bat` and the native-script forms) with the name they introduce (`Mahathir bin Mohamad` → family `bin Mohamad`, as today, and patronymic `bin Mohamad`); RU's and TR_AZ's rules fill it where a caller turns them on (`Ivan Ivanovich Ivanov` under RU → middle `Ivanovich` and patronymic `Ivanovich`); an Icelandic ending rule fills it under an `is` pack only, since the default cannot tell `-dóttir` and `-son` from Swedish surnames (ORCID's Swedish records end their family box in `-son` or `-dóttir` in 2,184 of 20,000). A matronymic, written the same way, fills it too. SUPERSEDES: the RFC's earlier statement of decision 8's value, that `family` can be empty for a name with a patronymic. Measured: filers put an Icelandic patronymic in the family box (ORCID 1,647 of 2,583 Icelandic records, PatentsView 535 of 722, PSC 101 of 125), and CLDR's `is` file's 3 Icelandic samples put it in `surname` or `surname-core` in all 9 of their renderings (its one foreign sample puts a `-dóttir` word in `surname2`), while Wikidata's `patronymic-is` and `patronymic-my` strata record 108 names with a patronymic and no family name, of which the parse puts the patronymic in `family` 107 times. An East Slavic patronymic stands in the given part: CLDR's 2 `ru` samples put it in `given2` in all 6 renderings; PatentsView's Russian given boxes end in one 8,423 times of 20,000, and the parse reads such a word as the middle 10,902 times of 10,962; 90 of the 112 Russian nationals in PSC who filled the middle-name box wrote one there. A `family` that keeps it and a `patronymic` that names it agree with both kinds of source. Weighed against: emptying `family` where the patronymic is recognized, which moves the field the fielded sources fill and changes `family` for every Icelandic and nasab name, a behavior change of a frozen property; and a neutral parent-names name for the view, which buys nothing the view can detect, since a matronymic and a patronymic are written alike. 4. **`generation` and `credentials` are views that partition `suffix` (#326)**, not roles, `generation` the tokens read as a generation or an agnomen, `credentials` the rest, the post-nominals that are neither (`Esq.`, the honorifics after a name) falling in `credentials`, CLDR's catch-all, with `Token.reading.kind` telling them apart for a caller who needs it. Measured: of the 1,054 committed corpus entries with a suffix, 33 hold a generation (or agnomen) and a listed credential together, 77 a generation beside any other suffix word, and 64 only listed post-nominals of other kinds (each suffix word classified by the RFC's Phase 4 recipe, joined tokens merged); of the 615 CLDR renderings with either field, the 75 whose suffix holds exactly CLDR's words split into CLDR's `generation` and `credentials` by the vocabulary's reading kind in all 75. Weighed against: two new `Role` members (item 1), and a third view for the honorifics, left open since CLDR has none. 5. **a `surname2` view, the family's second unit where rules/O6 read the family as two**, marked by a new stable tag where O6 takes its word, empty otherwise; `family` keeps both units as O6 makes it, and a family comma alone marks nothing (`Lloyd Webber, Andrew`). Measured: a two-word family box in ORCID's records from Spain 48.0%, Mexico 64.7%, Colombia 72.8% and Chile 50.5%, against 3.9% from the United States. Left open: a view for the first unit and its name, `surname` standing close to the `surnames` view that exists. Declined: redefining `surnames` as the two units, which would repeat `family`, the family holding both where O6 reads two, and lose what `surnames` does today, joining `middle` and `family` so that a double surname the default splits (`Gabriel García Márquez` → middle `García`) reads whole, the view rules/P2 names for that reading. - 6. **The render defaults follow the order the name was read in, a behavior change of the output rather than a break of the API.** `ParsedName` gains `order` (an addition, `ADDED_IN`, a 2.x pickle rendering as today, left out of comparison, which lets two equal names render differently, a cost accepted 2026-10-28: a 2.x pickle of `김민준` renders `'민준 김'` and a fresh parse `'김민준'`, and the two compare equal); the default `str()` and `render()` spec put the family first where `order` does, and join two words that stand one directly after the other, in the same order, in the original and the rendering with the kind of separator that stood between them there, normalized, decided 2026-10-28 (nothing; one space for any whitespace; an interpunct or a nakaguro alone; one comma the parse reads as `, ` if ASCII, `، ` if Arabic and alone if full-width, so `John Smith ,Jr.` renders `'John Smith, Jr.'`; the written whitespace staying in `ParsedName.original` and the token spans), the spec's decoration between the two words' fields standing instead (the nickname's quotes, the `) ` after a maiden name), and any other pair or separator (a delimiter, a consumed marker word, two or more commas, a hyphen, a slash, a zero-width mark) taking the spec's own joiner; `initials()` gives nothing for a word in a script with no initials. A name with a family comma or written given-first keeps today's word order and its separators by the same rule, a suffix comma alone leaving a family-first name family-first (`威廉·莎士比亚` keeps its dot; 20 of the 36 names whose interpunct `str()` drops are read given-first). A family comma is never kept, the rendering reversing its pair (`Smith, John` → `'John Smith'`), and one directly between the family and its suffix is, decided 2026-10-28 (`John Smith, Jr.` → `'John Smith, Jr.'`, today `'John Smith Jr.'`; 214 of the 1,003 committed corpus entries whose suffix follows the family write one there). Accepted with it: `Smith, John, Jr.`, whose suffix the rendering moves away from the given name before it, keeps `'John Smith Jr.'`, so two writings of one name render differently. Measured over the committed corpus's distinct entries, the counts reading the ASCII comma (the RFC gives the full-width ones): 728 without a comma read family-first, `str()` renders all 728 differently from their text (`김민준` → `'민준 김'`), 150 of them with the family's last word touching the given name's first, and it drops the U+00B7 interpunct from 36 of the 41 holding one. This carries out decisions/3-0-reevaluations' render-default clean slate. No signature changes, but `str()`'s output changes for every name read family-first, every name whose written separator is not a space, and every name with one comma directly between its family and its suffix, and no 2.x release can warn about a `str()` result, so it is taken as an improvement for 3.0's release notes, not a break needing a warning. + 6. **The render defaults follow the order the name was read in, a behavior change of the output rather than a break of the API.** `ParsedName` gains `order` (an addition, `ADDED_IN`, a 2.x pickle rendering as today, left out of comparison, which lets two equal names render differently, a cost accepted 2026-10-28: a 2.x pickle of `김민준` renders `'민준 김'` and a fresh parse `'김민준'`, and the two compare equal); the default `str()` and `render()` spec put the family first where `order` does, and join two words that stand one directly after the other, in the same order, in the original and the rendering with the kind of separator that stood between them there, normalized, decided 2026-10-28 (nothing; one space for any whitespace; an interpunct or a nakaguro the parse read as a separator alone, a spaced interpunct being a name word, rules/T3; one comma the parse reads as `, ` if ASCII, `، ` if Arabic and alone if full-width, so `John Smith ,Jr.` renders `'John Smith, Jr.'`; the written whitespace staying in `ParsedName.original` and the token spans), the spec's decoration between the two words' fields standing instead (the nickname's quotes, the `) ` after a maiden name), and any other pair or separator (a delimiter, a consumed marker word, two or more commas, a hyphen, a slash, a zero-width or directional mark) taking the spec's own joiner; `initials()` gives nothing for a word in a script with no initials. A name with a family comma or written given-first keeps today's word order and its separators by the same rule, a suffix comma alone leaving a family-first name family-first (`威廉·莎士比亚` keeps its dot; 20 of the 36 names whose interpunct `str()` drops are read given-first). A family comma is never kept, the rendering reversing its pair (`Smith, John` → `'John Smith'`), and one directly between the family and its suffix is, decided 2026-10-28 (`John Smith, Jr.` → `'John Smith, Jr.'`, today `'John Smith Jr.'`; 214 of the 1,003 committed corpus entries whose suffix follows the family write one there). Accepted with it: `Smith, John, Jr.`, whose suffix the rendering moves away from the given name before it, keeps `'John Smith Jr.'`, so two writings of one name render differently. Measured over the committed corpus's distinct entries, the counts reading the ASCII comma (the RFC gives the full-width and Arabic ones: 36 of the 728 hold a family comma the parse reads and keep today's word order): 728 without a comma read family-first, `str()` renders all 728 differently from their text (`김민준` → `'민준 김'`), 150 of them with the family's last word touching the given name's first, and it drops the U+00B7 interpunct from 36 of the 41 holding one. This carries out decisions/3-0-reevaluations' render-default clean slate. No signature changes, but `str()`'s output changes for every name read family-first, every name whose words are written with nothing, an interpunct or a nakaguro between them, and every name with one comma directly between its family and its suffix, and no 2.x release can warn about a `str()` result, so it is taken as an improvement for 3.0's release notes, not a break needing a warning. 7. **`middle` keeps its meaning (#704)**, every word of the given part after the given name, the given name one unit unless a rule joins words into it, as rules/P5 already does (`Abdul Aziz Tariq` → given `Abdul Aziz`); a compound given name is a `[rules]` rule a caller or a pack turns on, CLDR's `given2` is `middle`, and `Kaur` and an English name beside a Chinese one stay in `middle`, an accepted cost. Measured: in PSC, whose form has a middle-name box, 2,499 of 1,040,708 forename boxes hold two or more words, and of 6,101 Spanish nationals 25 do while 2,621 fill the middle box; ORCID's and PatentsView's given box holds middles too, so it cannot separate the two. The `Maria do Carmo` shape, a particle inside a given box of three or more words with a family box filled, reads the given box's words into the family in 1,812 of 1,923 ORCID records and 1,165 of 1,219 PatentsView records from Brazil, Portugal, Spain, Mexico, Argentina, Colombia, Chile and Italy, which is that rule's case. Weighed against: #704's option (b), one given unit by default, which changes `middle` for every English name of three words. 8. **A language file's `[rules]` compiles into its shipped `Locale`'s policy patch**, each rule kind a `Policy` field (an addition), so RU and TR_AZ are rebuilt from their files and `Locale` gains no field for rules. With it, **phase 3's open API dependency is answered**: its draft (branch `feat/vocabulary-phase3-design`, unmerged) carries a choice of languages on `Locale.languages`, assuming `Locale` and `Policy` keep their 2.x shape after the facade's removal, and the 2026-10-28 freeze keeps them so. The rest of that draft, strict as an argument of the whole choice included, stays phase 3's to decide; its entry, when it lands, says the dependency is answered. 9. **The work order**, one PR each, measured on its own base, the snapshot recording each new view where it is non-empty: 4a `Token.reading` and `ReadingKind.PATRONYMIC` (neutral; `languages` after 3a); 4b `generation` and `credentials` (neutral); 4c `patronymic` (neutral); 4d `surname2` (neutral); 4e the render defaults and `order`; 4f `[rules]` into the policy patch, with or after 3b; 4g the compound given name, after 4f; 4h the shape inventory per order (#469). Interleaved with phase 3 (its sub-phases the draft's): 4a beside 3a; 4b to 4d before the packs that need them, `cs` (3f) and `pt-BR` (3h) on 4b, `es` and `pt` (3g) on 4d and 4g; an `is` pack on 3b, 4c and 4f. diff --git a/docs/design/vocabulary-model-rfc.md b/docs/design/vocabulary-model-rfc.md index 6e9455082..5234a17b0 100644 --- a/docs/design/vocabulary-model-rfc.md +++ b/docs/design/vocabulary-model-rfc.md @@ -250,10 +250,10 @@ ORCID's and PatentsView's given box holds the middle names too, so a two-word gi **Render defaults** (decisions/3-0-reevaluations' clean slate). The default `str()` reads every name given-first and joins its words with spaces, so a name read family-first is turned round and a written separator is lost: `str(parse("김민준"))` is `'민준 김'`, `str(parse("王 小明"))` is `'小明 王'` and `str(parse("威廉·莎士比亚"))` is `'威廉 莎士比亚'`. Over the committed corpus's distinct entries, each under its declared order (2026-10-28, `c8357e74`), 728 without a comma read family-first (the family's first word written before the given name's), `str()` renders every one differently from the text, and in 150 of them the family's last word touches the given name's first, written unspaced; of 41 holding the U+00B7 interpunct, `str()` drops it from 36. So: - `ParsedName` gains `order`, the order the name was read under (its script's or the declared `name_order`, a family comma aside), an addition with `ADDED_IN` metadata whose missing value, in a 2.x pickle, renders as today, and left out of comparison as `Token.reading` is, so a loaded 2.x pickle still equals a fresh parse of its text. The cost, accepted 2026-10-28: two equal names can render differently, where today every rendering is derived from what `==` compares (a 2.x pickle of `김민준` renders `'민준 김'` and a fresh parse `'김민준'`, and a set keeps one of them); the views read from `Token.reading` share it. -- `str()` and `render()`'s default spec put the family first where `order` does, and two words that stand one directly after the other, in the same order, in the original and in the rendering keep the kind of separator that stood between them in the original, written in a normalized form (decided 2026-10-28): nothing stays nothing; whitespace of any width or length becomes one ASCII space; an interpunct or a nakaguro stands alone, with no space around it; and one comma the parse reads as one becomes `, ` if ASCII and `، ` if Arabic (U+060C), no space before it and one after, and stands alone if full-width (U+FF0C), which carries its own width in CJK text, so `John Smith ,Jr.` and `John Smith, Jr.` render `'John Smith, Jr.'` and `John Smith， Jr.` renders `'John Smith，Jr.'`. The written whitespace is not lost: `ParsedName.original` and the token spans keep it; the spec's decoration between the two fields the words belong to stands instead (the nickname's quotes, the `) ` after the maiden name, so `Jane Smith née Jones, PhD` keeps `'Jane Smith (Jones) PhD'`; 38 committed entries put a suffix directly after an undelimited maiden name); and any other pair, or any other separator (a delimiter, a marker word the parse consumed, two or more commas, a hyphen, a slash, a zero-width or directional mark), takes the spec's own joiner, as today, so `John (Jack) Smith` keeps today's quoted nickname and `str(parse("김민준"))` is `'김민준'` and `str(parse("威廉·莎士比亚"))` keeps the dot. A name with a family comma or written given-first keeps today's word order, and its separators are kept by the same rule (a comma before a suffix alone does not make a family-first name given-first: `Wang Xiaoming, Jr.` under FAMILY_FIRST renders `'Wang Xiaoming, Jr.'`), so the given-first transcription `威廉·莎士比亚` keeps its dot too (20 of the 36 names whose interpunct `str()` drops are read given-first, one of them with a comma). A family comma is never kept, since the rendering reverses the pair it stands between (`Smith, John` → `'John Smith'`), while a comma directly between the family and its suffix is (`John Smith, Jr.` → `'John Smith, Jr.'`, where `str()` gives `'John Smith Jr.'` today): of the 1,003 committed corpus entries whose suffix starts after the family's last word ends, 214 have nothing between the two but one comma and whitespace (2026-10-28, `c8357e74`, each entry under its declared order; these counts read the ASCII comma, and 7 more write a full-width one there, while 36 of the 728 family-first names above hold a family comma the parse reads, 5 full-width and 31 Arabic (30 of the Arabic in `corpus_cldr.jsonl`), and so keep today's word order under the rule); 456 hold a comma somewhere between, most with a given name or title standing there too. Pairs the rendering separates, as `John` and `Jr.` in `Smith, John, Jr.`, take the spec's own joiner, as today, so `Smith, John, Jr.` keeps rendering `'John Smith Jr.'` while `John Smith, Jr.` becomes `'John Smith, Jr.'`: two writings of one name rendering differently, a consequence of following the text accepted with the choice. +- `str()` and `render()`'s default spec put the family first where `order` does, and two words that stand one directly after the other, in the same order, in the original and in the rendering keep the kind of separator that stood between them in the original, written in a normalized form (decided 2026-10-28): nothing stays nothing; whitespace of any width or length becomes one ASCII space; an interpunct or a nakaguro the parse read as a separator stands alone, with no space around it (an interpunct written with spaces round it is not one, rules/T3 dividing at it only when flanked, so `威廉 · 莎士比亚` reads the dot as a name word and keeps rendering with its spaces, where `威廉·莎士比亚` keeps the dot unspaced; a spaced nakaguro is a separator, so `ジョン ・ スミス` renders `'ジョン・スミス'`); and one comma the parse reads as one becomes `, ` if ASCII and `، ` if Arabic (U+060C), no space before it and one after, and stands alone if full-width (U+FF0C), which carries its own width in CJK text, so `John Smith ,Jr.` and `John Smith, Jr.` render `'John Smith, Jr.'` and `John Smith， Jr.` renders `'John Smith，Jr.'`. Over that, the spec's decoration between the two fields the words belong to stands in place of the separator (the nickname's quotes, the `) ` after the maiden name, so `Jane Smith née Jones, PhD` keeps `'Jane Smith (Jones) PhD'`; 38 committed entries put a suffix directly after an undelimited maiden name); and any other pair, or any other separator (a delimiter, a marker word the parse consumed, two or more commas, a hyphen, a slash, a zero-width or directional mark), takes the spec's own joiner, as today (the written whitespace is not lost: `ParsedName.original` and the token spans keep it), so `John (Jack) Smith` keeps today's quoted nickname and `str(parse("김민준"))` is `'김민준'` and `str(parse("威廉·莎士比亚"))` keeps the dot. A name with a family comma or written given-first keeps today's word order, and its separators are kept by the same rule (a comma before a suffix alone does not make a family-first name given-first: `Wang Xiaoming, Jr.` under FAMILY_FIRST renders `'Wang Xiaoming, Jr.'`), so the given-first transcription `威廉·莎士比亚` keeps its dot too (20 of the 36 names whose interpunct `str()` drops are read given-first, 4 of them with a comma the parse reads (1 ASCII, 3 full-width)). A family comma is never kept, since the rendering reverses the pair it stands between (`Smith, John` → `'John Smith'`), while a comma directly between the family and its suffix is (`John Smith, Jr.` → `'John Smith, Jr.'`, where `str()` gives `'John Smith Jr.'` today): of the 1,003 committed corpus entries whose suffix starts after the family's last word ends, 214 have nothing between the two but one comma and whitespace (2026-10-28, `c8357e74`, each entry under its declared order; these counts read the ASCII comma, and 7 more write a full-width one there, while 36 of the 728 family-first names above hold a family comma the parse reads, 5 full-width and 31 Arabic (30 of the Arabic in `corpus_cldr.jsonl`), and so keep today's word order under the rule); 456 hold a comma somewhere between, most with a given name or title standing there too. Pairs the rendering separates, as `John` and `Jr.` in `Smith, John, Jr.`, take the spec's own joiner, as today, so `Smith, John, Jr.` keeps rendering `'John Smith Jr.'` while `John Smith, Jr.` becomes `'John Smith, Jr.'`: two writings of one name rendering differently, a consequence of following the text accepted with the choice. - `initials()` gives nothing for a word in a script with no initials (Han, Hangul, kana, `_policy._NO_INITIALS`), where today it gives the first character with a period (`'王.'`). -These change no signature but change `str()`'s output for every name read family-first, every name whose written separator is not a space, and every name written with one comma directly between its family and its suffix, which no 2.x release can warn about; Derek took them as an improvement for 3.0's release notes rather than a break (item 6 of the entry). +These change no signature but change `str()`'s output for every name read family-first, every name whose words are written with nothing, an interpunct or a nakaguro between them, and every name written with one comma directly between its family and its suffix, which no 2.x release can warn about; Derek took them as an improvement for 3.0's release notes rather than a break (item 6 of the entry). **What changes in the API**, by the 3.0 freeze: @@ -310,7 +310,7 @@ Each sub-phase is one PR, measured on its own base as phase 2's and phase 3's ar A shape question, to settle before the 3.0 freeze makes every field permanent: -- **Whether `Lexicon` keeps a public field for each compiled set**, or answers word, kind and property queries from phase 3's reading index and keeps its sets internal. `dataclasses.fields(Lexicon)` gives 22 public fields on 2026-10-28, 11 of them added since 2.0.0's 11 (`surnames`, `honorific_tails`, `conjunctions_ambiguous`, `after_given_titles`, `suffix_words_ambiguous`, `agnomens`, `given_name_abbreviations`, `titles_ambiguous`, `title_modifiers`, `title_styles`, `title_phrases`), each a new parse behavior's set, and a rule keyed on a title kind or on one language's reading of a kind (phase 3's draft, "a property a language's own reading should take only under its pack") would add more. Keeping the fields: each is frozen API, shown in the bounded repr (`Lexicon(default + titles: +1)`), and pickled, where `Lexicon`'s own `__setstate__`, unlike the `ADDED_IN` guard the other types share, refuses any pickle whose fields differ, so every new field breaks every older `Lexicon` pickle, and with it every pickled `Parser` and locale pack, until that guard learns the marker. Turning them internal: a break that has to land in 3.0, where a query added beside the fields is an addition that can come later; a caller who configures vocabulary by the field today (`Lexicon.add(titles=...)`, `dataclasses.replace`) needs another way in, for words that carry no kind and no language; and phase 3's draft loads the reading index only when a caller chooses a language (about 11 ms to execute with no bytecode cache, by the draft's timing; the per-word index this RFC weighs for `Reading.languages` above costs about 0.4 ms of a 19.5 ms import), so a `Lexicon` answering from it loads it for every caller. The parser need not pay either way: a frozenset test per word costs no frame and an internal cache keeps that path, while a query method called per word would add one per token, the cost the frame budget counts as scaling with the input. +- **Whether `Lexicon` keeps a public field for each compiled set**, or answers word, kind and property queries from phase 3's reading index and keeps its sets internal. `dataclasses.fields(Lexicon)` gives 22 public fields on 2026-10-28, 11 of them added since 2.0.0's 11 (`surnames`, `honorific_tails`, `conjunctions_ambiguous`, `after_given_titles`, `suffix_words_ambiguous`, `agnomens`, `given_name_abbreviations`, `titles_ambiguous`, `title_modifiers`, `title_styles`, `title_phrases`), each a new parse behavior's set, and a rule keyed on a title kind or on one language's reading of a kind (phase 3's draft, "a property a language's own reading should take only under its pack") would add more. Keeping the fields: each is frozen API, shown in the bounded repr (`Lexicon(default + titles: +1)`), and pickled, where `Lexicon`'s own `__setstate__`, unlike the `ADDED_IN` guard the other types share, refuses any pickle whose fields differ, so every new field breaks every older `Lexicon` pickle, and with it every pickled `Parser` and locale pack, until that guard learns the marker. Turning them internal: a break that has to land in 3.0, where a query added beside the fields is an addition that can come later; a caller who configures vocabulary by the field today (`Lexicon.add(titles=...)`, `dataclasses.replace`) needs another way in, for words that carry no kind and no language; and phase 3's draft loads the reading index only when a caller chooses a language (about 11 ms to execute with no bytecode cache, by the draft's timing; decisions/A1 timed a per-word index of similar size with bytecode cached, as an installed wheel has it, at about 0.35 ms of an 18.2 ms import, so most of that gap is likely the cache rather than the design, unmeasured for the draft's index), so a `Lexicon` answering from it loads it for every caller. The parser need not pay either way: a frozenset test per word costs no frame and an internal cache keeps that path, while a query method called per word would add one per token, the cost the frame budget counts as scaling with the input. Value questions inside the shapes above: From 96454680265721b7f8f2f449a5eb2dfcbf5dcc51 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Wed, 28 Oct 2026 14:06:14 -0700 Subject: [PATCH 23/39] Normalize the ideographic space and correct the index's import cost (#754) U+3000 is whitespace like any other, so it renders as one ASCII space; the full-width comma's own-width reason is scoped to the comma. The Japanese comma is noted as a parser question outside the design. The per-word index's import cost now matches decisions/A1's measurement (0.35 ms of 18.2 ms), where two copies had 0.4 of 19.5. Co-Authored-By: Claude Opus 5.5 --- docs/design/decisions/vocabulary-model.md | 4 ++-- docs/design/vocabulary-model-rfc.md | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/design/decisions/vocabulary-model.md b/docs/design/decisions/vocabulary-model.md index d4dd3cfbd..2f97977d8 100644 --- a/docs/design/decisions/vocabulary-model.md +++ b/docs/design/decisions/vocabulary-model.md @@ -242,7 +242,7 @@ Open: #913 (the title words phase 2j left, read as titles though borne as names) - The words, with their counts on each entry: `benj`, `chas`, `edw`, `geo`, `jas`, `jno`, `richd`, `robt`, `saml`, `thos` and `wm` in en.toml (the last keeping its refused title, whose reason now names this reading), `jo` and `ma` in fil.toml. What moved, the vetting and the grid are decisions/H2's entry of this date. Recompute the list: `uv run python -c "from nameparser import Lexicon; print(sorted(Lexicon.default().given_name_abbreviations))"`. Adding the field changes `Lexicon`'s pickle layout, as phases 2h and 2i did. - **2026-10-26 #752, phase 2l — THE BRACKET ESCAPE READS THE DERIVED CLASH**, carrying out item 16 of the 2026-10-23 #752 entry above (#810). A member of the ambiguous class, the class 2a derives from name readings, alone in brackets or quotes after the name is the suffix where its writing settles it (`John Smith (MA)`, `John Smith (M.A)` → suffix), read in one pass, and stays the nickname otherwise (`John Smith (Ed)`, `JOHN SMITH (MA)`, `Jeffrey (JD) Bricken`). It is recorded as rules/S13, a rule of its own rather than the amendment to S1 item 16 planned, the position it reads being evidence S1's signal does not read; the writing is S3's and S6's evidence, which S13 cites. The record, the measurements and what was declined are decisions/S13's 2026-10-26 #810 entry. -- **2026-10-26 #752, phase 2m — THE READING NOT TAKEN SURFACES AS `Ambiguity.alternative`, WITHOUT ITS LANGUAGES**, carrying out item 13 of the 2026-10-23 #752 entry above and item 7's source with it: a word the credential's run took by position alone reports its name reading's source as `position`, a fourth source beside the three the item named, and a word with a declared name reading reports `declared`. The pickle guard was widened as item 13 asked, every skew the narrow guard refused still refused. The reading's languages were left out: the per-word index costs about 0.4 ms of a 19.5 ms import and describes the default vocabulary, not a pack's or a caller's lexicon, so they wait for 3.0 with `Token.reading`. The record, the measurements and what was declined are decisions/A1's 2026-10-26 #752 entry. +- **2026-10-26 #752, phase 2m — THE READING NOT TAKEN SURFACES AS `Ambiguity.alternative`, WITHOUT ITS LANGUAGES**, carrying out item 13 of the 2026-10-23 #752 entry above and item 7's source with it: a word the credential's run took by position alone reports its name reading's source as `position`, a fourth source beside the three the item named, and a word with a declared name reading reports `declared`. The pickle guard was widened as item 13 asked, every skew the narrow guard refused still refused. The reading's languages were left out: the per-word index costs about 0.35 ms of an 18.2 ms import (decisions/A1) and describes the default vocabulary, not a pack's or a caller's lexicon, so they wait for 3.0 with `Token.reading`. The record, the measurements and what was declined are decisions/A1's 2026-10-26 #752 entry. - **2026-10-28 #754 (Derek) — PHASE 4'S OUTPUT MODEL IS AGREED: THE SEVEN ROLES STAY AND EVERY NEW FIELD IS A VIEW, MOST READ FROM `Token.reading`; A PATRONYMIC NAMES ITS WORDS WITHOUT EMPTYING `family`; THE RENDER DEFAULTS FOLLOW THE ORDER A NAME WAS READ IN.** Derek agreed every item in chat on 2026-10-28, items 6 and 4 and the `surnames` line of item 5 first, then the rest. The design overview is docs/design/vocabulary-model-rfc.md's Phase 4 section. Decision 8 (a patronymic field in 3.0) is kept and its stated value revised (item 3); decisions 9 and 14 are kept as written. The core API is frozen for 3.0 except for additions (decisions/3-0-reevaluations, 2026-10-28 #925, THE CORE API), and each item says whether it is one. The inputs are #754 and its comment of 2026-10-23, #326, #704 and #694. Measurements were taken 2026-10-28 on master `c8357e74` (Python 3.11.17) over the fielded local corpora (`orcid-researchers`, `psc-individuals`, `patentsview-inventors`), the committed fields files and the committed corpus's distinct entries (`snapshot.corpus_entries()`, each parsed under its declared order), with throwaway scripts that are not kept, so each figure is evidence for the decision and each sub-phase re-measures on its own base. Local-corpus figures are counts. Recompute the box counts by reading each source's records with `local_corpora.read_records`, counting its boxes by word count and word ending per country or nationality, and the role the parse gives the counted word with `parse()`; the ending tests are the RFC's Phase 4 table's, and other tests give counts a few percent apart. 1. **`Role` keeps its seven members, and every field asked for is a view.** A role partitions the tokens, and each field asked for (a second surname, a patronymic, generation and credentials, a second given name) subdivides a role that exists, so each is computed from the tokens, a property and a render key, except the second given name, which is the `middle` that exists (item 7), as `family_base` and `given_names` are (rules/R1, rules/R2). `as_dict()`, `comparison_key()` and the seven role properties are unchanged, and every view is an addition. Weighed against: a `Role` member for any of them, a break of the frozen `ParsedName` (`as_dict()` gains a key, `comparison_key()` a component, so a stored key stops matching, and the role its words leave returns less text than in 2.x), with no 2.x feature release left to warn in. @@ -250,7 +250,7 @@ Open: #913 (the title words phase 2j left, read as titles though borne as names) 3. **A `patronymic` view, filled by a word or a rule that says patronymic, with `family` and `middle` unchanged.** The default fills it from the vocabulary's `patronymic` particles (`bin`, `binti`, `ibn`, `bint`, `bar`, `bat` and the native-script forms) with the name they introduce (`Mahathir bin Mohamad` → family `bin Mohamad`, as today, and patronymic `bin Mohamad`); RU's and TR_AZ's rules fill it where a caller turns them on (`Ivan Ivanovich Ivanov` under RU → middle `Ivanovich` and patronymic `Ivanovich`); an Icelandic ending rule fills it under an `is` pack only, since the default cannot tell `-dóttir` and `-son` from Swedish surnames (ORCID's Swedish records end their family box in `-son` or `-dóttir` in 2,184 of 20,000). A matronymic, written the same way, fills it too. SUPERSEDES: the RFC's earlier statement of decision 8's value, that `family` can be empty for a name with a patronymic. Measured: filers put an Icelandic patronymic in the family box (ORCID 1,647 of 2,583 Icelandic records, PatentsView 535 of 722, PSC 101 of 125), and CLDR's `is` file's 3 Icelandic samples put it in `surname` or `surname-core` in all 9 of their renderings (its one foreign sample puts a `-dóttir` word in `surname2`), while Wikidata's `patronymic-is` and `patronymic-my` strata record 108 names with a patronymic and no family name, of which the parse puts the patronymic in `family` 107 times. An East Slavic patronymic stands in the given part: CLDR's 2 `ru` samples put it in `given2` in all 6 renderings; PatentsView's Russian given boxes end in one 8,423 times of 20,000, and the parse reads such a word as the middle 10,902 times of 10,962; 90 of the 112 Russian nationals in PSC who filled the middle-name box wrote one there. A `family` that keeps it and a `patronymic` that names it agree with both kinds of source. Weighed against: emptying `family` where the patronymic is recognized, which moves the field the fielded sources fill and changes `family` for every Icelandic and nasab name, a behavior change of a frozen property; and a neutral parent-names name for the view, which buys nothing the view can detect, since a matronymic and a patronymic are written alike. 4. **`generation` and `credentials` are views that partition `suffix` (#326)**, not roles, `generation` the tokens read as a generation or an agnomen, `credentials` the rest, the post-nominals that are neither (`Esq.`, the honorifics after a name) falling in `credentials`, CLDR's catch-all, with `Token.reading.kind` telling them apart for a caller who needs it. Measured: of the 1,054 committed corpus entries with a suffix, 33 hold a generation (or agnomen) and a listed credential together, 77 a generation beside any other suffix word, and 64 only listed post-nominals of other kinds (each suffix word classified by the RFC's Phase 4 recipe, joined tokens merged); of the 615 CLDR renderings with either field, the 75 whose suffix holds exactly CLDR's words split into CLDR's `generation` and `credentials` by the vocabulary's reading kind in all 75. Weighed against: two new `Role` members (item 1), and a third view for the honorifics, left open since CLDR has none. 5. **a `surname2` view, the family's second unit where rules/O6 read the family as two**, marked by a new stable tag where O6 takes its word, empty otherwise; `family` keeps both units as O6 makes it, and a family comma alone marks nothing (`Lloyd Webber, Andrew`). Measured: a two-word family box in ORCID's records from Spain 48.0%, Mexico 64.7%, Colombia 72.8% and Chile 50.5%, against 3.9% from the United States. Left open: a view for the first unit and its name, `surname` standing close to the `surnames` view that exists. Declined: redefining `surnames` as the two units, which would repeat `family`, the family holding both where O6 reads two, and lose what `surnames` does today, joining `middle` and `family` so that a double surname the default splits (`Gabriel García Márquez` → middle `García`) reads whole, the view rules/P2 names for that reading. - 6. **The render defaults follow the order the name was read in, a behavior change of the output rather than a break of the API.** `ParsedName` gains `order` (an addition, `ADDED_IN`, a 2.x pickle rendering as today, left out of comparison, which lets two equal names render differently, a cost accepted 2026-10-28: a 2.x pickle of `김민준` renders `'민준 김'` and a fresh parse `'김민준'`, and the two compare equal); the default `str()` and `render()` spec put the family first where `order` does, and join two words that stand one directly after the other, in the same order, in the original and the rendering with the kind of separator that stood between them there, normalized, decided 2026-10-28 (nothing; one space for any whitespace; an interpunct or a nakaguro the parse read as a separator alone, a spaced interpunct being a name word, rules/T3; one comma the parse reads as `, ` if ASCII, `، ` if Arabic and alone if full-width, so `John Smith ,Jr.` renders `'John Smith, Jr.'`; the written whitespace staying in `ParsedName.original` and the token spans), the spec's decoration between the two words' fields standing instead (the nickname's quotes, the `) ` after a maiden name), and any other pair or separator (a delimiter, a consumed marker word, two or more commas, a hyphen, a slash, a zero-width or directional mark) taking the spec's own joiner; `initials()` gives nothing for a word in a script with no initials. A name with a family comma or written given-first keeps today's word order and its separators by the same rule, a suffix comma alone leaving a family-first name family-first (`威廉·莎士比亚` keeps its dot; 20 of the 36 names whose interpunct `str()` drops are read given-first). A family comma is never kept, the rendering reversing its pair (`Smith, John` → `'John Smith'`), and one directly between the family and its suffix is, decided 2026-10-28 (`John Smith, Jr.` → `'John Smith, Jr.'`, today `'John Smith Jr.'`; 214 of the 1,003 committed corpus entries whose suffix follows the family write one there). Accepted with it: `Smith, John, Jr.`, whose suffix the rendering moves away from the given name before it, keeps `'John Smith Jr.'`, so two writings of one name render differently. Measured over the committed corpus's distinct entries, the counts reading the ASCII comma (the RFC gives the full-width and Arabic ones: 36 of the 728 hold a family comma the parse reads and keep today's word order): 728 without a comma read family-first, `str()` renders all 728 differently from their text (`김민준` → `'민준 김'`), 150 of them with the family's last word touching the given name's first, and it drops the U+00B7 interpunct from 36 of the 41 holding one. This carries out decisions/3-0-reevaluations' render-default clean slate. No signature changes, but `str()`'s output changes for every name read family-first, every name whose words are written with nothing, an interpunct or a nakaguro between them, and every name with one comma directly between its family and its suffix, and no 2.x release can warn about a `str()` result, so it is taken as an improvement for 3.0's release notes, not a break needing a warning. + 6. **The render defaults follow the order the name was read in, a behavior change of the output rather than a break of the API.** `ParsedName` gains `order` (an addition, `ADDED_IN`, a 2.x pickle rendering as today, left out of comparison, which lets two equal names render differently, a cost accepted 2026-10-28: a 2.x pickle of `김민준` renders `'민준 김'` and a fresh parse `'김민준'`, and the two compare equal); the default `str()` and `render()` spec put the family first where `order` does, and join two words that stand one directly after the other, in the same order, in the original and the rendering with the kind of separator that stood between them there, normalized, decided 2026-10-28 (nothing; one space for any whitespace, the ideographic space U+3000 included (`山田　太郎` → `'山田 太郎'`, as today); an interpunct or a nakaguro the parse read as a separator alone, a spaced interpunct being a name word, rules/T3; one comma the parse reads as `, ` if ASCII, `، ` if Arabic and alone if full-width, its own width a reason for the comma and not for whitespace, so `John Smith ,Jr.` renders `'John Smith, Jr.'`; the written whitespace staying in `ParsedName.original` and the token spans), the spec's decoration between the two words' fields standing instead (the nickname's quotes, the `) ` after a maiden name), and any other pair or separator (a delimiter, a consumed marker word, two or more commas, a hyphen, a slash, a zero-width or directional mark) taking the spec's own joiner; `initials()` gives nothing for a word in a script with no initials. A name with a family comma or written given-first keeps today's word order and its separators by the same rule, a suffix comma alone leaving a family-first name family-first (`威廉·莎士比亚` keeps its dot; 20 of the 36 names whose interpunct `str()` drops are read given-first). A family comma is never kept, the rendering reversing its pair (`Smith, John` → `'John Smith'`), and one directly between the family and its suffix is, decided 2026-10-28 (`John Smith, Jr.` → `'John Smith, Jr.'`, today `'John Smith Jr.'`; 214 of the 1,003 committed corpus entries whose suffix follows the family write one there). Accepted with it: `Smith, John, Jr.`, whose suffix the rendering moves away from the given name before it, keeps `'John Smith Jr.'`, so two writings of one name render differently. Measured over the committed corpus's distinct entries, the counts reading the ASCII comma (the RFC gives the full-width and Arabic ones: 36 of the 728 hold a family comma the parse reads and keep today's word order): 728 without a comma read family-first, `str()` renders all 728 differently from their text (`김민준` → `'민준 김'`), 150 of them with the family's last word touching the given name's first, and it drops the U+00B7 interpunct from 36 of the 41 holding one. This carries out decisions/3-0-reevaluations' render-default clean slate. No signature changes, but `str()`'s output changes for every name read family-first, every name whose words are written with nothing, an interpunct or a nakaguro between them, and every name with one comma directly between its family and its suffix, and no 2.x release can warn about a `str()` result, so it is taken as an improvement for 3.0's release notes, not a break needing a warning. 7. **`middle` keeps its meaning (#704)**, every word of the given part after the given name, the given name one unit unless a rule joins words into it, as rules/P5 already does (`Abdul Aziz Tariq` → given `Abdul Aziz`); a compound given name is a `[rules]` rule a caller or a pack turns on, CLDR's `given2` is `middle`, and `Kaur` and an English name beside a Chinese one stay in `middle`, an accepted cost. Measured: in PSC, whose form has a middle-name box, 2,499 of 1,040,708 forename boxes hold two or more words, and of 6,101 Spanish nationals 25 do while 2,621 fill the middle box; ORCID's and PatentsView's given box holds middles too, so it cannot separate the two. The `Maria do Carmo` shape, a particle inside a given box of three or more words with a family box filled, reads the given box's words into the family in 1,812 of 1,923 ORCID records and 1,165 of 1,219 PatentsView records from Brazil, Portugal, Spain, Mexico, Argentina, Colombia, Chile and Italy, which is that rule's case. Weighed against: #704's option (b), one given unit by default, which changes `middle` for every English name of three words. 8. **A language file's `[rules]` compiles into its shipped `Locale`'s policy patch**, each rule kind a `Policy` field (an addition), so RU and TR_AZ are rebuilt from their files and `Locale` gains no field for rules. With it, **phase 3's open API dependency is answered**: its draft (branch `feat/vocabulary-phase3-design`, unmerged) carries a choice of languages on `Locale.languages`, assuming `Locale` and `Policy` keep their 2.x shape after the facade's removal, and the 2026-10-28 freeze keeps them so. The rest of that draft, strict as an argument of the whole choice included, stays phase 3's to decide; its entry, when it lands, says the dependency is answered. 9. **The work order**, one PR each, measured on its own base, the snapshot recording each new view where it is non-empty: 4a `Token.reading` and `ReadingKind.PATRONYMIC` (neutral; `languages` after 3a); 4b `generation` and `credentials` (neutral); 4c `patronymic` (neutral); 4d `surname2` (neutral); 4e the render defaults and `order`; 4f `[rules]` into the policy patch, with or after 3b; 4g the compound given name, after 4f; 4h the shape inventory per order (#469). Interleaved with phase 3 (its sub-phases the draft's): 4a beside 3a; 4b to 4d before the packs that need them, `cs` (3f) and `pt-BR` (3h) on 4b, `es` and `pt` (3g) on 4d and 4g; an `is` pack on 3b, 4c and 4f. diff --git a/docs/design/vocabulary-model-rfc.md b/docs/design/vocabulary-model-rfc.md index 5234a17b0..9b9b26af2 100644 --- a/docs/design/vocabulary-model-rfc.md +++ b/docs/design/vocabulary-model-rfc.md @@ -134,7 +134,7 @@ Phase 2l carried this out as [rules/S13](rules/S13-bracketed-member-settled-by-i ### Readings on the parse -A caller should be able to see the reading behind a parse. **An `Ambiguity` carries the reading the parse did not take** (`Ambiguity.alternative`: its kind, the field it would have put the token in, and whether it came from the vocabulary, a declared name or a shape), and **a token carries the reading it did** (`Token.reading`), so a word whose reading forks nothing still says what it was read as, without a report nobody would act on (phase 4's 4a; a native spelling such as `do`'s Vietnamese `Đỗ` is not part of it). A word the credential's run takes by position alone is told from a word with a declared name reading by that source, so it needs no new ambiguity kind. `Ambiguity.alternative` shipped in 2.7 (decisions/A1, 2026-10-26 #752): a `Reading` of a kind, the field the word would land in with every other word read as the parse read it, and a source, which is `vocabulary`, `declared`, `shape` or `position`, the fourth being the reading of an ordinary word nothing lists, which is what a word the credential's run took by position alone has. The pickle guard was widened to accept a field added later. A `languages` field, naming the files that hold a reading, was weighed and left out: it needs the per-word reading index, which costs about 0.4 ms of a 19.5 ms import and could not describe a pack's or a caller's lexicon, so it waits with `Token.reading` for 3.0. +A caller should be able to see the reading behind a parse. **An `Ambiguity` carries the reading the parse did not take** (`Ambiguity.alternative`: its kind, the field it would have put the token in, and whether it came from the vocabulary, a declared name or a shape), and **a token carries the reading it did** (`Token.reading`), so a word whose reading forks nothing still says what it was read as, without a report nobody would act on (phase 4's 4a; a native spelling such as `do`'s Vietnamese `Đỗ` is not part of it). A word the credential's run takes by position alone is told from a word with a declared name reading by that source, so it needs no new ambiguity kind. `Ambiguity.alternative` shipped in 2.7 (decisions/A1, 2026-10-26 #752): a `Reading` of a kind, the field the word would land in with every other word read as the parse read it, and a source, which is `vocabulary`, `declared`, `shape` or `position`, the fourth being the reading of an ordinary word nothing lists, which is what a word the credential's run took by position alone has. The pickle guard was widened to accept a field added later. A `languages` field, naming the files that hold a reading, was weighed and left out: it needs the per-word reading index, which costs about 0.35 ms of an 18.2 ms import (decisions/A1) and could not describe a pack's or a caller's lexicon, so it waits with `Token.reading` for 3.0. ### The phase 1 reserved names @@ -250,7 +250,7 @@ ORCID's and PatentsView's given box holds the middle names too, so a two-word gi **Render defaults** (decisions/3-0-reevaluations' clean slate). The default `str()` reads every name given-first and joins its words with spaces, so a name read family-first is turned round and a written separator is lost: `str(parse("김민준"))` is `'민준 김'`, `str(parse("王 小明"))` is `'小明 王'` and `str(parse("威廉·莎士比亚"))` is `'威廉 莎士比亚'`. Over the committed corpus's distinct entries, each under its declared order (2026-10-28, `c8357e74`), 728 without a comma read family-first (the family's first word written before the given name's), `str()` renders every one differently from the text, and in 150 of them the family's last word touches the given name's first, written unspaced; of 41 holding the U+00B7 interpunct, `str()` drops it from 36. So: - `ParsedName` gains `order`, the order the name was read under (its script's or the declared `name_order`, a family comma aside), an addition with `ADDED_IN` metadata whose missing value, in a 2.x pickle, renders as today, and left out of comparison as `Token.reading` is, so a loaded 2.x pickle still equals a fresh parse of its text. The cost, accepted 2026-10-28: two equal names can render differently, where today every rendering is derived from what `==` compares (a 2.x pickle of `김민준` renders `'민준 김'` and a fresh parse `'김민준'`, and a set keeps one of them); the views read from `Token.reading` share it. -- `str()` and `render()`'s default spec put the family first where `order` does, and two words that stand one directly after the other, in the same order, in the original and in the rendering keep the kind of separator that stood between them in the original, written in a normalized form (decided 2026-10-28): nothing stays nothing; whitespace of any width or length becomes one ASCII space; an interpunct or a nakaguro the parse read as a separator stands alone, with no space around it (an interpunct written with spaces round it is not one, rules/T3 dividing at it only when flanked, so `威廉 · 莎士比亚` reads the dot as a name word and keeps rendering with its spaces, where `威廉·莎士比亚` keeps the dot unspaced; a spaced nakaguro is a separator, so `ジョン ・ スミス` renders `'ジョン・スミス'`); and one comma the parse reads as one becomes `, ` if ASCII and `، ` if Arabic (U+060C), no space before it and one after, and stands alone if full-width (U+FF0C), which carries its own width in CJK text, so `John Smith ,Jr.` and `John Smith, Jr.` render `'John Smith, Jr.'` and `John Smith， Jr.` renders `'John Smith，Jr.'`. Over that, the spec's decoration between the two fields the words belong to stands in place of the separator (the nickname's quotes, the `) ` after the maiden name, so `Jane Smith née Jones, PhD` keeps `'Jane Smith (Jones) PhD'`; 38 committed entries put a suffix directly after an undelimited maiden name); and any other pair, or any other separator (a delimiter, a marker word the parse consumed, two or more commas, a hyphen, a slash, a zero-width or directional mark), takes the spec's own joiner, as today (the written whitespace is not lost: `ParsedName.original` and the token spans keep it), so `John (Jack) Smith` keeps today's quoted nickname and `str(parse("김민준"))` is `'김민준'` and `str(parse("威廉·莎士比亚"))` keeps the dot. A name with a family comma or written given-first keeps today's word order, and its separators are kept by the same rule (a comma before a suffix alone does not make a family-first name given-first: `Wang Xiaoming, Jr.` under FAMILY_FIRST renders `'Wang Xiaoming, Jr.'`), so the given-first transcription `威廉·莎士比亚` keeps its dot too (20 of the 36 names whose interpunct `str()` drops are read given-first, 4 of them with a comma the parse reads (1 ASCII, 3 full-width)). A family comma is never kept, since the rendering reverses the pair it stands between (`Smith, John` → `'John Smith'`), while a comma directly between the family and its suffix is (`John Smith, Jr.` → `'John Smith, Jr.'`, where `str()` gives `'John Smith Jr.'` today): of the 1,003 committed corpus entries whose suffix starts after the family's last word ends, 214 have nothing between the two but one comma and whitespace (2026-10-28, `c8357e74`, each entry under its declared order; these counts read the ASCII comma, and 7 more write a full-width one there, while 36 of the 728 family-first names above hold a family comma the parse reads, 5 full-width and 31 Arabic (30 of the Arabic in `corpus_cldr.jsonl`), and so keep today's word order under the rule); 456 hold a comma somewhere between, most with a given name or title standing there too. Pairs the rendering separates, as `John` and `Jr.` in `Smith, John, Jr.`, take the spec's own joiner, as today, so `Smith, John, Jr.` keeps rendering `'John Smith Jr.'` while `John Smith, Jr.` becomes `'John Smith, Jr.'`: two writings of one name rendering differently, a consequence of following the text accepted with the choice. +- `str()` and `render()`'s default spec put the family first where `order` does, and two words that stand one directly after the other, in the same order, in the original and in the rendering keep the kind of separator that stood between them in the original, written in a normalized form (decided 2026-10-28): nothing stays nothing; whitespace of any width or length becomes one ASCII space, the ideographic space U+3000 included (decided 2026-10-28: `山田　太郎` renders `'山田 太郎'`, as `str()` joins it today, and a caller wanting native spacing passes a format spec of their own or reads `ParsedName.original`); an interpunct or a nakaguro the parse read as a separator stands alone, with no space around it (an interpunct written with spaces round it is not one, rules/T3 dividing at it only when flanked, so `威廉 · 莎士比亚` reads the dot as a name word and keeps rendering with its spaces, where `威廉·莎士比亚` keeps the dot unspaced; a spaced nakaguro is a separator, so `ジョン ・ スミス` renders `'ジョン・スミス'`); and one comma the parse reads as one becomes `, ` if ASCII and `، ` if Arabic (U+060C), no space before it and one after, and stands alone if full-width (U+FF0C), which carries its own width in CJK text, a reason that holds for the comma, a mark the rendering keeps, and not for whitespace, which the rendering normalizes, so `John Smith ,Jr.` and `John Smith, Jr.` render `'John Smith, Jr.'` and `John Smith， Jr.` renders `'John Smith，Jr.'`. The Japanese comma 、 (U+3001) is not one the parse reads as a comma, a parser question outside this design. Over that, the spec's decoration between the two fields the words belong to stands in place of the separator (the nickname's quotes, the `) ` after the maiden name, so `Jane Smith née Jones, PhD` keeps `'Jane Smith (Jones) PhD'`; 38 committed entries put a suffix directly after an undelimited maiden name); and any other pair, or any other separator (a delimiter, a marker word the parse consumed, two or more commas, a hyphen, a slash, a zero-width or directional mark), takes the spec's own joiner, as today (the written whitespace is not lost: `ParsedName.original` and the token spans keep it), so `John (Jack) Smith` keeps today's quoted nickname and `str(parse("김민준"))` is `'김민준'` and `str(parse("威廉·莎士比亚"))` keeps the dot. A name with a family comma or written given-first keeps today's word order, and its separators are kept by the same rule (a comma before a suffix alone does not make a family-first name given-first: `Wang Xiaoming, Jr.` under FAMILY_FIRST renders `'Wang Xiaoming, Jr.'`), so the given-first transcription `威廉·莎士比亚` keeps its dot too (20 of the 36 names whose interpunct `str()` drops are read given-first, 4 of them with a comma the parse reads (1 ASCII, 3 full-width)). A family comma is never kept, since the rendering reverses the pair it stands between (`Smith, John` → `'John Smith'`), while a comma directly between the family and its suffix is (`John Smith, Jr.` → `'John Smith, Jr.'`, where `str()` gives `'John Smith Jr.'` today): of the 1,003 committed corpus entries whose suffix starts after the family's last word ends, 214 have nothing between the two but one comma and whitespace (2026-10-28, `c8357e74`, each entry under its declared order; these counts read the ASCII comma, and 7 more write a full-width one there, while 36 of the 728 family-first names above hold a family comma the parse reads, 5 full-width and 31 Arabic (30 of the Arabic in `corpus_cldr.jsonl`), and so keep today's word order under the rule); 456 hold a comma somewhere between, most with a given name or title standing there too. Pairs the rendering separates, as `John` and `Jr.` in `Smith, John, Jr.`, take the spec's own joiner, as today, so `Smith, John, Jr.` keeps rendering `'John Smith Jr.'` while `John Smith, Jr.` becomes `'John Smith, Jr.'`: two writings of one name rendering differently, a consequence of following the text accepted with the choice. - `initials()` gives nothing for a word in a script with no initials (Han, Hangul, kana, `_policy._NO_INITIALS`), where today it gives the first character with a period (`'王.'`). These change no signature but change `str()`'s output for every name read family-first, every name whose words are written with nothing, an interpunct or a nakaguro between them, and every name written with one comma directly between its family and its suffix, which no 2.x release can warn about; Derek took them as an improvement for 3.0's release notes rather than a break (item 6 of the entry). From 87ed50236698587a6164cc966215f1b95afd0661 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Wed, 28 Oct 2026 14:10:39 -0700 Subject: [PATCH 24/39] Fix the eighth docs review's findings on phase 4's render rule (#754) The U+3000 example no longer reads as today's output, which is turned round; the decisions copy states the full-width comma's reason, says ASCII space, and carries the Japanese-comma note; the change list covers only separators between words the rendering keeps side by side. Co-Authored-By: Claude Opus 5.5 --- docs/design/decisions/vocabulary-model.md | 2 +- docs/design/vocabulary-model-rfc.md | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/design/decisions/vocabulary-model.md b/docs/design/decisions/vocabulary-model.md index 2f97977d8..502417363 100644 --- a/docs/design/decisions/vocabulary-model.md +++ b/docs/design/decisions/vocabulary-model.md @@ -250,7 +250,7 @@ Open: #913 (the title words phase 2j left, read as titles though borne as names) 3. **A `patronymic` view, filled by a word or a rule that says patronymic, with `family` and `middle` unchanged.** The default fills it from the vocabulary's `patronymic` particles (`bin`, `binti`, `ibn`, `bint`, `bar`, `bat` and the native-script forms) with the name they introduce (`Mahathir bin Mohamad` → family `bin Mohamad`, as today, and patronymic `bin Mohamad`); RU's and TR_AZ's rules fill it where a caller turns them on (`Ivan Ivanovich Ivanov` under RU → middle `Ivanovich` and patronymic `Ivanovich`); an Icelandic ending rule fills it under an `is` pack only, since the default cannot tell `-dóttir` and `-son` from Swedish surnames (ORCID's Swedish records end their family box in `-son` or `-dóttir` in 2,184 of 20,000). A matronymic, written the same way, fills it too. SUPERSEDES: the RFC's earlier statement of decision 8's value, that `family` can be empty for a name with a patronymic. Measured: filers put an Icelandic patronymic in the family box (ORCID 1,647 of 2,583 Icelandic records, PatentsView 535 of 722, PSC 101 of 125), and CLDR's `is` file's 3 Icelandic samples put it in `surname` or `surname-core` in all 9 of their renderings (its one foreign sample puts a `-dóttir` word in `surname2`), while Wikidata's `patronymic-is` and `patronymic-my` strata record 108 names with a patronymic and no family name, of which the parse puts the patronymic in `family` 107 times. An East Slavic patronymic stands in the given part: CLDR's 2 `ru` samples put it in `given2` in all 6 renderings; PatentsView's Russian given boxes end in one 8,423 times of 20,000, and the parse reads such a word as the middle 10,902 times of 10,962; 90 of the 112 Russian nationals in PSC who filled the middle-name box wrote one there. A `family` that keeps it and a `patronymic` that names it agree with both kinds of source. Weighed against: emptying `family` where the patronymic is recognized, which moves the field the fielded sources fill and changes `family` for every Icelandic and nasab name, a behavior change of a frozen property; and a neutral parent-names name for the view, which buys nothing the view can detect, since a matronymic and a patronymic are written alike. 4. **`generation` and `credentials` are views that partition `suffix` (#326)**, not roles, `generation` the tokens read as a generation or an agnomen, `credentials` the rest, the post-nominals that are neither (`Esq.`, the honorifics after a name) falling in `credentials`, CLDR's catch-all, with `Token.reading.kind` telling them apart for a caller who needs it. Measured: of the 1,054 committed corpus entries with a suffix, 33 hold a generation (or agnomen) and a listed credential together, 77 a generation beside any other suffix word, and 64 only listed post-nominals of other kinds (each suffix word classified by the RFC's Phase 4 recipe, joined tokens merged); of the 615 CLDR renderings with either field, the 75 whose suffix holds exactly CLDR's words split into CLDR's `generation` and `credentials` by the vocabulary's reading kind in all 75. Weighed against: two new `Role` members (item 1), and a third view for the honorifics, left open since CLDR has none. 5. **a `surname2` view, the family's second unit where rules/O6 read the family as two**, marked by a new stable tag where O6 takes its word, empty otherwise; `family` keeps both units as O6 makes it, and a family comma alone marks nothing (`Lloyd Webber, Andrew`). Measured: a two-word family box in ORCID's records from Spain 48.0%, Mexico 64.7%, Colombia 72.8% and Chile 50.5%, against 3.9% from the United States. Left open: a view for the first unit and its name, `surname` standing close to the `surnames` view that exists. Declined: redefining `surnames` as the two units, which would repeat `family`, the family holding both where O6 reads two, and lose what `surnames` does today, joining `middle` and `family` so that a double surname the default splits (`Gabriel García Márquez` → middle `García`) reads whole, the view rules/P2 names for that reading. - 6. **The render defaults follow the order the name was read in, a behavior change of the output rather than a break of the API.** `ParsedName` gains `order` (an addition, `ADDED_IN`, a 2.x pickle rendering as today, left out of comparison, which lets two equal names render differently, a cost accepted 2026-10-28: a 2.x pickle of `김민준` renders `'민준 김'` and a fresh parse `'김민준'`, and the two compare equal); the default `str()` and `render()` spec put the family first where `order` does, and join two words that stand one directly after the other, in the same order, in the original and the rendering with the kind of separator that stood between them there, normalized, decided 2026-10-28 (nothing; one space for any whitespace, the ideographic space U+3000 included (`山田　太郎` → `'山田 太郎'`, as today); an interpunct or a nakaguro the parse read as a separator alone, a spaced interpunct being a name word, rules/T3; one comma the parse reads as `, ` if ASCII, `، ` if Arabic and alone if full-width, its own width a reason for the comma and not for whitespace, so `John Smith ,Jr.` renders `'John Smith, Jr.'`; the written whitespace staying in `ParsedName.original` and the token spans), the spec's decoration between the two words' fields standing instead (the nickname's quotes, the `) ` after a maiden name), and any other pair or separator (a delimiter, a consumed marker word, two or more commas, a hyphen, a slash, a zero-width or directional mark) taking the spec's own joiner; `initials()` gives nothing for a word in a script with no initials. A name with a family comma or written given-first keeps today's word order and its separators by the same rule, a suffix comma alone leaving a family-first name family-first (`威廉·莎士比亚` keeps its dot; 20 of the 36 names whose interpunct `str()` drops are read given-first). A family comma is never kept, the rendering reversing its pair (`Smith, John` → `'John Smith'`), and one directly between the family and its suffix is, decided 2026-10-28 (`John Smith, Jr.` → `'John Smith, Jr.'`, today `'John Smith Jr.'`; 214 of the 1,003 committed corpus entries whose suffix follows the family write one there). Accepted with it: `Smith, John, Jr.`, whose suffix the rendering moves away from the given name before it, keeps `'John Smith Jr.'`, so two writings of one name render differently. Measured over the committed corpus's distinct entries, the counts reading the ASCII comma (the RFC gives the full-width and Arabic ones: 36 of the 728 hold a family comma the parse reads and keep today's word order): 728 without a comma read family-first, `str()` renders all 728 differently from their text (`김민준` → `'민준 김'`), 150 of them with the family's last word touching the given name's first, and it drops the U+00B7 interpunct from 36 of the 41 holding one. This carries out decisions/3-0-reevaluations' render-default clean slate. No signature changes, but `str()`'s output changes for every name read family-first, every name whose words are written with nothing, an interpunct or a nakaguro between them, and every name with one comma directly between its family and its suffix, and no 2.x release can warn about a `str()` result, so it is taken as an improvement for 3.0's release notes, not a break needing a warning. + 6. **The render defaults follow the order the name was read in, a behavior change of the output rather than a break of the API.** `ParsedName` gains `order` (an addition, `ADDED_IN`, a 2.x pickle rendering as today, left out of comparison, which lets two equal names render differently, a cost accepted 2026-10-28: a 2.x pickle of `김민준` renders `'민준 김'` and a fresh parse `'김민준'`, and the two compare equal); the default `str()` and `render()` spec put the family first where `order` does, and join two words that stand one directly after the other, in the same order, in the original and the rendering with the kind of separator that stood between them there, normalized, decided 2026-10-28 (nothing; one ASCII space for any whitespace, the ideographic space U+3000 included (`山田　太郎` → `'山田 太郎'`, where today gives `'太郎 山田'`); an interpunct or a nakaguro the parse read as a separator alone, a spaced interpunct being a name word, rules/T3; one comma the parse reads as `, ` if ASCII, `، ` if Arabic and alone if full-width, since a full-width comma carries its own width in CJK text, a reason for the comma the rendering keeps and not for the whitespace it normalizes (the Japanese comma U+3001 is no comma to the parse, a parser question outside this design), so `John Smith ,Jr.` renders `'John Smith, Jr.'`; the written whitespace staying in `ParsedName.original` and the token spans), the spec's decoration between the two words' fields standing instead (the nickname's quotes, the `) ` after a maiden name), and any other pair or separator (a delimiter, a consumed marker word, two or more commas, a hyphen, a slash, a zero-width or directional mark) taking the spec's own joiner; `initials()` gives nothing for a word in a script with no initials. A name with a family comma or written given-first keeps today's word order and its separators by the same rule, a suffix comma alone leaving a family-first name family-first (`威廉·莎士比亚` keeps its dot; 20 of the 36 names whose interpunct `str()` drops are read given-first). A family comma is never kept, the rendering reversing its pair (`Smith, John` → `'John Smith'`), and one directly between the family and its suffix is, decided 2026-10-28 (`John Smith, Jr.` → `'John Smith, Jr.'`, today `'John Smith Jr.'`; 214 of the 1,003 committed corpus entries whose suffix follows the family write one there). Accepted with it: `Smith, John, Jr.`, whose suffix the rendering moves away from the given name before it, keeps `'John Smith Jr.'`, so two writings of one name render differently. Measured over the committed corpus's distinct entries, the counts reading the ASCII comma (the RFC gives the full-width and Arabic ones: 36 of the 728 hold a family comma the parse reads and keep today's word order): 728 without a comma read family-first, `str()` renders all 728 differently from their text (`김민준` → `'민준 김'`), 150 of them with the family's last word touching the given name's first, and it drops the U+00B7 interpunct from 36 of the 41 holding one. This carries out decisions/3-0-reevaluations' render-default clean slate. No signature changes, but `str()`'s output changes for every name read family-first, every name whose words are written with nothing, an interpunct or a nakaguro between two words the rendering keeps side by side, and every name with one comma directly between its family and its suffix, and no 2.x release can warn about a `str()` result, so it is taken as an improvement for 3.0's release notes, not a break needing a warning. 7. **`middle` keeps its meaning (#704)**, every word of the given part after the given name, the given name one unit unless a rule joins words into it, as rules/P5 already does (`Abdul Aziz Tariq` → given `Abdul Aziz`); a compound given name is a `[rules]` rule a caller or a pack turns on, CLDR's `given2` is `middle`, and `Kaur` and an English name beside a Chinese one stay in `middle`, an accepted cost. Measured: in PSC, whose form has a middle-name box, 2,499 of 1,040,708 forename boxes hold two or more words, and of 6,101 Spanish nationals 25 do while 2,621 fill the middle box; ORCID's and PatentsView's given box holds middles too, so it cannot separate the two. The `Maria do Carmo` shape, a particle inside a given box of three or more words with a family box filled, reads the given box's words into the family in 1,812 of 1,923 ORCID records and 1,165 of 1,219 PatentsView records from Brazil, Portugal, Spain, Mexico, Argentina, Colombia, Chile and Italy, which is that rule's case. Weighed against: #704's option (b), one given unit by default, which changes `middle` for every English name of three words. 8. **A language file's `[rules]` compiles into its shipped `Locale`'s policy patch**, each rule kind a `Policy` field (an addition), so RU and TR_AZ are rebuilt from their files and `Locale` gains no field for rules. With it, **phase 3's open API dependency is answered**: its draft (branch `feat/vocabulary-phase3-design`, unmerged) carries a choice of languages on `Locale.languages`, assuming `Locale` and `Policy` keep their 2.x shape after the facade's removal, and the 2026-10-28 freeze keeps them so. The rest of that draft, strict as an argument of the whole choice included, stays phase 3's to decide; its entry, when it lands, says the dependency is answered. 9. **The work order**, one PR each, measured on its own base, the snapshot recording each new view where it is non-empty: 4a `Token.reading` and `ReadingKind.PATRONYMIC` (neutral; `languages` after 3a); 4b `generation` and `credentials` (neutral); 4c `patronymic` (neutral); 4d `surname2` (neutral); 4e the render defaults and `order`; 4f `[rules]` into the policy patch, with or after 3b; 4g the compound given name, after 4f; 4h the shape inventory per order (#469). Interleaved with phase 3 (its sub-phases the draft's): 4a beside 3a; 4b to 4d before the packs that need them, `cs` (3f) and `pt-BR` (3h) on 4b, `es` and `pt` (3g) on 4d and 4g; an `is` pack on 3b, 4c and 4f. diff --git a/docs/design/vocabulary-model-rfc.md b/docs/design/vocabulary-model-rfc.md index 9b9b26af2..e9b223de3 100644 --- a/docs/design/vocabulary-model-rfc.md +++ b/docs/design/vocabulary-model-rfc.md @@ -250,10 +250,10 @@ ORCID's and PatentsView's given box holds the middle names too, so a two-word gi **Render defaults** (decisions/3-0-reevaluations' clean slate). The default `str()` reads every name given-first and joins its words with spaces, so a name read family-first is turned round and a written separator is lost: `str(parse("김민준"))` is `'민준 김'`, `str(parse("王 小明"))` is `'小明 王'` and `str(parse("威廉·莎士比亚"))` is `'威廉 莎士比亚'`. Over the committed corpus's distinct entries, each under its declared order (2026-10-28, `c8357e74`), 728 without a comma read family-first (the family's first word written before the given name's), `str()` renders every one differently from the text, and in 150 of them the family's last word touches the given name's first, written unspaced; of 41 holding the U+00B7 interpunct, `str()` drops it from 36. So: - `ParsedName` gains `order`, the order the name was read under (its script's or the declared `name_order`, a family comma aside), an addition with `ADDED_IN` metadata whose missing value, in a 2.x pickle, renders as today, and left out of comparison as `Token.reading` is, so a loaded 2.x pickle still equals a fresh parse of its text. The cost, accepted 2026-10-28: two equal names can render differently, where today every rendering is derived from what `==` compares (a 2.x pickle of `김민준` renders `'민준 김'` and a fresh parse `'김민준'`, and a set keeps one of them); the views read from `Token.reading` share it. -- `str()` and `render()`'s default spec put the family first where `order` does, and two words that stand one directly after the other, in the same order, in the original and in the rendering keep the kind of separator that stood between them in the original, written in a normalized form (decided 2026-10-28): nothing stays nothing; whitespace of any width or length becomes one ASCII space, the ideographic space U+3000 included (decided 2026-10-28: `山田　太郎` renders `'山田 太郎'`, as `str()` joins it today, and a caller wanting native spacing passes a format spec of their own or reads `ParsedName.original`); an interpunct or a nakaguro the parse read as a separator stands alone, with no space around it (an interpunct written with spaces round it is not one, rules/T3 dividing at it only when flanked, so `威廉 · 莎士比亚` reads the dot as a name word and keeps rendering with its spaces, where `威廉·莎士比亚` keeps the dot unspaced; a spaced nakaguro is a separator, so `ジョン ・ スミス` renders `'ジョン・スミス'`); and one comma the parse reads as one becomes `, ` if ASCII and `، ` if Arabic (U+060C), no space before it and one after, and stands alone if full-width (U+FF0C), which carries its own width in CJK text, a reason that holds for the comma, a mark the rendering keeps, and not for whitespace, which the rendering normalizes, so `John Smith ,Jr.` and `John Smith, Jr.` render `'John Smith, Jr.'` and `John Smith， Jr.` renders `'John Smith，Jr.'`. The Japanese comma 、 (U+3001) is not one the parse reads as a comma, a parser question outside this design. Over that, the spec's decoration between the two fields the words belong to stands in place of the separator (the nickname's quotes, the `) ` after the maiden name, so `Jane Smith née Jones, PhD` keeps `'Jane Smith (Jones) PhD'`; 38 committed entries put a suffix directly after an undelimited maiden name); and any other pair, or any other separator (a delimiter, a marker word the parse consumed, two or more commas, a hyphen, a slash, a zero-width or directional mark), takes the spec's own joiner, as today (the written whitespace is not lost: `ParsedName.original` and the token spans keep it), so `John (Jack) Smith` keeps today's quoted nickname and `str(parse("김민준"))` is `'김민준'` and `str(parse("威廉·莎士比亚"))` keeps the dot. A name with a family comma or written given-first keeps today's word order, and its separators are kept by the same rule (a comma before a suffix alone does not make a family-first name given-first: `Wang Xiaoming, Jr.` under FAMILY_FIRST renders `'Wang Xiaoming, Jr.'`), so the given-first transcription `威廉·莎士比亚` keeps its dot too (20 of the 36 names whose interpunct `str()` drops are read given-first, 4 of them with a comma the parse reads (1 ASCII, 3 full-width)). A family comma is never kept, since the rendering reverses the pair it stands between (`Smith, John` → `'John Smith'`), while a comma directly between the family and its suffix is (`John Smith, Jr.` → `'John Smith, Jr.'`, where `str()` gives `'John Smith Jr.'` today): of the 1,003 committed corpus entries whose suffix starts after the family's last word ends, 214 have nothing between the two but one comma and whitespace (2026-10-28, `c8357e74`, each entry under its declared order; these counts read the ASCII comma, and 7 more write a full-width one there, while 36 of the 728 family-first names above hold a family comma the parse reads, 5 full-width and 31 Arabic (30 of the Arabic in `corpus_cldr.jsonl`), and so keep today's word order under the rule); 456 hold a comma somewhere between, most with a given name or title standing there too. Pairs the rendering separates, as `John` and `Jr.` in `Smith, John, Jr.`, take the spec's own joiner, as today, so `Smith, John, Jr.` keeps rendering `'John Smith Jr.'` while `John Smith, Jr.` becomes `'John Smith, Jr.'`: two writings of one name rendering differently, a consequence of following the text accepted with the choice. +- `str()` and `render()`'s default spec put the family first where `order` does, and two words that stand one directly after the other, in the same order, in the original and in the rendering keep the kind of separator that stood between them in the original, written in a normalized form (decided 2026-10-28): nothing stays nothing; whitespace of any width or length becomes one ASCII space, the ideographic space U+3000 included (decided 2026-10-28: `山田　太郎` renders `'山田 太郎'`, the same ASCII space `str()` joins with today, where today's order gives `'太郎 山田'`, and a caller wanting native spacing passes a format spec of their own or reads `ParsedName.original`); an interpunct or a nakaguro the parse read as a separator stands alone, with no space around it (an interpunct written with spaces round it is not one, rules/T3 dividing at it only when flanked, so `威廉 · 莎士比亚` reads the dot as a name word and keeps rendering with its spaces, where `威廉·莎士比亚` keeps the dot unspaced; a spaced nakaguro is a separator, so `ジョン ・ スミス` renders `'ジョン・スミス'`); and one comma the parse reads as one becomes `, ` if ASCII and `، ` if Arabic (U+060C), no space before it and one after, and stands alone if full-width (U+FF0C), which carries its own width in CJK text, a reason that holds for the comma, a mark the rendering keeps, and not for whitespace, which the rendering normalizes, so `John Smith ,Jr.` and `John Smith, Jr.` render `'John Smith, Jr.'` and `John Smith， Jr.` renders `'John Smith，Jr.'`. The Japanese comma 、 (U+3001) is not one the parse reads as a comma, a parser question outside this design. Over that, the spec's decoration between the two fields the words belong to stands in place of the separator (the nickname's quotes, the `) ` after the maiden name, so `Jane Smith née Jones, PhD` keeps `'Jane Smith (Jones) PhD'`; 38 committed entries put a suffix directly after an undelimited maiden name); and any other pair, or any other separator (a delimiter, a marker word the parse consumed, two or more commas, a hyphen, a slash, a zero-width or directional mark), takes the spec's own joiner, as today (the written whitespace is not lost: `ParsedName.original` and the token spans keep it), so `John (Jack) Smith` keeps today's quoted nickname and `str(parse("김민준"))` is `'김민준'` and `str(parse("威廉·莎士比亚"))` keeps the dot. A name with a family comma or written given-first keeps today's word order, and its separators are kept by the same rule (a comma before a suffix alone does not make a family-first name given-first: `Wang Xiaoming, Jr.` under FAMILY_FIRST renders `'Wang Xiaoming, Jr.'`), so the given-first transcription `威廉·莎士比亚` keeps its dot too (20 of the 36 names whose interpunct `str()` drops are read given-first, 4 of them with a comma the parse reads (1 ASCII, 3 full-width)). A family comma is never kept, since the rendering reverses the pair it stands between (`Smith, John` → `'John Smith'`), while a comma directly between the family and its suffix is (`John Smith, Jr.` → `'John Smith, Jr.'`, where `str()` gives `'John Smith Jr.'` today): of the 1,003 committed corpus entries whose suffix starts after the family's last word ends, 214 have nothing between the two but one comma and whitespace (2026-10-28, `c8357e74`, each entry under its declared order; these counts read the ASCII comma, and 7 more write a full-width one there, while 36 of the 728 family-first names above hold a family comma the parse reads, 5 full-width and 31 Arabic (30 of the Arabic in `corpus_cldr.jsonl`), and so keep today's word order under the rule); 456 hold a comma somewhere between, most with a given name or title standing there too. Pairs the rendering separates, as `John` and `Jr.` in `Smith, John, Jr.`, take the spec's own joiner, as today, so `Smith, John, Jr.` keeps rendering `'John Smith Jr.'` while `John Smith, Jr.` becomes `'John Smith, Jr.'`: two writings of one name rendering differently, a consequence of following the text accepted with the choice. - `initials()` gives nothing for a word in a script with no initials (Han, Hangul, kana, `_policy._NO_INITIALS`), where today it gives the first character with a period (`'王.'`). -These change no signature but change `str()`'s output for every name read family-first, every name whose words are written with nothing, an interpunct or a nakaguro between them, and every name written with one comma directly between its family and its suffix, which no 2.x release can warn about; Derek took them as an improvement for 3.0's release notes rather than a break (item 6 of the entry). +These change no signature but change `str()`'s output for every name read family-first, every name whose words are written with nothing, an interpunct or a nakaguro between two words the rendering keeps side by side (not `김, 민준씨`, whose family comma moves `김` between `민준` and `씨`), and every name written with one comma directly between its family and its suffix, which no 2.x release can warn about; Derek took them as an improvement for 3.0's release notes rather than a break (item 6 of the entry). **What changes in the API**, by the 3.0 freeze: From 1dfbdb63d06481a0026dfa2798d135d4113cb165 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Wed, 28 Oct 2026 14:12:19 -0700 Subject: [PATCH 25/39] Propose how phase 4's render default reads order (#754) render(spec=None): None is the order-aware default str() passes, and a caller's spec renders exactly as written, so 2.x specs keep their meaning. The family-first layout keeps title first and suffix last with the name roles in order's sequence, checked against CLDR's surnameFirst samples. Both marked proposed. Co-Authored-By: Claude Opus 5.5 --- docs/design/decisions/vocabulary-model.md | 2 +- docs/design/vocabulary-model-rfc.md | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/design/decisions/vocabulary-model.md b/docs/design/decisions/vocabulary-model.md index 502417363..3877e6813 100644 --- a/docs/design/decisions/vocabulary-model.md +++ b/docs/design/decisions/vocabulary-model.md @@ -250,7 +250,7 @@ Open: #913 (the title words phase 2j left, read as titles though borne as names) 3. **A `patronymic` view, filled by a word or a rule that says patronymic, with `family` and `middle` unchanged.** The default fills it from the vocabulary's `patronymic` particles (`bin`, `binti`, `ibn`, `bint`, `bar`, `bat` and the native-script forms) with the name they introduce (`Mahathir bin Mohamad` → family `bin Mohamad`, as today, and patronymic `bin Mohamad`); RU's and TR_AZ's rules fill it where a caller turns them on (`Ivan Ivanovich Ivanov` under RU → middle `Ivanovich` and patronymic `Ivanovich`); an Icelandic ending rule fills it under an `is` pack only, since the default cannot tell `-dóttir` and `-son` from Swedish surnames (ORCID's Swedish records end their family box in `-son` or `-dóttir` in 2,184 of 20,000). A matronymic, written the same way, fills it too. SUPERSEDES: the RFC's earlier statement of decision 8's value, that `family` can be empty for a name with a patronymic. Measured: filers put an Icelandic patronymic in the family box (ORCID 1,647 of 2,583 Icelandic records, PatentsView 535 of 722, PSC 101 of 125), and CLDR's `is` file's 3 Icelandic samples put it in `surname` or `surname-core` in all 9 of their renderings (its one foreign sample puts a `-dóttir` word in `surname2`), while Wikidata's `patronymic-is` and `patronymic-my` strata record 108 names with a patronymic and no family name, of which the parse puts the patronymic in `family` 107 times. An East Slavic patronymic stands in the given part: CLDR's 2 `ru` samples put it in `given2` in all 6 renderings; PatentsView's Russian given boxes end in one 8,423 times of 20,000, and the parse reads such a word as the middle 10,902 times of 10,962; 90 of the 112 Russian nationals in PSC who filled the middle-name box wrote one there. A `family` that keeps it and a `patronymic` that names it agree with both kinds of source. Weighed against: emptying `family` where the patronymic is recognized, which moves the field the fielded sources fill and changes `family` for every Icelandic and nasab name, a behavior change of a frozen property; and a neutral parent-names name for the view, which buys nothing the view can detect, since a matronymic and a patronymic are written alike. 4. **`generation` and `credentials` are views that partition `suffix` (#326)**, not roles, `generation` the tokens read as a generation or an agnomen, `credentials` the rest, the post-nominals that are neither (`Esq.`, the honorifics after a name) falling in `credentials`, CLDR's catch-all, with `Token.reading.kind` telling them apart for a caller who needs it. Measured: of the 1,054 committed corpus entries with a suffix, 33 hold a generation (or agnomen) and a listed credential together, 77 a generation beside any other suffix word, and 64 only listed post-nominals of other kinds (each suffix word classified by the RFC's Phase 4 recipe, joined tokens merged); of the 615 CLDR renderings with either field, the 75 whose suffix holds exactly CLDR's words split into CLDR's `generation` and `credentials` by the vocabulary's reading kind in all 75. Weighed against: two new `Role` members (item 1), and a third view for the honorifics, left open since CLDR has none. 5. **a `surname2` view, the family's second unit where rules/O6 read the family as two**, marked by a new stable tag where O6 takes its word, empty otherwise; `family` keeps both units as O6 makes it, and a family comma alone marks nothing (`Lloyd Webber, Andrew`). Measured: a two-word family box in ORCID's records from Spain 48.0%, Mexico 64.7%, Colombia 72.8% and Chile 50.5%, against 3.9% from the United States. Left open: a view for the first unit and its name, `surname` standing close to the `surnames` view that exists. Declined: redefining `surnames` as the two units, which would repeat `family`, the family holding both where O6 reads two, and lose what `surnames` does today, joining `middle` and `family` so that a double surname the default splits (`Gabriel García Márquez` → middle `García`) reads whole, the view rules/P2 names for that reading. - 6. **The render defaults follow the order the name was read in, a behavior change of the output rather than a break of the API.** `ParsedName` gains `order` (an addition, `ADDED_IN`, a 2.x pickle rendering as today, left out of comparison, which lets two equal names render differently, a cost accepted 2026-10-28: a 2.x pickle of `김민준` renders `'민준 김'` and a fresh parse `'김민준'`, and the two compare equal); the default `str()` and `render()` spec put the family first where `order` does, and join two words that stand one directly after the other, in the same order, in the original and the rendering with the kind of separator that stood between them there, normalized, decided 2026-10-28 (nothing; one ASCII space for any whitespace, the ideographic space U+3000 included (`山田　太郎` → `'山田 太郎'`, where today gives `'太郎 山田'`); an interpunct or a nakaguro the parse read as a separator alone, a spaced interpunct being a name word, rules/T3; one comma the parse reads as `, ` if ASCII, `، ` if Arabic and alone if full-width, since a full-width comma carries its own width in CJK text, a reason for the comma the rendering keeps and not for the whitespace it normalizes (the Japanese comma U+3001 is no comma to the parse, a parser question outside this design), so `John Smith ,Jr.` renders `'John Smith, Jr.'`; the written whitespace staying in `ParsedName.original` and the token spans), the spec's decoration between the two words' fields standing instead (the nickname's quotes, the `) ` after a maiden name), and any other pair or separator (a delimiter, a consumed marker word, two or more commas, a hyphen, a slash, a zero-width or directional mark) taking the spec's own joiner; `initials()` gives nothing for a word in a script with no initials. A name with a family comma or written given-first keeps today's word order and its separators by the same rule, a suffix comma alone leaving a family-first name family-first (`威廉·莎士比亚` keeps its dot; 20 of the 36 names whose interpunct `str()` drops are read given-first). A family comma is never kept, the rendering reversing its pair (`Smith, John` → `'John Smith'`), and one directly between the family and its suffix is, decided 2026-10-28 (`John Smith, Jr.` → `'John Smith, Jr.'`, today `'John Smith Jr.'`; 214 of the 1,003 committed corpus entries whose suffix follows the family write one there). Accepted with it: `Smith, John, Jr.`, whose suffix the rendering moves away from the given name before it, keeps `'John Smith Jr.'`, so two writings of one name render differently. Measured over the committed corpus's distinct entries, the counts reading the ASCII comma (the RFC gives the full-width and Arabic ones: 36 of the 728 hold a family comma the parse reads and keep today's word order): 728 without a comma read family-first, `str()` renders all 728 differently from their text (`김민준` → `'민준 김'`), 150 of them with the family's last word touching the given name's first, and it drops the U+00B7 interpunct from 36 of the 41 holding one. This carries out decisions/3-0-reevaluations' render-default clean slate. No signature changes, but `str()`'s output changes for every name read family-first, every name whose words are written with nothing, an interpunct or a nakaguro between two words the rendering keeps side by side, and every name with one comma directly between its family and its suffix, and no 2.x release can warn about a `str()` result, so it is taken as an improvement for 3.0's release notes, not a break needing a warning. + 6. **The render defaults follow the order the name was read in, a behavior change of the output rather than a break of the API.** `ParsedName` gains `order` (an addition, `ADDED_IN`, a 2.x pickle rendering as today, left out of comparison, which lets two equal names render differently, a cost accepted 2026-10-28: a 2.x pickle of `김민준` renders `'민준 김'` and a fresh parse `'김민준'`, and the two compare equal); the default `str()` and `render()` spec put the family first where `order` does, and join two words that stand one directly after the other, in the same order, in the original and the rendering with the kind of separator that stood between them there, normalized, decided 2026-10-28 (nothing; one ASCII space for any whitespace, the ideographic space U+3000 included (`山田　太郎` → `'山田 太郎'`, where today gives `'太郎 山田'`); an interpunct or a nakaguro the parse read as a separator alone, a spaced interpunct being a name word, rules/T3; one comma the parse reads as `, ` if ASCII, `، ` if Arabic and alone if full-width, since a full-width comma carries its own width in CJK text, a reason for the comma the rendering keeps and not for the whitespace it normalizes (the Japanese comma U+3001 is no comma to the parse, a parser question outside this design), so `John Smith ,Jr.` renders `'John Smith, Jr.'`; the written whitespace staying in `ParsedName.original` and the token spans), the spec's decoration between the two words' fields standing instead (the nickname's quotes, the `) ` after a maiden name), and any other pair or separator (a delimiter, a consumed marker word, two or more commas, a hyphen, a slash, a zero-width or directional mark) taking the spec's own joiner; `initials()` gives nothing for a word in a script with no initials. A name with a family comma or written given-first keeps today's word order and its separators by the same rule, a suffix comma alone leaving a family-first name family-first (`威廉·莎士比亚` keeps its dot; 20 of the 36 names whose interpunct `str()` drops are read given-first). A family comma is never kept, the rendering reversing its pair (`Smith, John` → `'John Smith'`), and one directly between the family and its suffix is, decided 2026-10-28 (`John Smith, Jr.` → `'John Smith, Jr.'`, today `'John Smith Jr.'`; 214 of the 1,003 committed corpus entries whose suffix follows the family write one there). Accepted with it: `Smith, John, Jr.`, whose suffix the rendering moves away from the given name before it, keeps `'John Smith Jr.'`, so two writings of one name render differently. Measured over the committed corpus's distinct entries, the counts reading the ASCII comma (the RFC gives the full-width and Arabic ones: 36 of the 728 hold a family comma the parse reads and keep today's word order): 728 without a comma read family-first, `str()` renders all 728 differently from their text (`김민준` → `'민준 김'`), 150 of them with the family's last word touching the given name's first, and it drops the U+00B7 interpunct from 36 of the 41 holding one. This carries out decisions/3-0-reevaluations' render-default clean slate. Proposed with it, not yet decided (the RFC's Phase 4 Render defaults): `render(spec: str | None = None)`, `None` the order-aware default `str()` passes and a caller's spec rendering exactly as written, so `render("{given} {family}")` keeps its 2.x meaning and a caller passing 2.x's default string keeps given-first output; and a family-first layout with the title first, the suffix last and the name roles in `order`'s sequence, each with its decoration. No signature changes, but `str()`'s output changes for every name read family-first, every name whose words are written with nothing, an interpunct or a nakaguro between two words the rendering keeps side by side, and every name with one comma directly between its family and its suffix, and no 2.x release can warn about a `str()` result, so it is taken as an improvement for 3.0's release notes, not a break needing a warning. 7. **`middle` keeps its meaning (#704)**, every word of the given part after the given name, the given name one unit unless a rule joins words into it, as rules/P5 already does (`Abdul Aziz Tariq` → given `Abdul Aziz`); a compound given name is a `[rules]` rule a caller or a pack turns on, CLDR's `given2` is `middle`, and `Kaur` and an English name beside a Chinese one stay in `middle`, an accepted cost. Measured: in PSC, whose form has a middle-name box, 2,499 of 1,040,708 forename boxes hold two or more words, and of 6,101 Spanish nationals 25 do while 2,621 fill the middle box; ORCID's and PatentsView's given box holds middles too, so it cannot separate the two. The `Maria do Carmo` shape, a particle inside a given box of three or more words with a family box filled, reads the given box's words into the family in 1,812 of 1,923 ORCID records and 1,165 of 1,219 PatentsView records from Brazil, Portugal, Spain, Mexico, Argentina, Colombia, Chile and Italy, which is that rule's case. Weighed against: #704's option (b), one given unit by default, which changes `middle` for every English name of three words. 8. **A language file's `[rules]` compiles into its shipped `Locale`'s policy patch**, each rule kind a `Policy` field (an addition), so RU and TR_AZ are rebuilt from their files and `Locale` gains no field for rules. With it, **phase 3's open API dependency is answered**: its draft (branch `feat/vocabulary-phase3-design`, unmerged) carries a choice of languages on `Locale.languages`, assuming `Locale` and `Policy` keep their 2.x shape after the facade's removal, and the 2026-10-28 freeze keeps them so. The rest of that draft, strict as an argument of the whole choice included, stays phase 3's to decide; its entry, when it lands, says the dependency is answered. 9. **The work order**, one PR each, measured on its own base, the snapshot recording each new view where it is non-empty: 4a `Token.reading` and `ReadingKind.PATRONYMIC` (neutral; `languages` after 3a); 4b `generation` and `credentials` (neutral); 4c `patronymic` (neutral); 4d `surname2` (neutral); 4e the render defaults and `order`; 4f `[rules]` into the policy patch, with or after 3b; 4g the compound given name, after 4f; 4h the shape inventory per order (#469). Interleaved with phase 3 (its sub-phases the draft's): 4a beside 3a; 4b to 4d before the packs that need them, `cs` (3f) and `pt-BR` (3h) on 4b, `es` and `pt` (3g) on 4d and 4g; an `is` pack on 3b, 4c and 4f. diff --git a/docs/design/vocabulary-model-rfc.md b/docs/design/vocabulary-model-rfc.md index e9b223de3..e9c797a44 100644 --- a/docs/design/vocabulary-model-rfc.md +++ b/docs/design/vocabulary-model-rfc.md @@ -251,6 +251,7 @@ ORCID's and PatentsView's given box holds the middle names too, so a two-word gi - `ParsedName` gains `order`, the order the name was read under (its script's or the declared `name_order`, a family comma aside), an addition with `ADDED_IN` metadata whose missing value, in a 2.x pickle, renders as today, and left out of comparison as `Token.reading` is, so a loaded 2.x pickle still equals a fresh parse of its text. The cost, accepted 2026-10-28: two equal names can render differently, where today every rendering is derived from what `==` compares (a 2.x pickle of `김민준` renders `'민준 김'` and a fresh parse `'김민준'`, and a set keeps one of them); the views read from `Token.reading` share it. - `str()` and `render()`'s default spec put the family first where `order` does, and two words that stand one directly after the other, in the same order, in the original and in the rendering keep the kind of separator that stood between them in the original, written in a normalized form (decided 2026-10-28): nothing stays nothing; whitespace of any width or length becomes one ASCII space, the ideographic space U+3000 included (decided 2026-10-28: `山田　太郎` renders `'山田 太郎'`, the same ASCII space `str()` joins with today, where today's order gives `'太郎 山田'`, and a caller wanting native spacing passes a format spec of their own or reads `ParsedName.original`); an interpunct or a nakaguro the parse read as a separator stands alone, with no space around it (an interpunct written with spaces round it is not one, rules/T3 dividing at it only when flanked, so `威廉 · 莎士比亚` reads the dot as a name word and keeps rendering with its spaces, where `威廉·莎士比亚` keeps the dot unspaced; a spaced nakaguro is a separator, so `ジョン ・ スミス` renders `'ジョン・スミス'`); and one comma the parse reads as one becomes `, ` if ASCII and `، ` if Arabic (U+060C), no space before it and one after, and stands alone if full-width (U+FF0C), which carries its own width in CJK text, a reason that holds for the comma, a mark the rendering keeps, and not for whitespace, which the rendering normalizes, so `John Smith ,Jr.` and `John Smith, Jr.` render `'John Smith, Jr.'` and `John Smith， Jr.` renders `'John Smith，Jr.'`. The Japanese comma 、 (U+3001) is not one the parse reads as a comma, a parser question outside this design. Over that, the spec's decoration between the two fields the words belong to stands in place of the separator (the nickname's quotes, the `) ` after the maiden name, so `Jane Smith née Jones, PhD` keeps `'Jane Smith (Jones) PhD'`; 38 committed entries put a suffix directly after an undelimited maiden name); and any other pair, or any other separator (a delimiter, a marker word the parse consumed, two or more commas, a hyphen, a slash, a zero-width or directional mark), takes the spec's own joiner, as today (the written whitespace is not lost: `ParsedName.original` and the token spans keep it), so `John (Jack) Smith` keeps today's quoted nickname and `str(parse("김민준"))` is `'김민준'` and `str(parse("威廉·莎士比亚"))` keeps the dot. A name with a family comma or written given-first keeps today's word order, and its separators are kept by the same rule (a comma before a suffix alone does not make a family-first name given-first: `Wang Xiaoming, Jr.` under FAMILY_FIRST renders `'Wang Xiaoming, Jr.'`), so the given-first transcription `威廉·莎士比亚` keeps its dot too (20 of the 36 names whose interpunct `str()` drops are read given-first, 4 of them with a comma the parse reads (1 ASCII, 3 full-width)). A family comma is never kept, since the rendering reverses the pair it stands between (`Smith, John` → `'John Smith'`), while a comma directly between the family and its suffix is (`John Smith, Jr.` → `'John Smith, Jr.'`, where `str()` gives `'John Smith Jr.'` today): of the 1,003 committed corpus entries whose suffix starts after the family's last word ends, 214 have nothing between the two but one comma and whitespace (2026-10-28, `c8357e74`, each entry under its declared order; these counts read the ASCII comma, and 7 more write a full-width one there, while 36 of the 728 family-first names above hold a family comma the parse reads, 5 full-width and 31 Arabic (30 of the Arabic in `corpus_cldr.jsonl`), and so keep today's word order under the rule); 456 hold a comma somewhere between, most with a given name or title standing there too. Pairs the rendering separates, as `John` and `Jr.` in `Smith, John, Jr.`, take the spec's own joiner, as today, so `Smith, John, Jr.` keeps rendering `'John Smith Jr.'` while `John Smith, Jr.` becomes `'John Smith, Jr.'`: two writings of one name rendering differently, a consequence of following the text accepted with the choice. +- **How, proposed and not yet decided.** Today `str(name)` is `name.render()`, whose default is one fixed given-first spec, `'{title} {given} "{nickname}" {middle} {family} ({maiden}) {suffix}'`, which is why `김민준` renders `'민준 김'`; a default that reads `order` and the written separators cannot stay one string. So `render(spec: str | None = None)`: `None`, what `str()` passes, is the order-aware default, and a spec a caller writes renders exactly as written, its order and its literal joiners, with no separator rule applied, so `render("{given} {family}")` means in 3.0 what it meant in 2.x. The default's layout for a family-first `order` puts the title first and the suffix last, as the given-first spec does, and the three name roles between them in `order`'s own sequence, each keeping its decoration beside it: `'{title} {family} ({maiden}) {given} "{nickname}" {middle} {suffix}'` for FAMILY_FIRST and `'{title} {family} ({maiden}) {middle} {given} "{nickname}" {suffix}'` for FAMILY_FIRST_GIVEN_LAST, the separator rule then joining adjacent words (`王小明先生`, read family `王小明` and suffix `先生`, renders `'王小明先生'` where today gives `'王小明 先生'`; `Dr. Wang Xiaoming Jr.` under FAMILY_FIRST renders `'Dr. Wang Xiaoming Jr.'` where today gives `'Dr. Xiaoming Wang Jr.'`). CLDR's surnameFirst samples (`tools/differential/cldr_fields.jsonl`, release 48) agree on the core order and not on the rest: `ja` (`佐藤グレース恵子さん`) and `vi` (`ĐBQH Nguyễn Văn Anh`) put the second given name before the first, as FAMILY_FIRST_GIVEN_LAST does, and `ko` (`김민규헨리`) puts it after, as FAMILY_FIRST does; but `pt` puts the title after the family (`da Silva Sr. José Pedro Eduardo Jr., MP`), `zh` puts the generation first (`小彭德威东升议员先生`), and `vi` the credentials first. So the default keeps title and suffix where the given-first default has them, and a locale's own placement is the render patterns' job (below). For 3.0's notes: a caller who copied 2.x's default string and passes it keeps given-first, space-joined output; only a caller who passes nothing, `str()` included, gets the new default. `None` leaves both answers to how a pack's pattern is reached as open as they were: a spec string still means a spec, so `render("sorting")` still returns `'sorting'`, and a pattern is asked for by a new method or argument under (a) or through the parser under (b). - `initials()` gives nothing for a word in a script with no initials (Han, Hangul, kana, `_policy._NO_INITIALS`), where today it gives the first character with a period (`'王.'`). These change no signature but change `str()`'s output for every name read family-first, every name whose words are written with nothing, an interpunct or a nakaguro between two words the rendering keeps side by side (not `김, 민준씨`, whose family comma moves `김` between `민준` and `씨`), and every name written with one comma directly between its family and its suffix, which no 2.x release can warn about; Derek took them as an improvement for 3.0's release notes rather than a break (item 6 of the entry). From 783c235ead3c0a9b03272f4c75d5597e4c08966a Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Wed, 28 Oct 2026 14:17:16 -0700 Subject: [PATCH 26/39] Fix the ninth docs review's findings on the render proposal (#754) The proposed render(spec=None) is named a signature change, not an addition, in both copies and the API table. A family-comma name keeps a given-first order, so it renders by today's spec. The CLDR comparison now says most samples put the title after the family (50 of 70), so the title's place in the family-first default stays open. Co-Authored-By: Claude Opus 5.5 --- docs/design/decisions/vocabulary-model.md | 2 +- docs/design/vocabulary-model-rfc.md | 5 +++-- 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/design/decisions/vocabulary-model.md b/docs/design/decisions/vocabulary-model.md index 3877e6813..fdc7d744b 100644 --- a/docs/design/decisions/vocabulary-model.md +++ b/docs/design/decisions/vocabulary-model.md @@ -250,7 +250,7 @@ Open: #913 (the title words phase 2j left, read as titles though borne as names) 3. **A `patronymic` view, filled by a word or a rule that says patronymic, with `family` and `middle` unchanged.** The default fills it from the vocabulary's `patronymic` particles (`bin`, `binti`, `ibn`, `bint`, `bar`, `bat` and the native-script forms) with the name they introduce (`Mahathir bin Mohamad` → family `bin Mohamad`, as today, and patronymic `bin Mohamad`); RU's and TR_AZ's rules fill it where a caller turns them on (`Ivan Ivanovich Ivanov` under RU → middle `Ivanovich` and patronymic `Ivanovich`); an Icelandic ending rule fills it under an `is` pack only, since the default cannot tell `-dóttir` and `-son` from Swedish surnames (ORCID's Swedish records end their family box in `-son` or `-dóttir` in 2,184 of 20,000). A matronymic, written the same way, fills it too. SUPERSEDES: the RFC's earlier statement of decision 8's value, that `family` can be empty for a name with a patronymic. Measured: filers put an Icelandic patronymic in the family box (ORCID 1,647 of 2,583 Icelandic records, PatentsView 535 of 722, PSC 101 of 125), and CLDR's `is` file's 3 Icelandic samples put it in `surname` or `surname-core` in all 9 of their renderings (its one foreign sample puts a `-dóttir` word in `surname2`), while Wikidata's `patronymic-is` and `patronymic-my` strata record 108 names with a patronymic and no family name, of which the parse puts the patronymic in `family` 107 times. An East Slavic patronymic stands in the given part: CLDR's 2 `ru` samples put it in `given2` in all 6 renderings; PatentsView's Russian given boxes end in one 8,423 times of 20,000, and the parse reads such a word as the middle 10,902 times of 10,962; 90 of the 112 Russian nationals in PSC who filled the middle-name box wrote one there. A `family` that keeps it and a `patronymic` that names it agree with both kinds of source. Weighed against: emptying `family` where the patronymic is recognized, which moves the field the fielded sources fill and changes `family` for every Icelandic and nasab name, a behavior change of a frozen property; and a neutral parent-names name for the view, which buys nothing the view can detect, since a matronymic and a patronymic are written alike. 4. **`generation` and `credentials` are views that partition `suffix` (#326)**, not roles, `generation` the tokens read as a generation or an agnomen, `credentials` the rest, the post-nominals that are neither (`Esq.`, the honorifics after a name) falling in `credentials`, CLDR's catch-all, with `Token.reading.kind` telling them apart for a caller who needs it. Measured: of the 1,054 committed corpus entries with a suffix, 33 hold a generation (or agnomen) and a listed credential together, 77 a generation beside any other suffix word, and 64 only listed post-nominals of other kinds (each suffix word classified by the RFC's Phase 4 recipe, joined tokens merged); of the 615 CLDR renderings with either field, the 75 whose suffix holds exactly CLDR's words split into CLDR's `generation` and `credentials` by the vocabulary's reading kind in all 75. Weighed against: two new `Role` members (item 1), and a third view for the honorifics, left open since CLDR has none. 5. **a `surname2` view, the family's second unit where rules/O6 read the family as two**, marked by a new stable tag where O6 takes its word, empty otherwise; `family` keeps both units as O6 makes it, and a family comma alone marks nothing (`Lloyd Webber, Andrew`). Measured: a two-word family box in ORCID's records from Spain 48.0%, Mexico 64.7%, Colombia 72.8% and Chile 50.5%, against 3.9% from the United States. Left open: a view for the first unit and its name, `surname` standing close to the `surnames` view that exists. Declined: redefining `surnames` as the two units, which would repeat `family`, the family holding both where O6 reads two, and lose what `surnames` does today, joining `middle` and `family` so that a double surname the default splits (`Gabriel García Márquez` → middle `García`) reads whole, the view rules/P2 names for that reading. - 6. **The render defaults follow the order the name was read in, a behavior change of the output rather than a break of the API.** `ParsedName` gains `order` (an addition, `ADDED_IN`, a 2.x pickle rendering as today, left out of comparison, which lets two equal names render differently, a cost accepted 2026-10-28: a 2.x pickle of `김민준` renders `'민준 김'` and a fresh parse `'김민준'`, and the two compare equal); the default `str()` and `render()` spec put the family first where `order` does, and join two words that stand one directly after the other, in the same order, in the original and the rendering with the kind of separator that stood between them there, normalized, decided 2026-10-28 (nothing; one ASCII space for any whitespace, the ideographic space U+3000 included (`山田　太郎` → `'山田 太郎'`, where today gives `'太郎 山田'`); an interpunct or a nakaguro the parse read as a separator alone, a spaced interpunct being a name word, rules/T3; one comma the parse reads as `, ` if ASCII, `، ` if Arabic and alone if full-width, since a full-width comma carries its own width in CJK text, a reason for the comma the rendering keeps and not for the whitespace it normalizes (the Japanese comma U+3001 is no comma to the parse, a parser question outside this design), so `John Smith ,Jr.` renders `'John Smith, Jr.'`; the written whitespace staying in `ParsedName.original` and the token spans), the spec's decoration between the two words' fields standing instead (the nickname's quotes, the `) ` after a maiden name), and any other pair or separator (a delimiter, a consumed marker word, two or more commas, a hyphen, a slash, a zero-width or directional mark) taking the spec's own joiner; `initials()` gives nothing for a word in a script with no initials. A name with a family comma or written given-first keeps today's word order and its separators by the same rule, a suffix comma alone leaving a family-first name family-first (`威廉·莎士比亚` keeps its dot; 20 of the 36 names whose interpunct `str()` drops are read given-first). A family comma is never kept, the rendering reversing its pair (`Smith, John` → `'John Smith'`), and one directly between the family and its suffix is, decided 2026-10-28 (`John Smith, Jr.` → `'John Smith, Jr.'`, today `'John Smith Jr.'`; 214 of the 1,003 committed corpus entries whose suffix follows the family write one there). Accepted with it: `Smith, John, Jr.`, whose suffix the rendering moves away from the given name before it, keeps `'John Smith Jr.'`, so two writings of one name render differently. Measured over the committed corpus's distinct entries, the counts reading the ASCII comma (the RFC gives the full-width and Arabic ones: 36 of the 728 hold a family comma the parse reads and keep today's word order): 728 without a comma read family-first, `str()` renders all 728 differently from their text (`김민준` → `'민준 김'`), 150 of them with the family's last word touching the given name's first, and it drops the U+00B7 interpunct from 36 of the 41 holding one. This carries out decisions/3-0-reevaluations' render-default clean slate. Proposed with it, not yet decided (the RFC's Phase 4 Render defaults): `render(spec: str | None = None)`, `None` the order-aware default `str()` passes and a caller's spec rendering exactly as written, so `render("{given} {family}")` keeps its 2.x meaning and a caller passing 2.x's default string keeps given-first output; and a family-first layout with the title first, the suffix last and the name roles in `order`'s sequence, each with its decoration. No signature changes, but `str()`'s output changes for every name read family-first, every name whose words are written with nothing, an interpunct or a nakaguro between two words the rendering keeps side by side, and every name with one comma directly between its family and its suffix, and no 2.x release can warn about a `str()` result, so it is taken as an improvement for 3.0's release notes, not a break needing a warning. + 6. **The render defaults follow the order the name was read in, a behavior change of the output rather than a break of the API.** `ParsedName` gains `order` (an addition, `ADDED_IN`, a 2.x pickle rendering as today, left out of comparison, which lets two equal names render differently, a cost accepted 2026-10-28: a 2.x pickle of `김민준` renders `'민준 김'` and a fresh parse `'김민준'`, and the two compare equal); the default `str()` and `render()` spec put the family first where `order` does, and join two words that stand one directly after the other, in the same order, in the original and the rendering with the kind of separator that stood between them there, normalized, decided 2026-10-28 (nothing; one ASCII space for any whitespace, the ideographic space U+3000 included (`山田　太郎` → `'山田 太郎'`, where today gives `'太郎 山田'`); an interpunct or a nakaguro the parse read as a separator alone, a spaced interpunct being a name word, rules/T3; one comma the parse reads as `, ` if ASCII, `، ` if Arabic and alone if full-width, since a full-width comma carries its own width in CJK text, a reason for the comma the rendering keeps and not for the whitespace it normalizes (the Japanese comma U+3001 is no comma to the parse, a parser question outside this design), so `John Smith ,Jr.` renders `'John Smith, Jr.'`; the written whitespace staying in `ParsedName.original` and the token spans), the spec's decoration between the two words' fields standing instead (the nickname's quotes, the `) ` after a maiden name), and any other pair or separator (a delimiter, a consumed marker word, two or more commas, a hyphen, a slash, a zero-width or directional mark) taking the spec's own joiner; `initials()` gives nothing for a word in a script with no initials. A name with a family comma or written given-first keeps today's word order and its separators by the same rule, a suffix comma alone leaving a family-first name family-first (`威廉·莎士比亚` keeps its dot; 20 of the 36 names whose interpunct `str()` drops are read given-first). A family comma is never kept, the rendering reversing its pair (`Smith, John` → `'John Smith'`), and one directly between the family and its suffix is, decided 2026-10-28 (`John Smith, Jr.` → `'John Smith, Jr.'`, today `'John Smith Jr.'`; 214 of the 1,003 committed corpus entries whose suffix follows the family write one there). Accepted with it: `Smith, John, Jr.`, whose suffix the rendering moves away from the given name before it, keeps `'John Smith Jr.'`, so two writings of one name render differently. Measured over the committed corpus's distinct entries, the counts reading the ASCII comma (the RFC gives the full-width and Arabic ones: 36 of the 728 hold a family comma the parse reads and keep today's word order): 728 without a comma read family-first, `str()` renders all 728 differently from their text (`김민준` → `'민준 김'`), 150 of them with the family's last word touching the given name's first, and it drops the U+00B7 interpunct from 36 of the 41 holding one. This carries out decisions/3-0-reevaluations' render-default clean slate. Proposed with it, not yet decided (the RFC's Phase 4 Render defaults): `render(spec: str | None = None)`, `None` the order-aware default `str()` passes and a caller's spec rendering exactly as written, so `render("{given} {family}")` keeps its 2.x meaning and a caller passing 2.x's default string keeps given-first output; and a family-first layout with the title first, the suffix last and the name roles in `order`'s sequence, each with its decoration, a family-comma name keeping a given-first `order`. The signature is a change, not an addition as the freeze defines one (no working call changes meaning, `render(None)` raising today, but the default's introspection and a subclass override typed `spec: str` do), and the title's place is open, 50 of CLDR's 70 surnameFirst renderings with a title putting it after the surname. Apart from that proposed signature, no signature changes, but `str()`'s output changes for every name read family-first, every name whose words are written with nothing, an interpunct or a nakaguro between two words the rendering keeps side by side, and every name with one comma directly between its family and its suffix, and no 2.x release can warn about a `str()` result, so it is taken as an improvement for 3.0's release notes, not a break needing a warning. 7. **`middle` keeps its meaning (#704)**, every word of the given part after the given name, the given name one unit unless a rule joins words into it, as rules/P5 already does (`Abdul Aziz Tariq` → given `Abdul Aziz`); a compound given name is a `[rules]` rule a caller or a pack turns on, CLDR's `given2` is `middle`, and `Kaur` and an English name beside a Chinese one stay in `middle`, an accepted cost. Measured: in PSC, whose form has a middle-name box, 2,499 of 1,040,708 forename boxes hold two or more words, and of 6,101 Spanish nationals 25 do while 2,621 fill the middle box; ORCID's and PatentsView's given box holds middles too, so it cannot separate the two. The `Maria do Carmo` shape, a particle inside a given box of three or more words with a family box filled, reads the given box's words into the family in 1,812 of 1,923 ORCID records and 1,165 of 1,219 PatentsView records from Brazil, Portugal, Spain, Mexico, Argentina, Colombia, Chile and Italy, which is that rule's case. Weighed against: #704's option (b), one given unit by default, which changes `middle` for every English name of three words. 8. **A language file's `[rules]` compiles into its shipped `Locale`'s policy patch**, each rule kind a `Policy` field (an addition), so RU and TR_AZ are rebuilt from their files and `Locale` gains no field for rules. With it, **phase 3's open API dependency is answered**: its draft (branch `feat/vocabulary-phase3-design`, unmerged) carries a choice of languages on `Locale.languages`, assuming `Locale` and `Policy` keep their 2.x shape after the facade's removal, and the 2026-10-28 freeze keeps them so. The rest of that draft, strict as an argument of the whole choice included, stays phase 3's to decide; its entry, when it lands, says the dependency is answered. 9. **The work order**, one PR each, measured on its own base, the snapshot recording each new view where it is non-empty: 4a `Token.reading` and `ReadingKind.PATRONYMIC` (neutral; `languages` after 3a); 4b `generation` and `credentials` (neutral); 4c `patronymic` (neutral); 4d `surname2` (neutral); 4e the render defaults and `order`; 4f `[rules]` into the policy patch, with or after 3b; 4g the compound given name, after 4f; 4h the shape inventory per order (#469). Interleaved with phase 3 (its sub-phases the draft's): 4a beside 3a; 4b to 4d before the packs that need them, `cs` (3f) and `pt-BR` (3h) on 4b, `es` and `pt` (3g) on 4d and 4g; an `is` pack on 3b, 4c and 4f. diff --git a/docs/design/vocabulary-model-rfc.md b/docs/design/vocabulary-model-rfc.md index e9c797a44..647338e5b 100644 --- a/docs/design/vocabulary-model-rfc.md +++ b/docs/design/vocabulary-model-rfc.md @@ -251,10 +251,10 @@ ORCID's and PatentsView's given box holds the middle names too, so a two-word gi - `ParsedName` gains `order`, the order the name was read under (its script's or the declared `name_order`, a family comma aside), an addition with `ADDED_IN` metadata whose missing value, in a 2.x pickle, renders as today, and left out of comparison as `Token.reading` is, so a loaded 2.x pickle still equals a fresh parse of its text. The cost, accepted 2026-10-28: two equal names can render differently, where today every rendering is derived from what `==` compares (a 2.x pickle of `김민준` renders `'민준 김'` and a fresh parse `'김민준'`, and a set keeps one of them); the views read from `Token.reading` share it. - `str()` and `render()`'s default spec put the family first where `order` does, and two words that stand one directly after the other, in the same order, in the original and in the rendering keep the kind of separator that stood between them in the original, written in a normalized form (decided 2026-10-28): nothing stays nothing; whitespace of any width or length becomes one ASCII space, the ideographic space U+3000 included (decided 2026-10-28: `山田　太郎` renders `'山田 太郎'`, the same ASCII space `str()` joins with today, where today's order gives `'太郎 山田'`, and a caller wanting native spacing passes a format spec of their own or reads `ParsedName.original`); an interpunct or a nakaguro the parse read as a separator stands alone, with no space around it (an interpunct written with spaces round it is not one, rules/T3 dividing at it only when flanked, so `威廉 · 莎士比亚` reads the dot as a name word and keeps rendering with its spaces, where `威廉·莎士比亚` keeps the dot unspaced; a spaced nakaguro is a separator, so `ジョン ・ スミス` renders `'ジョン・スミス'`); and one comma the parse reads as one becomes `, ` if ASCII and `، ` if Arabic (U+060C), no space before it and one after, and stands alone if full-width (U+FF0C), which carries its own width in CJK text, a reason that holds for the comma, a mark the rendering keeps, and not for whitespace, which the rendering normalizes, so `John Smith ,Jr.` and `John Smith, Jr.` render `'John Smith, Jr.'` and `John Smith， Jr.` renders `'John Smith，Jr.'`. The Japanese comma 、 (U+3001) is not one the parse reads as a comma, a parser question outside this design. Over that, the spec's decoration between the two fields the words belong to stands in place of the separator (the nickname's quotes, the `) ` after the maiden name, so `Jane Smith née Jones, PhD` keeps `'Jane Smith (Jones) PhD'`; 38 committed entries put a suffix directly after an undelimited maiden name); and any other pair, or any other separator (a delimiter, a marker word the parse consumed, two or more commas, a hyphen, a slash, a zero-width or directional mark), takes the spec's own joiner, as today (the written whitespace is not lost: `ParsedName.original` and the token spans keep it), so `John (Jack) Smith` keeps today's quoted nickname and `str(parse("김민준"))` is `'김민준'` and `str(parse("威廉·莎士比亚"))` keeps the dot. A name with a family comma or written given-first keeps today's word order, and its separators are kept by the same rule (a comma before a suffix alone does not make a family-first name given-first: `Wang Xiaoming, Jr.` under FAMILY_FIRST renders `'Wang Xiaoming, Jr.'`), so the given-first transcription `威廉·莎士比亚` keeps its dot too (20 of the 36 names whose interpunct `str()` drops are read given-first, 4 of them with a comma the parse reads (1 ASCII, 3 full-width)). A family comma is never kept, since the rendering reverses the pair it stands between (`Smith, John` → `'John Smith'`), while a comma directly between the family and its suffix is (`John Smith, Jr.` → `'John Smith, Jr.'`, where `str()` gives `'John Smith Jr.'` today): of the 1,003 committed corpus entries whose suffix starts after the family's last word ends, 214 have nothing between the two but one comma and whitespace (2026-10-28, `c8357e74`, each entry under its declared order; these counts read the ASCII comma, and 7 more write a full-width one there, while 36 of the 728 family-first names above hold a family comma the parse reads, 5 full-width and 31 Arabic (30 of the Arabic in `corpus_cldr.jsonl`), and so keep today's word order under the rule); 456 hold a comma somewhere between, most with a given name or title standing there too. Pairs the rendering separates, as `John` and `Jr.` in `Smith, John, Jr.`, take the spec's own joiner, as today, so `Smith, John, Jr.` keeps rendering `'John Smith Jr.'` while `John Smith, Jr.` becomes `'John Smith, Jr.'`: two writings of one name rendering differently, a consequence of following the text accepted with the choice. -- **How, proposed and not yet decided.** Today `str(name)` is `name.render()`, whose default is one fixed given-first spec, `'{title} {given} "{nickname}" {middle} {family} ({maiden}) {suffix}'`, which is why `김민준` renders `'민준 김'`; a default that reads `order` and the written separators cannot stay one string. So `render(spec: str | None = None)`: `None`, what `str()` passes, is the order-aware default, and a spec a caller writes renders exactly as written, its order and its literal joiners, with no separator rule applied, so `render("{given} {family}")` means in 3.0 what it meant in 2.x. The default's layout for a family-first `order` puts the title first and the suffix last, as the given-first spec does, and the three name roles between them in `order`'s own sequence, each keeping its decoration beside it: `'{title} {family} ({maiden}) {given} "{nickname}" {middle} {suffix}'` for FAMILY_FIRST and `'{title} {family} ({maiden}) {middle} {given} "{nickname}" {suffix}'` for FAMILY_FIRST_GIVEN_LAST, the separator rule then joining adjacent words (`王小明先生`, read family `王小明` and suffix `先生`, renders `'王小明先生'` where today gives `'王小明 先生'`; `Dr. Wang Xiaoming Jr.` under FAMILY_FIRST renders `'Dr. Wang Xiaoming Jr.'` where today gives `'Dr. Xiaoming Wang Jr.'`). CLDR's surnameFirst samples (`tools/differential/cldr_fields.jsonl`, release 48) agree on the core order and not on the rest: `ja` (`佐藤グレース恵子さん`) and `vi` (`ĐBQH Nguyễn Văn Anh`) put the second given name before the first, as FAMILY_FIRST_GIVEN_LAST does, and `ko` (`김민규헨리`) puts it after, as FAMILY_FIRST does; but `pt` puts the title after the family (`da Silva Sr. José Pedro Eduardo Jr., MP`), `zh` puts the generation first (`小彭德威东升议员先生`), and `vi` the credentials first. So the default keeps title and suffix where the given-first default has them, and a locale's own placement is the render patterns' job (below). For 3.0's notes: a caller who copied 2.x's default string and passes it keeps given-first, space-joined output; only a caller who passes nothing, `str()` included, gets the new default. `None` leaves both answers to how a pack's pattern is reached as open as they were: a spec string still means a spec, so `render("sorting")` still returns `'sorting'`, and a pattern is asked for by a new method or argument under (a) or through the parser under (b). +- **How, proposed and not yet decided.** Today `str(name)` is `name.render()`, whose default is one fixed given-first spec, `'{title} {given} "{nickname}" {middle} {family} ({maiden}) {suffix}'`, which is why `김민준` renders `'민준 김'`; a default that reads `order` and the written separators cannot stay one string. So `render(spec: str | None = None)`: `None`, what `str()` passes, is the order-aware default, and a spec a caller writes renders exactly as written, its order and its literal joiners, with no separator rule applied, so `render("{given} {family}")` means in 3.0 what it meant in 2.x. That is a signature change, not an addition as the freeze defines one (decisions/3-0-reevaluations, THE CORE API), and needs its own decision: no call that works today changes meaning, since `render(None)` raises `TypeError` today, but introspecting the default gives `None` rather than a spec, and a subclass overriding `render(self, spec: str = ...)` becomes an incompatible override to a type checker. The default's layout is picked by `order`, and a name with a family comma has a given-first `order` ("a family comma aside", above), so a given-first name and a family-comma name render by today's spec with the separator rule (`Smith, John` under FAMILY_FIRST still renders `'John Smith'`). For a family-first `order` the layout puts the title first and the suffix last, as the given-first spec does, and the three name roles between them in `order`'s own sequence, each keeping its decoration beside it: `'{title} {family} ({maiden}) {given} "{nickname}" {middle} {suffix}'` for FAMILY_FIRST and `'{title} {family} ({maiden}) {middle} {given} "{nickname}" {suffix}'` for FAMILY_FIRST_GIVEN_LAST, the separator rule then joining adjacent words (`王小明先生`, read family `王小明` and suffix `先生`, renders `'王小明先生'` where today gives `'王小明 先生'`; `Dr. Wang Xiaoming Jr.` under FAMILY_FIRST renders `'Dr. Wang Xiaoming Jr.'` where today gives `'Dr. Xiaoming Wang Jr.'`). CLDR's surnameFirst samples (`tools/differential/cldr_fields.jsonl`, release 48) agree on the core order and not on the rest: `ja` (`佐藤グレース恵子さん`) and `vi` (`ĐBQH Nguyễn Văn Anh`) put the second given name before the first, as FAMILY_FIRST_GIVEN_LAST does, and `ko` (`김민규헨리`) puts it after, as FAMILY_FIRST does; but most put the title after the family, not first: `pt` after it (`da Silva Sr. José Pedro Eduardo Jr., MP`), `ja` and `zh` last, and over every file's nativeFull surnameFirst rendering holding a title and a surname, 50 of 70 put the title after the surname (2026-10-28, positions of the field values in the text); `zh` also puts the generation first (`小彭德威东升议员先生`), and `vi` the credentials first. So keeping the title first, where the given-first default and the written name have it, goes against most of CLDR, and whether the family-first default should move it after the family is part of what is to be decided; a locale's own placement is the render patterns' job (below). The suffix last agrees with `ja` and `zh`, whose honorifics the parse reads as suffixes. For 3.0's notes: a caller who copied 2.x's default string and passes it keeps given-first, space-joined output; only a caller who passes nothing, `str()` included, gets the new default. `None` leaves both answers to how a pack's pattern is reached as open as they were: a spec string still means a spec, so `render("sorting")` still returns `'sorting'`, and a pattern is asked for by a new method or argument under (a) or through the parser under (b). - `initials()` gives nothing for a word in a script with no initials (Han, Hangul, kana, `_policy._NO_INITIALS`), where today it gives the first character with a period (`'王.'`). -These change no signature but change `str()`'s output for every name read family-first, every name whose words are written with nothing, an interpunct or a nakaguro between two words the rendering keeps side by side (not `김, 민준씨`, whose family comma moves `김` between `민준` and `씨`), and every name written with one comma directly between its family and its suffix, which no 2.x release can warn about; Derek took them as an improvement for 3.0's release notes rather than a break (item 6 of the entry). +Apart from that proposed signature, these change no signature but change `str()`'s output for every name read family-first, every name whose words are written with nothing, an interpunct or a nakaguro between two words the rendering keeps side by side (not `김, 민준씨`, whose family comma moves `김` between `민준` and `씨`), and every name written with one comma directly between its family and its suffix, which no 2.x release can warn about; Derek took them as an improvement for 3.0's release notes rather than a break (item 6 of the entry). **What changes in the API**, by the 3.0 freeze: @@ -267,6 +267,7 @@ These change no signature but change `str()`'s output for every name read family | new `Policy` fields for rules, and `Locale.languages` (phase 3) | additions | | `Role`, `as_dict()`, `comparison_key()`, the seven role properties | unchanged | | the default `str()`, `render()` spec and `initials()` output | behavior change, accepted for 3.0 | +| `render(spec: str \| None = None)`, proposed | a signature change, not an addition; undecided | | a `Role` member for any of these views | a break, declined | **Phase 3's open API dependency answered**: phase 3's draft (branch `feat/vocabulary-phase3-design`, not yet merged) carries a choice of languages on `Locale.languages` and notes it assumes `Locale` and `Policy` keep their 2.x shape once the facade is gone; the 2026-10-28 freeze answers that, both keeping it with additions. The rest of that item, strict as an argument of the whole choice included, is phase 3's to decide, and its entry is to say the dependency is answered when it lands. With the `[rules]` proposal above, a shipped language pack is a `Locale` whose `languages` choose its vocabulary and whose `policy` carries its rules, so `Locale` needs no third field. From 2eaeb57a89191638a95adc8d8a12ea6cad8ca641 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Wed, 28 Oct 2026 17:12:14 -0700 Subject: [PATCH 27/39] Phase 3 proposal: strict's precondition gets a sub-phase, 3i (#753) Nothing in 3a-3h said whose names a name reading's are, which strict waits on. Record it as open beside the strict paragraph and in the decisions entry's item 4: an attribute naming each name reading's bearers' languages, filled word by word, since a box count says where a word was typed, not whose name it is (do's ORCID family box is led by Brazil's particle, none of its 7,227 boxes do alone). Proposed as 3i, after 3a, neutral; nothing decided. Co-Authored-By: Claude Opus 5.5 --- docs/design/decisions/vocabulary-model.md | 4 ++-- docs/design/vocabulary-model-rfc.md | 5 ++++- 2 files changed, 6 insertions(+), 3 deletions(-) diff --git a/docs/design/decisions/vocabulary-model.md b/docs/design/decisions/vocabulary-model.md index 377ede506..0fe7235b2 100644 --- a/docs/design/decisions/vocabulary-model.md +++ b/docs/design/decisions/vocabulary-model.md @@ -241,12 +241,12 @@ Open: #913 (the title words phase 2j left, read as titles though borne as names) 1. **Proposed: a pack is a choice of languages carried by `Locale`.** `Locale` gains `languages`, a set of BCP 47 tags; `parser_for` unions the sets of the packs it is given, so the choice is order-independent while policy patches still fold in order; strict is an argument of the whole choice (`parser_for(locales.NE, locales.HI, strict=True)`), since a strict pack beside a preferring one has no meaning, reshaping decision 6's "strict variant". The shipped language packs are `Locale` constants with an empty lexicon fragment (`locales.NE`, `locales.PT_BR`). Open: this assumes `Locale` and `Policy` keep their 2.x shape once the facade is gone, which 3.0 has not decided. Weighed against: `Parser(languages=…)` beside `Locale`, two ways to choose one thing, and a `Policy` field, `Policy` being behavior and vocabulary being `Lexicon`'s. 2. **Proposed: one mechanism.** The chosen languages select which readings the compiler compiles, the default being that compiler with every language and no choice. Turning on, prefer and strict are derived from the file each reading sits in, and nothing is written per pack beyond its language's readings: a chosen language's `default = false` readings join; prefer drops other languages' vocabulary readings of a word on which a chosen language holds one, never `mul.toml`'s (`Haddad, Abd`, suffix `Abd` today, reads given `Abd` with the en credential gone); strict drops other languages' name readings and the markers they derive. A clash a property records with a SHAPE reading is settled for the chosen language's own reading, decision 20 applied: `may_be_initial` does not hold on it, so a `pt` choice reads `ANA SILVA E SOUZA` family `SILVA E SOUZA` where the default reads middle `SILVA E` with a report. This answers decisions/3-0-reevaluations' 2026-09-13 bullet that a pack cannot remove `e`, and its 2026-08-17 bullet's "overrides rather than additions": a pack overrides by selection, never by a removal list. That removes base entries all the same, so mechanisms.md#LOCALE-PACKS-PURE-DATA's add-only contract is amended with it (item 12). The shape-clash rule follows from decision 7, which decision 20 permits; it reaches every reading `may_be_initial` stands on (pt and it `e`, ca and pl `i`, the Italian one unjudged), and it forecloses an `nl` pack marking `y` (the 2026-08-18 bullet there), which waits with `fru` below. New beside decision 6: `mul.toml`'s exemption from prefer. Left open rather than built: a property a language's own reading should take only under its pack, Swedish `fru` and Icelandic `frú` as after-given salutations (rules/H8), refused in the default for the Cameroonian surname, the two readings in the inventory needing it. 3. **Proposed: `default = false` is allowed on `given` and `family` readings, and turned on they derive markers like any name reading, never settling (decision 15 unchanged).** Measured on nepal-candidates' 126,981 Devanagari and romanized pairs, comparing every word's role across the two spellings: 123,440 read alike in the default; `given` readings on `श्री`, `guru` and `गुरु`, which make each a title only inside a title phrase (rules/H9), give 123,636 (198 fixed, 2 broken, both pairs whose romanization opens with a listed title); a `family` reading on `dc` as well, making the credential ambiguous, gives 123,639, the count still leaving `Ram Bahadur DC` a suffix; dropping `dc`'s credential, which settling by a name reading would do, gives 123,658, 19 more. Settling is not proposed: the 19 pairs are the price of decision 15, whose `do` case it protects. `bk`, `kc`, `gc` and `bc` read as the family in every shape tried (spoken order, a family comma, `FAMILY_FIRST`, dotted and Title case), so a reading would move nothing and none is proposed. Recompute: the readings moved into or out of `Lexicon` fields by hand, the pairs compared by `tuple(t.role for t in parser.parse(s).tokens)`. - 4. **Proposed: strict waits, out of the first packs, until name readings say whose names they are.** A name reading sits in the file whose source counted it: 113 of the 211 name readings are in en.toml, `do`'s Vietnamese Đỗ readings among them, vi.toml holding no `do`, so strict built on the files drops them under a `vi` choice, and simulated, `Minh Do` reads suffix `Do` and `Do Van Minh` family `Do Van Minh`, decision 15's harm. A census count belongs to no one language either. The candidates needing strict wait with it, `nl`'s prepositions among them, which need more than strict: turned on as possibly-given particles, `in` and `to` read `Park In Hwan` family `In Hwan` and `Nguyen To Nga` family `To Nga` (the marker acts at the head only), and check 11 accepts `may_be_given = false` in place of the Korean and Vietnamese readings. Recompute: count `given` and `family` readings per file with tools/vocab/build.py's `load`; simulate by removing `do` from `suffix_acronyms_ambiguous` and `particles_ambiguous`. + 4. **Proposed: strict waits, out of the first packs, until name readings say whose names they are.** A name reading sits in the file whose source counted it: 113 of the 211 name readings are in en.toml, `do`'s Vietnamese Đỗ readings among them, vi.toml holding no `do`, so strict built on the files drops them under a `vi` choice, and simulated, `Minh Do` reads suffix `Do` and `Do Van Minh` family `Do Van Minh`, decision 15's harm. A census count belongs to no one language either. The candidates needing strict wait with it, `nl`'s prepositions among them, which need more than strict: turned on as possibly-given particles, `in` and `to` read `Park In Hwan` family `In Hwan` and `Nguyen To Nga` family `To Nga` (the marker acts at the head only), and check 11 accepts `may_be_given = false` in place of the Korean and Vietnamese readings. Recompute: count `given` and `family` readings per file with tools/vocab/build.py's `load`; simulate by removing `do` from `suffix_acronyms_ambiguous` and `particles_ambiguous`. Open: none of 3a–3h says whose names they are. Moving readings between files would not, a word being borne in several languages (`do` Vietnamese and Korean, `kim` Korean and Scandinavian), so the RFC points at an attribute on each name reading naming its bearers' languages, compiling to nothing until strict reads it, the first provenance anything would compute a reading from. It cannot be filled by script: 172 of the 211 count a US source and 21 none, per-country evidence is note prose in 40, and a box count says where a word was typed, not whose name it is (`do`'s ORCID family box is led by Brazil, 7,227 against Vietnam's 876, none of Brazil's `do` alone against Vietnam's 618; by share Vietnam leads, 1.90% to 0.81%; 2026-10-28, master `c8357e74`). The attribute names languages, so no country-to-language table is read at run time (item 6). Proposed: a sub-phase of its own, 3i, after 3a, independent of the output model, moving no default parse; strict after it. 5. **Proposed: the languages reach run time as a reading index.** A second generated module, one row per reading (text, tag, kind, behavioral properties), loaded only when a language is chosen; the compiler moves from tools/vocab/build.py into the package so build and pack share one implementation, and a test holds that compiling the index with no choice reproduces `_compiled.py`. It is the per-word index item 13 of the 2026-10-23 #752 entry says `Token.reading` needs. A choice is applied as a delta against the default's compile, so it composes with a caller's `base=` lexicon and a `Locale`'s own fragment. Measured: 2,140 readings over 1,797 words, 106 in more than one file; the index about 93 KB of source and 11 ms to execute without a bytecode cache, the compile 9 ms, both once per choice and only for a caller who makes one; loading the TOML 39 ms. Weighed against: per-language deltas compiled at build time, which do not compose (under prefer `{pt, en}` `do` keeps its particle and its credential, while Portuguese's delta drops the credential and English's the particle); every combination precompiled, combinatorial; the TOML read at run time, slower and either re-running the build's checks or skipping them. 6. **Proposed: a chosen tag selects its file and its ancestors', never its descendants, and no macrolanguage table is read.** `pt-BR` selects `pt-BR.toml` and `pt.toml`; `pt` only `pt.toml`, so a caller in Portugal does not turn on the Brazilian agnomens (#843). The two files share no word today. Malay and Indonesian, which share `puan`, `putra` and `putri`, are chosen together by a caller who has both. The TOML names `prefer` and `strict` stay refused, the message saying a strength is the caller's choice and never data. 7. **Proposed: a pack is measured three ways.** The default does not move (the committed snapshot, and the index invariant of item 5); a vocabulary pack's deviation surface, the words whose compiled readings its choice changes, is derived from the index, and tests/v2/test_locales.py's non-interference gate requires the default's reading for every corpus name without such a word, the policy and segmentation packs (JA, RU, TR_AZ, ZH) keeping their written `DEVIATES`; and `snapshot.py`, `--local` with `--base` included, takes a choice and prints what it moves by field, beside each pack's own ground truth (nepal-candidates' script pairs for `ne`, the CLDR agreement per pack, PSC's nationality slice). This answers decisions/3-0-reevaluations' 2026-09-13 bullet that the harness cannot see a pack. 8. **Proposed: third-party language files are deferred past phase 3.** A file joining the pool at run time would skip the checks that run across files (check 6's exclusions, check 11's particle statement), the ones that refuse a third-party `nl` `in` written with neither a `given` reading nor `may_be_given = false`. `Locale(lexicon=…)`, or its successor, stays the way to add vocabulary. 9. **Proposed: every exclusion a pack turns on becomes a `default = false` reading,** the shape cs `st` already has, so that once the packs land an `excluded` key means wrong everywhere: the eight whose reason opens "pack-shaped" in 3c (cs `ml`, hr, sk and sl `st` and `ml`, fr `père`), and the rest in the PR of the pack that turns each on (`pan`, `roz`, `nic`, `shri`, `sig`, `jaun`, `bt`, `bay`, `sb`). Recompute: `grep -h 'pack-shaped' nameparser/vocab/*.toml | wc -l`. 10. **Proposed: what stays out of phase 3.** ZH's surname list and the RU and TR_AZ endings are rule data for phase 4's `[rules]`, and a Python pack gains `languages` only where a measurement shows what it moves; the candidates that wait on a position rule (Turkish `Bey` and `Hanım`, Hungarian `úr`, Thai `นาย`, French `M.`) are phase 4's; same-language clashes (`kumari`, `sayyid`) take phase 2j's route in the default, since no pack settles one. - 11. **Proposed: the work order,** shape first and one PR each, measured on its own base: 3a the index and shared compiler with name readings allowed `default = false` (neutral); 3b the choice, tag selection, turning on, prefer, the shape clash, the derived deviation surface, `snapshot.py`'s choice and the amended contract (neutral); 3c the pack-shaped exclusions converted (neutral); then the packs, `ne` (3d), `hi` and `bn`'s Latin forms (3e), `cs`, `sk`, `sl` and `hr` (3f), `es` and `pt` with a connective that counts only inside a title run (3g), and `ga`/`gd` and `pt-BR` (3h). 3g's connective is a rule with its own design: with `profesora` added and no rule, `Profesora de Historia Ana López` reads family `de Historia Ana López` (the rules/P1 fold) and `Ministro da Fazenda Carlos Souza` given `da`, while `Doctora de la Vega` must keep family `de la Vega`, the two separated by whether a name stands behind the phrase; it must reach only readings a pack turns on. + 11. **Proposed: the work order,** shape first and one PR each, measured on its own base: 3a the index and shared compiler with name readings allowed `default = false` (neutral); 3b the choice, tag selection, turning on, prefer, the shape clash, the derived deviation surface, `snapshot.py`'s choice and the amended contract (neutral); 3c the pack-shaped exclusions converted (neutral); then the packs, `ne` (3d), `hi` and `bn`'s Latin forms (3e), `cs`, `sk`, `sl` and `hr` (3f), `es` and `pt` with a connective that counts only inside a title run (3g), and `ga`/`gd` and `pt-BR` (3h); 3i, the bearers' languages of each name reading (item 4), after 3a, ahead of strict. 3g's connective is a rule with its own design: with `profesora` added and no rule, `Profesora de Historia Ana López` reads family `de Historia Ana López` (the rules/P1 fold) and `Ministro da Fazenda Carlos Souza` given `da`, while `Doctora de la Vega` must keep family `de la Vega`, the two separated by whether a name stands behind the phrase; it must reach only readings a pack turns on. 12. **Proposed: phase 3 is 3.0 work, opt-in, the default's parses unmoved, with phase 4's shape questions settled before or alongside its mechanism** (the 2026-10-28 re-plan: 2.8 is the last 2.x feature release, and 3.0 ships phases 3 and 4 with the facade's removal). The output model decides what a pack can express (a patronymic field, a second surname, #326's credentials apart from generations, #704's `middle`). 3a and 3c are independent of it, and so are 3b's selection semantics; 3b's API waits on item 1's open dependency; `cs` and `pt-BR` wait on #326 and `es`/`pt` on a second surname, while `ne` does not. One standing text is amended with 3b, mechanisms.md#LOCALE-PACKS-PURE-DATA's contract that a pack is add-only and declares its deviations in its docstring (a pack adds, and selects among the default's readings by language, a vocabulary pack's deviations derived); decisions/3-0-reevaluations' Standing line that removing base vocabulary through a pack is part of 3.0's pack design is carried out. diff --git a/docs/design/vocabulary-model-rfc.md b/docs/design/vocabulary-model-rfc.md index f7179a75d..841e13074 100644 --- a/docs/design/vocabulary-model-rfc.md +++ b/docs/design/vocabulary-model-rfc.md @@ -215,6 +215,8 @@ The 2 broken are pairs whose romanization opens with a listed title, which reads **Strict cannot be built on today's files.** A name reading sits in the file whose source counted it, not the file of the language whose names carry it: 113 of the 211 name readings are in en.toml (2026-10-27), `do`'s Vietnamese Đỗ readings among them, and vi.toml holds no `do`. A strict `vi` choice built on the files would drop exactly those readings, and with them the gate: simulated, `Minh Do` reads suffix `Do` and `Do Van Minh` family `Do Van Minh`, the harm decision 15 is there to prevent. Nor does a census count belong to one language. So strict is proposed to wait, out of the first packs, until name readings say whose names they are, and the candidates that need it wait with it. The `nl` prepositions show they would need more than strict: turned on as possibly-given particles, `in` and `to` already read `Park In Hwan` family `In Hwan` and `Nguyen To Nga` family `To Nga`, the marker acting at the head only, and check 11 lets their author write `may_be_given = false` instead of declaring the Korean and Vietnamese readings. +**Open: nothing in the work order says whose names a name reading's are.** Strict waits on it, and none of 3a–3h does it. Moving readings between files would not: a census count belongs to no one language, and a word is borne in several, `do` as a Vietnamese and a Korean name, `kim` as a Korean surname and a Scandinavian given name. What it points at is an attribute on each name reading naming the languages of its bearers (`do`'s family reading `vi` and `ko`; the key's name still to choose), compiling to nothing until strict reads it. The counts to fill it exist, but none fills it mechanically. Of the 211 name readings, 172 count a US source and 21 none, and what per-country evidence they carry is prose in a `note`, as `do`'s nam_dict bands are (Korea 8, Vietnam 6, the Netherlands 1), in 40 of them by a text search. `name_usage.py --boxes` gives `orcid-researchers`' and `patentsview-inventors`' counts by box and country, and a box count says where a word was typed, not whose name it is: `do`'s family box in ORCID is led by Brazil, 7,227 holders against Vietnam's 876, and none of Brazil's is `do` alone (2,629 open with it, 4,598 hold it inside), the particle of a Brazilian surname, where Vietnam has 618 boxes of `do` alone and Korea 151. A share of each country's holders ranks them the other way (Vietnam 1.90%, Brazil 0.81%; PatentsView 1.78% and 0.64%), and `kim` reads cleanly either way, its family box 17.67% of ORCID's Korean holders and 13.30% of PatentsView's, its given box 0.93% of PatentsView's Danish inventors (2026-10-28, master `c8357e74`). So the fill is a person reading each of the 157 words with name readings, by share and by whole box, not a script. Two further consequences: the attribute names languages, filled from countries by that reader, since a country-to-language table read at run time is the macrolanguage table the tag matching below refuses; and it is the first provenance anything computes a reading from, where counts today are evidence nothing reads. Proposed: a sub-phase of its own, 3i, after 3a, independent of the output model and moving no default parse, with strict and the candidates waiting on it after it. Recompute: count the name readings per file and per count key with `tomllib` over `nameparser/vocab/*.toml`; `name_usage.py do kim --no-wikidata --boxes`; the whole-box split by reading the cached ORCID extract's `family` and `country` keys. + ### Where the provenance lives at runtime The compiled module holds sets, and a set has forgotten which file each word came from. The proposal compiles a second module beside it: a **reading index**, one row per reading (its text, its file's tag, its kind and its behavioral properties), loaded only when a caller chooses a language. The compiler moves from tools/vocab/build.py into the package so that the build and a pack share one implementation, and a test holds the invariant that compiling the index with no choice reproduces `nameparser/config/_compiled.py` exactly. The same index is what item 13 of decisions/vocabulary-model's 2026-10-23 #752 entry says `Token.reading` needs in 3.0. @@ -250,7 +252,7 @@ The snapshot and `compare.py` parse under the default, so neither can see a pack ### The order of work -Shape first, then the packs, each one PR measured on its own base, as phase 2's were. The first three move no default parse, and no pack exists until 3d. Strict is not in this order (above). +Shape first, then the packs, each one PR measured on its own base, as phase 2's were. The first three move no default parse, and no pack exists until 3d. Strict is not in this order (above); 3i, the bearers' languages it would read, is. | Sub-phase | What | Needs | |---|---|---| @@ -262,6 +264,7 @@ Shape first, then the packs, each one PR measured on its own base, as phase 2's | 3f | `cs`, `sk`, `sl`, `hr`: the generations of 3c, `pan`, `roz` and the academic abbreviations #848 left | 3b, 3c | | 3g | `es` and `pt`: the occupational titles and `Don`/`Doña`/`Dona`, with a connective that counts only inside a title run, its own rule; `pt` also settles the one-case `e` | 3b | | 3h | `ga` and `gd` (`nic`, `mhic`, `mic`, `uí`), `pt-BR` (`F`, `N`, `sb`); `nl`'s prepositions wait with strict | 3b | +| 3i | the languages of each name reading's bearers, an attribute compiling to nothing, filled word by word from per-country shares and whole-box counts (above); strict and its candidates after it | 3a (neutral) | 3g's connective needs a design of its own, because the pair that separates it is a count: `Profesora de Historia Ana López` should read title `Profesora de Historia`, while `Doctora de la Vega`, the same title, preposition and following word, has no name behind it and must keep family `de la Vega`. With `profesora` added and no rule, the first reads family `de Historia Ana López` (the rules/P1 fold), and `Ministro da Fazenda Carlos Souza` reads given `da`. The rule must reach only readings a pack turns on, so the default's parses stay where they are. From a4c1d9129b240d1a0ff247f33a00ab97856166dc Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Wed, 28 Oct 2026 17:15:29 -0700 Subject: [PATCH 28/39] Phase 3 proposal brought in line with phase 4's agreed output model (#753) - The reading index loads for every caller, since Reading.languages (4a) is filled from it on every parse; lazy loading would need a Reading to carry its word as a hidden key. Cached import measured at about 0.45 ms against about 20 ms for import nameparser; the earlier 11 ms was taken under PYTHONDONTWRITEBYTECODE. - The Locale/Policy dependency is answered by the 2026-10-28 freeze. - Work order: 3f and 3h need 4b, 3g needs 4d and 4g, 4f lands with or after 3b; RU and TR_AZ wait for 4f. - The shape-questions-first paragraph is history: the output model was agreed 2026-10-28 (#754). Co-Authored-By: Claude Opus 5.5 --- docs/design/decisions/vocabulary-model.md | 8 ++++---- docs/design/vocabulary-model-rfc.md | 18 +++++++++--------- 2 files changed, 13 insertions(+), 13 deletions(-) diff --git a/docs/design/decisions/vocabulary-model.md b/docs/design/decisions/vocabulary-model.md index 0fe7235b2..3033ae928 100644 --- a/docs/design/decisions/vocabulary-model.md +++ b/docs/design/decisions/vocabulary-model.md @@ -238,15 +238,15 @@ Open: #913 (the title words phase 2j left, read as titles though borne as names) - **2026-10-26 #752, phase 2m — THE READING NOT TAKEN SURFACES AS `Ambiguity.alternative`, WITHOUT ITS LANGUAGES**, carrying out item 13 of the 2026-10-23 #752 entry above and item 7's source with it: a word the credential's run took by position alone reports its name reading's source as `position`, a fourth source beside the three the item named, and a word with a declared name reading reports `declared`. The pickle guard was widened as item 13 asked, every skew the narrow guard refused still refused. The reading's languages were left out: the per-word index costs about 0.4 ms of a 19.5 ms import and describes the default vocabulary, not a pack's or a caller's lexicon, so they wait for 3.0 with `Token.reading`. The record, the measurements and what was declined are decisions/A1's 2026-10-26 #752 entry. - **2026-10-27 #753 — PHASE 3'S DESIGN IS PROPOSED, NOT AGREED: A CHOICE OF LANGUAGES SELECTS THE POOL, ONE MECHANISM FOR TURNING READINGS ON AND PREFER; NAME READINGS IN A PACK KEEP THE GATE ON; A READING INDEX CARRIES THE LANGUAGES TO RUN TIME; STRICT WAITS.** Every item below is a proposal awaiting Derek's decision, and this head is rewritten when he makes it. The design overview is docs/design/vocabulary-model-rfc.md's Phase 3 section. Decisions 15 and 20 are kept as written; decision 6's strict variant is reshaped to a property of the whole choice and deferred (items 1 and 4). Phase 3 is 3.0 work since the 2026-10-28 re-plan (item 12). The inputs are #753's comments of 2026-10-27: the inventory of 411 kept-out candidate readings with their verdicts, and the probe of foreign titles and function-word collisions. Measurements were taken 2026-10-27 on master `cfe8bf92` (Python 3.11.17) with throwaway Lexicon edits standing in for a pack, so each is evidence for the proposal and each sub-phase re-measures on its own base. Local-corpus figures are counts. - 1. **Proposed: a pack is a choice of languages carried by `Locale`.** `Locale` gains `languages`, a set of BCP 47 tags; `parser_for` unions the sets of the packs it is given, so the choice is order-independent while policy patches still fold in order; strict is an argument of the whole choice (`parser_for(locales.NE, locales.HI, strict=True)`), since a strict pack beside a preferring one has no meaning, reshaping decision 6's "strict variant". The shipped language packs are `Locale` constants with an empty lexicon fragment (`locales.NE`, `locales.PT_BR`). Open: this assumes `Locale` and `Policy` keep their 2.x shape once the facade is gone, which 3.0 has not decided. Weighed against: `Parser(languages=…)` beside `Locale`, two ways to choose one thing, and a `Policy` field, `Policy` being behavior and vocabulary being `Lexicon`'s. + 1. **Proposed: a pack is a choice of languages carried by `Locale`.** `Locale` gains `languages`, a set of BCP 47 tags; `parser_for` unions the sets of the packs it is given, so the choice is order-independent while policy patches still fold in order; strict is an argument of the whole choice (`parser_for(locales.NE, locales.HI, strict=True)`), since a strict pack beside a preferring one has no meaning, reshaping decision 6's "strict variant". The shipped language packs are `Locale` constants with an empty lexicon fragment (`locales.NE`, `locales.PT_BR`). The dependency this rested on, `Locale` and `Policy` keeping their 2.x shape once the facade is gone, is answered by the 2026-10-28 3.0 freeze: both keep it, with additions. Weighed against: `Parser(languages=…)` beside `Locale`, two ways to choose one thing, and a `Policy` field, `Policy` being behavior and vocabulary being `Lexicon`'s. 2. **Proposed: one mechanism.** The chosen languages select which readings the compiler compiles, the default being that compiler with every language and no choice. Turning on, prefer and strict are derived from the file each reading sits in, and nothing is written per pack beyond its language's readings: a chosen language's `default = false` readings join; prefer drops other languages' vocabulary readings of a word on which a chosen language holds one, never `mul.toml`'s (`Haddad, Abd`, suffix `Abd` today, reads given `Abd` with the en credential gone); strict drops other languages' name readings and the markers they derive. A clash a property records with a SHAPE reading is settled for the chosen language's own reading, decision 20 applied: `may_be_initial` does not hold on it, so a `pt` choice reads `ANA SILVA E SOUZA` family `SILVA E SOUZA` where the default reads middle `SILVA E` with a report. This answers decisions/3-0-reevaluations' 2026-09-13 bullet that a pack cannot remove `e`, and its 2026-08-17 bullet's "overrides rather than additions": a pack overrides by selection, never by a removal list. That removes base entries all the same, so mechanisms.md#LOCALE-PACKS-PURE-DATA's add-only contract is amended with it (item 12). The shape-clash rule follows from decision 7, which decision 20 permits; it reaches every reading `may_be_initial` stands on (pt and it `e`, ca and pl `i`, the Italian one unjudged), and it forecloses an `nl` pack marking `y` (the 2026-08-18 bullet there), which waits with `fru` below. New beside decision 6: `mul.toml`'s exemption from prefer. Left open rather than built: a property a language's own reading should take only under its pack, Swedish `fru` and Icelandic `frú` as after-given salutations (rules/H8), refused in the default for the Cameroonian surname, the two readings in the inventory needing it. 3. **Proposed: `default = false` is allowed on `given` and `family` readings, and turned on they derive markers like any name reading, never settling (decision 15 unchanged).** Measured on nepal-candidates' 126,981 Devanagari and romanized pairs, comparing every word's role across the two spellings: 123,440 read alike in the default; `given` readings on `श्री`, `guru` and `गुरु`, which make each a title only inside a title phrase (rules/H9), give 123,636 (198 fixed, 2 broken, both pairs whose romanization opens with a listed title); a `family` reading on `dc` as well, making the credential ambiguous, gives 123,639, the count still leaving `Ram Bahadur DC` a suffix; dropping `dc`'s credential, which settling by a name reading would do, gives 123,658, 19 more. Settling is not proposed: the 19 pairs are the price of decision 15, whose `do` case it protects. `bk`, `kc`, `gc` and `bc` read as the family in every shape tried (spoken order, a family comma, `FAMILY_FIRST`, dotted and Title case), so a reading would move nothing and none is proposed. Recompute: the readings moved into or out of `Lexicon` fields by hand, the pairs compared by `tuple(t.role for t in parser.parse(s).tokens)`. 4. **Proposed: strict waits, out of the first packs, until name readings say whose names they are.** A name reading sits in the file whose source counted it: 113 of the 211 name readings are in en.toml, `do`'s Vietnamese Đỗ readings among them, vi.toml holding no `do`, so strict built on the files drops them under a `vi` choice, and simulated, `Minh Do` reads suffix `Do` and `Do Van Minh` family `Do Van Minh`, decision 15's harm. A census count belongs to no one language either. The candidates needing strict wait with it, `nl`'s prepositions among them, which need more than strict: turned on as possibly-given particles, `in` and `to` read `Park In Hwan` family `In Hwan` and `Nguyen To Nga` family `To Nga` (the marker acts at the head only), and check 11 accepts `may_be_given = false` in place of the Korean and Vietnamese readings. Recompute: count `given` and `family` readings per file with tools/vocab/build.py's `load`; simulate by removing `do` from `suffix_acronyms_ambiguous` and `particles_ambiguous`. Open: none of 3a–3h says whose names they are. Moving readings between files would not, a word being borne in several languages (`do` Vietnamese and Korean, `kim` Korean and Scandinavian), so the RFC points at an attribute on each name reading naming its bearers' languages, compiling to nothing until strict reads it, the first provenance anything would compute a reading from. It cannot be filled by script: 172 of the 211 count a US source and 21 none, per-country evidence is note prose in 40, and a box count says where a word was typed, not whose name it is (`do`'s ORCID family box is led by Brazil, 7,227 against Vietnam's 876, none of Brazil's `do` alone against Vietnam's 618; by share Vietnam leads, 1.90% to 0.81%; 2026-10-28, master `c8357e74`). The attribute names languages, so no country-to-language table is read at run time (item 6). Proposed: a sub-phase of its own, 3i, after 3a, independent of the output model, moving no default parse; strict after it. - 5. **Proposed: the languages reach run time as a reading index.** A second generated module, one row per reading (text, tag, kind, behavioral properties), loaded only when a language is chosen; the compiler moves from tools/vocab/build.py into the package so build and pack share one implementation, and a test holds that compiling the index with no choice reproduces `_compiled.py`. It is the per-word index item 13 of the 2026-10-23 #752 entry says `Token.reading` needs. A choice is applied as a delta against the default's compile, so it composes with a caller's `base=` lexicon and a `Locale`'s own fragment. Measured: 2,140 readings over 1,797 words, 106 in more than one file; the index about 93 KB of source and 11 ms to execute without a bytecode cache, the compile 9 ms, both once per choice and only for a caller who makes one; loading the TOML 39 ms. Weighed against: per-language deltas compiled at build time, which do not compose (under prefer `{pt, en}` `do` keeps its particle and its credential, while Portuguese's delta drops the credential and English's the particle); every combination precompiled, combinatorial; the TOML read at run time, slower and either re-running the build's checks or skipping them. + 5. **Proposed: the languages reach run time as a reading index.** A second generated module, one row per reading (text, tag, kind, behavioral properties), loaded for every caller with `Lexicon.default()`, since phase 4's `Reading.languages` (4a) is filled from it on every parse, rather than on first access to `reading.languages`, which would need a `Reading`, which holds no word, to carry one as a hidden key; the compiler moves from tools/vocab/build.py into the package so build and pack share one implementation, and a test holds that compiling the index with no choice reproduces `_compiled.py`. It is the per-word index item 13 of the 2026-10-23 #752 entry says `Token.reading` needs. A choice is applied as a delta against the default's compile, so it composes with a caller's `base=` lexicon and a `Locale`'s own fragment. Measured: 2,140 readings over 1,797 words, 106 in more than one file; the index about 93 KB of source, about 0.45 ms to import with its bytecode cached against about 20 ms for a cached `import nameparser` (2026-10-28, `c8357e74`; about 11 ms without the cache, the 2026-10-27 figure, taken under `PYTHONDONTWRITEBYTECODE`), once per process for every caller; the compile 9 ms, once per choice and only for a caller who makes one; loading the TOML 39 ms. Weighed against: per-language deltas compiled at build time, which do not compose (under prefer `{pt, en}` `do` keeps its particle and its credential, while Portuguese's delta drops the credential and English's the particle); every combination precompiled, combinatorial; the TOML read at run time, slower and either re-running the build's checks or skipping them. 6. **Proposed: a chosen tag selects its file and its ancestors', never its descendants, and no macrolanguage table is read.** `pt-BR` selects `pt-BR.toml` and `pt.toml`; `pt` only `pt.toml`, so a caller in Portugal does not turn on the Brazilian agnomens (#843). The two files share no word today. Malay and Indonesian, which share `puan`, `putra` and `putri`, are chosen together by a caller who has both. The TOML names `prefer` and `strict` stay refused, the message saying a strength is the caller's choice and never data. 7. **Proposed: a pack is measured three ways.** The default does not move (the committed snapshot, and the index invariant of item 5); a vocabulary pack's deviation surface, the words whose compiled readings its choice changes, is derived from the index, and tests/v2/test_locales.py's non-interference gate requires the default's reading for every corpus name without such a word, the policy and segmentation packs (JA, RU, TR_AZ, ZH) keeping their written `DEVIATES`; and `snapshot.py`, `--local` with `--base` included, takes a choice and prints what it moves by field, beside each pack's own ground truth (nepal-candidates' script pairs for `ne`, the CLDR agreement per pack, PSC's nationality slice). This answers decisions/3-0-reevaluations' 2026-09-13 bullet that the harness cannot see a pack. 8. **Proposed: third-party language files are deferred past phase 3.** A file joining the pool at run time would skip the checks that run across files (check 6's exclusions, check 11's particle statement), the ones that refuse a third-party `nl` `in` written with neither a `given` reading nor `may_be_given = false`. `Locale(lexicon=…)`, or its successor, stays the way to add vocabulary. 9. **Proposed: every exclusion a pack turns on becomes a `default = false` reading,** the shape cs `st` already has, so that once the packs land an `excluded` key means wrong everywhere: the eight whose reason opens "pack-shaped" in 3c (cs `ml`, hr, sk and sl `st` and `ml`, fr `père`), and the rest in the PR of the pack that turns each on (`pan`, `roz`, `nic`, `shri`, `sig`, `jaun`, `bt`, `bay`, `sb`). Recompute: `grep -h 'pack-shaped' nameparser/vocab/*.toml | wc -l`. - 10. **Proposed: what stays out of phase 3.** ZH's surname list and the RU and TR_AZ endings are rule data for phase 4's `[rules]`, and a Python pack gains `languages` only where a measurement shows what it moves; the candidates that wait on a position rule (Turkish `Bey` and `Hanım`, Hungarian `úr`, Thai `นาย`, French `M.`) are phase 4's; same-language clashes (`kumari`, `sayyid`) take phase 2j's route in the default, since no pack settles one. + 10. **Proposed: what stays out of phase 3.** ZH's surname list and the RU and TR_AZ endings are rule data for phase 4's `[rules]` (its 4f), and a Python pack gains `languages` only where a measurement shows what it moves; the candidates that wait on a position rule (Turkish `Bey` and `Hanım`, Hungarian `úr`, Thai `นาย`, French `M.`) are phase 4's; same-language clashes (`kumari`, `sayyid`) take phase 2j's route in the default, since no pack settles one. 11. **Proposed: the work order,** shape first and one PR each, measured on its own base: 3a the index and shared compiler with name readings allowed `default = false` (neutral); 3b the choice, tag selection, turning on, prefer, the shape clash, the derived deviation surface, `snapshot.py`'s choice and the amended contract (neutral); 3c the pack-shaped exclusions converted (neutral); then the packs, `ne` (3d), `hi` and `bn`'s Latin forms (3e), `cs`, `sk`, `sl` and `hr` (3f), `es` and `pt` with a connective that counts only inside a title run (3g), and `ga`/`gd` and `pt-BR` (3h); 3i, the bearers' languages of each name reading (item 4), after 3a, ahead of strict. 3g's connective is a rule with its own design: with `profesora` added and no rule, `Profesora de Historia Ana López` reads family `de Historia Ana López` (the rules/P1 fold) and `Ministro da Fazenda Carlos Souza` given `da`, while `Doctora de la Vega` must keep family `de la Vega`, the two separated by whether a name stands behind the phrase; it must reach only readings a pack turns on. - 12. **Proposed: phase 3 is 3.0 work, opt-in, the default's parses unmoved, with phase 4's shape questions settled before or alongside its mechanism** (the 2026-10-28 re-plan: 2.8 is the last 2.x feature release, and 3.0 ships phases 3 and 4 with the facade's removal). The output model decides what a pack can express (a patronymic field, a second surname, #326's credentials apart from generations, #704's `middle`). 3a and 3c are independent of it, and so are 3b's selection semantics; 3b's API waits on item 1's open dependency; `cs` and `pt-BR` wait on #326 and `es`/`pt` on a second surname, while `ne` does not. One standing text is amended with 3b, mechanisms.md#LOCALE-PACKS-PURE-DATA's contract that a pack is add-only and declares its deviations in its docstring (a pack adds, and selects among the default's readings by language, a vocabulary pack's deviations derived); decisions/3-0-reevaluations' Standing line that removing base vocabulary through a pack is part of 3.0's pack design is carried out. + 12. **Proposed: phase 3 is 3.0 work, opt-in, the default's parses unmoved, with phase 4's shape questions settled first** (the 2026-10-28 re-plan: 2.8 is the last 2.x feature release, and 3.0 ships phases 3 and 4 with the facade's removal). The output model decides what a pack can express (a patronymic field, a second surname, #326's credentials apart from generations, #704's `middle`). The output model was agreed on 2026-10-28 (the 2026-10-28 #754 entry). 3a and 3c are independent of it, and so are 3b's selection semantics, its API's dependency answered (item 1); 3f and 3h need 4b's `generation` view, 3g needs 4d's `surname2` and 4g's compound given name, and 4f, which compiles `[rules]` into a `Locale`'s policy patch, lands with or after 3b; `ne` needs none. One standing text is amended with 3b, mechanisms.md#LOCALE-PACKS-PURE-DATA's contract that a pack is add-only and declares its deviations in its docstring (a pack adds, and selects among the default's readings by language, a vocabulary pack's deviations derived); decisions/3-0-reevaluations' Standing line that removing base vocabulary through a pack is part of 3.0's pack design is carried out. diff --git a/docs/design/vocabulary-model-rfc.md b/docs/design/vocabulary-model-rfc.md index 841e13074..a5fa35d57 100644 --- a/docs/design/vocabulary-model-rfc.md +++ b/docs/design/vocabulary-model-rfc.md @@ -166,7 +166,7 @@ Each sub-phase is one PR, measured on its own base: the committed snapshot read Status: **proposed, not agreed** (2026-10-27, #753). The record is decisions/vocabulary-model's 2026-10-27 #753 entry, whose items are each marked proposed until Derek decides them; this section is the proposal those items describe. Phase 3 is 3.0 work (re-planned 2026-10-28): 2.8 is the last 2.x feature release, and 3.0 ships phases 3 and 4 with the facade's removal. Packs stay opt-in, so a caller who chooses no language parses as the default does, and nothing in this phase moves the default's parses. One standing text would be amended with it: mechanisms.md#LOCALE-PACKS-PURE-DATA's contract that a pack is add-only and declares its deviations in its docstring (a pack would add, and select among the default's readings by language, its deviations derived for a vocabulary pack). decisions/3-0-reevaluations' line that removing base vocabulary through a pack is part of 3.0's pack design is carried out rather than moved. -**Phase 4's shape questions come first, or alongside.** The output model decides what a pack can express: a patronymic field, a second surname, credentials apart from generation suffixes (#326), and what `middle` means (#704). A pack reading of a kind with no field to land in is a reading nothing can surface, so those questions are to be settled before or with phase 3's mechanism, by the same shape-before-value rule. Of the work order below, 3a (the reading index and the shared compiler) and 3c (the pack-shaped exclusions converted) are independent of the output model: the index carries whatever kinds the files hold, and 3c moves data between two encodings that compile alike. 3b's selection semantics are independent too, but its API is not: whether a choice rides on `Locale.languages` depends on what `Locale` and `Policy` look like once the facade is gone, so Q1 below is an open dependency. The packs depend on the output model in part: `cs`'s generations and `pt-BR`'s agnomens on #326, `es` and `pt` on a second surname; `ne`'s readings do not. +**Phase 4's shape questions came first.** The output model decides what a pack can express: a patronymic field, a second surname, credentials apart from generation suffixes (#326), and what `middle` means (#704). A pack reading of a kind with no field to land in is a reading nothing can surface, so those questions were to be settled before or with phase 3's mechanism, by the same shape-before-value rule, and the output model was agreed on 2026-10-28 (decisions/vocabulary-model's 2026-10-28 #754 entry): the seven roles stay, and `generation`, `credentials`, `patronymic` and `surname2` are views read from `Token.reading` or rules/O6's tag. Of the work order below, 3a (the reading index and the shared compiler) and 3c (the pack-shaped exclusions converted) are independent of the output model: the index carries whatever kinds the files hold, and 3c moves data between two encodings that compile alike. 3b's selection semantics are independent too, and its API's dependency, what `Locale` and `Policy` look like once the facade is gone, is answered (below). The packs depend on the output model in part, each on the phase 4 sub-phase that builds its view: `cs`'s generations and `pt-BR`'s agnomens on 4b's `generation` (3f and 3h), `es` and `pt` on 4d's `surname2` and 4g's compound given name (3g); `ne`'s readings do not. The semantics were fixed in 2026-10-15/16, and phase 3 keeps them. A pack is a choice of languages, in two strengths (decision 6): @@ -219,9 +219,9 @@ The 2 broken are pairs whose romanization opens with a listed title, which reads ### Where the provenance lives at runtime -The compiled module holds sets, and a set has forgotten which file each word came from. The proposal compiles a second module beside it: a **reading index**, one row per reading (its text, its file's tag, its kind and its behavioral properties), loaded only when a caller chooses a language. The compiler moves from tools/vocab/build.py into the package so that the build and a pack share one implementation, and a test holds the invariant that compiling the index with no choice reproduces `nameparser/config/_compiled.py` exactly. The same index is what item 13 of decisions/vocabulary-model's 2026-10-23 #752 entry says `Token.reading` needs in 3.0. +The compiled module holds sets, and a set has forgotten which file each word came from. The proposal compiles a second module beside it: a **reading index**, one row per reading (its text, its file's tag, its kind and its behavioral properties), loaded for every caller, with `Lexicon.default()`, because phase 4's `Reading.languages` (its sub-phase 4a) is filled from it on every parse. Loading it on first access to `reading.languages` instead was weighed and set aside: a `Reading` holds its kind, role and source but not its word, so a lazy field would have to carry the word as a hidden key on a frozen, slotted, pickled type, to save a cost the measurement below puts at about 2% of the import. The compiler moves from tools/vocab/build.py into the package so that the build and a pack share one implementation, and a test holds the invariant that compiling the index with no choice reproduces `nameparser/config/_compiled.py` exactly. The same index is what item 13 of decisions/vocabulary-model's 2026-10-23 #752 entry says `Token.reading` needs in 3.0. -Measured 2026-10-27 on `cfe8bf92`: the pool holds 2,140 readings over 1,797 words, 106 of them in more than one file. An index of their behavioral properties is about 93 KB of generated source, executing in about 11 ms with no bytecode cache, and compiling the pool takes about 9 ms, both paid once per choice and only by a caller who makes one. Reading the TOML instead costs about 39 ms. Recompute: the build's `read_dir`, `load` and `compile_pool`, timed. +Measured 2026-10-27 on `cfe8bf92`: the pool holds 2,140 readings over 1,797 words, 106 of them in more than one file. An index of their behavioral properties is about 93 KB of generated source. Imported with its bytecode cached, as an installed wheel has it, it takes about 0.45 ms (median of nine fresh processes, 2026-10-28 on `c8357e74`, Python 3.11.17), against about 20 ms for `import nameparser` cached the same way; with no cache it takes about 11 ms, the figure this section gave on 2026-10-27, measured in a shell that sets `PYTHONDONTWRITEBYTECODE`. That load is paid once per process by every caller. Compiling the pool takes about 9 ms, paid once per choice and only by a caller who makes one. Reading the TOML instead costs about 39 ms. Recompute: the build's `read_dir`, `load` and `compile_pool`, timed; for the import, write the index as a module, `py_compile` it, and time its import in fresh processes beside a cached `import nameparser`. Two cheaper-looking forms do not work. Per-language deltas compiled at build time do not compose: under prefer `{pt, en}`, `do` keeps both its particle and its credential, but Portuguese's delta removes the credential and English's removes the particle. And precompiling every combination of languages is combinatorial. @@ -229,7 +229,7 @@ A choice is applied as a delta against the default's compile, so it composes wit ### The API and tags -Open dependency: this assumes `Locale` and `Policy` keep their 2.x shape after the facade's removal, which is undecided. A choice is carried by `Locale`, which gains a `languages` field, a set of BCP 47 tags. `parser_for` unions the `languages` of the packs it is given, so the choice is order-independent while their policy patches still fold in order. Strict, once it can be built, is a property of the whole choice, not of one pack, and an argument: `parser_for(locales.NE, locales.HI, strict=True)`. The shipped language packs are `Locale` constants like the four today (`locales.NE`, `locales.get("ne")`), each with an empty lexicon fragment; a code keeps `Locale`'s `[a-z0-9_]+`, so `pt-BR` is `locales.PT_BR`. +This rested on `Locale` and `Policy` keeping their 2.x shape after the facade's removal, and the 2026-10-28 3.0 freeze answers it: both keep it, with additions. A choice is carried by `Locale`, which gains a `languages` field, a set of BCP 47 tags. `parser_for` unions the `languages` of the packs it is given, so the choice is order-independent while their policy patches still fold in order. Strict, once it can be built, is a property of the whole choice, not of one pack, and an argument: `parser_for(locales.NE, locales.HI, strict=True)`. The shipped language packs are `Locale` constants like the four today (`locales.NE`, `locales.get("ne")`), each with an empty lexicon fragment; a code keeps `Locale`'s `[a-z0-9_]+`, so `pt-BR` is `locales.PT_BR`. A chosen tag selects its own file and its ancestors', never its descendants': `pt-BR` selects `pt-BR.toml` and `pt.toml`, and `pt` selects only `pt.toml`, so a caller in Portugal does not turn on the Brazilian agnomens (#843 left the single-letter `F` and `N` to a `pt-BR` pack). No macrolanguage table is read: Malay and Indonesian are separate tags, though BCP 47 counts Indonesian within the Malay macrolanguage, and a caller with names in both chooses both. The Czech, Slovak, Slovenian and Croatian files each hold the words they share, so each language is a pack of its own. Script subtags need no matching rule yet, since no file carries one. @@ -246,13 +246,13 @@ The snapshot and `compare.py` parse under the default, so neither can see a pack ### What stays out of phase 3 - **Third-party language files.** A file joining the pool at run time would skip the checks that run across files, an exclusion in one file refusing a reading in any other (check 6) and check 11's particle statement. A third-party `nl` file adding the preposition `in` without a `given` reading or `may_be_given = false` is refused by the build and would ship on the unsafe side at run time. `Locale(lexicon=…)`, or its successor once the facade is gone, stays the way to add vocabulary; a public file format is a later question. -- **The Python packs' rule data.** ZH's surnames (556 on 2026-10-27) are a segmentation list, outside the per-word model, and the RU and TR_AZ markers are endings inside rules (`-ovich`, `oğlu`); both wait for phase 4's `[rules]`. A pack gets a `languages` field only where measuring it shows what it moves. +- **The Python packs' rule data.** ZH's surnames (556 on 2026-10-27) are a segmentation list, outside the per-word model, and the RU and TR_AZ markers are endings inside rules (`-ovich`, `oğlu`); both wait for phase 4's `[rules]`, compiled into a `Locale`'s policy patch in its sub-phase 4f. A pack gets a `languages` field only where measuring it shows what it moves. - **Candidates that wait on a rule, not a word:** Turkish `Bey` and `Hanım` after the given name, Hungarian `úr` after the name, Thai `นาย` glued at the head, French `M.` against the initial it spells. Each is a position rule, phase 4's. - **Same-language clashes.** No pack settles one (the `fils` argument): the Indic and Arabic titles that are names in their own language (`kumari`, `sayyid`) take phase 2j's route, a given reading and the title phrase, as default work. ### The order of work -Shape first, then the packs, each one PR measured on its own base, as phase 2's were. The first three move no default parse, and no pack exists until 3d. Strict is not in this order (above); 3i, the bearers' languages it would read, is. +Shape first, then the packs, each one PR measured on its own base, as phase 2's were, interleaved with phase 4's sub-phases as its work order says: a pack needing a view waits for the phase 4 sub-phase that builds it, and 4f, which compiles `[rules]` into a `Locale`'s policy patch, lands with or after 3b, both deciding what a shipped `Locale` holds. The first three move no default parse, and no pack exists until 3d. Strict is not in this order (above); 3i, the bearers' languages it would read, is. | Sub-phase | What | Needs | |---|---|---| @@ -261,9 +261,9 @@ Shape first, then the packs, each one PR measured on its own base, as phase 2's | 3c | the eight exclusions whose reason opens "pack-shaped" become `default = false` readings: cs `ml`, hr, sk and sl `st` and `ml`, fr `père`; every other exclusion a pack turns on (`pan`, `roz`, `nic`, `shri`, `sig`, `jaun`, `bt`, `bay`, `sb`) converts in the pack's own PR, so that after the packs an exclusion means wrong everywhere | — (neutral) | | 3d | `ne`: `given` on `श्री`, `guru` and `गुरु`, measured on nepal-candidates' script pairs | 3b | | 3e | `hi` and `bn`, the Latin forms (`smt`, `shrimati`, `sushri`, `shriyut`, `shri`, `adhyapak`), the ones whose collisions prefer cannot settle left for strict (#345) | 3b | -| 3f | `cs`, `sk`, `sl`, `hr`: the generations of 3c, `pan`, `roz` and the academic abbreviations #848 left | 3b, 3c | -| 3g | `es` and `pt`: the occupational titles and `Don`/`Doña`/`Dona`, with a connective that counts only inside a title run, its own rule; `pt` also settles the one-case `e` | 3b | -| 3h | `ga` and `gd` (`nic`, `mhic`, `mic`, `uí`), `pt-BR` (`F`, `N`, `sb`); `nl`'s prepositions wait with strict | 3b | +| 3f | `cs`, `sk`, `sl`, `hr`: the generations of 3c, `pan`, `roz` and the academic abbreviations #848 left | 3b, 3c, 4b | +| 3g | `es` and `pt`: the occupational titles and `Don`/`Doña`/`Dona`, with a connective that counts only inside a title run, its own rule; `pt` also settles the one-case `e` | 3b, 4d, 4g | +| 3h | `ga` and `gd` (`nic`, `mhic`, `mic`, `uí`), `pt-BR` (`F`, `N`, `sb`); `nl`'s prepositions wait with strict | 3b, 4b | | 3i | the languages of each name reading's bearers, an attribute compiling to nothing, filled word by word from per-country shares and whole-box counts (above); strict and its candidates after it | 3a (neutral) | 3g's connective needs a design of its own, because the pair that separates it is a count: `Profesora de Historia Ana López` should read title `Profesora de Historia`, while `Doctora de la Vega`, the same title, preposition and following word, has no name behind it and must keep family `de la Vega`. With `profesora` added and no rule, the first reads family `de Historia Ana López` (the rules/P1 fold), and `Ministro da Fazenda Carlos Souza` reads given `da`. The rule must reach only readings a pack turns on, so the default's parses stay where they are. From b0a987cbca5059e8cf02924b5abea1012246c627 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Wed, 28 Oct 2026 18:03:22 -0700 Subject: [PATCH 29/39] Record the render signature decision (#754) render(spec: str | None = None) is taken: None is the order-aware default str() passes. The family-first layout's title place stays open. Co-Authored-By: Claude Opus 5.5 --- docs/design/decisions/vocabulary-model.md | 4 ++-- docs/design/vocabulary-model-rfc.md | 6 +++--- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/design/decisions/vocabulary-model.md b/docs/design/decisions/vocabulary-model.md index 537a2b1d0..0df997de1 100644 --- a/docs/design/decisions/vocabulary-model.md +++ b/docs/design/decisions/vocabulary-model.md @@ -11,7 +11,7 @@ The vocabulary was a set of base sets plus marker subsets (`particles` and `part - `e`, `y` and `i` stay in the default; choosing a language is to settle their one-case fork, which no shipped pack does yet — 2026-10-15/16. - A patronymic field is to come in 3.0, as a view that names its words without emptying `family`, and the particle kinds already draw the `clan`/`patronymic` line for it: a patronymic is followed by a parent's name, a clan prefix by a family name the lineage shares — 2026-10-15/16; 2026-10-19 #749; 2026-10-28 #754 (Derek). - A `Locale` may carry CLDR-shaped render patterns, display and sorting, a later addition whose shape is agreed; how a call reaches one is open — 2026-10-28 #754, render patterns. -- 3.0's output model keeps the seven roles and adds every new field as a view, read from `Token.reading`, the reading each token took, or for `surname2` from rules/O6's tag: `generation` and `credentials` partitioning `suffix`, `patronymic`, and `surname2` where rules/O6 read two family names; `middle` and `surnames` keep their meaning, a compound given name being a rule a caller or pack turns on; the default `str()` renders a name in the order it was read in with its written separators in a normalized form; a language file's `[rules]` compile into its `Locale`'s policy patch — 2026-10-28 #754 (Derek). +- 3.0's output model keeps the seven roles and adds every new field as a view, read from `Token.reading`, the reading each token took, or for `surname2` from rules/O6's tag: `generation` and `credentials` partitioning `suffix`, `patronymic`, and `surname2` where rules/O6 read two family names; `middle` and `surnames` keep their meaning, a compound given name being a rule a caller or pack turns on; the default `str()` renders a name in the order it was read in with its written separators in a normalized form, through `render(spec: str | None = None)`, `None` being that default; a language file's `[rules]` compile into its `Locale`'s policy patch — 2026-10-28 #754 (Derek). - Gender is recorded only where a word gives a definitive clue: a feminine form or one half of a fixed pair, never an unmarked noun — 2026-10-15/16; 2026-10-19 #750, title kinds. - Title-phrase modifiers count as titles only beside a head title, at the head of the name only and never in a comma part (`Wing Lau` → given 'Wing', `John Smith, Vice` → title 'Vice'), save a modifier that may be a given name, which the head's test reads in every part (`Lau, Wing Kai` → given 'Wing'), rules/H9 — 2026-10-15/16; 2026-10-19 #750, title kinds; 2026-10-23 #752; 2026-10-25 #752, phase 2i; 2026-10-26 #752, phase 2j, `wing`. - Name readings are hand-declared, and only on a word some language file claims as vocabulary or refuses, so the vocabulary carries no surname list (the segmentation lists, Korean and the ZH pack's, are outside the per-word model) — 2026-10-15/16; 2026-10-19. @@ -263,7 +263,7 @@ Open: #913 (the title words phase 2j left, read as titles though borne as names) 3. **A `patronymic` view, filled by a word or a rule that says patronymic, with `family` and `middle` unchanged.** The default fills it from the vocabulary's `patronymic` particles (`bin`, `binti`, `ibn`, `bint`, `bar`, `bat` and the native-script forms) with the name they introduce (`Mahathir bin Mohamad` → family `bin Mohamad`, as today, and patronymic `bin Mohamad`); RU's and TR_AZ's rules fill it where a caller turns them on (`Ivan Ivanovich Ivanov` under RU → middle `Ivanovich` and patronymic `Ivanovich`); an Icelandic ending rule fills it under an `is` pack only, since the default cannot tell `-dóttir` and `-son` from Swedish surnames (ORCID's Swedish records end their family box in `-son` or `-dóttir` in 2,184 of 20,000). A matronymic, written the same way, fills it too. SUPERSEDES: the RFC's earlier statement of decision 8's value, that `family` can be empty for a name with a patronymic. Measured: filers put an Icelandic patronymic in the family box (ORCID 1,647 of 2,583 Icelandic records, PatentsView 535 of 722, PSC 101 of 125), and CLDR's `is` file's 3 Icelandic samples put it in `surname` or `surname-core` in all 9 of their renderings (its one foreign sample puts a `-dóttir` word in `surname2`), while Wikidata's `patronymic-is` and `patronymic-my` strata record 108 names with a patronymic and no family name, of which the parse puts the patronymic in `family` 107 times. An East Slavic patronymic stands in the given part: CLDR's 2 `ru` samples put it in `given2` in all 6 renderings; PatentsView's Russian given boxes end in one 8,423 times of 20,000, and the parse reads such a word as the middle 10,902 times of 10,962; 90 of the 112 Russian nationals in PSC who filled the middle-name box wrote one there. A `family` that keeps it and a `patronymic` that names it agree with both kinds of source. Weighed against: emptying `family` where the patronymic is recognized, which moves the field the fielded sources fill and changes `family` for every Icelandic and nasab name, a behavior change of a frozen property; and a neutral parent-names name for the view, which buys nothing the view can detect, since a matronymic and a patronymic are written alike. 4. **`generation` and `credentials` are views that partition `suffix` (#326)**, not roles, `generation` the tokens read as a generation or an agnomen, `credentials` the rest, the post-nominals that are neither (`Esq.`, the honorifics after a name) falling in `credentials`, CLDR's catch-all, with `Token.reading.kind` telling them apart for a caller who needs it. Measured: of the 1,054 committed corpus entries with a suffix, 33 hold a generation (or agnomen) and a listed credential together, 77 a generation beside any other suffix word, and 64 only listed post-nominals of other kinds (each suffix word classified by the RFC's Phase 4 recipe, joined tokens merged); of the 615 CLDR renderings with either field, the 75 whose suffix holds exactly CLDR's words split into CLDR's `generation` and `credentials` by the vocabulary's reading kind in all 75. Weighed against: two new `Role` members (item 1), and a third view for the honorifics, left open since CLDR has none. 5. **a `surname2` view, the family's second unit where rules/O6 read the family as two**, marked by a new stable tag where O6 takes its word, empty otherwise; `family` keeps both units as O6 makes it, and a family comma alone marks nothing (`Lloyd Webber, Andrew`). Measured: a two-word family box in ORCID's records from Spain 48.0%, Mexico 64.7%, Colombia 72.8% and Chile 50.5%, against 3.9% from the United States. Left open: a view for the first unit and its name, `surname` standing close to the `surnames` view that exists. Declined: redefining `surnames` as the two units, which would repeat `family`, the family holding both where O6 reads two, and lose what `surnames` does today, joining `middle` and `family` so that a double surname the default splits (`Gabriel García Márquez` → middle `García`) reads whole, the view rules/P2 names for that reading. - 6. **The render defaults follow the order the name was read in, a behavior change of the output rather than a break of the API.** `ParsedName` gains `order` (an addition, `ADDED_IN`, a 2.x pickle rendering as today, left out of comparison, which lets two equal names render differently, a cost accepted 2026-10-28: a 2.x pickle of `김민준` renders `'민준 김'` and a fresh parse `'김민준'`, and the two compare equal); the default `str()` and `render()` spec put the family first where `order` does, and join two words that stand one directly after the other, in the same order, in the original and the rendering with the kind of separator that stood between them there, normalized, decided 2026-10-28 (nothing; one ASCII space for any whitespace, the ideographic space U+3000 included (`山田　太郎` → `'山田 太郎'`, where today gives `'太郎 山田'`); an interpunct or a nakaguro the parse read as a separator alone, a spaced interpunct being a name word, rules/T3; one comma the parse reads as `, ` if ASCII, `، ` if Arabic and alone if full-width, since a full-width comma carries its own width in CJK text, a reason for the comma the rendering keeps and not for the whitespace it normalizes (the Japanese comma U+3001 is no comma to the parse, a parser question outside this design), so `John Smith ,Jr.` renders `'John Smith, Jr.'`; the written whitespace staying in `ParsedName.original` and the token spans), the spec's decoration between the two words' fields standing instead (the nickname's quotes, the `) ` after a maiden name), and any other pair or separator (a delimiter, a consumed marker word, two or more commas, a hyphen, a slash, a zero-width or directional mark) taking the spec's own joiner; `initials()` gives nothing for a word in a script with no initials. A name with a family comma or written given-first keeps today's word order and its separators by the same rule, a suffix comma alone leaving a family-first name family-first (`威廉·莎士比亚` keeps its dot; 20 of the 36 names whose interpunct `str()` drops are read given-first). A family comma is never kept, the rendering reversing its pair (`Smith, John` → `'John Smith'`), and one directly between the family and its suffix is, decided 2026-10-28 (`John Smith, Jr.` → `'John Smith, Jr.'`, today `'John Smith Jr.'`; 214 of the 1,003 committed corpus entries whose suffix follows the family write one there). Accepted with it: `Smith, John, Jr.`, whose suffix the rendering moves away from the given name before it, keeps `'John Smith Jr.'`, so two writings of one name render differently. Measured over the committed corpus's distinct entries, the counts reading the ASCII comma (the RFC gives the full-width and Arabic ones: 36 of the 728 hold a family comma the parse reads and keep today's word order): 728 without a comma read family-first, `str()` renders all 728 differently from their text (`김민준` → `'민준 김'`), 150 of them with the family's last word touching the given name's first, and it drops the U+00B7 interpunct from 36 of the 41 holding one. This carries out decisions/3-0-reevaluations' render-default clean slate. Proposed with it, not yet decided (the RFC's Phase 4 Render defaults): `render(spec: str | None = None)`, `None` the order-aware default `str()` passes and a caller's spec rendering exactly as written, so `render("{given} {family}")` keeps its 2.x meaning and a caller passing 2.x's default string keeps given-first output; and a family-first layout with the title first, the suffix last and the name roles in `order`'s sequence, each with its decoration, a family-comma name keeping a given-first `order`. The signature is a change, not an addition as the freeze defines one (no working call changes meaning, `render(None)` raising today, but the default's introspection and a subclass override typed `spec: str` do), and the title's place is open, 50 of CLDR's 70 surnameFirst renderings with a title putting it after the surname. Apart from that proposed signature, no signature changes, but `str()`'s output changes for every name read family-first, every name whose words are written with nothing, an interpunct or a nakaguro between two words the rendering keeps side by side, and every name with one comma directly between its family and its suffix, and no 2.x release can warn about a `str()` result, so it is taken as an improvement for 3.0's release notes, not a break needing a warning. + 6. **The render defaults follow the order the name was read in, a behavior change of the output rather than a break of the API.** `ParsedName` gains `order` (an addition, `ADDED_IN`, a 2.x pickle rendering as today, left out of comparison, which lets two equal names render differently, a cost accepted 2026-10-28: a 2.x pickle of `김민준` renders `'민준 김'` and a fresh parse `'김민준'`, and the two compare equal); the default `str()` and `render()` spec put the family first where `order` does, and join two words that stand one directly after the other, in the same order, in the original and the rendering with the kind of separator that stood between them there, normalized, decided 2026-10-28 (nothing; one ASCII space for any whitespace, the ideographic space U+3000 included (`山田　太郎` → `'山田 太郎'`, where today gives `'太郎 山田'`); an interpunct or a nakaguro the parse read as a separator alone, a spaced interpunct being a name word, rules/T3; one comma the parse reads as `, ` if ASCII, `، ` if Arabic and alone if full-width, since a full-width comma carries its own width in CJK text, a reason for the comma the rendering keeps and not for the whitespace it normalizes (the Japanese comma U+3001 is no comma to the parse, a parser question outside this design), so `John Smith ,Jr.` renders `'John Smith, Jr.'`; the written whitespace staying in `ParsedName.original` and the token spans), the spec's decoration between the two words' fields standing instead (the nickname's quotes, the `) ` after a maiden name), and any other pair or separator (a delimiter, a consumed marker word, two or more commas, a hyphen, a slash, a zero-width or directional mark) taking the spec's own joiner; `initials()` gives nothing for a word in a script with no initials. A name with a family comma or written given-first keeps today's word order and its separators by the same rule, a suffix comma alone leaving a family-first name family-first (`威廉·莎士比亚` keeps its dot; 20 of the 36 names whose interpunct `str()` drops are read given-first). A family comma is never kept, the rendering reversing its pair (`Smith, John` → `'John Smith'`), and one directly between the family and its suffix is, decided 2026-10-28 (`John Smith, Jr.` → `'John Smith, Jr.'`, today `'John Smith Jr.'`; 214 of the 1,003 committed corpus entries whose suffix follows the family write one there). Accepted with it: `Smith, John, Jr.`, whose suffix the rendering moves away from the given name before it, keeps `'John Smith Jr.'`, so two writings of one name render differently. Measured over the committed corpus's distinct entries, the counts reading the ASCII comma (the RFC gives the full-width and Arabic ones: 36 of the 728 hold a family comma the parse reads and keep today's word order): 728 without a comma read family-first, `str()` renders all 728 differently from their text (`김민준` → `'민준 김'`), 150 of them with the family's last word touching the given name's first, and it drops the U+00B7 interpunct from 36 of the 41 holding one. This carries out decisions/3-0-reevaluations' render-default clean slate. Decided with it on 2026-10-28 (the RFC's Phase 4 Render defaults): `render(spec: str | None = None)`, `None` the order-aware default `str()` passes and a caller's spec rendering exactly as written, so `render("{given} {family}")` keeps its 2.x meaning and a caller passing 2.x's default string keeps given-first output. Proposed and not yet decided: a family-first layout with the title first, the suffix last and the name roles in `order`'s sequence, each with its decoration, a family-comma name keeping a given-first `order`. The signature is a change, not an addition as the freeze defines one (no working call changes meaning, `render(None)` raising today, but the default's introspection and a subclass override typed `spec: str` do), Derek taking it as worth that, and the title's place is open, 50 of CLDR's 70 surnameFirst renderings with a title putting it after the surname. Apart from that signature, no signature changes, but `str()`'s output changes for every name read family-first, every name whose words are written with nothing, an interpunct or a nakaguro between two words the rendering keeps side by side, and every name with one comma directly between its family and its suffix, and no 2.x release can warn about a `str()` result, so it is taken as an improvement for 3.0's release notes, not a break needing a warning. 7. **`middle` keeps its meaning (#704)**, every word of the given part after the given name, the given name one unit unless a rule joins words into it, as rules/P5 already does (`Abdul Aziz Tariq` → given `Abdul Aziz`); a compound given name is a `[rules]` rule a caller or a pack turns on, CLDR's `given2` is `middle`, and `Kaur` and an English name beside a Chinese one stay in `middle`, an accepted cost. Measured: in PSC, whose form has a middle-name box, 2,499 of 1,040,708 forename boxes hold two or more words, and of 6,101 Spanish nationals 25 do while 2,621 fill the middle box; ORCID's and PatentsView's given box holds middles too, so it cannot separate the two. The `Maria do Carmo` shape, a particle inside a given box of three or more words with a family box filled, reads the given box's words into the family in 1,812 of 1,923 ORCID records and 1,165 of 1,219 PatentsView records from Brazil, Portugal, Spain, Mexico, Argentina, Colombia, Chile and Italy, which is that rule's case. Weighed against: #704's option (b), one given unit by default, which changes `middle` for every English name of three words. 8. **A language file's `[rules]` compiles into its shipped `Locale`'s policy patch**, each rule kind a `Policy` field (an addition), so RU and TR_AZ are rebuilt from their files and `Locale` gains no field for rules. With it, **phase 3's open API dependency is answered**: its design (the 2026-10-27 #753 entry, item 1) carries a choice of languages on `Locale.languages`, assuming `Locale` and `Policy` keep their 2.x shape after the facade's removal, and the 2026-10-28 freeze keeps them so. The rest of that design, strict as an argument of the whole choice included, stays phase 3's to decide; its RFC section records the dependency as answered. 9. **The work order**, one PR each, measured on its own base, the snapshot recording each new view where it is non-empty: 4a `Token.reading` and `ReadingKind.PATRONYMIC` (neutral; `languages` after 3a); 4b `generation` and `credentials` (neutral); 4c `patronymic` (neutral); 4d `surname2` (neutral); 4e the render defaults and `order`; 4f `[rules]` into the policy patch, with or after 3b; 4g the compound given name, after 4f; 4h the shape inventory per order (#469). Interleaved with phase 3 (its sub-phases the 2026-10-27 #753 entry's item 11): 4a beside 3a; 4b to 4d before the packs that need them, `cs` (3f) and `pt-BR` (3h) on 4b, `es` and `pt` (3g) on 4d and 4g; an `is` pack on 3b, 4c and 4f; 3i, the bearers' languages, on 3a alone. diff --git a/docs/design/vocabulary-model-rfc.md b/docs/design/vocabulary-model-rfc.md index 67bac5015..27da2403e 100644 --- a/docs/design/vocabulary-model-rfc.md +++ b/docs/design/vocabulary-model-rfc.md @@ -339,10 +339,10 @@ ORCID's and PatentsView's given box holds the middle names too, so a two-word gi - `ParsedName` gains `order`, the order the name was read under (its script's or the declared `name_order`, a family comma aside), an addition with `ADDED_IN` metadata whose missing value, in a 2.x pickle, renders as today, and left out of comparison as `Token.reading` is, so a loaded 2.x pickle still equals a fresh parse of its text. The cost, accepted 2026-10-28: two equal names can render differently, where today every rendering is derived from what `==` compares (a 2.x pickle of `김민준` renders `'민준 김'` and a fresh parse `'김민준'`, and a set keeps one of them); the views read from `Token.reading` share it. - `str()` and `render()`'s default spec put the family first where `order` does, and two words that stand one directly after the other, in the same order, in the original and in the rendering keep the kind of separator that stood between them in the original, written in a normalized form (decided 2026-10-28): nothing stays nothing; whitespace of any width or length becomes one ASCII space, the ideographic space U+3000 included (decided 2026-10-28: `山田　太郎` renders `'山田 太郎'`, the same ASCII space `str()` joins with today, where today's order gives `'太郎 山田'`, and a caller wanting native spacing passes a format spec of their own or reads `ParsedName.original`); an interpunct or a nakaguro the parse read as a separator stands alone, with no space around it (an interpunct written with spaces round it is not one, rules/T3 dividing at it only when flanked, so `威廉 · 莎士比亚` reads the dot as a name word and keeps rendering with its spaces, where `威廉·莎士比亚` keeps the dot unspaced; a spaced nakaguro is a separator, so `ジョン ・ スミス` renders `'ジョン・スミス'`); and one comma the parse reads as one becomes `, ` if ASCII and `، ` if Arabic (U+060C), no space before it and one after, and stands alone if full-width (U+FF0C), which carries its own width in CJK text, a reason that holds for the comma, a mark the rendering keeps, and not for whitespace, which the rendering normalizes, so `John Smith ,Jr.` and `John Smith, Jr.` render `'John Smith, Jr.'` and `John Smith， Jr.` renders `'John Smith，Jr.'`. The Japanese comma 、 (U+3001) is not one the parse reads as a comma, a parser question outside this design. Over that, the spec's decoration between the two fields the words belong to stands in place of the separator (the nickname's quotes, the `) ` after the maiden name, so `Jane Smith née Jones, PhD` keeps `'Jane Smith (Jones) PhD'`; 38 committed entries put a suffix directly after an undelimited maiden name); and any other pair, or any other separator (a delimiter, a marker word the parse consumed, two or more commas, a hyphen, a slash, a zero-width or directional mark), takes the spec's own joiner, as today (the written whitespace is not lost: `ParsedName.original` and the token spans keep it), so `John (Jack) Smith` keeps today's quoted nickname and `str(parse("김민준"))` is `'김민준'` and `str(parse("威廉·莎士比亚"))` keeps the dot. A name with a family comma or written given-first keeps today's word order, and its separators are kept by the same rule (a comma before a suffix alone does not make a family-first name given-first: `Wang Xiaoming, Jr.` under FAMILY_FIRST renders `'Wang Xiaoming, Jr.'`), so the given-first transcription `威廉·莎士比亚` keeps its dot too (20 of the 36 names whose interpunct `str()` drops are read given-first, 4 of them with a comma the parse reads (1 ASCII, 3 full-width)). A family comma is never kept, since the rendering reverses the pair it stands between (`Smith, John` → `'John Smith'`), while a comma directly between the family and its suffix is (`John Smith, Jr.` → `'John Smith, Jr.'`, where `str()` gives `'John Smith Jr.'` today): of the 1,003 committed corpus entries whose suffix starts after the family's last word ends, 214 have nothing between the two but one comma and whitespace (2026-10-28, `c8357e74`, each entry under its declared order; these counts read the ASCII comma, and 7 more write a full-width one there, while 36 of the 728 family-first names above hold a family comma the parse reads, 5 full-width and 31 Arabic (30 of the Arabic in `corpus_cldr.jsonl`), and so keep today's word order under the rule); 456 hold a comma somewhere between, most with a given name or title standing there too. Pairs the rendering separates, as `John` and `Jr.` in `Smith, John, Jr.`, take the spec's own joiner, as today, so `Smith, John, Jr.` keeps rendering `'John Smith Jr.'` while `John Smith, Jr.` becomes `'John Smith, Jr.'`: two writings of one name rendering differently, a consequence of following the text accepted with the choice. -- **How, proposed and not yet decided.** Today `str(name)` is `name.render()`, whose default is one fixed given-first spec, `'{title} {given} "{nickname}" {middle} {family} ({maiden}) {suffix}'`, which is why `김민준` renders `'민준 김'`; a default that reads `order` and the written separators cannot stay one string. So `render(spec: str | None = None)`: `None`, what `str()` passes, is the order-aware default, and a spec a caller writes renders exactly as written, its order and its literal joiners, with no separator rule applied, so `render("{given} {family}")` means in 3.0 what it meant in 2.x. That is a signature change, not an addition as the freeze defines one (decisions/3-0-reevaluations, THE CORE API), and needs its own decision: no call that works today changes meaning, since `render(None)` raises `TypeError` today, but introspecting the default gives `None` rather than a spec, and a subclass overriding `render(self, spec: str = ...)` becomes an incompatible override to a type checker. The default's layout is picked by `order`, and a name with a family comma has a given-first `order` ("a family comma aside", above), so a given-first name and a family-comma name render by today's spec with the separator rule (`Smith, John` under FAMILY_FIRST still renders `'John Smith'`). For a family-first `order` the layout puts the title first and the suffix last, as the given-first spec does, and the three name roles between them in `order`'s own sequence, each keeping its decoration beside it: `'{title} {family} ({maiden}) {given} "{nickname}" {middle} {suffix}'` for FAMILY_FIRST and `'{title} {family} ({maiden}) {middle} {given} "{nickname}" {suffix}'` for FAMILY_FIRST_GIVEN_LAST, the separator rule then joining adjacent words (`王小明先生`, read family `王小明` and suffix `先生`, renders `'王小明先生'` where today gives `'王小明 先生'`; `Dr. Wang Xiaoming Jr.` under FAMILY_FIRST renders `'Dr. Wang Xiaoming Jr.'` where today gives `'Dr. Xiaoming Wang Jr.'`). CLDR's surnameFirst samples (`tools/differential/cldr_fields.jsonl`, release 48) agree on the core order and not on the rest: `ja` (`佐藤グレース恵子さん`) and `vi` (`ĐBQH Nguyễn Văn Anh`) put the second given name before the first, as FAMILY_FIRST_GIVEN_LAST does, and `ko` (`김민규헨리`) puts it after, as FAMILY_FIRST does; but most put the title after the family, not first: `pt` after it (`da Silva Sr. José Pedro Eduardo Jr., MP`), `ja` and `zh` last, and over every file's nativeFull surnameFirst rendering holding a title and a surname, 50 of 70 put the title after the surname (2026-10-28, positions of the field values in the text); `zh` also puts the generation first (`小彭德威东升议员先生`), and `vi` the credentials first. So keeping the title first, where the given-first default and the written name have it, goes against most of CLDR, and whether the family-first default should move it after the family is part of what is to be decided; a locale's own placement is the render patterns' job (below). The suffix last agrees with `ja` and `zh`, whose honorifics the parse reads as suffixes. For 3.0's notes: a caller who copied 2.x's default string and passes it keeps given-first, space-joined output; only a caller who passes nothing, `str()` included, gets the new default. `None` leaves both answers to how a pack's pattern is reached as open as they were: a spec string still means a spec, so `render("sorting")` still returns `'sorting'`, and a pattern is asked for by a new method or argument under (a) or through the parser under (b). +- **How: the signature decided 2026-10-28, the family-first layout's title place open.** Today `str(name)` is `name.render()`, whose default is one fixed given-first spec, `'{title} {given} "{nickname}" {middle} {family} ({maiden}) {suffix}'`, which is why `김민준` renders `'민준 김'`; a default that reads `order` and the written separators cannot stay one string. So `render(spec: str | None = None)`: `None`, what `str()` passes, is the order-aware default, and a spec a caller writes renders exactly as written, its order and its literal joiners, with no separator rule applied, so `render("{given} {family}")` means in 3.0 what it meant in 2.x. That is a signature change, not an addition as the freeze defines one (decisions/3-0-reevaluations, THE CORE API), and Derek took it on 2026-10-28: no call that works today changes meaning, since `render(None)` raises `TypeError` today, but introspecting the default gives `None` rather than a spec, and a subclass overriding `render(self, spec: str = ...)` becomes an incompatible override to a type checker. The default's layout is picked by `order`, and a name with a family comma has a given-first `order` ("a family comma aside", above), so a given-first name and a family-comma name render by today's spec with the separator rule (`Smith, John` under FAMILY_FIRST still renders `'John Smith'`). For a family-first `order` the layout puts the title first and the suffix last, as the given-first spec does, and the three name roles between them in `order`'s own sequence, each keeping its decoration beside it: `'{title} {family} ({maiden}) {given} "{nickname}" {middle} {suffix}'` for FAMILY_FIRST and `'{title} {family} ({maiden}) {middle} {given} "{nickname}" {suffix}'` for FAMILY_FIRST_GIVEN_LAST, the separator rule then joining adjacent words (`王小明先生`, read family `王小明` and suffix `先生`, renders `'王小明先生'` where today gives `'王小明 先生'`; `Dr. Wang Xiaoming Jr.` under FAMILY_FIRST renders `'Dr. Wang Xiaoming Jr.'` where today gives `'Dr. Xiaoming Wang Jr.'`). CLDR's surnameFirst samples (`tools/differential/cldr_fields.jsonl`, release 48) agree on the core order and not on the rest: `ja` (`佐藤グレース恵子さん`) and `vi` (`ĐBQH Nguyễn Văn Anh`) put the second given name before the first, as FAMILY_FIRST_GIVEN_LAST does, and `ko` (`김민규헨리`) puts it after, as FAMILY_FIRST does; but most put the title after the family, not first: `pt` after it (`da Silva Sr. José Pedro Eduardo Jr., MP`), `ja` and `zh` last, and over every file's nativeFull surnameFirst rendering holding a title and a surname, 50 of 70 put the title after the surname (2026-10-28, positions of the field values in the text); `zh` also puts the generation first (`小彭德威东升议员先生`), and `vi` the credentials first. So keeping the title first, where the given-first default and the written name have it, goes against most of CLDR, and whether the family-first default should move it after the family is part of what is to be decided; a locale's own placement is the render patterns' job (below). The suffix last agrees with `ja` and `zh`, whose honorifics the parse reads as suffixes. For 3.0's notes: a caller who copied 2.x's default string and passes it keeps given-first, space-joined output; only a caller who passes nothing, `str()` included, gets the new default. `None` leaves both answers to how a pack's pattern is reached as open as they were: a spec string still means a spec, so `render("sorting")` still returns `'sorting'`, and a pattern is asked for by a new method or argument under (a) or through the parser under (b). - `initials()` gives nothing for a word in a script with no initials (Han, Hangul, kana, `_policy._NO_INITIALS`), where today it gives the first character with a period (`'王.'`). -Apart from that proposed signature, these change no signature but change `str()`'s output for every name read family-first, every name whose words are written with nothing, an interpunct or a nakaguro between two words the rendering keeps side by side (not `김, 민준씨`, whose family comma moves `김` between `민준` and `씨`), and every name written with one comma directly between its family and its suffix, which no 2.x release can warn about; Derek took them as an improvement for 3.0's release notes rather than a break (item 6 of the entry). +Apart from that signature, these change no signature but change `str()`'s output for every name read family-first, every name whose words are written with nothing, an interpunct or a nakaguro between two words the rendering keeps side by side (not `김, 민준씨`, whose family comma moves `김` between `민준` and `씨`), and every name written with one comma directly between its family and its suffix, which no 2.x release can warn about; Derek took them as an improvement for 3.0's release notes rather than a break (item 6 of the entry). **What changes in the API**, by the 3.0 freeze: @@ -355,7 +355,7 @@ Apart from that proposed signature, these change no signature but change `str()` | new `Policy` fields for rules, and `Locale.languages` (phase 3) | additions | | `Role`, `as_dict()`, `comparison_key()`, the seven role properties | unchanged | | the default `str()`, `render()` spec and `initials()` output | behavior change, accepted for 3.0 | -| `render(spec: str \| None = None)`, proposed | a signature change, not an addition; undecided | +| `render(spec: str \| None = None)` | a signature change, not an addition; decided 2026-10-28 | | a `Role` member for any of these views | a break, declined | **Phase 3's open API dependency answered**: phase 3's design (above) carries a choice of languages on `Locale.languages` and notes it assumes `Locale` and `Policy` keep their 2.x shape once the facade is gone; the 2026-10-28 freeze answers that, both keeping it with additions. The rest of that item, strict as an argument of the whole choice included, is phase 3's to decide, and its section records the dependency as answered. With the `[rules]` proposal above, a shipped language pack is a `Locale` whose `languages` choose its vocabulary and whose `policy` carries its rules, so `Locale` needs no third field. From 5685eae8c757d465004d8fcc628fac943bbe3ba8 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Wed, 28 Oct 2026 18:14:09 -0700 Subject: [PATCH 30/39] Record the family-first title place and the Lexicon surface (#754) The family-first default keeps the title first, the one place the parse reads every title back from; reading titles where CLDR writes them is a separate parser issue. Vocabulary is configured by reading in 3.0 and the compiled sets become internal, the first break decided under the core API freeze; its 2.x warning is open. Co-Authored-By: Claude Opus 5.5 --- docs/design/decisions/3-0-reevaluations.md | 2 ++ docs/design/decisions/vocabulary-model.md | 4 +++- docs/design/vocabulary-model-rfc.md | 6 +++--- 3 files changed, 8 insertions(+), 4 deletions(-) diff --git a/docs/design/decisions/3-0-reevaluations.md b/docs/design/decisions/3-0-reevaluations.md index c3f4d9725..be514212c 100644 --- a/docs/design/decisions/3-0-reevaluations.md +++ b/docs/design/decisions/3-0-reevaluations.md @@ -13,6 +13,7 @@ Promoted 2026-08-15 from session memory (Derek's 2026-07-30 ask; promotion appro - Reading positionally where no script licenses family-first is the safe direction on its own, family-first being the marked case that needs affirmative evidence — 2026-08-15. - From 2.8, the last 2.x feature release, the uses 3.0 removes warn with a `DeprecationWarning` on every call at the caller's line naming code that runs: constructing `HumanName` or a `Constants`, loading either from a pickle, mutating the shared `CONSTANTS`, and reading `nameparser.config`'s three managers or `nameparser.util.lc`; importing `nameparser`, `nameparser.parser` or `HumanName` stays quiet, and so does reading the shared `CONSTANTS`, which no warning reaches — 2026-10-28 #925 (Derek), WARNINGS. - The 3.0 core API — `parse`, `Parser`, `Lexicon`, `Policy`, `Locale` and `ParsedName` — is frozen except for additions, so anything 3.0 later breaks there is a new decision needing its own 2.x warning — 2026-10-28 #925 (Derek), THE CORE API. +- `Lexicon`'s compiled-set fields are the first break decided under that freeze: 3.0 configures vocabulary by reading and keeps the sets internal; its 2.x warning is open — 2026-10-28 #754 (Derek), THE LEXICON SURFACE. - The deprecation bridge's mechanics are the template for every remaining 2.x→3.0 shim: PEP 562 module `__getattr__` plus `__all__`, a warning per read location rather than per process, and the `TYPE_CHECKING` split — 2026-08-15. - A scoped config override through contextvars (`with nameparser.config.use(c):`) stays parked for 3.0, the shared `CONSTANTS` surviving 2.x as the one config channel that reaches parses inside code a caller does not own — 2026-07-06. - The 1.3.0-era legacy-pickle skip of `suffixes_prefixes_titles` rides in the shim's `__setstate__` and dies with it — 2026-08-16. @@ -75,3 +76,4 @@ Open: #753 (phase 3, language packs, where the three pack bullets land). - The shared-`CONSTANTS` mutation message drops the private `Constants` it offered as an escape, which 3.0 removes too, and names `Parser(lexicon=Lexicon.default().add(titles={'dean'}))`; the hook-override message (#280) adds that `HumanName` is removed in 3.0. Every warning is a `DeprecationWarning` rather than a `FutureWarning`, emitted on every call at the caller's line and never once per process (a process flag lets a dependency's first `HumanName` consume the warning meant for the caller, the reason the 2026-08-15 bridge template warns per read location, and turns this suite's `filterwarnings = error` into an order-dependent failure under `--dist worksteal`). A `DeprecationWarning` is hidden by default outside `__main__`, so the release log, README and docs/migrate.rst announce the removal loudly instead. Every message names code that type-checks and runs, pinned by tests/v2/test_deprecations.py. `force_mixed_case_capitalization`, a removal candidate since 2026-08-29, is a `Constants` attribute, so removing `Constants` removes it and the construction warning is its removal warning; that follows from the facade's removal rather than being decided apart from it. Phase 4's output model is announced in one line of docs/migrate.rst ("details with the 3.0 alphas"), nothing undecided listed. - 2026-10-28 #925 (Derek), THE CORE API — THE 3.0 CORE API IS FROZEN EXCEPT FOR ADDITIONS: `parse`, `Parser`, `Lexicon`, `Policy`, `Locale` and `ParsedName` keep their 2.x shape through 3.0, gaining fields, members and methods but losing and renaming none. Anything 3.0 later finds it must break there is a new decision, and it needs a 2.x release warning about it first, the bridge discipline in this file's preamble; Phase 3's draft, which assumes `Locale` and `Policy` keep their 2.x shape after the facade's removal, rests on this entry. +- 2026-10-28 #754 (Derek), THE LEXICON SURFACE — `Lexicon`'S COMPILED-SET FIELDS GO INTERNAL IN 3.0, THE FIRST BREAK DECIDED UNDER THE CORE API FREEZE ABOVE. A caller configures vocabulary by reading instead; the reasons and the weighing are decisions/vocabulary-model's 2026-10-28 #754 entry on the `Lexicon` surface. The freeze asks a break for a 2.x release warning first, and 2.8 was the last 2.x feature release, so how that warning ships (a 2.8.x patch, or the release notes alone) is open. diff --git a/docs/design/decisions/vocabulary-model.md b/docs/design/decisions/vocabulary-model.md index 0df997de1..c12a7c111 100644 --- a/docs/design/decisions/vocabulary-model.md +++ b/docs/design/decisions/vocabulary-model.md @@ -10,6 +10,7 @@ The vocabulary was a set of base sets plus marker subsets (`particles` and `part - The default loads every language and keeps working for names of any language. Choosing a language is to narrow it in phase 3 (#753): a pack is to settle only the clashes where its language has a vocabulary reading, a strict pack to drop other languages' name-level readings too, and a pack's preferred reading to be able to outrank one a shape rule derives — 2026-10-15/16. - `e`, `y` and `i` stay in the default; choosing a language is to settle their one-case fork, which no shipped pack does yet — 2026-10-15/16. - A patronymic field is to come in 3.0, as a view that names its words without emptying `family`, and the particle kinds already draw the `clan`/`patronymic` line for it: a patronymic is followed by a parent's name, a clan prefix by a family name the lineage shares — 2026-10-15/16; 2026-10-19 #749; 2026-10-28 #754 (Derek). +- A caller configures vocabulary by reading, `Lexicon` deriving the sets and gates the build derives, and the compiled sets become internal in 3.0 — 2026-10-28 #754, the `Lexicon` surface (Derek). - A `Locale` may carry CLDR-shaped render patterns, display and sorting, a later addition whose shape is agreed; how a call reaches one is open — 2026-10-28 #754, render patterns. - 3.0's output model keeps the seven roles and adds every new field as a view, read from `Token.reading`, the reading each token took, or for `surname2` from rules/O6's tag: `generation` and `credentials` partitioning `suffix`, `patronymic`, and `surname2` where rules/O6 read two family names; `middle` and `surnames` keep their meaning, a compound given name being a rule a caller or pack turns on; the default `str()` renders a name in the order it was read in with its written separators in a normalized form, through `render(spec: str | None = None)`, `None` being that default; a language file's `[rules]` compile into its `Locale`'s policy patch — 2026-10-28 #754 (Derek). - Gender is recorded only where a word gives a definitive clue: a feminine form or one half of a fixed pair, never an unmarked noun — 2026-10-15/16; 2026-10-19 #750, title kinds. @@ -263,9 +264,10 @@ Open: #913 (the title words phase 2j left, read as titles though borne as names) 3. **A `patronymic` view, filled by a word or a rule that says patronymic, with `family` and `middle` unchanged.** The default fills it from the vocabulary's `patronymic` particles (`bin`, `binti`, `ibn`, `bint`, `bar`, `bat` and the native-script forms) with the name they introduce (`Mahathir bin Mohamad` → family `bin Mohamad`, as today, and patronymic `bin Mohamad`); RU's and TR_AZ's rules fill it where a caller turns them on (`Ivan Ivanovich Ivanov` under RU → middle `Ivanovich` and patronymic `Ivanovich`); an Icelandic ending rule fills it under an `is` pack only, since the default cannot tell `-dóttir` and `-son` from Swedish surnames (ORCID's Swedish records end their family box in `-son` or `-dóttir` in 2,184 of 20,000). A matronymic, written the same way, fills it too. SUPERSEDES: the RFC's earlier statement of decision 8's value, that `family` can be empty for a name with a patronymic. Measured: filers put an Icelandic patronymic in the family box (ORCID 1,647 of 2,583 Icelandic records, PatentsView 535 of 722, PSC 101 of 125), and CLDR's `is` file's 3 Icelandic samples put it in `surname` or `surname-core` in all 9 of their renderings (its one foreign sample puts a `-dóttir` word in `surname2`), while Wikidata's `patronymic-is` and `patronymic-my` strata record 108 names with a patronymic and no family name, of which the parse puts the patronymic in `family` 107 times. An East Slavic patronymic stands in the given part: CLDR's 2 `ru` samples put it in `given2` in all 6 renderings; PatentsView's Russian given boxes end in one 8,423 times of 20,000, and the parse reads such a word as the middle 10,902 times of 10,962; 90 of the 112 Russian nationals in PSC who filled the middle-name box wrote one there. A `family` that keeps it and a `patronymic` that names it agree with both kinds of source. Weighed against: emptying `family` where the patronymic is recognized, which moves the field the fielded sources fill and changes `family` for every Icelandic and nasab name, a behavior change of a frozen property; and a neutral parent-names name for the view, which buys nothing the view can detect, since a matronymic and a patronymic are written alike. 4. **`generation` and `credentials` are views that partition `suffix` (#326)**, not roles, `generation` the tokens read as a generation or an agnomen, `credentials` the rest, the post-nominals that are neither (`Esq.`, the honorifics after a name) falling in `credentials`, CLDR's catch-all, with `Token.reading.kind` telling them apart for a caller who needs it. Measured: of the 1,054 committed corpus entries with a suffix, 33 hold a generation (or agnomen) and a listed credential together, 77 a generation beside any other suffix word, and 64 only listed post-nominals of other kinds (each suffix word classified by the RFC's Phase 4 recipe, joined tokens merged); of the 615 CLDR renderings with either field, the 75 whose suffix holds exactly CLDR's words split into CLDR's `generation` and `credentials` by the vocabulary's reading kind in all 75. Weighed against: two new `Role` members (item 1), and a third view for the honorifics, left open since CLDR has none. 5. **a `surname2` view, the family's second unit where rules/O6 read the family as two**, marked by a new stable tag where O6 takes its word, empty otherwise; `family` keeps both units as O6 makes it, and a family comma alone marks nothing (`Lloyd Webber, Andrew`). Measured: a two-word family box in ORCID's records from Spain 48.0%, Mexico 64.7%, Colombia 72.8% and Chile 50.5%, against 3.9% from the United States. Left open: a view for the first unit and its name, `surname` standing close to the `surnames` view that exists. Declined: redefining `surnames` as the two units, which would repeat `family`, the family holding both where O6 reads two, and lose what `surnames` does today, joining `middle` and `family` so that a double surname the default splits (`Gabriel García Márquez` → middle `García`) reads whole, the view rules/P2 names for that reading. - 6. **The render defaults follow the order the name was read in, a behavior change of the output rather than a break of the API.** `ParsedName` gains `order` (an addition, `ADDED_IN`, a 2.x pickle rendering as today, left out of comparison, which lets two equal names render differently, a cost accepted 2026-10-28: a 2.x pickle of `김민준` renders `'민준 김'` and a fresh parse `'김민준'`, and the two compare equal); the default `str()` and `render()` spec put the family first where `order` does, and join two words that stand one directly after the other, in the same order, in the original and the rendering with the kind of separator that stood between them there, normalized, decided 2026-10-28 (nothing; one ASCII space for any whitespace, the ideographic space U+3000 included (`山田　太郎` → `'山田 太郎'`, where today gives `'太郎 山田'`); an interpunct or a nakaguro the parse read as a separator alone, a spaced interpunct being a name word, rules/T3; one comma the parse reads as `, ` if ASCII, `، ` if Arabic and alone if full-width, since a full-width comma carries its own width in CJK text, a reason for the comma the rendering keeps and not for the whitespace it normalizes (the Japanese comma U+3001 is no comma to the parse, a parser question outside this design), so `John Smith ,Jr.` renders `'John Smith, Jr.'`; the written whitespace staying in `ParsedName.original` and the token spans), the spec's decoration between the two words' fields standing instead (the nickname's quotes, the `) ` after a maiden name), and any other pair or separator (a delimiter, a consumed marker word, two or more commas, a hyphen, a slash, a zero-width or directional mark) taking the spec's own joiner; `initials()` gives nothing for a word in a script with no initials. A name with a family comma or written given-first keeps today's word order and its separators by the same rule, a suffix comma alone leaving a family-first name family-first (`威廉·莎士比亚` keeps its dot; 20 of the 36 names whose interpunct `str()` drops are read given-first). A family comma is never kept, the rendering reversing its pair (`Smith, John` → `'John Smith'`), and one directly between the family and its suffix is, decided 2026-10-28 (`John Smith, Jr.` → `'John Smith, Jr.'`, today `'John Smith Jr.'`; 214 of the 1,003 committed corpus entries whose suffix follows the family write one there). Accepted with it: `Smith, John, Jr.`, whose suffix the rendering moves away from the given name before it, keeps `'John Smith Jr.'`, so two writings of one name render differently. Measured over the committed corpus's distinct entries, the counts reading the ASCII comma (the RFC gives the full-width and Arabic ones: 36 of the 728 hold a family comma the parse reads and keep today's word order): 728 without a comma read family-first, `str()` renders all 728 differently from their text (`김민준` → `'민준 김'`), 150 of them with the family's last word touching the given name's first, and it drops the U+00B7 interpunct from 36 of the 41 holding one. This carries out decisions/3-0-reevaluations' render-default clean slate. Decided with it on 2026-10-28 (the RFC's Phase 4 Render defaults): `render(spec: str | None = None)`, `None` the order-aware default `str()` passes and a caller's spec rendering exactly as written, so `render("{given} {family}")` keeps its 2.x meaning and a caller passing 2.x's default string keeps given-first output. Proposed and not yet decided: a family-first layout with the title first, the suffix last and the name roles in `order`'s sequence, each with its decoration, a family-comma name keeping a given-first `order`. The signature is a change, not an addition as the freeze defines one (no working call changes meaning, `render(None)` raising today, but the default's introspection and a subclass override typed `spec: str` do), Derek taking it as worth that, and the title's place is open, 50 of CLDR's 70 surnameFirst renderings with a title putting it after the surname. Apart from that signature, no signature changes, but `str()`'s output changes for every name read family-first, every name whose words are written with nothing, an interpunct or a nakaguro between two words the rendering keeps side by side, and every name with one comma directly between its family and its suffix, and no 2.x release can warn about a `str()` result, so it is taken as an improvement for 3.0's release notes, not a break needing a warning. + 6. **The render defaults follow the order the name was read in, a behavior change of the output rather than a break of the API.** `ParsedName` gains `order` (an addition, `ADDED_IN`, a 2.x pickle rendering as today, left out of comparison, which lets two equal names render differently, a cost accepted 2026-10-28: a 2.x pickle of `김민준` renders `'민준 김'` and a fresh parse `'김민준'`, and the two compare equal); the default `str()` and `render()` spec put the family first where `order` does, and join two words that stand one directly after the other, in the same order, in the original and the rendering with the kind of separator that stood between them there, normalized, decided 2026-10-28 (nothing; one ASCII space for any whitespace, the ideographic space U+3000 included (`山田　太郎` → `'山田 太郎'`, where today gives `'太郎 山田'`); an interpunct or a nakaguro the parse read as a separator alone, a spaced interpunct being a name word, rules/T3; one comma the parse reads as `, ` if ASCII, `، ` if Arabic and alone if full-width, since a full-width comma carries its own width in CJK text, a reason for the comma the rendering keeps and not for the whitespace it normalizes (the Japanese comma U+3001 is no comma to the parse, a parser question outside this design), so `John Smith ,Jr.` renders `'John Smith, Jr.'`; the written whitespace staying in `ParsedName.original` and the token spans), the spec's decoration between the two words' fields standing instead (the nickname's quotes, the `) ` after a maiden name), and any other pair or separator (a delimiter, a consumed marker word, two or more commas, a hyphen, a slash, a zero-width or directional mark) taking the spec's own joiner; `initials()` gives nothing for a word in a script with no initials. A name with a family comma or written given-first keeps today's word order and its separators by the same rule, a suffix comma alone leaving a family-first name family-first (`威廉·莎士比亚` keeps its dot; 20 of the 36 names whose interpunct `str()` drops are read given-first). A family comma is never kept, the rendering reversing its pair (`Smith, John` → `'John Smith'`), and one directly between the family and its suffix is, decided 2026-10-28 (`John Smith, Jr.` → `'John Smith, Jr.'`, today `'John Smith Jr.'`; 214 of the 1,003 committed corpus entries whose suffix follows the family write one there). Accepted with it: `Smith, John, Jr.`, whose suffix the rendering moves away from the given name before it, keeps `'John Smith Jr.'`, so two writings of one name render differently. Measured over the committed corpus's distinct entries, the counts reading the ASCII comma (the RFC gives the full-width and Arabic ones: 36 of the 728 hold a family comma the parse reads and keep today's word order): 728 without a comma read family-first, `str()` renders all 728 differently from their text (`김민준` → `'민준 김'`), 150 of them with the family's last word touching the given name's first, and it drops the U+00B7 interpunct from 36 of the 41 holding one. This carries out decisions/3-0-reevaluations' render-default clean slate. Decided with it on 2026-10-28 (the RFC's Phase 4 Render defaults): `render(spec: str | None = None)`, `None` the order-aware default `str()` passes and a caller's spec rendering exactly as written, so `render("{given} {family}")` keeps its 2.x meaning and a caller passing 2.x's default string keeps given-first output. Decided too: a family-first layout with the title first, the suffix last and the name roles in `order`'s sequence, each with its decoration, a family-comma name keeping a given-first `order`. The signature is a change, not an addition as the freeze defines one (no working call changes meaning, `render(None)` raising today, but the default's introspection and a subclass override typed `spec: str` do), Derek taking it as worth that. The title stays first though 50 of CLDR's 70 surnameFirst renderings with a title put it after the surname, because the default renders only an order the parse reads back, and under a family-first order the parse reads a bare title only at the front (`Wang Dr Xiaoming` → given `Dr`, `Wang Xiaoming Senator` → middle `Senator`); reading a title where CLDR writes it is a separate parser issue, and the default may follow it. Apart from that signature, no signature changes, but `str()`'s output changes for every name read family-first, every name whose words are written with nothing, an interpunct or a nakaguro between two words the rendering keeps side by side, and every name with one comma directly between its family and its suffix, and no 2.x release can warn about a `str()` result, so it is taken as an improvement for 3.0's release notes, not a break needing a warning. 7. **`middle` keeps its meaning (#704)**, every word of the given part after the given name, the given name one unit unless a rule joins words into it, as rules/P5 already does (`Abdul Aziz Tariq` → given `Abdul Aziz`); a compound given name is a `[rules]` rule a caller or a pack turns on, CLDR's `given2` is `middle`, and `Kaur` and an English name beside a Chinese one stay in `middle`, an accepted cost. Measured: in PSC, whose form has a middle-name box, 2,499 of 1,040,708 forename boxes hold two or more words, and of 6,101 Spanish nationals 25 do while 2,621 fill the middle box; ORCID's and PatentsView's given box holds middles too, so it cannot separate the two. The `Maria do Carmo` shape, a particle inside a given box of three or more words with a family box filled, reads the given box's words into the family in 1,812 of 1,923 ORCID records and 1,165 of 1,219 PatentsView records from Brazil, Portugal, Spain, Mexico, Argentina, Colombia, Chile and Italy, which is that rule's case. Weighed against: #704's option (b), one given unit by default, which changes `middle` for every English name of three words. 8. **A language file's `[rules]` compiles into its shipped `Locale`'s policy patch**, each rule kind a `Policy` field (an addition), so RU and TR_AZ are rebuilt from their files and `Locale` gains no field for rules. With it, **phase 3's open API dependency is answered**: its design (the 2026-10-27 #753 entry, item 1) carries a choice of languages on `Locale.languages`, assuming `Locale` and `Policy` keep their 2.x shape after the facade's removal, and the 2026-10-28 freeze keeps them so. The rest of that design, strict as an argument of the whole choice included, stays phase 3's to decide; its RFC section records the dependency as answered. 9. **The work order**, one PR each, measured on its own base, the snapshot recording each new view where it is non-empty: 4a `Token.reading` and `ReadingKind.PATRONYMIC` (neutral; `languages` after 3a); 4b `generation` and `credentials` (neutral); 4c `patronymic` (neutral); 4d `surname2` (neutral); 4e the render defaults and `order`; 4f `[rules]` into the policy patch, with or after 3b; 4g the compound given name, after 4f; 4h the shape inventory per order (#469). Interleaved with phase 3 (its sub-phases the 2026-10-27 #753 entry's item 11): 4a beside 3a; 4b to 4d before the packs that need them, `cs` (3f) and `pt-BR` (3h) on 4b, `es` and `pt` (3g) on 4d and 4g; an `is` pack on 3b, 4c and 4f; 3i, the bearers' languages, on 3a alone. 10. **What stays out of the output model.** Gender (decision 9) waits, `Token.reading` able to carry it later as one more addition. Names of given names only are a rule's question, not a field's: ORCID's empty family box holds 68,006 of 1,831,533 records, 15.9% to 18.7% of those from India, Pakistan, Bangladesh and Indonesia, and the parse fills `family` in 51,367 of the 51,415 with two or more given-box words. #694's two-word surname after a given name (`Tariq Abdul Aziz`) is a parse rule for the default, outside phase 4. Phase 1's four reserved names said to be phase 4's (`separators`, `bare_only`, `leading_only`, `trailing_only`) have no sub-phase yet. - **2026-10-28 #754, render patterns — A `Locale` MAY CARRY CLDR-SHAPED RENDER PATTERNS (A GIVEN-FIRST AND A SURNAME-FIRST DISPLAY PATTERN AND A SORTING PATTERN), A SHAPE AGREED NOW FOR A FEATURE THAT COMES LATER AS AN ADDITION.** The RFC's Phase 4 section, "Render patterns a pack may carry", holds the shape: the pack field, a table of Phase 4's views against CLDR's person-name fields, and the gap it shows, CLDR's `surname-prefix` and `surname-core` belonging to the first surname where `family_particles` and `family_base` read the whole family, which the per-surname units close. CLDR is a starting point, not the authority: its `pt` sorting pattern files under the first surname and leaves `surname2` out, which is not ABNT's last-surname entry (decisions/R2). Proposed there, not decided: a casefolded key method beside `comparison_key()`, collation left to the caller; and, open for a decision, whether a pattern is reached through a convention recorded on the name or through the parser. +- **2026-10-28 #754, the `Lexicon` surface (Derek) — A CALLER CONFIGURES VOCABULARY BY READING, AND THE COMPILED SETS BECOME INTERNAL.** Derek decided in chat on 2026-10-28, choosing it over keeping a public field per compiled set and over keeping 2.0.0's eleven while hiding the eleven added since. A caller states what a word may be in the language files' terms (a `given` reading on `guru`, no credential reading on `dc`) and `Lexicon` derives the sets and gates as the build does. The reasons: the set mechanics are hard to keep straight even for the maintainer, a reading is what a caller can know about a word, and it is the form a pack contribution takes. Today the same configuration needs rules/H9's set algebra, `Lexicon.default().remove(titles={'guru'}).add(titles_ambiguous={'guru'})` (`Guru Prasad Sharma` → given `Guru`) and `remove(suffix_acronyms={'dc'})` (`Ram Bahadur DC` → family `DC`), while `add(suffix_acronyms_ambiguous={'dc'})` leaves suffix `DC` (2026-10-28, run on this branch's base). A caller's own removal may settle a clash: decision 15 binds the default and the packs, not a caller's data. `dataclasses.fields(Lexicon)` gives 22 public fields on 2026-10-28, eleven since 2.0.0's eleven; the RFC's Phase 4 "Left open" holds the weighing. A break of the frozen `Lexicon`, recorded in decisions/3-0-reevaluations. Open: the method names, whether a caller's own language file can join the pool (phase 3's item 8 defers third-party files), and the 2.x warning the freeze asks for, 2.8 being the last 2.x feature release. diff --git a/docs/design/vocabulary-model-rfc.md b/docs/design/vocabulary-model-rfc.md index 27da2403e..b99bb2da4 100644 --- a/docs/design/vocabulary-model-rfc.md +++ b/docs/design/vocabulary-model-rfc.md @@ -339,7 +339,7 @@ ORCID's and PatentsView's given box holds the middle names too, so a two-word gi - `ParsedName` gains `order`, the order the name was read under (its script's or the declared `name_order`, a family comma aside), an addition with `ADDED_IN` metadata whose missing value, in a 2.x pickle, renders as today, and left out of comparison as `Token.reading` is, so a loaded 2.x pickle still equals a fresh parse of its text. The cost, accepted 2026-10-28: two equal names can render differently, where today every rendering is derived from what `==` compares (a 2.x pickle of `김민준` renders `'민준 김'` and a fresh parse `'김민준'`, and a set keeps one of them); the views read from `Token.reading` share it. - `str()` and `render()`'s default spec put the family first where `order` does, and two words that stand one directly after the other, in the same order, in the original and in the rendering keep the kind of separator that stood between them in the original, written in a normalized form (decided 2026-10-28): nothing stays nothing; whitespace of any width or length becomes one ASCII space, the ideographic space U+3000 included (decided 2026-10-28: `山田　太郎` renders `'山田 太郎'`, the same ASCII space `str()` joins with today, where today's order gives `'太郎 山田'`, and a caller wanting native spacing passes a format spec of their own or reads `ParsedName.original`); an interpunct or a nakaguro the parse read as a separator stands alone, with no space around it (an interpunct written with spaces round it is not one, rules/T3 dividing at it only when flanked, so `威廉 · 莎士比亚` reads the dot as a name word and keeps rendering with its spaces, where `威廉·莎士比亚` keeps the dot unspaced; a spaced nakaguro is a separator, so `ジョン ・ スミス` renders `'ジョン・スミス'`); and one comma the parse reads as one becomes `, ` if ASCII and `، ` if Arabic (U+060C), no space before it and one after, and stands alone if full-width (U+FF0C), which carries its own width in CJK text, a reason that holds for the comma, a mark the rendering keeps, and not for whitespace, which the rendering normalizes, so `John Smith ,Jr.` and `John Smith, Jr.` render `'John Smith, Jr.'` and `John Smith， Jr.` renders `'John Smith，Jr.'`. The Japanese comma 、 (U+3001) is not one the parse reads as a comma, a parser question outside this design. Over that, the spec's decoration between the two fields the words belong to stands in place of the separator (the nickname's quotes, the `) ` after the maiden name, so `Jane Smith née Jones, PhD` keeps `'Jane Smith (Jones) PhD'`; 38 committed entries put a suffix directly after an undelimited maiden name); and any other pair, or any other separator (a delimiter, a marker word the parse consumed, two or more commas, a hyphen, a slash, a zero-width or directional mark), takes the spec's own joiner, as today (the written whitespace is not lost: `ParsedName.original` and the token spans keep it), so `John (Jack) Smith` keeps today's quoted nickname and `str(parse("김민준"))` is `'김민준'` and `str(parse("威廉·莎士比亚"))` keeps the dot. A name with a family comma or written given-first keeps today's word order, and its separators are kept by the same rule (a comma before a suffix alone does not make a family-first name given-first: `Wang Xiaoming, Jr.` under FAMILY_FIRST renders `'Wang Xiaoming, Jr.'`), so the given-first transcription `威廉·莎士比亚` keeps its dot too (20 of the 36 names whose interpunct `str()` drops are read given-first, 4 of them with a comma the parse reads (1 ASCII, 3 full-width)). A family comma is never kept, since the rendering reverses the pair it stands between (`Smith, John` → `'John Smith'`), while a comma directly between the family and its suffix is (`John Smith, Jr.` → `'John Smith, Jr.'`, where `str()` gives `'John Smith Jr.'` today): of the 1,003 committed corpus entries whose suffix starts after the family's last word ends, 214 have nothing between the two but one comma and whitespace (2026-10-28, `c8357e74`, each entry under its declared order; these counts read the ASCII comma, and 7 more write a full-width one there, while 36 of the 728 family-first names above hold a family comma the parse reads, 5 full-width and 31 Arabic (30 of the Arabic in `corpus_cldr.jsonl`), and so keep today's word order under the rule); 456 hold a comma somewhere between, most with a given name or title standing there too. Pairs the rendering separates, as `John` and `Jr.` in `Smith, John, Jr.`, take the spec's own joiner, as today, so `Smith, John, Jr.` keeps rendering `'John Smith Jr.'` while `John Smith, Jr.` becomes `'John Smith, Jr.'`: two writings of one name rendering differently, a consequence of following the text accepted with the choice. -- **How: the signature decided 2026-10-28, the family-first layout's title place open.** Today `str(name)` is `name.render()`, whose default is one fixed given-first spec, `'{title} {given} "{nickname}" {middle} {family} ({maiden}) {suffix}'`, which is why `김민준` renders `'민준 김'`; a default that reads `order` and the written separators cannot stay one string. So `render(spec: str | None = None)`: `None`, what `str()` passes, is the order-aware default, and a spec a caller writes renders exactly as written, its order and its literal joiners, with no separator rule applied, so `render("{given} {family}")` means in 3.0 what it meant in 2.x. That is a signature change, not an addition as the freeze defines one (decisions/3-0-reevaluations, THE CORE API), and Derek took it on 2026-10-28: no call that works today changes meaning, since `render(None)` raises `TypeError` today, but introspecting the default gives `None` rather than a spec, and a subclass overriding `render(self, spec: str = ...)` becomes an incompatible override to a type checker. The default's layout is picked by `order`, and a name with a family comma has a given-first `order` ("a family comma aside", above), so a given-first name and a family-comma name render by today's spec with the separator rule (`Smith, John` under FAMILY_FIRST still renders `'John Smith'`). For a family-first `order` the layout puts the title first and the suffix last, as the given-first spec does, and the three name roles between them in `order`'s own sequence, each keeping its decoration beside it: `'{title} {family} ({maiden}) {given} "{nickname}" {middle} {suffix}'` for FAMILY_FIRST and `'{title} {family} ({maiden}) {middle} {given} "{nickname}" {suffix}'` for FAMILY_FIRST_GIVEN_LAST, the separator rule then joining adjacent words (`王小明先生`, read family `王小明` and suffix `先生`, renders `'王小明先生'` where today gives `'王小明 先生'`; `Dr. Wang Xiaoming Jr.` under FAMILY_FIRST renders `'Dr. Wang Xiaoming Jr.'` where today gives `'Dr. Xiaoming Wang Jr.'`). CLDR's surnameFirst samples (`tools/differential/cldr_fields.jsonl`, release 48) agree on the core order and not on the rest: `ja` (`佐藤グレース恵子さん`) and `vi` (`ĐBQH Nguyễn Văn Anh`) put the second given name before the first, as FAMILY_FIRST_GIVEN_LAST does, and `ko` (`김민규헨리`) puts it after, as FAMILY_FIRST does; but most put the title after the family, not first: `pt` after it (`da Silva Sr. José Pedro Eduardo Jr., MP`), `ja` and `zh` last, and over every file's nativeFull surnameFirst rendering holding a title and a surname, 50 of 70 put the title after the surname (2026-10-28, positions of the field values in the text); `zh` also puts the generation first (`小彭德威东升议员先生`), and `vi` the credentials first. So keeping the title first, where the given-first default and the written name have it, goes against most of CLDR, and whether the family-first default should move it after the family is part of what is to be decided; a locale's own placement is the render patterns' job (below). The suffix last agrees with `ja` and `zh`, whose honorifics the parse reads as suffixes. For 3.0's notes: a caller who copied 2.x's default string and passes it keeps given-first, space-joined output; only a caller who passes nothing, `str()` included, gets the new default. `None` leaves both answers to how a pack's pattern is reached as open as they were: a spec string still means a spec, so `render("sorting")` still returns `'sorting'`, and a pattern is asked for by a new method or argument under (a) or through the parser under (b). +- **How, decided 2026-10-28.** Today `str(name)` is `name.render()`, whose default is one fixed given-first spec, `'{title} {given} "{nickname}" {middle} {family} ({maiden}) {suffix}'`, which is why `김민준` renders `'민준 김'`; a default that reads `order` and the written separators cannot stay one string. So `render(spec: str | None = None)`: `None`, what `str()` passes, is the order-aware default, and a spec a caller writes renders exactly as written, its order and its literal joiners, with no separator rule applied, so `render("{given} {family}")` means in 3.0 what it meant in 2.x. That is a signature change, not an addition as the freeze defines one (decisions/3-0-reevaluations, THE CORE API), and Derek took it on 2026-10-28: no call that works today changes meaning, since `render(None)` raises `TypeError` today, but introspecting the default gives `None` rather than a spec, and a subclass overriding `render(self, spec: str = ...)` becomes an incompatible override to a type checker. The default's layout is picked by `order`, and a name with a family comma has a given-first `order` ("a family comma aside", above), so a given-first name and a family-comma name render by today's spec with the separator rule (`Smith, John` under FAMILY_FIRST still renders `'John Smith'`). For a family-first `order` the layout puts the title first and the suffix last, as the given-first spec does, and the three name roles between them in `order`'s own sequence, each keeping its decoration beside it: `'{title} {family} ({maiden}) {given} "{nickname}" {middle} {suffix}'` for FAMILY_FIRST and `'{title} {family} ({maiden}) {middle} {given} "{nickname}" {suffix}'` for FAMILY_FIRST_GIVEN_LAST, the separator rule then joining adjacent words (`王小明先生`, read family `王小明` and suffix `先生`, renders `'王小明先生'` where today gives `'王小明 先生'`; `Dr. Wang Xiaoming Jr.` under FAMILY_FIRST renders `'Dr. Wang Xiaoming Jr.'` where today gives `'Dr. Xiaoming Wang Jr.'`). CLDR's surnameFirst samples (`tools/differential/cldr_fields.jsonl`, release 48) agree on the core order and not on the rest: `ja` (`佐藤グレース恵子さん`) and `vi` (`ĐBQH Nguyễn Văn Anh`) put the second given name before the first, as FAMILY_FIRST_GIVEN_LAST does, and `ko` (`김민규헨리`) puts it after, as FAMILY_FIRST does; but most put the title after the family, not first: `pt` after it (`da Silva Sr. José Pedro Eduardo Jr., MP`), `ja` and `zh` last, and over every file's nativeFull surnameFirst rendering holding a title and a surname, 50 of 70 put the title after the surname (2026-10-28, positions of the field values in the text); `zh` also puts the generation first (`小彭德威东升议员先生`), and `vi` the credentials first. Keeping the title first, where the given-first default and the written name have it, goes against most of CLDR, but it is the one place the parse reads every title back from under a family-first order: `Senator Wang Xiaoming` reads title `Senator`, while `Wang Dr Xiaoming` reads given `Dr` and `Wang Xiaoming Senator` middle `Senator`, a bare title ending a name being a name word (rules/H5), only a period-marked one (`Wang Xiaoming Dr.`) reading as the title there (2026-10-28, under `Policy(name_order=FAMILY_FIRST)`). Derek decided on 2026-10-28 that the default renders only an order the parse reads back, so the title stays first; reading a title where CLDR writes it is the parser's question, a separate issue, and the default may follow once the parse reads it there. A locale's own placement is the render patterns' job (below). The suffix last agrees with `ja` and `zh`, whose honorifics the parse reads as suffixes. For 3.0's notes: a caller who copied 2.x's default string and passes it keeps given-first, space-joined output; only a caller who passes nothing, `str()` included, gets the new default. `None` leaves both answers to how a pack's pattern is reached as open as they were: a spec string still means a spec, so `render("sorting")` still returns `'sorting'`, and a pattern is asked for by a new method or argument under (a) or through the parser under (b). - `initials()` gives nothing for a word in a script with no initials (Han, Hangul, kana, `_policy._NO_INITIALS`), where today it gives the first character with a period (`'王.'`). Apart from that signature, these change no signature but change `str()`'s output for every name read family-first, every name whose words are written with nothing, an interpunct or a nakaguro between two words the rendering keeps side by side (not `김, 민준씨`, whose family comma moves `김` between `민준` and `씨`), and every name written with one comma directly between its family and its suffix, which no 2.x release can warn about; Derek took them as an improvement for 3.0's release notes rather than a break (item 6 of the entry). @@ -398,9 +398,9 @@ Each sub-phase is one PR, measured on its own base as phase 2's and phase 3's ar **Left open.** With no sub-phase yet: phase 1's four reserved names said to be phase 4's (`separators` with phase 4's glued words, and `bare_only`, `leading_only` and `trailing_only`, which this RFC's phase 2 section found not needed), whose build messages still name this phase. -A shape question, to settle before the 3.0 freeze makes every field permanent: +A shape question, decided 2026-10-28: -- **Whether `Lexicon` keeps a public field for each compiled set**, or answers word, kind and property queries from phase 3's reading index and keeps its sets internal. `dataclasses.fields(Lexicon)` gives 22 public fields on 2026-10-28, 11 of them added since 2.0.0's 11 (`surnames`, `honorific_tails`, `conjunctions_ambiguous`, `after_given_titles`, `suffix_words_ambiguous`, `agnomens`, `given_name_abbreviations`, `titles_ambiguous`, `title_modifiers`, `title_styles`, `title_phrases`), each a new parse behavior's set, and a rule keyed on a title kind or on one language's reading of a kind (phase 3's design, "a property a language's own reading should take only under its pack") would add more. Keeping the fields: each is frozen API, shown in the bounded repr (`Lexicon(default + titles: +1)`), and pickled, where `Lexicon`'s own `__setstate__`, unlike the `ADDED_IN` guard the other types share, refuses any pickle whose fields differ, so every new field breaks every older `Lexicon` pickle, and with it every pickled `Parser` and locale pack, until that guard learns the marker. Turning them internal: a break that has to land in 3.0, where a query added beside the fields is an addition that can come later; a caller who configures vocabulary by the field today (`Lexicon.add(titles=...)`, `dataclasses.replace`) needs another way in, for words that carry no kind and no language; and phase 3's reading index, which a `Lexicon` answering queries would read, is already loaded for every caller with `Lexicon.default()` (about 0.45 ms with its bytecode cached, by phase 3's timing above), so answering from it adds no load. The parser need not pay either way: a frozenset test per word costs no frame and an internal cache keeps that path, while a query method called per word would add one per token, the cost the frame budget counts as scaling with the input. +- **Whether `Lexicon` keeps a public field for each compiled set**, or answers word, kind and property queries from phase 3's reading index and keeps its sets internal. `dataclasses.fields(Lexicon)` gives 22 public fields on 2026-10-28, 11 of them added since 2.0.0's 11 (`surnames`, `honorific_tails`, `conjunctions_ambiguous`, `after_given_titles`, `suffix_words_ambiguous`, `agnomens`, `given_name_abbreviations`, `titles_ambiguous`, `title_modifiers`, `title_styles`, `title_phrases`), each a new parse behavior's set, and a rule keyed on a title kind or on one language's reading of a kind (phase 3's design, "a property a language's own reading should take only under its pack") would add more. Keeping the fields: each is frozen API, shown in the bounded repr (`Lexicon(default + titles: +1)`), and pickled, where `Lexicon`'s own `__setstate__`, unlike the `ADDED_IN` guard the other types share, refuses any pickle whose fields differ, so every new field breaks every older `Lexicon` pickle, and with it every pickled `Parser` and locale pack, until that guard learns the marker. Turning them internal: a break that has to land in 3.0, where a query added beside the fields is an addition that can come later; a caller who configures vocabulary by the field today (`Lexicon.add(titles=...)`, `dataclasses.replace`) needs another way in, for words that carry no kind and no language; and phase 3's reading index, which a `Lexicon` answering queries would read, is already loaded for every caller with `Lexicon.default()` (about 0.45 ms with its bytecode cached, by phase 3's timing above), so answering from it adds no load. The parser need not pay either way: a frozenset test per word costs no frame and an internal cache keeps that path, while a query method called per word would add one per token, the cost the frame budget counts as scaling with the input. **Derek decided on 2026-10-28 that a caller configures vocabulary by reading and the compiled sets become internal** (decisions/vocabulary-model's 2026-10-28 #754 entry on the `Lexicon` surface). A caller states what a word may be in the language files' terms, a `given` reading on `guru`, no credential reading on `dc`, and `Lexicon` derives the sets and gates the build derives, so making `Guru` a possible given name needs no knowledge of rules/H9's sets, where today it is `remove(titles={'guru'}).add(titles_ambiguous={'guru'})`, and `add(given_name_titles=...)` or `add(suffix_acronyms_ambiguous=...)`, the plausible wrong guesses, do something else. A reading is also what a caller can know about a word and what a pack contribution states. Removing a reading from a caller's own lexicon may settle a clash; decision 15 binds the default and the packs, not a caller's choice about their own data. It breaks the frozen `Lexicon` (decisions/3-0-reevaluations); the method names, whether a caller's own language file can join the pool later (phase 3 defers it), and the 2.x warning the freeze asks for are open. Value questions inside the shapes above: From 1261d9a611ce94d509df7e8d951977ef3ded8a9a Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Wed, 28 Oct 2026 18:22:41 -0700 Subject: [PATCH 31/39] Cite #938 for reading titles where CLDR writes them (#754) Co-Authored-By: Claude Opus 5.5 --- docs/design/decisions/vocabulary-model.md | 2 +- docs/design/vocabulary-model-rfc.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/design/decisions/vocabulary-model.md b/docs/design/decisions/vocabulary-model.md index c12a7c111..a87454427 100644 --- a/docs/design/decisions/vocabulary-model.md +++ b/docs/design/decisions/vocabulary-model.md @@ -264,7 +264,7 @@ Open: #913 (the title words phase 2j left, read as titles though borne as names) 3. **A `patronymic` view, filled by a word or a rule that says patronymic, with `family` and `middle` unchanged.** The default fills it from the vocabulary's `patronymic` particles (`bin`, `binti`, `ibn`, `bint`, `bar`, `bat` and the native-script forms) with the name they introduce (`Mahathir bin Mohamad` → family `bin Mohamad`, as today, and patronymic `bin Mohamad`); RU's and TR_AZ's rules fill it where a caller turns them on (`Ivan Ivanovich Ivanov` under RU → middle `Ivanovich` and patronymic `Ivanovich`); an Icelandic ending rule fills it under an `is` pack only, since the default cannot tell `-dóttir` and `-son` from Swedish surnames (ORCID's Swedish records end their family box in `-son` or `-dóttir` in 2,184 of 20,000). A matronymic, written the same way, fills it too. SUPERSEDES: the RFC's earlier statement of decision 8's value, that `family` can be empty for a name with a patronymic. Measured: filers put an Icelandic patronymic in the family box (ORCID 1,647 of 2,583 Icelandic records, PatentsView 535 of 722, PSC 101 of 125), and CLDR's `is` file's 3 Icelandic samples put it in `surname` or `surname-core` in all 9 of their renderings (its one foreign sample puts a `-dóttir` word in `surname2`), while Wikidata's `patronymic-is` and `patronymic-my` strata record 108 names with a patronymic and no family name, of which the parse puts the patronymic in `family` 107 times. An East Slavic patronymic stands in the given part: CLDR's 2 `ru` samples put it in `given2` in all 6 renderings; PatentsView's Russian given boxes end in one 8,423 times of 20,000, and the parse reads such a word as the middle 10,902 times of 10,962; 90 of the 112 Russian nationals in PSC who filled the middle-name box wrote one there. A `family` that keeps it and a `patronymic` that names it agree with both kinds of source. Weighed against: emptying `family` where the patronymic is recognized, which moves the field the fielded sources fill and changes `family` for every Icelandic and nasab name, a behavior change of a frozen property; and a neutral parent-names name for the view, which buys nothing the view can detect, since a matronymic and a patronymic are written alike. 4. **`generation` and `credentials` are views that partition `suffix` (#326)**, not roles, `generation` the tokens read as a generation or an agnomen, `credentials` the rest, the post-nominals that are neither (`Esq.`, the honorifics after a name) falling in `credentials`, CLDR's catch-all, with `Token.reading.kind` telling them apart for a caller who needs it. Measured: of the 1,054 committed corpus entries with a suffix, 33 hold a generation (or agnomen) and a listed credential together, 77 a generation beside any other suffix word, and 64 only listed post-nominals of other kinds (each suffix word classified by the RFC's Phase 4 recipe, joined tokens merged); of the 615 CLDR renderings with either field, the 75 whose suffix holds exactly CLDR's words split into CLDR's `generation` and `credentials` by the vocabulary's reading kind in all 75. Weighed against: two new `Role` members (item 1), and a third view for the honorifics, left open since CLDR has none. 5. **a `surname2` view, the family's second unit where rules/O6 read the family as two**, marked by a new stable tag where O6 takes its word, empty otherwise; `family` keeps both units as O6 makes it, and a family comma alone marks nothing (`Lloyd Webber, Andrew`). Measured: a two-word family box in ORCID's records from Spain 48.0%, Mexico 64.7%, Colombia 72.8% and Chile 50.5%, against 3.9% from the United States. Left open: a view for the first unit and its name, `surname` standing close to the `surnames` view that exists. Declined: redefining `surnames` as the two units, which would repeat `family`, the family holding both where O6 reads two, and lose what `surnames` does today, joining `middle` and `family` so that a double surname the default splits (`Gabriel García Márquez` → middle `García`) reads whole, the view rules/P2 names for that reading. - 6. **The render defaults follow the order the name was read in, a behavior change of the output rather than a break of the API.** `ParsedName` gains `order` (an addition, `ADDED_IN`, a 2.x pickle rendering as today, left out of comparison, which lets two equal names render differently, a cost accepted 2026-10-28: a 2.x pickle of `김민준` renders `'민준 김'` and a fresh parse `'김민준'`, and the two compare equal); the default `str()` and `render()` spec put the family first where `order` does, and join two words that stand one directly after the other, in the same order, in the original and the rendering with the kind of separator that stood between them there, normalized, decided 2026-10-28 (nothing; one ASCII space for any whitespace, the ideographic space U+3000 included (`山田　太郎` → `'山田 太郎'`, where today gives `'太郎 山田'`); an interpunct or a nakaguro the parse read as a separator alone, a spaced interpunct being a name word, rules/T3; one comma the parse reads as `, ` if ASCII, `، ` if Arabic and alone if full-width, since a full-width comma carries its own width in CJK text, a reason for the comma the rendering keeps and not for the whitespace it normalizes (the Japanese comma U+3001 is no comma to the parse, a parser question outside this design), so `John Smith ,Jr.` renders `'John Smith, Jr.'`; the written whitespace staying in `ParsedName.original` and the token spans), the spec's decoration between the two words' fields standing instead (the nickname's quotes, the `) ` after a maiden name), and any other pair or separator (a delimiter, a consumed marker word, two or more commas, a hyphen, a slash, a zero-width or directional mark) taking the spec's own joiner; `initials()` gives nothing for a word in a script with no initials. A name with a family comma or written given-first keeps today's word order and its separators by the same rule, a suffix comma alone leaving a family-first name family-first (`威廉·莎士比亚` keeps its dot; 20 of the 36 names whose interpunct `str()` drops are read given-first). A family comma is never kept, the rendering reversing its pair (`Smith, John` → `'John Smith'`), and one directly between the family and its suffix is, decided 2026-10-28 (`John Smith, Jr.` → `'John Smith, Jr.'`, today `'John Smith Jr.'`; 214 of the 1,003 committed corpus entries whose suffix follows the family write one there). Accepted with it: `Smith, John, Jr.`, whose suffix the rendering moves away from the given name before it, keeps `'John Smith Jr.'`, so two writings of one name render differently. Measured over the committed corpus's distinct entries, the counts reading the ASCII comma (the RFC gives the full-width and Arabic ones: 36 of the 728 hold a family comma the parse reads and keep today's word order): 728 without a comma read family-first, `str()` renders all 728 differently from their text (`김민준` → `'민준 김'`), 150 of them with the family's last word touching the given name's first, and it drops the U+00B7 interpunct from 36 of the 41 holding one. This carries out decisions/3-0-reevaluations' render-default clean slate. Decided with it on 2026-10-28 (the RFC's Phase 4 Render defaults): `render(spec: str | None = None)`, `None` the order-aware default `str()` passes and a caller's spec rendering exactly as written, so `render("{given} {family}")` keeps its 2.x meaning and a caller passing 2.x's default string keeps given-first output. Decided too: a family-first layout with the title first, the suffix last and the name roles in `order`'s sequence, each with its decoration, a family-comma name keeping a given-first `order`. The signature is a change, not an addition as the freeze defines one (no working call changes meaning, `render(None)` raising today, but the default's introspection and a subclass override typed `spec: str` do), Derek taking it as worth that. The title stays first though 50 of CLDR's 70 surnameFirst renderings with a title put it after the surname, because the default renders only an order the parse reads back, and under a family-first order the parse reads a bare title only at the front (`Wang Dr Xiaoming` → given `Dr`, `Wang Xiaoming Senator` → middle `Senator`); reading a title where CLDR writes it is a separate parser issue, and the default may follow it. Apart from that signature, no signature changes, but `str()`'s output changes for every name read family-first, every name whose words are written with nothing, an interpunct or a nakaguro between two words the rendering keeps side by side, and every name with one comma directly between its family and its suffix, and no 2.x release can warn about a `str()` result, so it is taken as an improvement for 3.0's release notes, not a break needing a warning. + 6. **The render defaults follow the order the name was read in, a behavior change of the output rather than a break of the API.** `ParsedName` gains `order` (an addition, `ADDED_IN`, a 2.x pickle rendering as today, left out of comparison, which lets two equal names render differently, a cost accepted 2026-10-28: a 2.x pickle of `김민준` renders `'민준 김'` and a fresh parse `'김민준'`, and the two compare equal); the default `str()` and `render()` spec put the family first where `order` does, and join two words that stand one directly after the other, in the same order, in the original and the rendering with the kind of separator that stood between them there, normalized, decided 2026-10-28 (nothing; one ASCII space for any whitespace, the ideographic space U+3000 included (`山田　太郎` → `'山田 太郎'`, where today gives `'太郎 山田'`); an interpunct or a nakaguro the parse read as a separator alone, a spaced interpunct being a name word, rules/T3; one comma the parse reads as `, ` if ASCII, `، ` if Arabic and alone if full-width, since a full-width comma carries its own width in CJK text, a reason for the comma the rendering keeps and not for the whitespace it normalizes (the Japanese comma U+3001 is no comma to the parse, a parser question outside this design), so `John Smith ,Jr.` renders `'John Smith, Jr.'`; the written whitespace staying in `ParsedName.original` and the token spans), the spec's decoration between the two words' fields standing instead (the nickname's quotes, the `) ` after a maiden name), and any other pair or separator (a delimiter, a consumed marker word, two or more commas, a hyphen, a slash, a zero-width or directional mark) taking the spec's own joiner; `initials()` gives nothing for a word in a script with no initials. A name with a family comma or written given-first keeps today's word order and its separators by the same rule, a suffix comma alone leaving a family-first name family-first (`威廉·莎士比亚` keeps its dot; 20 of the 36 names whose interpunct `str()` drops are read given-first). A family comma is never kept, the rendering reversing its pair (`Smith, John` → `'John Smith'`), and one directly between the family and its suffix is, decided 2026-10-28 (`John Smith, Jr.` → `'John Smith, Jr.'`, today `'John Smith Jr.'`; 214 of the 1,003 committed corpus entries whose suffix follows the family write one there). Accepted with it: `Smith, John, Jr.`, whose suffix the rendering moves away from the given name before it, keeps `'John Smith Jr.'`, so two writings of one name render differently. Measured over the committed corpus's distinct entries, the counts reading the ASCII comma (the RFC gives the full-width and Arabic ones: 36 of the 728 hold a family comma the parse reads and keep today's word order): 728 without a comma read family-first, `str()` renders all 728 differently from their text (`김민준` → `'민준 김'`), 150 of them with the family's last word touching the given name's first, and it drops the U+00B7 interpunct from 36 of the 41 holding one. This carries out decisions/3-0-reevaluations' render-default clean slate. Decided with it on 2026-10-28 (the RFC's Phase 4 Render defaults): `render(spec: str | None = None)`, `None` the order-aware default `str()` passes and a caller's spec rendering exactly as written, so `render("{given} {family}")` keeps its 2.x meaning and a caller passing 2.x's default string keeps given-first output. Decided too: a family-first layout with the title first, the suffix last and the name roles in `order`'s sequence, each with its decoration, a family-comma name keeping a given-first `order`. The signature is a change, not an addition as the freeze defines one (no working call changes meaning, `render(None)` raising today, but the default's introspection and a subclass override typed `spec: str` do), Derek taking it as worth that. The title stays first though 50 of CLDR's 70 surnameFirst renderings with a title put it after the surname, because the default renders only an order the parse reads back, and under a family-first order the parse reads a bare title only at the front (`Wang Dr Xiaoming` → given `Dr`, `Wang Xiaoming Senator` → middle `Senator`); reading a title where CLDR writes it is a parser question, #938, and the default may follow it. Apart from that signature, no signature changes, but `str()`'s output changes for every name read family-first, every name whose words are written with nothing, an interpunct or a nakaguro between two words the rendering keeps side by side, and every name with one comma directly between its family and its suffix, and no 2.x release can warn about a `str()` result, so it is taken as an improvement for 3.0's release notes, not a break needing a warning. 7. **`middle` keeps its meaning (#704)**, every word of the given part after the given name, the given name one unit unless a rule joins words into it, as rules/P5 already does (`Abdul Aziz Tariq` → given `Abdul Aziz`); a compound given name is a `[rules]` rule a caller or a pack turns on, CLDR's `given2` is `middle`, and `Kaur` and an English name beside a Chinese one stay in `middle`, an accepted cost. Measured: in PSC, whose form has a middle-name box, 2,499 of 1,040,708 forename boxes hold two or more words, and of 6,101 Spanish nationals 25 do while 2,621 fill the middle box; ORCID's and PatentsView's given box holds middles too, so it cannot separate the two. The `Maria do Carmo` shape, a particle inside a given box of three or more words with a family box filled, reads the given box's words into the family in 1,812 of 1,923 ORCID records and 1,165 of 1,219 PatentsView records from Brazil, Portugal, Spain, Mexico, Argentina, Colombia, Chile and Italy, which is that rule's case. Weighed against: #704's option (b), one given unit by default, which changes `middle` for every English name of three words. 8. **A language file's `[rules]` compiles into its shipped `Locale`'s policy patch**, each rule kind a `Policy` field (an addition), so RU and TR_AZ are rebuilt from their files and `Locale` gains no field for rules. With it, **phase 3's open API dependency is answered**: its design (the 2026-10-27 #753 entry, item 1) carries a choice of languages on `Locale.languages`, assuming `Locale` and `Policy` keep their 2.x shape after the facade's removal, and the 2026-10-28 freeze keeps them so. The rest of that design, strict as an argument of the whole choice included, stays phase 3's to decide; its RFC section records the dependency as answered. 9. **The work order**, one PR each, measured on its own base, the snapshot recording each new view where it is non-empty: 4a `Token.reading` and `ReadingKind.PATRONYMIC` (neutral; `languages` after 3a); 4b `generation` and `credentials` (neutral); 4c `patronymic` (neutral); 4d `surname2` (neutral); 4e the render defaults and `order`; 4f `[rules]` into the policy patch, with or after 3b; 4g the compound given name, after 4f; 4h the shape inventory per order (#469). Interleaved with phase 3 (its sub-phases the 2026-10-27 #753 entry's item 11): 4a beside 3a; 4b to 4d before the packs that need them, `cs` (3f) and `pt-BR` (3h) on 4b, `es` and `pt` (3g) on 4d and 4g; an `is` pack on 3b, 4c and 4f; 3i, the bearers' languages, on 3a alone. diff --git a/docs/design/vocabulary-model-rfc.md b/docs/design/vocabulary-model-rfc.md index b99bb2da4..1ae9ed7fe 100644 --- a/docs/design/vocabulary-model-rfc.md +++ b/docs/design/vocabulary-model-rfc.md @@ -339,7 +339,7 @@ ORCID's and PatentsView's given box holds the middle names too, so a two-word gi - `ParsedName` gains `order`, the order the name was read under (its script's or the declared `name_order`, a family comma aside), an addition with `ADDED_IN` metadata whose missing value, in a 2.x pickle, renders as today, and left out of comparison as `Token.reading` is, so a loaded 2.x pickle still equals a fresh parse of its text. The cost, accepted 2026-10-28: two equal names can render differently, where today every rendering is derived from what `==` compares (a 2.x pickle of `김민준` renders `'민준 김'` and a fresh parse `'김민준'`, and a set keeps one of them); the views read from `Token.reading` share it. - `str()` and `render()`'s default spec put the family first where `order` does, and two words that stand one directly after the other, in the same order, in the original and in the rendering keep the kind of separator that stood between them in the original, written in a normalized form (decided 2026-10-28): nothing stays nothing; whitespace of any width or length becomes one ASCII space, the ideographic space U+3000 included (decided 2026-10-28: `山田　太郎` renders `'山田 太郎'`, the same ASCII space `str()` joins with today, where today's order gives `'太郎 山田'`, and a caller wanting native spacing passes a format spec of their own or reads `ParsedName.original`); an interpunct or a nakaguro the parse read as a separator stands alone, with no space around it (an interpunct written with spaces round it is not one, rules/T3 dividing at it only when flanked, so `威廉 · 莎士比亚` reads the dot as a name word and keeps rendering with its spaces, where `威廉·莎士比亚` keeps the dot unspaced; a spaced nakaguro is a separator, so `ジョン ・ スミス` renders `'ジョン・スミス'`); and one comma the parse reads as one becomes `, ` if ASCII and `، ` if Arabic (U+060C), no space before it and one after, and stands alone if full-width (U+FF0C), which carries its own width in CJK text, a reason that holds for the comma, a mark the rendering keeps, and not for whitespace, which the rendering normalizes, so `John Smith ,Jr.` and `John Smith, Jr.` render `'John Smith, Jr.'` and `John Smith， Jr.` renders `'John Smith，Jr.'`. The Japanese comma 、 (U+3001) is not one the parse reads as a comma, a parser question outside this design. Over that, the spec's decoration between the two fields the words belong to stands in place of the separator (the nickname's quotes, the `) ` after the maiden name, so `Jane Smith née Jones, PhD` keeps `'Jane Smith (Jones) PhD'`; 38 committed entries put a suffix directly after an undelimited maiden name); and any other pair, or any other separator (a delimiter, a marker word the parse consumed, two or more commas, a hyphen, a slash, a zero-width or directional mark), takes the spec's own joiner, as today (the written whitespace is not lost: `ParsedName.original` and the token spans keep it), so `John (Jack) Smith` keeps today's quoted nickname and `str(parse("김민준"))` is `'김민준'` and `str(parse("威廉·莎士比亚"))` keeps the dot. A name with a family comma or written given-first keeps today's word order, and its separators are kept by the same rule (a comma before a suffix alone does not make a family-first name given-first: `Wang Xiaoming, Jr.` under FAMILY_FIRST renders `'Wang Xiaoming, Jr.'`), so the given-first transcription `威廉·莎士比亚` keeps its dot too (20 of the 36 names whose interpunct `str()` drops are read given-first, 4 of them with a comma the parse reads (1 ASCII, 3 full-width)). A family comma is never kept, since the rendering reverses the pair it stands between (`Smith, John` → `'John Smith'`), while a comma directly between the family and its suffix is (`John Smith, Jr.` → `'John Smith, Jr.'`, where `str()` gives `'John Smith Jr.'` today): of the 1,003 committed corpus entries whose suffix starts after the family's last word ends, 214 have nothing between the two but one comma and whitespace (2026-10-28, `c8357e74`, each entry under its declared order; these counts read the ASCII comma, and 7 more write a full-width one there, while 36 of the 728 family-first names above hold a family comma the parse reads, 5 full-width and 31 Arabic (30 of the Arabic in `corpus_cldr.jsonl`), and so keep today's word order under the rule); 456 hold a comma somewhere between, most with a given name or title standing there too. Pairs the rendering separates, as `John` and `Jr.` in `Smith, John, Jr.`, take the spec's own joiner, as today, so `Smith, John, Jr.` keeps rendering `'John Smith Jr.'` while `John Smith, Jr.` becomes `'John Smith, Jr.'`: two writings of one name rendering differently, a consequence of following the text accepted with the choice. -- **How, decided 2026-10-28.** Today `str(name)` is `name.render()`, whose default is one fixed given-first spec, `'{title} {given} "{nickname}" {middle} {family} ({maiden}) {suffix}'`, which is why `김민준` renders `'민준 김'`; a default that reads `order` and the written separators cannot stay one string. So `render(spec: str | None = None)`: `None`, what `str()` passes, is the order-aware default, and a spec a caller writes renders exactly as written, its order and its literal joiners, with no separator rule applied, so `render("{given} {family}")` means in 3.0 what it meant in 2.x. That is a signature change, not an addition as the freeze defines one (decisions/3-0-reevaluations, THE CORE API), and Derek took it on 2026-10-28: no call that works today changes meaning, since `render(None)` raises `TypeError` today, but introspecting the default gives `None` rather than a spec, and a subclass overriding `render(self, spec: str = ...)` becomes an incompatible override to a type checker. The default's layout is picked by `order`, and a name with a family comma has a given-first `order` ("a family comma aside", above), so a given-first name and a family-comma name render by today's spec with the separator rule (`Smith, John` under FAMILY_FIRST still renders `'John Smith'`). For a family-first `order` the layout puts the title first and the suffix last, as the given-first spec does, and the three name roles between them in `order`'s own sequence, each keeping its decoration beside it: `'{title} {family} ({maiden}) {given} "{nickname}" {middle} {suffix}'` for FAMILY_FIRST and `'{title} {family} ({maiden}) {middle} {given} "{nickname}" {suffix}'` for FAMILY_FIRST_GIVEN_LAST, the separator rule then joining adjacent words (`王小明先生`, read family `王小明` and suffix `先生`, renders `'王小明先生'` where today gives `'王小明 先生'`; `Dr. Wang Xiaoming Jr.` under FAMILY_FIRST renders `'Dr. Wang Xiaoming Jr.'` where today gives `'Dr. Xiaoming Wang Jr.'`). CLDR's surnameFirst samples (`tools/differential/cldr_fields.jsonl`, release 48) agree on the core order and not on the rest: `ja` (`佐藤グレース恵子さん`) and `vi` (`ĐBQH Nguyễn Văn Anh`) put the second given name before the first, as FAMILY_FIRST_GIVEN_LAST does, and `ko` (`김민규헨리`) puts it after, as FAMILY_FIRST does; but most put the title after the family, not first: `pt` after it (`da Silva Sr. José Pedro Eduardo Jr., MP`), `ja` and `zh` last, and over every file's nativeFull surnameFirst rendering holding a title and a surname, 50 of 70 put the title after the surname (2026-10-28, positions of the field values in the text); `zh` also puts the generation first (`小彭德威东升议员先生`), and `vi` the credentials first. Keeping the title first, where the given-first default and the written name have it, goes against most of CLDR, but it is the one place the parse reads every title back from under a family-first order: `Senator Wang Xiaoming` reads title `Senator`, while `Wang Dr Xiaoming` reads given `Dr` and `Wang Xiaoming Senator` middle `Senator`, a bare title ending a name being a name word (rules/H5), only a period-marked one (`Wang Xiaoming Dr.`) reading as the title there (2026-10-28, under `Policy(name_order=FAMILY_FIRST)`). Derek decided on 2026-10-28 that the default renders only an order the parse reads back, so the title stays first; reading a title where CLDR writes it is the parser's question, a separate issue, and the default may follow once the parse reads it there. A locale's own placement is the render patterns' job (below). The suffix last agrees with `ja` and `zh`, whose honorifics the parse reads as suffixes. For 3.0's notes: a caller who copied 2.x's default string and passes it keeps given-first, space-joined output; only a caller who passes nothing, `str()` included, gets the new default. `None` leaves both answers to how a pack's pattern is reached as open as they were: a spec string still means a spec, so `render("sorting")` still returns `'sorting'`, and a pattern is asked for by a new method or argument under (a) or through the parser under (b). +- **How, decided 2026-10-28.** Today `str(name)` is `name.render()`, whose default is one fixed given-first spec, `'{title} {given} "{nickname}" {middle} {family} ({maiden}) {suffix}'`, which is why `김민준` renders `'민준 김'`; a default that reads `order` and the written separators cannot stay one string. So `render(spec: str | None = None)`: `None`, what `str()` passes, is the order-aware default, and a spec a caller writes renders exactly as written, its order and its literal joiners, with no separator rule applied, so `render("{given} {family}")` means in 3.0 what it meant in 2.x. That is a signature change, not an addition as the freeze defines one (decisions/3-0-reevaluations, THE CORE API), and Derek took it on 2026-10-28: no call that works today changes meaning, since `render(None)` raises `TypeError` today, but introspecting the default gives `None` rather than a spec, and a subclass overriding `render(self, spec: str = ...)` becomes an incompatible override to a type checker. The default's layout is picked by `order`, and a name with a family comma has a given-first `order` ("a family comma aside", above), so a given-first name and a family-comma name render by today's spec with the separator rule (`Smith, John` under FAMILY_FIRST still renders `'John Smith'`). For a family-first `order` the layout puts the title first and the suffix last, as the given-first spec does, and the three name roles between them in `order`'s own sequence, each keeping its decoration beside it: `'{title} {family} ({maiden}) {given} "{nickname}" {middle} {suffix}'` for FAMILY_FIRST and `'{title} {family} ({maiden}) {middle} {given} "{nickname}" {suffix}'` for FAMILY_FIRST_GIVEN_LAST, the separator rule then joining adjacent words (`王小明先生`, read family `王小明` and suffix `先生`, renders `'王小明先生'` where today gives `'王小明 先生'`; `Dr. Wang Xiaoming Jr.` under FAMILY_FIRST renders `'Dr. Wang Xiaoming Jr.'` where today gives `'Dr. Xiaoming Wang Jr.'`). CLDR's surnameFirst samples (`tools/differential/cldr_fields.jsonl`, release 48) agree on the core order and not on the rest: `ja` (`佐藤グレース恵子さん`) and `vi` (`ĐBQH Nguyễn Văn Anh`) put the second given name before the first, as FAMILY_FIRST_GIVEN_LAST does, and `ko` (`김민규헨리`) puts it after, as FAMILY_FIRST does; but most put the title after the family, not first: `pt` after it (`da Silva Sr. José Pedro Eduardo Jr., MP`), `ja` and `zh` last, and over every file's nativeFull surnameFirst rendering holding a title and a surname, 50 of 70 put the title after the surname (2026-10-28, positions of the field values in the text); `zh` also puts the generation first (`小彭德威东升议员先生`), and `vi` the credentials first. Keeping the title first, where the given-first default and the written name have it, goes against most of CLDR, but it is the one place the parse reads every title back from under a family-first order: `Senator Wang Xiaoming` reads title `Senator`, while `Wang Dr Xiaoming` reads given `Dr` and `Wang Xiaoming Senator` middle `Senator`, a bare title ending a name being a name word (rules/H5), only a period-marked one (`Wang Xiaoming Dr.`) reading as the title there (2026-10-28, under `Policy(name_order=FAMILY_FIRST)`). Derek decided on 2026-10-28 that the default renders only an order the parse reads back, so the title stays first; reading a title where CLDR writes it is the parser's question, #938, and the default may follow once the parse reads it there. A locale's own placement is the render patterns' job (below). The suffix last agrees with `ja` and `zh`, whose honorifics the parse reads as suffixes. For 3.0's notes: a caller who copied 2.x's default string and passes it keeps given-first, space-joined output; only a caller who passes nothing, `str()` included, gets the new default. `None` leaves both answers to how a pack's pattern is reached as open as they were: a spec string still means a spec, so `render("sorting")` still returns `'sorting'`, and a pattern is asked for by a new method or argument under (a) or through the parser under (b). - `initials()` gives nothing for a word in a script with no initials (Han, Hangul, kana, `_policy._NO_INITIALS`), where today it gives the first character with a period (`'王.'`). Apart from that signature, these change no signature but change `str()`'s output for every name read family-first, every name whose words are written with nothing, an interpunct or a nakaguro between two words the rendering keeps side by side (not `김, 민준씨`, whose family comma moves `김` between `민준` and `씨`), and every name written with one comma directly between its family and its suffix, which no 2.x release can warn about; Derek took them as an improvement for 3.0's release notes rather than a break (item 6 of the entry). From ece0673b335f6631eb3c07b0d6faf9a968067426 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Wed, 28 Oct 2026 18:37:04 -0700 Subject: [PATCH 32/39] Record six of phase 3's twelve items as decided (#753) Items 1, 2, 4, 5, 6 and 8 are accepted as written: the choice of languages on Locale.languages, one mechanism, strict waiting for 3i, the reading index loaded for every caller, tag selection by file and ancestors, and third-party files deferred. Items 3, 7 and 9-12 stay proposed. The add-only pack contract is amended in 3b's PR, as the items tie it there. A pack's lexicon fragment and Locale(lexicon=...)'s successor now point at the Lexicon-surface entry. Co-Authored-By: Claude Opus 5.5 --- docs/design/decisions/vocabulary-model.md | 15 ++++++++------- docs/design/vocabulary-model-rfc.md | 16 ++++++++-------- 2 files changed, 16 insertions(+), 15 deletions(-) diff --git a/docs/design/decisions/vocabulary-model.md b/docs/design/decisions/vocabulary-model.md index a87454427..c98af472e 100644 --- a/docs/design/decisions/vocabulary-model.md +++ b/docs/design/decisions/vocabulary-model.md @@ -10,6 +10,7 @@ The vocabulary was a set of base sets plus marker subsets (`particles` and `part - The default loads every language and keeps working for names of any language. Choosing a language is to narrow it in phase 3 (#753): a pack is to settle only the clashes where its language has a vocabulary reading, a strict pack to drop other languages' name-level readings too, and a pack's preferred reading to be able to outrank one a shape rule derives — 2026-10-15/16. - `e`, `y` and `i` stay in the default; choosing a language is to settle their one-case fork, which no shipped pack does yet — 2026-10-15/16. - A patronymic field is to come in 3.0, as a view that names its words without emptying `family`, and the particle kinds already draw the `clan`/`patronymic` line for it: a patronymic is followed by a parent's name, a clan prefix by a family name the lineage shares — 2026-10-15/16; 2026-10-19 #749; 2026-10-28 #754 (Derek). +- A language pack is a choice of languages on `Locale.languages` selecting which readings compile, a tag choosing its own file and its ancestors', and a reading index loaded for every caller carries the languages to run time; strict waits for 3i, and third-party language files are deferred — 2026-10-27 #753 (Derek, items 1, 2, 4, 5, 6 and 8). - A caller configures vocabulary by reading, `Lexicon` deriving the sets and gates the build derives, and the compiled sets become internal in 3.0 — 2026-10-28 #754, the `Lexicon` surface (Derek). - A `Locale` may carry CLDR-shaped render patterns, display and sorting, a later addition whose shape is agreed; how a call reaches one is open — 2026-10-28 #754, render patterns. - 3.0's output model keeps the seven roles and adds every new field as a view, read from `Token.reading`, the reading each token took, or for `surname2` from rules/O6's tag: `generation` and `credentials` partitioning `suffix`, `patronymic`, and `surname2` where rules/O6 read two family names; `middle` and `surnames` keep their meaning, a compound given name being a rule a caller or pack turns on; the default `str()` renders a name in the order it was read in with its written separators in a normalized form, through `render(spec: str | None = None)`, `None` being that default; a language file's `[rules]` compile into its `Locale`'s policy patch — 2026-10-28 #754 (Derek). @@ -245,15 +246,15 @@ Open: #913 (the title words phase 2j left, read as titles though borne as names) - **2026-10-26 #752, phase 2l — THE BRACKET ESCAPE READS THE DERIVED CLASH**, carrying out item 16 of the 2026-10-23 #752 entry above (#810). A member of the ambiguous class, the class 2a derives from name readings, alone in brackets or quotes after the name is the suffix where its writing settles it (`John Smith (MA)`, `John Smith (M.A)` → suffix), read in one pass, and stays the nickname otherwise (`John Smith (Ed)`, `JOHN SMITH (MA)`, `Jeffrey (JD) Bricken`). It is recorded as rules/S13, a rule of its own rather than the amendment to S1 item 16 planned, the position it reads being evidence S1's signal does not read; the writing is S3's and S6's evidence, which S13 cites. The record, the measurements and what was declined are decisions/S13's 2026-10-26 #810 entry. - **2026-10-26 #752, phase 2m — THE READING NOT TAKEN SURFACES AS `Ambiguity.alternative`, WITHOUT ITS LANGUAGES**, carrying out item 13 of the 2026-10-23 #752 entry above and item 7's source with it: a word the credential's run took by position alone reports its name reading's source as `position`, a fourth source beside the three the item named, and a word with a declared name reading reports `declared`. The pickle guard was widened as item 13 asked, every skew the narrow guard refused still refused. The reading's languages were left out: the per-word index costs about 0.35 ms of an 18.2 ms import (decisions/A1) and describes the default vocabulary, not a pack's or a caller's lexicon, so they wait for 3.0 with `Token.reading`. The record, the measurements and what was declined are decisions/A1's 2026-10-26 #752 entry. -- **2026-10-27 #753 — PHASE 3'S DESIGN IS PROPOSED, NOT AGREED: A CHOICE OF LANGUAGES SELECTS THE POOL, ONE MECHANISM FOR TURNING READINGS ON AND PREFER; NAME READINGS IN A PACK KEEP THE GATE ON; A READING INDEX CARRIES THE LANGUAGES TO RUN TIME; STRICT WAITS.** Every item below is a proposal awaiting Derek's decision, and this head is rewritten when he makes it. The design overview is docs/design/vocabulary-model-rfc.md's Phase 3 section. Decisions 15 and 20 are kept as written; decision 6's strict variant is reshaped to a property of the whole choice and deferred (items 1 and 4). Phase 3 is 3.0 work since the 2026-10-28 re-plan (item 12). The inputs are #753's comments of 2026-10-27: the inventory of 411 kept-out candidate readings with their verdicts, and the probe of foreign titles and function-word collisions. Measurements were taken 2026-10-27 on master `cfe8bf92` (Python 3.11.17) with throwaway Lexicon edits standing in for a pack, so each is evidence for the proposal and each sub-phase re-measures on its own base. Local-corpus figures are counts. - 1. **Proposed: a pack is a choice of languages carried by `Locale`.** `Locale` gains `languages`, a set of BCP 47 tags; `parser_for` unions the sets of the packs it is given, so the choice is order-independent while policy patches still fold in order; strict is an argument of the whole choice (`parser_for(locales.NE, locales.HI, strict=True)`), since a strict pack beside a preferring one has no meaning, reshaping decision 6's "strict variant". The shipped language packs are `Locale` constants with an empty lexicon fragment (`locales.NE`, `locales.PT_BR`). The dependency this rested on, `Locale` and `Policy` keeping their 2.x shape once the facade is gone, is answered by the 2026-10-28 3.0 freeze: both keep it, with additions. Weighed against: `Parser(languages=…)` beside `Locale`, two ways to choose one thing, and a `Policy` field, `Policy` being behavior and vocabulary being `Lexicon`'s. - 2. **Proposed: one mechanism.** The chosen languages select which readings the compiler compiles, the default being that compiler with every language and no choice. Turning on, prefer and strict are derived from the file each reading sits in, and nothing is written per pack beyond its language's readings: a chosen language's `default = false` readings join; prefer drops other languages' vocabulary readings of a word on which a chosen language holds one, never `mul.toml`'s (`Haddad, Abd`, suffix `Abd` today, reads given `Abd` with the en credential gone); strict drops other languages' name readings and the markers they derive. A clash a property records with a SHAPE reading is settled for the chosen language's own reading, decision 20 applied: `may_be_initial` does not hold on it, so a `pt` choice reads `ANA SILVA E SOUZA` family `SILVA E SOUZA` where the default reads middle `SILVA E` with a report. This answers decisions/3-0-reevaluations' 2026-09-13 bullet that a pack cannot remove `e`, and its 2026-08-17 bullet's "overrides rather than additions": a pack overrides by selection, never by a removal list. That removes base entries all the same, so mechanisms.md#LOCALE-PACKS-PURE-DATA's add-only contract is amended with it (item 12). The shape-clash rule follows from decision 7, which decision 20 permits; it reaches every reading `may_be_initial` stands on (pt and it `e`, ca and pl `i`, the Italian one unjudged), and it forecloses an `nl` pack marking `y` (the 2026-08-18 bullet there), which waits with `fru` below. New beside decision 6: `mul.toml`'s exemption from prefer. Left open rather than built: a property a language's own reading should take only under its pack, Swedish `fru` and Icelandic `frú` as after-given salutations (rules/H8), refused in the default for the Cameroonian surname, the two readings in the inventory needing it. +- **2026-10-27 #753 (Derek, items 1, 2, 4, 5, 6 and 8) — PHASE 3'S DESIGN IS PARTLY AGREED: A CHOICE OF LANGUAGES SELECTS THE POOL, ONE MECHANISM FOR TURNING READINGS ON AND PREFER, A READING INDEX CARRIES THE LANGUAGES TO RUN TIME, AND STRICT WAITS ARE DECIDED; NAME READINGS IN A PACK KEEPING THE GATE ON, THE MEASURES, THE EXCLUSIONS AND THE WORK ORDER ARE STILL PROPOSED.** Derek accepted items 1, 2, 4, 5, 6 and 8 as written in chat on 2026-10-28; items 3, 7 and 9 to 12 are proposals awaiting his decision, and this head is rewritten when he makes them. The design overview is docs/design/vocabulary-model-rfc.md's Phase 3 section. Decisions 15 and 20 are kept as written; decision 6's strict variant is reshaped to a property of the whole choice and deferred (items 1 and 4). Phase 3 is 3.0 work since the 2026-10-28 re-plan (item 12). The inputs are #753's comments of 2026-10-27: the inventory of 411 kept-out candidate readings with their verdicts, and the probe of foreign titles and function-word collisions. Measurements were taken 2026-10-27 on master `cfe8bf92` (Python 3.11.17) with throwaway Lexicon edits standing in for a pack, so each is evidence for the proposal and each sub-phase re-measures on its own base. Local-corpus figures are counts. + 1. **A pack is a choice of languages carried by `Locale`.** `Locale` gains `languages`, a set of BCP 47 tags; `parser_for` unions the sets of the packs it is given, so the choice is order-independent while policy patches still fold in order; strict is an argument of the whole choice (`parser_for(locales.NE, locales.HI, strict=True)`), since a strict pack beside a preferring one has no meaning, reshaping decision 6's "strict variant". The shipped language packs are `Locale` constants with an empty lexicon fragment (`locales.NE`, `locales.PT_BR`); what a fragment holds once 3.0 configures vocabulary by reading is the 2026-10-28 #754 entry on the `Lexicon` surface. The dependency this rested on, `Locale` and `Policy` keeping their 2.x shape once the facade is gone, is answered by the 2026-10-28 3.0 freeze: both keep it, with additions. Weighed against: `Parser(languages=…)` beside `Locale`, two ways to choose one thing, and a `Policy` field, `Policy` being behavior and vocabulary being `Lexicon`'s. + 2. **One mechanism.** The chosen languages select which readings the compiler compiles, the default being that compiler with every language and no choice. Turning on, prefer and strict are derived from the file each reading sits in, and nothing is written per pack beyond its language's readings: a chosen language's `default = false` readings join; prefer drops other languages' vocabulary readings of a word on which a chosen language holds one, never `mul.toml`'s (`Haddad, Abd`, suffix `Abd` today, reads given `Abd` with the en credential gone); strict drops other languages' name readings and the markers they derive. A clash a property records with a SHAPE reading is settled for the chosen language's own reading, decision 20 applied: `may_be_initial` does not hold on it, so a `pt` choice reads `ANA SILVA E SOUZA` family `SILVA E SOUZA` where the default reads middle `SILVA E` with a report. This answers decisions/3-0-reevaluations' 2026-09-13 bullet that a pack cannot remove `e`, and its 2026-08-17 bullet's "overrides rather than additions": a pack overrides by selection, never by a removal list. That removes base entries all the same, so mechanisms.md#LOCALE-PACKS-PURE-DATA's add-only contract is amended, in 3b's PR, which builds the selection (items 11 and 12), not by this decision. The shape-clash rule follows from decision 7, which decision 20 permits; it reaches every reading `may_be_initial` stands on (pt and it `e`, ca and pl `i`, the Italian one unjudged), and it forecloses an `nl` pack marking `y` (the 2026-08-18 bullet there), which waits with `fru` below. New beside decision 6: `mul.toml`'s exemption from prefer. Left open rather than built: a property a language's own reading should take only under its pack, Swedish `fru` and Icelandic `frú` as after-given salutations (rules/H8), refused in the default for the Cameroonian surname, the two readings in the inventory needing it. 3. **Proposed: `default = false` is allowed on `given` and `family` readings, and turned on they derive markers like any name reading, never settling (decision 15 unchanged).** Measured on nepal-candidates' 126,981 Devanagari and romanized pairs, comparing every word's role across the two spellings: 123,440 read alike in the default; `given` readings on `श्री`, `guru` and `गुरु`, which make each a title only inside a title phrase (rules/H9), give 123,636 (198 fixed, 2 broken, both pairs whose romanization opens with a listed title); a `family` reading on `dc` as well, making the credential ambiguous, gives 123,639, the count still leaving `Ram Bahadur DC` a suffix; dropping `dc`'s credential, which settling by a name reading would do, gives 123,658, 19 more. Settling is not proposed: the 19 pairs are the price of decision 15, whose `do` case it protects. `bk`, `kc`, `gc` and `bc` read as the family in every shape tried (spoken order, a family comma, `FAMILY_FIRST`, dotted and Title case), so a reading would move nothing and none is proposed. Recompute: the readings moved into or out of `Lexicon` fields by hand, the pairs compared by `tuple(t.role for t in parser.parse(s).tokens)`. - 4. **Proposed: strict waits, out of the first packs, until name readings say whose names they are.** A name reading sits in the file whose source counted it: 113 of the 211 name readings are in en.toml, `do`'s Vietnamese Đỗ readings among them, vi.toml holding no `do`, so strict built on the files drops them under a `vi` choice, and simulated, `Minh Do` reads suffix `Do` and `Do Van Minh` family `Do Van Minh`, decision 15's harm. A census count belongs to no one language either. The candidates needing strict wait with it, `nl`'s prepositions among them, which need more than strict: turned on as possibly-given particles, `in` and `to` read `Park In Hwan` family `In Hwan` and `Nguyen To Nga` family `To Nga` (the marker acts at the head only), and check 11 accepts `may_be_given = false` in place of the Korean and Vietnamese readings. Recompute: count `given` and `family` readings per file with tools/vocab/build.py's `load`; simulate by removing `do` from `suffix_acronyms_ambiguous` and `particles_ambiguous`. Open: none of 3a–3h says whose names they are. Moving readings between files would not, a word being borne in several languages (`do` Vietnamese and Korean, `kim` Korean and Scandinavian), so the RFC points at an attribute on each name reading naming its bearers' languages, compiling to nothing until strict reads it, the first provenance anything would compute a reading from. It cannot be filled by script: 172 of the 211 count a US source and 21 none, per-country evidence is note prose in 40, and a box count says where a word was typed, not whose name it is (`do`'s ORCID family box is led by Brazil, 7,227 against Vietnam's 876, none of Brazil's `do` alone against Vietnam's 618; by share Vietnam leads, 1.90% to 0.81%; 2026-10-28, master `c8357e74`). The attribute names languages, so no country-to-language table is read at run time (item 6). Proposed: a sub-phase of its own, 3i, after 3a, independent of the output model, moving no default parse; strict after it. - 5. **Proposed: the languages reach run time as a reading index.** A second generated module, one row per reading (text, tag, kind, behavioral properties), loaded for every caller with `Lexicon.default()`, since phase 4's `Reading.languages` (4a) is filled from it on every parse, rather than on first access to `reading.languages`, which would need a `Reading`, which holds no word, to carry one as a hidden key; the compiler moves from tools/vocab/build.py into the package so build and pack share one implementation, and a test holds that compiling the index with no choice reproduces `_compiled.py`. It is the per-word index item 13 of the 2026-10-23 #752 entry says `Token.reading` needs. A choice is applied as a delta against the default's compile, so it composes with a caller's `base=` lexicon and a `Locale`'s own fragment. Measured: 2,140 readings over 1,797 words, 106 in more than one file; the index about 93 KB of source, about 0.45 ms to import with its bytecode cached against about 20 ms for a cached `import nameparser` (2026-10-28, `c8357e74`; about 11 ms without the cache, the 2026-10-27 figure, taken under `PYTHONDONTWRITEBYTECODE`), once per process for every caller; the compile 9 ms, once per choice and only for a caller who makes one; loading the TOML 39 ms. Weighed against: per-language deltas compiled at build time, which do not compose (under prefer `{pt, en}` `do` keeps its particle and its credential, while Portuguese's delta drops the credential and English's the particle); every combination precompiled, combinatorial; the TOML read at run time, slower and either re-running the build's checks or skipping them. - 6. **Proposed: a chosen tag selects its file and its ancestors', never its descendants, and no macrolanguage table is read.** `pt-BR` selects `pt-BR.toml` and `pt.toml`; `pt` only `pt.toml`, so a caller in Portugal does not turn on the Brazilian agnomens (#843). The two files share no word today. Malay and Indonesian, which share `puan`, `putra` and `putri`, are chosen together by a caller who has both. The TOML names `prefer` and `strict` stay refused, the message saying a strength is the caller's choice and never data. + 4. **Strict waits, out of the first packs, until name readings say whose names they are.** A name reading sits in the file whose source counted it: 113 of the 211 name readings are in en.toml, `do`'s Vietnamese Đỗ readings among them, vi.toml holding no `do`, so strict built on the files drops them under a `vi` choice, and simulated, `Minh Do` reads suffix `Do` and `Do Van Minh` family `Do Van Minh`, decision 15's harm. A census count belongs to no one language either. The candidates needing strict wait with it, `nl`'s prepositions among them, which need more than strict: turned on as possibly-given particles, `in` and `to` read `Park In Hwan` family `In Hwan` and `Nguyen To Nga` family `To Nga` (the marker acts at the head only), and check 11 accepts `may_be_given = false` in place of the Korean and Vietnamese readings. Recompute: count `given` and `family` readings per file with tools/vocab/build.py's `load`; simulate by removing `do` from `suffix_acronyms_ambiguous` and `particles_ambiguous`. Open: none of 3a–3h says whose names they are. Moving readings between files would not, a word being borne in several languages (`do` Vietnamese and Korean, `kim` Korean and Scandinavian), so the RFC points at an attribute on each name reading naming its bearers' languages, compiling to nothing until strict reads it, the first provenance anything would compute a reading from. It cannot be filled by script: 172 of the 211 count a US source and 21 none, per-country evidence is note prose in 40, and a box count says where a word was typed, not whose name it is (`do`'s ORCID family box is led by Brazil, 7,227 against Vietnam's 876, none of Brazil's `do` alone against Vietnam's 618; by share Vietnam leads, 1.90% to 0.81%; 2026-10-28, master `c8357e74`). The attribute names languages, so no country-to-language table is read at run time (item 6). A sub-phase of its own, 3i, after 3a, independent of the output model, moving no default parse; strict after it. + 5. **The languages reach run time as a reading index.** A second generated module, one row per reading (text, tag, kind, behavioral properties), loaded for every caller with `Lexicon.default()`, since phase 4's `Reading.languages` (4a) is filled from it on every parse, rather than on first access to `reading.languages`, which would need a `Reading`, which holds no word, to carry one as a hidden key; the compiler moves from tools/vocab/build.py into the package so build and pack share one implementation, and a test holds that compiling the index with no choice reproduces `_compiled.py`. It is the per-word index item 13 of the 2026-10-23 #752 entry says `Token.reading` needs. A choice is applied as a delta against the default's compile, so it composes with a caller's `base=` lexicon and a `Locale`'s own fragment. Measured: 2,140 readings over 1,797 words, 106 in more than one file; the index about 93 KB of source, about 0.45 ms to import with its bytecode cached against about 20 ms for a cached `import nameparser` (2026-10-28, `c8357e74`; about 11 ms without the cache, the 2026-10-27 figure, taken under `PYTHONDONTWRITEBYTECODE`), once per process for every caller; the compile 9 ms, once per choice and only for a caller who makes one; loading the TOML 39 ms. Weighed against: per-language deltas compiled at build time, which do not compose (under prefer `{pt, en}` `do` keeps its particle and its credential, while Portuguese's delta drops the credential and English's the particle); every combination precompiled, combinatorial; the TOML read at run time, slower and either re-running the build's checks or skipping them. + 6. **A chosen tag selects its file and its ancestors', never its descendants, and no macrolanguage table is read.** `pt-BR` selects `pt-BR.toml` and `pt.toml`; `pt` only `pt.toml`, so a caller in Portugal does not turn on the Brazilian agnomens (#843). The two files share no word today. Malay and Indonesian, which share `puan`, `putra` and `putri`, are chosen together by a caller who has both. The TOML names `prefer` and `strict` stay refused, the message saying a strength is the caller's choice and never data. 7. **Proposed: a pack is measured three ways.** The default does not move (the committed snapshot, and the index invariant of item 5); a vocabulary pack's deviation surface, the words whose compiled readings its choice changes, is derived from the index, and tests/v2/test_locales.py's non-interference gate requires the default's reading for every corpus name without such a word, the policy and segmentation packs (JA, RU, TR_AZ, ZH) keeping their written `DEVIATES`; and `snapshot.py`, `--local` with `--base` included, takes a choice and prints what it moves by field, beside each pack's own ground truth (nepal-candidates' script pairs for `ne`, the CLDR agreement per pack, PSC's nationality slice). This answers decisions/3-0-reevaluations' 2026-09-13 bullet that the harness cannot see a pack. - 8. **Proposed: third-party language files are deferred past phase 3.** A file joining the pool at run time would skip the checks that run across files (check 6's exclusions, check 11's particle statement), the ones that refuse a third-party `nl` `in` written with neither a `given` reading nor `may_be_given = false`. `Locale(lexicon=…)`, or its successor, stays the way to add vocabulary. + 8. **Third-party language files are deferred past phase 3.** A file joining the pool at run time would skip the checks that run across files (check 6's exclusions, check 11's particle statement), the ones that refuse a third-party `nl` `in` written with neither a `given` reading nor `may_be_given = false`. `Locale(lexicon=…)`, or its successor, stays the way to add vocabulary, that successor being configuration by reading (the 2026-10-28 #754 entry on the `Lexicon` surface). 9. **Proposed: every exclusion a pack turns on becomes a `default = false` reading,** the shape cs `st` already has, so that once the packs land an `excluded` key means wrong everywhere: the eight whose reason opens "pack-shaped" in 3c (cs `ml`, hr, sk and sl `st` and `ml`, fr `père`), and the rest in the PR of the pack that turns each on (`pan`, `roz`, `nic`, `shri`, `sig`, `jaun`, `bt`, `bay`, `sb`). Recompute: `grep -h 'pack-shaped' nameparser/vocab/*.toml | wc -l`. 10. **Proposed: what stays out of phase 3.** ZH's surname list and the RU and TR_AZ endings are rule data for phase 4's `[rules]` (its 4f), and a Python pack gains `languages` only where a measurement shows what it moves; the candidates that wait on a position rule (Turkish `Bey` and `Hanım`, Hungarian `úr`, Thai `นาย`, French `M.`) are phase 4's; same-language clashes (`kumari`, `sayyid`) take phase 2j's route in the default, since no pack settles one. 11. **Proposed: the work order,** shape first and one PR each, measured on its own base: 3a the index and shared compiler with name readings allowed `default = false` (neutral); 3b the choice, tag selection, turning on, prefer, the shape clash, the derived deviation surface, `snapshot.py`'s choice and the amended contract (neutral); 3c the pack-shaped exclusions converted (neutral); then the packs, `ne` (3d), `hi` and `bn`'s Latin forms (3e), `cs`, `sk`, `sl` and `hr` (3f), `es` and `pt` with a connective that counts only inside a title run (3g), and `ga`/`gd` and `pt-BR` (3h); 3i, the bearers' languages of each name reading (item 4), after 3a, ahead of strict. 3g's connective is a rule with its own design: with `profesora` added and no rule, `Profesora de Historia Ana López` reads family `de Historia Ana López` (the rules/P1 fold) and `Ministro da Fazenda Carlos Souza` given `da`, while `Doctora de la Vega` must keep family `de la Vega`, the two separated by whether a name stands behind the phrase; it must reach only readings a pack turns on. diff --git a/docs/design/vocabulary-model-rfc.md b/docs/design/vocabulary-model-rfc.md index 1ae9ed7fe..03ef2acba 100644 --- a/docs/design/vocabulary-model-rfc.md +++ b/docs/design/vocabulary-model-rfc.md @@ -26,7 +26,7 @@ Four phases, in a fixed order: |---|---|---|---| | 1 | The per-word TOML format, compiled to today's sets | no | shipped in 2.6 (#737, #744); #749–#751 gave the words in `und.toml` their languages, leaving two particles of unrecorded origin | | 2 | Ambiguity derived from clashing readings; the ambiguity gate extended to every reading kind | yes, the first phase that does | #752, design agreed 2026-10-23; 2a, the two name-clash markers derived, landed without moving a parse (decisions/vocabulary-model, 2026-10-25) | -| 3 | Language packs that prefer or restrict their language | yes, for callers who choose one | #753, for 3.0, needs 1 and 2; design proposed 2026-10-27, not agreed | +| 3 | Language packs that prefer or restrict their language | yes, for callers who choose one | #753, for 3.0, needs 1 and 2; design proposed 2026-10-27, its mechanism, index and API agreed 2026-10-28, the rest proposed | | 4 | Order and shape; the rules a pack can choose; the output model | yes | #754, 3.0 work; its output model agreed 2026-10-28, ahead of or alongside phase 3's mechanism | Two commitments hold through all four. **The default loads every language and keeps working for names of any language, as today**; choosing a language narrows it (decision 5 in decisions/vocabulary-model). And **languages own vocabulary and defaults, the core owns structure**: a pack configures rules the core implements, and never ships code (mechanisms.md#LOCALE-PACKS-PURE-DATA). @@ -164,7 +164,7 @@ Each sub-phase is one PR, measured on its own base: the committed snapshot read ## Phase 3: packs that prefer or restrict -Status: **proposed, not agreed** (2026-10-27, #753). The record is decisions/vocabulary-model's 2026-10-27 #753 entry, whose items are each marked proposed until Derek decides them; this section is the proposal those items describe. Phase 3 is 3.0 work (re-planned 2026-10-28): 2.8 is the last 2.x feature release, and 3.0 ships phases 3 and 4 with the facade's removal. Packs stay opt-in, so a caller who chooses no language parses as the default does, and nothing in this phase moves the default's parses. One standing text would be amended with it: mechanisms.md#LOCALE-PACKS-PURE-DATA's contract that a pack is add-only and declares its deviations in its docstring (a pack would add, and select among the default's readings by language, its deviations derived for a vocabulary pack). decisions/3-0-reevaluations' line that removing base vocabulary through a pack is part of 3.0's pack design is carried out rather than moved. +Status: **partly agreed** (proposed 2026-10-27, #753). Derek accepted on 2026-10-28 the choice of languages on `Locale.languages`, the one mechanism, strict waiting for 3i, the reading index loaded for every caller, tag selection by file and ancestors, and the deferral of third-party language files; name readings turned on in a pack, how a pack is measured, the exclusions converted, what stays out, the work order and the phase's place in 3.0 are still proposed. The record is decisions/vocabulary-model's 2026-10-27 #753 entry, whose items are each marked decided or proposed; this section describes them all. Phase 3 is 3.0 work (re-planned 2026-10-28): 2.8 is the last 2.x feature release, and 3.0 ships phases 3 and 4 with the facade's removal. Packs stay opt-in, so a caller who chooses no language parses as the default does, and nothing in this phase moves the default's parses. One standing text is amended with it, in 3b's PR, which builds the selection: mechanisms.md#LOCALE-PACKS-PURE-DATA's contract that a pack is add-only and declares its deviations in its docstring (a pack adds, and selects among the default's readings by language, its deviations derived for a vocabulary pack). decisions/3-0-reevaluations' line that removing base vocabulary through a pack is part of 3.0's pack design is carried out rather than moved. **Phase 4's shape questions came first.** The output model decides what a pack can express: a patronymic field, a second surname, credentials apart from generation suffixes (#326), and what `middle` means (#704). A pack reading of a kind with no field to land in is a reading nothing can surface, so those questions were to be settled before or with phase 3's mechanism, by the same shape-before-value rule, and the output model was agreed on 2026-10-28 (decisions/vocabulary-model's 2026-10-28 #754 entry): the seven roles stay, and `generation`, `credentials`, `patronymic` and `surname2` are views read from `Token.reading` or rules/O6's tag. Of the work order below, 3a (the reading index and the shared compiler) and 3c (the pack-shaped exclusions converted) are independent of the output model: the index carries whatever kinds the files hold, and 3c moves data between two encodings that compile alike. 3b's selection semantics are independent too, and its API's dependency, what `Locale` and `Policy` look like once the facade is gone, is answered (below). The packs depend on the output model in part, each on the phase 4 sub-phase that builds its view: `cs`'s generations and `pt-BR`'s agnomens on 4b's `generation` (3f and 3h), `es` and `pt` on 4d's `surname2` and 4g's compound given name (3g); `ne`'s readings do not. @@ -177,7 +177,7 @@ The asymmetry is deliberate (decision 15): **vocabulary readings settle a clash; Consequences that follow from the definitions: -- **Order does not matter.** A choice is a set of languages, so combining packs is commutative. Proposed below: strictness belongs to the whole choice rather than to one pack, reshaping decision 6's "strict variant", and strict waits until name readings say whose names they are. +- **Order does not matter.** A choice is a set of languages, so combining packs is commutative. Decided 2026-10-28 (below): strictness belongs to the whole choice rather than to one pack, reshaping decision 6's "strict variant", and strict waits until name readings say whose names they are. - **Two chosen languages that each hold a vocabulary reading both keep it**, and the word stays gated, as in the default. - **Non-Latin vocabulary stays on under any choice**, because it selects itself by script (`씨`, `さん`, Cyrillic titles). Almost every clash between languages is within one script: `do`, `e`, `van` are all Latin. - **Whether a word needs a pack depends on where its clash is.** `do` clashes between languages, so a pack settles it. `fils` clashes within French, a generation word and a French surname, so no pack settles it and the phase 2 gate does the work, reversing the premise of #684. @@ -213,13 +213,13 @@ Properties ride on their readings, so a choice that keeps a reading keeps its pr The 2 broken are pairs whose romanization opens with a listed title, which reads the `श्री` as a salutation (#870 counted `Mr` written for it twice). So the gate route delivers the `श्री` and `guru` fix without amending decision 15, and the 19 `dc` pairs it leaves are the price of keeping it. `bk`, `kc`, `gc` and `bc` need nothing: they read as the family in every shape tried (`Ram Bahadur BK`, `Ram Bahadur Kc`, `BK, Ram Bahadur`, `KC Sita` under `FAMILY_FIRST`), so a reading would move nothing. -**Strict cannot be built on today's files.** A name reading sits in the file whose source counted it, not the file of the language whose names carry it: 113 of the 211 name readings are in en.toml (2026-10-27), `do`'s Vietnamese Đỗ readings among them, and vi.toml holds no `do`. A strict `vi` choice built on the files would drop exactly those readings, and with them the gate: simulated, `Minh Do` reads suffix `Do` and `Do Van Minh` family `Do Van Minh`, the harm decision 15 is there to prevent. Nor does a census count belong to one language. So strict is proposed to wait, out of the first packs, until name readings say whose names they are, and the candidates that need it wait with it. The `nl` prepositions show they would need more than strict: turned on as possibly-given particles, `in` and `to` already read `Park In Hwan` family `In Hwan` and `Nguyen To Nga` family `To Nga`, the marker acting at the head only, and check 11 lets their author write `may_be_given = false` instead of declaring the Korean and Vietnamese readings. +**Strict cannot be built on today's files.** A name reading sits in the file whose source counted it, not the file of the language whose names carry it: 113 of the 211 name readings are in en.toml (2026-10-27), `do`'s Vietnamese Đỗ readings among them, and vi.toml holds no `do`. A strict `vi` choice built on the files would drop exactly those readings, and with them the gate: simulated, `Minh Do` reads suffix `Do` and `Do Van Minh` family `Do Van Minh`, the harm decision 15 is there to prevent. Nor does a census count belong to one language. So strict waits (decided 2026-10-28), out of the first packs, until name readings say whose names they are, and the candidates that need it wait with it. The `nl` prepositions show they would need more than strict: turned on as possibly-given particles, `in` and `to` already read `Park In Hwan` family `In Hwan` and `Nguyen To Nga` family `To Nga`, the marker acting at the head only, and check 11 lets their author write `may_be_given = false` instead of declaring the Korean and Vietnamese readings. -**Open: nothing in the work order says whose names a name reading's are.** Strict waits on it, and none of 3a–3h does it. Moving readings between files would not: a census count belongs to no one language, and a word is borne in several, `do` as a Vietnamese and a Korean name, `kim` as a Korean surname and a Scandinavian given name. What it points at is an attribute on each name reading naming the languages of its bearers (`do`'s family reading `vi` and `ko`; the key's name still to choose), compiling to nothing until strict reads it. The counts to fill it exist, but none fills it mechanically. Of the 211 name readings, 172 count a US source and 21 none, and what per-country evidence they carry is prose in a `note`, as `do`'s nam_dict bands are (Korea 8, Vietnam 6, the Netherlands 1), in 40 of them by a text search. `name_usage.py --boxes` gives `orcid-researchers`' and `patentsview-inventors`' counts by box and country, and a box count says where a word was typed, not whose name it is: `do`'s family box in ORCID is led by Brazil, 7,227 holders against Vietnam's 876, and none of Brazil's is `do` alone (2,629 open with it, 4,598 hold it inside), the particle of a Brazilian surname, where Vietnam has 618 boxes of `do` alone and Korea 151. A share of each country's holders ranks them the other way (Vietnam 1.90%, Brazil 0.81%; PatentsView 1.78% and 0.64%), and `kim` reads cleanly either way, its family box 17.67% of ORCID's Korean holders and 13.30% of PatentsView's, its given box 0.93% of PatentsView's Danish inventors (2026-10-28, master `c8357e74`). So the fill is a person reading each of the 157 words with name readings, by share and by whole box, not a script. Two further consequences: the attribute names languages, filled from countries by that reader, since a country-to-language table read at run time is the macrolanguage table the tag matching below refuses; and it is the first provenance anything computes a reading from, where counts today are evidence nothing reads. Proposed: a sub-phase of its own, 3i, after 3a, independent of the output model and moving no default parse, with strict and the candidates waiting on it after it. Recompute: count the name readings per file and per count key with `tomllib` over `nameparser/vocab/*.toml`; `name_usage.py do kim --no-wikidata --boxes`; the whole-box split by reading the cached ORCID extract's `family` and `country` keys. +**Open: nothing in the work order says whose names a name reading's are.** Strict waits on it, and none of 3a–3h does it. Moving readings between files would not: a census count belongs to no one language, and a word is borne in several, `do` as a Vietnamese and a Korean name, `kim` as a Korean surname and a Scandinavian given name. What it points at is an attribute on each name reading naming the languages of its bearers (`do`'s family reading `vi` and `ko`; the key's name still to choose), compiling to nothing until strict reads it. The counts to fill it exist, but none fills it mechanically. Of the 211 name readings, 172 count a US source and 21 none, and what per-country evidence they carry is prose in a `note`, as `do`'s nam_dict bands are (Korea 8, Vietnam 6, the Netherlands 1), in 40 of them by a text search. `name_usage.py --boxes` gives `orcid-researchers`' and `patentsview-inventors`' counts by box and country, and a box count says where a word was typed, not whose name it is: `do`'s family box in ORCID is led by Brazil, 7,227 holders against Vietnam's 876, and none of Brazil's is `do` alone (2,629 open with it, 4,598 hold it inside), the particle of a Brazilian surname, where Vietnam has 618 boxes of `do` alone and Korea 151. A share of each country's holders ranks them the other way (Vietnam 1.90%, Brazil 0.81%; PatentsView 1.78% and 0.64%), and `kim` reads cleanly either way, its family box 17.67% of ORCID's Korean holders and 13.30% of PatentsView's, its given box 0.93% of PatentsView's Danish inventors (2026-10-28, master `c8357e74`). So the fill is a person reading each of the 157 words with name readings, by share and by whole box, not a script. Two further consequences: the attribute names languages, filled from countries by that reader, since a country-to-language table read at run time is the macrolanguage table the tag matching below refuses; and it is the first provenance anything computes a reading from, where counts today are evidence nothing reads. Decided with it: a sub-phase of its own, 3i, after 3a, independent of the output model and moving no default parse, with strict and the candidates waiting on it after it. Recompute: count the name readings per file and per count key with `tomllib` over `nameparser/vocab/*.toml`; `name_usage.py do kim --no-wikidata --boxes`; the whole-box split by reading the cached ORCID extract's `family` and `country` keys. ### Where the provenance lives at runtime -The compiled module holds sets, and a set has forgotten which file each word came from. The proposal compiles a second module beside it: a **reading index**, one row per reading (its text, its file's tag, its kind and its behavioral properties), loaded for every caller, with `Lexicon.default()`, because phase 4's `Reading.languages` (its sub-phase 4a) is filled from it on every parse. Loading it on first access to `reading.languages` instead was weighed and set aside: a `Reading` holds its kind, role and source but not its word, so a lazy field would have to carry the word as a hidden key on a frozen, slotted, pickled type, to save a cost the measurement below puts at about 2% of the import. The compiler moves from tools/vocab/build.py into the package so that the build and a pack share one implementation, and a test holds the invariant that compiling the index with no choice reproduces `nameparser/config/_compiled.py` exactly. The same index is what item 13 of decisions/vocabulary-model's 2026-10-23 #752 entry says `Token.reading` needs in 3.0. +The compiled module holds sets, and a set has forgotten which file each word came from. Phase 3 compiles a second module beside it (decided 2026-10-28): a **reading index**, one row per reading (its text, its file's tag, its kind and its behavioral properties), loaded for every caller, with `Lexicon.default()`, because phase 4's `Reading.languages` (its sub-phase 4a) is filled from it on every parse. Loading it on first access to `reading.languages` instead was weighed and set aside: a `Reading` holds its kind, role and source but not its word, so a lazy field would have to carry the word as a hidden key on a frozen, slotted, pickled type, to save a cost the measurement below puts at about 2% of the import. The compiler moves from tools/vocab/build.py into the package so that the build and a pack share one implementation, and a test holds the invariant that compiling the index with no choice reproduces `nameparser/config/_compiled.py` exactly. The same index is what item 13 of decisions/vocabulary-model's 2026-10-23 #752 entry says `Token.reading` needs in 3.0. Measured 2026-10-27 on `cfe8bf92`: the pool holds 2,140 readings over 1,797 words, 106 of them in more than one file. An index of their behavioral properties is about 93 KB of generated source. Imported with its bytecode cached, as an installed wheel has it, it takes about 0.45 ms (median of nine fresh processes, 2026-10-28 on `c8357e74`, Python 3.11.17), against about 20 ms for `import nameparser` cached the same way; with no cache it takes about 11 ms, the figure this section gave on 2026-10-27, measured in a shell that sets `PYTHONDONTWRITEBYTECODE`. That load is paid once per process by every caller. Compiling the pool takes about 9 ms, paid once per choice and only by a caller who makes one. Reading the TOML instead costs about 39 ms. Recompute: the build's `read_dir`, `load` and `compile_pool`, timed; for the import, write the index as a module, `py_compile` it, and time its import in fresh processes beside a cached `import nameparser`. @@ -229,7 +229,7 @@ A choice is applied as a delta against the default's compile, so it composes wit ### The API and tags -This rested on `Locale` and `Policy` keeping their 2.x shape after the facade's removal, and the 2026-10-28 3.0 freeze answers it: both keep it, with additions. A choice is carried by `Locale`, which gains a `languages` field, a set of BCP 47 tags. `parser_for` unions the `languages` of the packs it is given, so the choice is order-independent while their policy patches still fold in order. Strict, once it can be built, is a property of the whole choice, not of one pack, and an argument: `parser_for(locales.NE, locales.HI, strict=True)`. The shipped language packs are `Locale` constants like the four today (`locales.NE`, `locales.get("ne")`), each with an empty lexicon fragment; a code keeps `Locale`'s `[a-z0-9_]+`, so `pt-BR` is `locales.PT_BR`. +This rested on `Locale` and `Policy` keeping their 2.x shape after the facade's removal, and the 2026-10-28 3.0 freeze answers it: both keep it, with additions. A choice is carried by `Locale`, which gains a `languages` field, a set of BCP 47 tags. `parser_for` unions the `languages` of the packs it is given, so the choice is order-independent while their policy patches still fold in order. Strict, once it can be built, is a property of the whole choice, not of one pack, and an argument: `parser_for(locales.NE, locales.HI, strict=True)`. The shipped language packs are `Locale` constants like the four today (`locales.NE`, `locales.get("ne")`), each with an empty lexicon fragment (what a fragment holds once 3.0 configures vocabulary by reading is decisions/vocabulary-model's the 2026-10-28 #754 entry on the `Lexicon` surface); a code keeps `Locale`'s `[a-z0-9_]+`, so `pt-BR` is `locales.PT_BR`. A chosen tag selects its own file and its ancestors', never its descendants': `pt-BR` selects `pt-BR.toml` and `pt.toml`, and `pt` selects only `pt.toml`, so a caller in Portugal does not turn on the Brazilian agnomens (#843 left the single-letter `F` and `N` to a `pt-BR` pack). No macrolanguage table is read: Malay and Indonesian are separate tags, though BCP 47 counts Indonesian within the Malay macrolanguage, and a caller with names in both chooses both. The Czech, Slovak, Slovenian and Croatian files each hold the words they share, so each language is a pack of its own. Script subtags need no matching rule yet, since no file carries one. @@ -245,7 +245,7 @@ The snapshot and `compare.py` parse under the default, so neither can see a pack ### What stays out of phase 3 -- **Third-party language files.** A file joining the pool at run time would skip the checks that run across files, an exclusion in one file refusing a reading in any other (check 6) and check 11's particle statement. A third-party `nl` file adding the preposition `in` without a `given` reading or `may_be_given = false` is refused by the build and would ship on the unsafe side at run time. `Locale(lexicon=…)`, or its successor once the facade is gone, stays the way to add vocabulary; a public file format is a later question. +- **Third-party language files.** A file joining the pool at run time would skip the checks that run across files, an exclusion in one file refusing a reading in any other (check 6) and check 11's particle statement. A third-party `nl` file adding the preposition `in` without a `given` reading or `may_be_given = false` is refused by the build and would ship on the unsafe side at run time. `Locale(lexicon=…)`, or its successor once the facade is gone, stays the way to add vocabulary, that successor being configuration by reading (decisions/vocabulary-model's the 2026-10-28 #754 entry on the `Lexicon` surface); a public file format is a later question. - **The Python packs' rule data.** ZH's surnames (556 on 2026-10-27) are a segmentation list, outside the per-word model, and the RU and TR_AZ markers are endings inside rules (`-ovich`, `oğlu`); both wait for phase 4's `[rules]`, compiled into a `Locale`'s policy patch in its sub-phase 4f. A pack gets a `languages` field only where measuring it shows what it moves. - **Candidates that wait on a rule, not a word:** Turkish `Bey` and `Hanım` after the given name, Hungarian `úr` after the name, Thai `นาย` glued at the head, French `M.` against the initial it spells. Each is a position rule, phase 4's. - **Same-language clashes.** No pack settles one (the `fils` argument): the Indic and Arabic titles that are names in their own language (`kumari`, `sayyid`) take phase 2j's route, a given reading and the title phrase, as default work. From eca13c329d4187e35ab7bc0e7eb0ef5186890d78 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Wed, 28 Oct 2026 18:46:32 -0700 Subject: [PATCH 33/39] Record phase 3's design as agreed (#753) Items 3, 7 and 9-12 are accepted as written, so all twelve are decided and the entry's head and the RFC's status say agreed. 3g's connective stays an open design inside the work order. The add-only pack contract is still amended in 3b's PR. Co-Authored-By: Claude Opus 5.5 --- docs/design/decisions/vocabulary-model.md | 16 ++++++++-------- docs/design/vocabulary-model-rfc.md | 8 ++++---- 2 files changed, 12 insertions(+), 12 deletions(-) diff --git a/docs/design/decisions/vocabulary-model.md b/docs/design/decisions/vocabulary-model.md index c98af472e..eeeedfd54 100644 --- a/docs/design/decisions/vocabulary-model.md +++ b/docs/design/decisions/vocabulary-model.md @@ -10,7 +10,7 @@ The vocabulary was a set of base sets plus marker subsets (`particles` and `part - The default loads every language and keeps working for names of any language. Choosing a language is to narrow it in phase 3 (#753): a pack is to settle only the clashes where its language has a vocabulary reading, a strict pack to drop other languages' name-level readings too, and a pack's preferred reading to be able to outrank one a shape rule derives — 2026-10-15/16. - `e`, `y` and `i` stay in the default; choosing a language is to settle their one-case fork, which no shipped pack does yet — 2026-10-15/16. - A patronymic field is to come in 3.0, as a view that names its words without emptying `family`, and the particle kinds already draw the `clan`/`patronymic` line for it: a patronymic is followed by a parent's name, a clan prefix by a family name the lineage shares — 2026-10-15/16; 2026-10-19 #749; 2026-10-28 #754 (Derek). -- A language pack is a choice of languages on `Locale.languages` selecting which readings compile, a tag choosing its own file and its ancestors', and a reading index loaded for every caller carries the languages to run time; strict waits for 3i, and third-party language files are deferred — 2026-10-27 #753 (Derek, items 1, 2, 4, 5, 6 and 8). +- A language pack is a choice of languages on `Locale.languages` selecting which readings compile, a tag choosing its own file and its ancestors', and a reading index loaded for every caller carries the languages to run time; strict waits for 3i, and third-party language files are deferred — 2026-10-27 #753 (Derek). - A caller configures vocabulary by reading, `Lexicon` deriving the sets and gates the build derives, and the compiled sets become internal in 3.0 — 2026-10-28 #754, the `Lexicon` surface (Derek). - A `Locale` may carry CLDR-shaped render patterns, display and sorting, a later addition whose shape is agreed; how a call reaches one is open — 2026-10-28 #754, render patterns. - 3.0's output model keeps the seven roles and adds every new field as a view, read from `Token.reading`, the reading each token took, or for `surname2` from rules/O6's tag: `generation` and `credentials` partitioning `suffix`, `patronymic`, and `surname2` where rules/O6 read two family names; `middle` and `surnames` keep their meaning, a compound given name being a rule a caller or pack turns on; the default `str()` renders a name in the order it was read in with its written separators in a normalized form, through `render(spec: str | None = None)`, `None` being that default; a language file's `[rules]` compile into its `Locale`'s policy patch — 2026-10-28 #754 (Derek). @@ -246,19 +246,19 @@ Open: #913 (the title words phase 2j left, read as titles though borne as names) - **2026-10-26 #752, phase 2l — THE BRACKET ESCAPE READS THE DERIVED CLASH**, carrying out item 16 of the 2026-10-23 #752 entry above (#810). A member of the ambiguous class, the class 2a derives from name readings, alone in brackets or quotes after the name is the suffix where its writing settles it (`John Smith (MA)`, `John Smith (M.A)` → suffix), read in one pass, and stays the nickname otherwise (`John Smith (Ed)`, `JOHN SMITH (MA)`, `Jeffrey (JD) Bricken`). It is recorded as rules/S13, a rule of its own rather than the amendment to S1 item 16 planned, the position it reads being evidence S1's signal does not read; the writing is S3's and S6's evidence, which S13 cites. The record, the measurements and what was declined are decisions/S13's 2026-10-26 #810 entry. - **2026-10-26 #752, phase 2m — THE READING NOT TAKEN SURFACES AS `Ambiguity.alternative`, WITHOUT ITS LANGUAGES**, carrying out item 13 of the 2026-10-23 #752 entry above and item 7's source with it: a word the credential's run took by position alone reports its name reading's source as `position`, a fourth source beside the three the item named, and a word with a declared name reading reports `declared`. The pickle guard was widened as item 13 asked, every skew the narrow guard refused still refused. The reading's languages were left out: the per-word index costs about 0.35 ms of an 18.2 ms import (decisions/A1) and describes the default vocabulary, not a pack's or a caller's lexicon, so they wait for 3.0 with `Token.reading`. The record, the measurements and what was declined are decisions/A1's 2026-10-26 #752 entry. -- **2026-10-27 #753 (Derek, items 1, 2, 4, 5, 6 and 8) — PHASE 3'S DESIGN IS PARTLY AGREED: A CHOICE OF LANGUAGES SELECTS THE POOL, ONE MECHANISM FOR TURNING READINGS ON AND PREFER, A READING INDEX CARRIES THE LANGUAGES TO RUN TIME, AND STRICT WAITS ARE DECIDED; NAME READINGS IN A PACK KEEPING THE GATE ON, THE MEASURES, THE EXCLUSIONS AND THE WORK ORDER ARE STILL PROPOSED.** Derek accepted items 1, 2, 4, 5, 6 and 8 as written in chat on 2026-10-28; items 3, 7 and 9 to 12 are proposals awaiting his decision, and this head is rewritten when he makes them. The design overview is docs/design/vocabulary-model-rfc.md's Phase 3 section. Decisions 15 and 20 are kept as written; decision 6's strict variant is reshaped to a property of the whole choice and deferred (items 1 and 4). Phase 3 is 3.0 work since the 2026-10-28 re-plan (item 12). The inputs are #753's comments of 2026-10-27: the inventory of 411 kept-out candidate readings with their verdicts, and the probe of foreign titles and function-word collisions. Measurements were taken 2026-10-27 on master `cfe8bf92` (Python 3.11.17) with throwaway Lexicon edits standing in for a pack, so each is evidence for the proposal and each sub-phase re-measures on its own base. Local-corpus figures are counts. +- **2026-10-27 #753 (Derek) — PHASE 3'S DESIGN IS AGREED: A CHOICE OF LANGUAGES SELECTS THE POOL, ONE MECHANISM FOR TURNING READINGS ON AND PREFER; NAME READINGS IN A PACK KEEP THE GATE ON; A READING INDEX CARRIES THE LANGUAGES TO RUN TIME; STRICT WAITS.** Derek accepted every item as written in chat on 2026-10-28, items 1, 2, 4, 5, 6 and 8 first, then 3, 7 and 9 to 12. One design inside item 11 stays open: 3g's connective, which accepting the work order did not decide. The design overview is docs/design/vocabulary-model-rfc.md's Phase 3 section. Decisions 15 and 20 are kept as written; decision 6's strict variant is reshaped to a property of the whole choice and deferred (items 1 and 4). Phase 3 is 3.0 work since the 2026-10-28 re-plan (item 12). The inputs are #753's comments of 2026-10-27: the inventory of 411 kept-out candidate readings with their verdicts, and the probe of foreign titles and function-word collisions. Measurements were taken 2026-10-27 on master `cfe8bf92` (Python 3.11.17) with throwaway Lexicon edits standing in for a pack, so each is evidence for the proposal and each sub-phase re-measures on its own base. Local-corpus figures are counts. 1. **A pack is a choice of languages carried by `Locale`.** `Locale` gains `languages`, a set of BCP 47 tags; `parser_for` unions the sets of the packs it is given, so the choice is order-independent while policy patches still fold in order; strict is an argument of the whole choice (`parser_for(locales.NE, locales.HI, strict=True)`), since a strict pack beside a preferring one has no meaning, reshaping decision 6's "strict variant". The shipped language packs are `Locale` constants with an empty lexicon fragment (`locales.NE`, `locales.PT_BR`); what a fragment holds once 3.0 configures vocabulary by reading is the 2026-10-28 #754 entry on the `Lexicon` surface. The dependency this rested on, `Locale` and `Policy` keeping their 2.x shape once the facade is gone, is answered by the 2026-10-28 3.0 freeze: both keep it, with additions. Weighed against: `Parser(languages=…)` beside `Locale`, two ways to choose one thing, and a `Policy` field, `Policy` being behavior and vocabulary being `Lexicon`'s. 2. **One mechanism.** The chosen languages select which readings the compiler compiles, the default being that compiler with every language and no choice. Turning on, prefer and strict are derived from the file each reading sits in, and nothing is written per pack beyond its language's readings: a chosen language's `default = false` readings join; prefer drops other languages' vocabulary readings of a word on which a chosen language holds one, never `mul.toml`'s (`Haddad, Abd`, suffix `Abd` today, reads given `Abd` with the en credential gone); strict drops other languages' name readings and the markers they derive. A clash a property records with a SHAPE reading is settled for the chosen language's own reading, decision 20 applied: `may_be_initial` does not hold on it, so a `pt` choice reads `ANA SILVA E SOUZA` family `SILVA E SOUZA` where the default reads middle `SILVA E` with a report. This answers decisions/3-0-reevaluations' 2026-09-13 bullet that a pack cannot remove `e`, and its 2026-08-17 bullet's "overrides rather than additions": a pack overrides by selection, never by a removal list. That removes base entries all the same, so mechanisms.md#LOCALE-PACKS-PURE-DATA's add-only contract is amended, in 3b's PR, which builds the selection (items 11 and 12), not by this decision. The shape-clash rule follows from decision 7, which decision 20 permits; it reaches every reading `may_be_initial` stands on (pt and it `e`, ca and pl `i`, the Italian one unjudged), and it forecloses an `nl` pack marking `y` (the 2026-08-18 bullet there), which waits with `fru` below. New beside decision 6: `mul.toml`'s exemption from prefer. Left open rather than built: a property a language's own reading should take only under its pack, Swedish `fru` and Icelandic `frú` as after-given salutations (rules/H8), refused in the default for the Cameroonian surname, the two readings in the inventory needing it. - 3. **Proposed: `default = false` is allowed on `given` and `family` readings, and turned on they derive markers like any name reading, never settling (decision 15 unchanged).** Measured on nepal-candidates' 126,981 Devanagari and romanized pairs, comparing every word's role across the two spellings: 123,440 read alike in the default; `given` readings on `श्री`, `guru` and `गुरु`, which make each a title only inside a title phrase (rules/H9), give 123,636 (198 fixed, 2 broken, both pairs whose romanization opens with a listed title); a `family` reading on `dc` as well, making the credential ambiguous, gives 123,639, the count still leaving `Ram Bahadur DC` a suffix; dropping `dc`'s credential, which settling by a name reading would do, gives 123,658, 19 more. Settling is not proposed: the 19 pairs are the price of decision 15, whose `do` case it protects. `bk`, `kc`, `gc` and `bc` read as the family in every shape tried (spoken order, a family comma, `FAMILY_FIRST`, dotted and Title case), so a reading would move nothing and none is proposed. Recompute: the readings moved into or out of `Lexicon` fields by hand, the pairs compared by `tuple(t.role for t in parser.parse(s).tokens)`. + 3. **`default = false` is allowed on `given` and `family` readings, and turned on they derive markers like any name reading, never settling (decision 15 unchanged).** Measured on nepal-candidates' 126,981 Devanagari and romanized pairs, comparing every word's role across the two spellings: 123,440 read alike in the default; `given` readings on `श्री`, `guru` and `गुरु`, which make each a title only inside a title phrase (rules/H9), give 123,636 (198 fixed, 2 broken, both pairs whose romanization opens with a listed title); a `family` reading on `dc` as well, making the credential ambiguous, gives 123,639, the count still leaving `Ram Bahadur DC` a suffix; dropping `dc`'s credential, which settling by a name reading would do, gives 123,658, 19 more. Settling is not taken: the 19 pairs are the accepted price of decision 15, whose `do` case it protects. `bk`, `kc`, `gc` and `bc` read as the family in every shape tried (spoken order, a family comma, `FAMILY_FIRST`, dotted and Title case), so a reading would move nothing and none is added. Recompute: the readings moved into or out of `Lexicon` fields by hand, the pairs compared by `tuple(t.role for t in parser.parse(s).tokens)`. 4. **Strict waits, out of the first packs, until name readings say whose names they are.** A name reading sits in the file whose source counted it: 113 of the 211 name readings are in en.toml, `do`'s Vietnamese Đỗ readings among them, vi.toml holding no `do`, so strict built on the files drops them under a `vi` choice, and simulated, `Minh Do` reads suffix `Do` and `Do Van Minh` family `Do Van Minh`, decision 15's harm. A census count belongs to no one language either. The candidates needing strict wait with it, `nl`'s prepositions among them, which need more than strict: turned on as possibly-given particles, `in` and `to` read `Park In Hwan` family `In Hwan` and `Nguyen To Nga` family `To Nga` (the marker acts at the head only), and check 11 accepts `may_be_given = false` in place of the Korean and Vietnamese readings. Recompute: count `given` and `family` readings per file with tools/vocab/build.py's `load`; simulate by removing `do` from `suffix_acronyms_ambiguous` and `particles_ambiguous`. Open: none of 3a–3h says whose names they are. Moving readings between files would not, a word being borne in several languages (`do` Vietnamese and Korean, `kim` Korean and Scandinavian), so the RFC points at an attribute on each name reading naming its bearers' languages, compiling to nothing until strict reads it, the first provenance anything would compute a reading from. It cannot be filled by script: 172 of the 211 count a US source and 21 none, per-country evidence is note prose in 40, and a box count says where a word was typed, not whose name it is (`do`'s ORCID family box is led by Brazil, 7,227 against Vietnam's 876, none of Brazil's `do` alone against Vietnam's 618; by share Vietnam leads, 1.90% to 0.81%; 2026-10-28, master `c8357e74`). The attribute names languages, so no country-to-language table is read at run time (item 6). A sub-phase of its own, 3i, after 3a, independent of the output model, moving no default parse; strict after it. 5. **The languages reach run time as a reading index.** A second generated module, one row per reading (text, tag, kind, behavioral properties), loaded for every caller with `Lexicon.default()`, since phase 4's `Reading.languages` (4a) is filled from it on every parse, rather than on first access to `reading.languages`, which would need a `Reading`, which holds no word, to carry one as a hidden key; the compiler moves from tools/vocab/build.py into the package so build and pack share one implementation, and a test holds that compiling the index with no choice reproduces `_compiled.py`. It is the per-word index item 13 of the 2026-10-23 #752 entry says `Token.reading` needs. A choice is applied as a delta against the default's compile, so it composes with a caller's `base=` lexicon and a `Locale`'s own fragment. Measured: 2,140 readings over 1,797 words, 106 in more than one file; the index about 93 KB of source, about 0.45 ms to import with its bytecode cached against about 20 ms for a cached `import nameparser` (2026-10-28, `c8357e74`; about 11 ms without the cache, the 2026-10-27 figure, taken under `PYTHONDONTWRITEBYTECODE`), once per process for every caller; the compile 9 ms, once per choice and only for a caller who makes one; loading the TOML 39 ms. Weighed against: per-language deltas compiled at build time, which do not compose (under prefer `{pt, en}` `do` keeps its particle and its credential, while Portuguese's delta drops the credential and English's the particle); every combination precompiled, combinatorial; the TOML read at run time, slower and either re-running the build's checks or skipping them. 6. **A chosen tag selects its file and its ancestors', never its descendants, and no macrolanguage table is read.** `pt-BR` selects `pt-BR.toml` and `pt.toml`; `pt` only `pt.toml`, so a caller in Portugal does not turn on the Brazilian agnomens (#843). The two files share no word today. Malay and Indonesian, which share `puan`, `putra` and `putri`, are chosen together by a caller who has both. The TOML names `prefer` and `strict` stay refused, the message saying a strength is the caller's choice and never data. - 7. **Proposed: a pack is measured three ways.** The default does not move (the committed snapshot, and the index invariant of item 5); a vocabulary pack's deviation surface, the words whose compiled readings its choice changes, is derived from the index, and tests/v2/test_locales.py's non-interference gate requires the default's reading for every corpus name without such a word, the policy and segmentation packs (JA, RU, TR_AZ, ZH) keeping their written `DEVIATES`; and `snapshot.py`, `--local` with `--base` included, takes a choice and prints what it moves by field, beside each pack's own ground truth (nepal-candidates' script pairs for `ne`, the CLDR agreement per pack, PSC's nationality slice). This answers decisions/3-0-reevaluations' 2026-09-13 bullet that the harness cannot see a pack. + 7. **A pack is measured three ways.** The default does not move (the committed snapshot, and the index invariant of item 5); a vocabulary pack's deviation surface, the words whose compiled readings its choice changes, is derived from the index, and tests/v2/test_locales.py's non-interference gate requires the default's reading for every corpus name without such a word, the policy and segmentation packs (JA, RU, TR_AZ, ZH) keeping their written `DEVIATES`; and `snapshot.py`, `--local` with `--base` included, takes a choice and prints what it moves by field, beside each pack's own ground truth (nepal-candidates' script pairs for `ne`, the CLDR agreement per pack, PSC's nationality slice). This answers decisions/3-0-reevaluations' 2026-09-13 bullet that the harness cannot see a pack. 8. **Third-party language files are deferred past phase 3.** A file joining the pool at run time would skip the checks that run across files (check 6's exclusions, check 11's particle statement), the ones that refuse a third-party `nl` `in` written with neither a `given` reading nor `may_be_given = false`. `Locale(lexicon=…)`, or its successor, stays the way to add vocabulary, that successor being configuration by reading (the 2026-10-28 #754 entry on the `Lexicon` surface). - 9. **Proposed: every exclusion a pack turns on becomes a `default = false` reading,** the shape cs `st` already has, so that once the packs land an `excluded` key means wrong everywhere: the eight whose reason opens "pack-shaped" in 3c (cs `ml`, hr, sk and sl `st` and `ml`, fr `père`), and the rest in the PR of the pack that turns each on (`pan`, `roz`, `nic`, `shri`, `sig`, `jaun`, `bt`, `bay`, `sb`). Recompute: `grep -h 'pack-shaped' nameparser/vocab/*.toml | wc -l`. - 10. **Proposed: what stays out of phase 3.** ZH's surname list and the RU and TR_AZ endings are rule data for phase 4's `[rules]` (its 4f), and a Python pack gains `languages` only where a measurement shows what it moves; the candidates that wait on a position rule (Turkish `Bey` and `Hanım`, Hungarian `úr`, Thai `นาย`, French `M.`) are phase 4's; same-language clashes (`kumari`, `sayyid`) take phase 2j's route in the default, since no pack settles one. - 11. **Proposed: the work order,** shape first and one PR each, measured on its own base: 3a the index and shared compiler with name readings allowed `default = false` (neutral); 3b the choice, tag selection, turning on, prefer, the shape clash, the derived deviation surface, `snapshot.py`'s choice and the amended contract (neutral); 3c the pack-shaped exclusions converted (neutral); then the packs, `ne` (3d), `hi` and `bn`'s Latin forms (3e), `cs`, `sk`, `sl` and `hr` (3f), `es` and `pt` with a connective that counts only inside a title run (3g), and `ga`/`gd` and `pt-BR` (3h); 3i, the bearers' languages of each name reading (item 4), after 3a, ahead of strict. 3g's connective is a rule with its own design: with `profesora` added and no rule, `Profesora de Historia Ana López` reads family `de Historia Ana López` (the rules/P1 fold) and `Ministro da Fazenda Carlos Souza` given `da`, while `Doctora de la Vega` must keep family `de la Vega`, the two separated by whether a name stands behind the phrase; it must reach only readings a pack turns on. - 12. **Proposed: phase 3 is 3.0 work, opt-in, the default's parses unmoved, with phase 4's shape questions settled first** (the 2026-10-28 re-plan: 2.8 is the last 2.x feature release, and 3.0 ships phases 3 and 4 with the facade's removal). The output model decides what a pack can express (a patronymic field, a second surname, #326's credentials apart from generations, #704's `middle`). The output model was agreed on 2026-10-28 (the 2026-10-28 #754 entry). 3a and 3c are independent of it, and so are 3b's selection semantics, its API's dependency answered (item 1); 3f and 3h need 4b's `generation` view, 3g needs 4d's `surname2` and 4g's compound given name, and 4f, which compiles `[rules]` into a `Locale`'s policy patch, lands with or after 3b; `ne` needs none. One standing text is amended with 3b, mechanisms.md#LOCALE-PACKS-PURE-DATA's contract that a pack is add-only and declares its deviations in its docstring (a pack adds, and selects among the default's readings by language, a vocabulary pack's deviations derived); decisions/3-0-reevaluations' Standing line that removing base vocabulary through a pack is part of 3.0's pack design is carried out. + 9. **Every exclusion a pack turns on becomes a `default = false` reading,** the shape cs `st` already has, so that once the packs land an `excluded` key means wrong everywhere: the eight whose reason opens "pack-shaped" in 3c (cs `ml`, hr, sk and sl `st` and `ml`, fr `père`), and the rest in the PR of the pack that turns each on (`pan`, `roz`, `nic`, `shri`, `sig`, `jaun`, `bt`, `bay`, `sb`). Recompute: `grep -h 'pack-shaped' nameparser/vocab/*.toml | wc -l`. + 10. **What stays out of phase 3.** ZH's surname list and the RU and TR_AZ endings are rule data for phase 4's `[rules]` (its 4f), and a Python pack gains `languages` only where a measurement shows what it moves; the candidates that wait on a position rule (Turkish `Bey` and `Hanım`, Hungarian `úr`, Thai `นาย`, French `M.`) are phase 4's; same-language clashes (`kumari`, `sayyid`) take phase 2j's route in the default, since no pack settles one. + 11. **The work order,** shape first and one PR each, measured on its own base: 3a the index and shared compiler with name readings allowed `default = false` (neutral); 3b the choice, tag selection, turning on, prefer, the shape clash, the derived deviation surface, `snapshot.py`'s choice and the amended contract (neutral); 3c the pack-shaped exclusions converted (neutral); then the packs, `ne` (3d), `hi` and `bn`'s Latin forms (3e), `cs`, `sk`, `sl` and `hr` (3f), `es` and `pt` with a connective that counts only inside a title run (3g), and `ga`/`gd` and `pt-BR` (3h); 3i, the bearers' languages of each name reading (item 4), after 3a, ahead of strict. 3g's connective is a rule with its own design, open, not decided with the work order: with `profesora` added and no rule, `Profesora de Historia Ana López` reads family `de Historia Ana López` (the rules/P1 fold) and `Ministro da Fazenda Carlos Souza` given `da`, while `Doctora de la Vega` must keep family `de la Vega`, the two separated by whether a name stands behind the phrase; it must reach only readings a pack turns on. + 12. **Phase 3 is 3.0 work, opt-in, the default's parses unmoved, with phase 4's shape questions settled first** (the 2026-10-28 re-plan: 2.8 is the last 2.x feature release, and 3.0 ships phases 3 and 4 with the facade's removal). The output model decides what a pack can express (a patronymic field, a second surname, #326's credentials apart from generations, #704's `middle`). The output model was agreed on 2026-10-28 (the 2026-10-28 #754 entry). 3a and 3c are independent of it, and so are 3b's selection semantics, its API's dependency answered (item 1); 3f and 3h need 4b's `generation` view, 3g needs 4d's `surname2` and 4g's compound given name, and 4f, which compiles `[rules]` into a `Locale`'s policy patch, lands with or after 3b; `ne` needs none. One standing text is amended with 3b, mechanisms.md#LOCALE-PACKS-PURE-DATA's contract that a pack is add-only and declares its deviations in its docstring (a pack adds, and selects among the default's readings by language, a vocabulary pack's deviations derived); decisions/3-0-reevaluations' Standing line that removing base vocabulary through a pack is part of 3.0's pack design is carried out. - **2026-10-28 #754 (Derek) — PHASE 4'S OUTPUT MODEL IS AGREED: THE SEVEN ROLES STAY AND EVERY NEW FIELD IS A VIEW, MOST READ FROM `Token.reading`; A PATRONYMIC NAMES ITS WORDS WITHOUT EMPTYING `family`; THE RENDER DEFAULTS FOLLOW THE ORDER A NAME WAS READ IN.** Derek agreed every item in chat on 2026-10-28, items 6 and 4 and the `surnames` line of item 5 first, then the rest. The design overview is docs/design/vocabulary-model-rfc.md's Phase 4 section. Decision 8 (a patronymic field in 3.0) is kept and its stated value revised (item 3); decisions 9 and 14 are kept as written. The core API is frozen for 3.0 except for additions (decisions/3-0-reevaluations, 2026-10-28 #925, THE CORE API), and each item says whether it is one. The inputs are #754 and its comment of 2026-10-23, #326, #704 and #694. Measurements were taken 2026-10-28 on master `c8357e74` (Python 3.11.17) over the fielded local corpora (`orcid-researchers`, `psc-individuals`, `patentsview-inventors`), the committed fields files and the committed corpus's distinct entries (`snapshot.corpus_entries()`, each parsed under its declared order), with throwaway scripts that are not kept, so each figure is evidence for the decision and each sub-phase re-measures on its own base. Local-corpus figures are counts. Recompute the box counts by reading each source's records with `local_corpora.read_records`, counting its boxes by word count and word ending per country or nationality, and the role the parse gives the counted word with `parse()`; the ending tests are the RFC's Phase 4 table's, and other tests give counts a few percent apart. 1. **`Role` keeps its seven members, and every field asked for is a view.** A role partitions the tokens, and each field asked for (a second surname, a patronymic, generation and credentials, a second given name) subdivides a role that exists, so each is computed from the tokens, a property and a render key, except the second given name, which is the `middle` that exists (item 7), as `family_base` and `given_names` are (rules/R1, rules/R2). `as_dict()`, `comparison_key()` and the seven role properties are unchanged, and every view is an addition. Weighed against: a `Role` member for any of them, a break of the frozen `ParsedName` (`as_dict()` gains a key, `comparison_key()` a component, so a stored key stops matching, and the role its words leave returns less text than in 2.x), with no 2.x feature release left to warn in. 2. **`Token.reading`, the reading each token took**, a `Reading` (decisions/A1's type), `None` for a token `replace()` builds, with `ADDED_IN` metadata so a 2.x pickle loads, and left out of comparison so `==` keeps its 2.x meaning. `Reading` gains `languages`, the files holding the reading, filled from phase 3's reading index (3a) and empty where no file holds it; `ReadingKind` gains `PATRONYMIC`. The views read it. It carries out the 2026-10-23 #752 entry's item 13 and the 2026-10-26 phase 2m entry's deferral of the languages. diff --git a/docs/design/vocabulary-model-rfc.md b/docs/design/vocabulary-model-rfc.md index 03ef2acba..5acb51855 100644 --- a/docs/design/vocabulary-model-rfc.md +++ b/docs/design/vocabulary-model-rfc.md @@ -26,7 +26,7 @@ Four phases, in a fixed order: |---|---|---|---| | 1 | The per-word TOML format, compiled to today's sets | no | shipped in 2.6 (#737, #744); #749–#751 gave the words in `und.toml` their languages, leaving two particles of unrecorded origin | | 2 | Ambiguity derived from clashing readings; the ambiguity gate extended to every reading kind | yes, the first phase that does | #752, design agreed 2026-10-23; 2a, the two name-clash markers derived, landed without moving a parse (decisions/vocabulary-model, 2026-10-25) | -| 3 | Language packs that prefer or restrict their language | yes, for callers who choose one | #753, for 3.0, needs 1 and 2; design proposed 2026-10-27, its mechanism, index and API agreed 2026-10-28, the rest proposed | +| 3 | Language packs that prefer or restrict their language | yes, for callers who choose one | #753, for 3.0, needs 1 and 2; design agreed 2026-10-28, 3g's connective open | | 4 | Order and shape; the rules a pack can choose; the output model | yes | #754, 3.0 work; its output model agreed 2026-10-28, ahead of or alongside phase 3's mechanism | Two commitments hold through all four. **The default loads every language and keeps working for names of any language, as today**; choosing a language narrows it (decision 5 in decisions/vocabulary-model). And **languages own vocabulary and defaults, the core owns structure**: a pack configures rules the core implements, and never ships code (mechanisms.md#LOCALE-PACKS-PURE-DATA). @@ -164,7 +164,7 @@ Each sub-phase is one PR, measured on its own base: the committed snapshot read ## Phase 3: packs that prefer or restrict -Status: **partly agreed** (proposed 2026-10-27, #753). Derek accepted on 2026-10-28 the choice of languages on `Locale.languages`, the one mechanism, strict waiting for 3i, the reading index loaded for every caller, tag selection by file and ancestors, and the deferral of third-party language files; name readings turned on in a pack, how a pack is measured, the exclusions converted, what stays out, the work order and the phase's place in 3.0 are still proposed. The record is decisions/vocabulary-model's 2026-10-27 #753 entry, whose items are each marked decided or proposed; this section describes them all. Phase 3 is 3.0 work (re-planned 2026-10-28): 2.8 is the last 2.x feature release, and 3.0 ships phases 3 and 4 with the facade's removal. Packs stay opt-in, so a caller who chooses no language parses as the default does, and nothing in this phase moves the default's parses. One standing text is amended with it, in 3b's PR, which builds the selection: mechanisms.md#LOCALE-PACKS-PURE-DATA's contract that a pack is add-only and declares its deviations in its docstring (a pack adds, and selects among the default's readings by language, its deviations derived for a vocabulary pack). decisions/3-0-reevaluations' line that removing base vocabulary through a pack is part of 3.0's pack design is carried out rather than moved. +Status: **agreed** (proposed 2026-10-27, #753; Derek accepted every item on 2026-10-28, six and then the other six). The record is decisions/vocabulary-model's 2026-10-27 #753 entry; this section describes its items. One design inside the work order is still open: 3g's connective (below), which accepting the work order did not decide. Phase 3 is 3.0 work (re-planned 2026-10-28): 2.8 is the last 2.x feature release, and 3.0 ships phases 3 and 4 with the facade's removal. Packs stay opt-in, so a caller who chooses no language parses as the default does, and nothing in this phase moves the default's parses. One standing text is amended with it, in 3b's PR, which builds the selection: mechanisms.md#LOCALE-PACKS-PURE-DATA's contract that a pack is add-only and declares its deviations in its docstring (a pack adds, and selects among the default's readings by language, its deviations derived for a vocabulary pack). decisions/3-0-reevaluations' line that removing base vocabulary through a pack is part of 3.0's pack design is carried out rather than moved. **Phase 4's shape questions came first.** The output model decides what a pack can express: a patronymic field, a second surname, credentials apart from generation suffixes (#326), and what `middle` means (#704). A pack reading of a kind with no field to land in is a reading nothing can surface, so those questions were to be settled before or with phase 3's mechanism, by the same shape-before-value rule, and the output model was agreed on 2026-10-28 (decisions/vocabulary-model's 2026-10-28 #754 entry): the seven roles stay, and `generation`, `credentials`, `patronymic` and `surname2` are views read from `Token.reading` or rules/O6's tag. Of the work order below, 3a (the reading index and the shared compiler) and 3c (the pack-shaped exclusions converted) are independent of the output model: the index carries whatever kinds the files hold, and 3c moves data between two encodings that compile alike. 3b's selection semantics are independent too, and its API's dependency, what `Locale` and `Policy` look like once the facade is gone, is answered (below). The packs depend on the output model in part, each on the phase 4 sub-phase that builds its view: `cs`'s generations and `pt-BR`'s agnomens on 4b's `generation` (3f and 3h), `es` and `pt` on 4d's `surname2` and 4g's compound given name (3g); `ne`'s readings do not. @@ -198,7 +198,7 @@ Everything a pack does is one operation: **the chosen languages select which rea The third row answers decisions/3-0-reevaluations' "a pack cannot remove `e`": `may_be_initial` records a clash between a conjunction and the initial a letter spells, and the initial is a shape reading, so a pack that prefers Portuguese settles it, as decision 7 says choosing a language does and decision 20 permits. It reaches every reading the property stands on, pt and it `e` and ca and pl `i`, the Italian one unjudged by the inventory; and it forecloses the other direction of the 2026-08-18 bullet there, an `nl` pack marking `y` as an initial, which waits with the `fru` question below. Two parts of the mechanism are new beside decision 6: the exemption of `mul.toml` from prefer (under `ca`, `i` keeps its mul generation), and strictness as a property of the whole choice. No property is toggled per pack, and no reading is written twice. -Properties ride on their readings, so a choice that keeps a reading keeps its properties. The one shape this cannot express is a property a language's own reading should take only under its pack: Swedish `fru` and Icelandic `frú` as after-given salutations (rules/H8), refused in the default because Fru is also a Cameroonian surname. The inventory found two readings needing it (2026-10-27), so the proposal builds nothing for them and leaves them open. +Properties ride on their readings, so a choice that keeps a reading keeps its properties. The one shape this cannot express is a property a language's own reading should take only under its pack: Swedish `fru` and Icelandic `frú` as after-given salutations (rules/H8), refused in the default because Fru is also a Cameroonian surname. The inventory found two readings needing it (2026-10-27), so phase 3 builds nothing for them and leaves them open. ### Name readings in a pack @@ -266,7 +266,7 @@ Shape first, then the packs, each one PR measured on its own base, as phase 2's | 3h | `ga` and `gd` (`nic`, `mhic`, `mic`, `uí`), `pt-BR` (`F`, `N`, `sb`); `nl`'s prepositions wait with strict | 3b, 4b | | 3i | the languages of each name reading's bearers, an attribute compiling to nothing, filled word by word from per-country shares and whole-box counts (above); strict and its candidates after it | 3a (neutral) | -3g's connective needs a design of its own, because the pair that separates it is a count: `Profesora de Historia Ana López` should read title `Profesora de Historia`, while `Doctora de la Vega`, the same title, preposition and following word, has no name behind it and must keep family `de la Vega`. With `profesora` added and no rule, the first reads family `de Historia Ana López` (the rules/P1 fold), and `Ministro da Fazenda Carlos Souza` reads given `da`. The rule must reach only readings a pack turns on, so the default's parses stay where they are. +**Open:** 3g's connective needs a design of its own, not decided with the work order, because the pair that separates it is a count: `Profesora de Historia Ana López` should read title `Profesora de Historia`, while `Doctora de la Vega`, the same title, preposition and following word, has no name behind it and must keep family `de la Vega`. With `profesora` added and no rule, the first reads family `de Historia Ana López` (the rules/P1 fold), and `Ministro da Fazenda Carlos Souza` reads given `da`. The rule must reach only readings a pack turns on, so the default's parses stay where they are. Languages with one to three candidates (`tr` `bay`, `it` `sig`, `lt` `jaun`, `ms` `bt`, `cy` `ab`/`ap`, `ber` `aït`) are a file each once 3b exists and need no sub-phase; `nb` and `sv`'s farm-name words wait for a measurement nobody has taken. From d97a910dc03d9616ef51ebde0eb0b3fcfa5fee6f Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Wed, 28 Oct 2026 19:00:21 -0700 Subject: [PATCH 34/39] Rebuild the rules corpus and snapshot for R2's new example (#754) R2's example 'da Silva do Amaral de Souza, Miguel' joins corpus_rules.jsonl and the snapshot as one new entry; no recorded reading moves, and every agreement record still matches. Co-Authored-By: Claude Opus 5.5 --- tools/differential/corpus_rules.jsonl | 1 + tools/differential/snapshot.jsonl | 1 + 2 files changed, 2 insertions(+) diff --git a/tools/differential/corpus_rules.jsonl b/tools/differential/corpus_rules.jsonl index 15d121b9d..beb2433f3 100644 --- a/tools/differential/corpus_rules.jsonl +++ b/tools/differential/corpus_rules.jsonl @@ -950,6 +950,7 @@ "abdul salam ahmed salem" "anh van do" "carod i" +"da Silva do Amaral de Souza, Miguel" "de" "de Anke van y" "de Mesnil Jean" diff --git a/tools/differential/snapshot.jsonl b/tools/differential/snapshot.jsonl index 1839b7cd6..38dc84d31 100644 --- a/tools/differential/snapshot.jsonl +++ b/tools/differential/snapshot.jsonl @@ -3752,6 +3752,7 @@ {"name": "bob v. de la macdole-eisenhower phd", "given": "bob", "middle": "v.", "family": "de la macdole-eisenhower", "suffix": "phd", "initials": "b. v. m.", "capitalized": "Bob V. de la MacDole-Eisenhower PhD"} {"name": "carod i", "given": "carod", "family": "i", "ambiguities": ["SUFFIX_OR_NAME"], "alternatives": ["SUFFIX_OR_NAME: generation as suffix (vocabulary)"], "initials": "c. i.", "capitalized": "Carod I"} {"name": "da Silva Sr. José Pedro Eduardo Jr., MP", "order": "FAMILY_FIRST", "given": "Silva", "middle": "Sr. José Pedro Eduardo", "family": "da", "suffix": "Jr., MP", "ambiguities": ["PARTICLE_OR_GIVEN"], "alternatives": ["PARTICLE_OR_GIVEN: particle as family (vocabulary)"], "initials": "S. S. J. P. E. d.", "capitalized_force": "Silva Sr. José Pedro Eduardo Da Jr., MP"} +{"name": "da Silva do Amaral de Souza, Miguel", "given": "Miguel", "family": "da Silva do Amaral de Souza", "initials": "M. S. A. S."} {"name": "de", "given": "de", "initials": "d.", "capitalized": "De"} {"name": "de Anke van y", "family": "de Anke van y", "initials": "A."} {"name": "de Bhulbh An tOll. An Dr. Brigid Aoibhe Sophia", "order": "FAMILY_FIRST", "given": "An", "middle": "tOll. An Dr. Brigid Aoibhe Sophia", "family": "de Bhulbh", "initials": "A. t. A. D. B. A. S. B.", "capitalized_force": "An Toll. An Dr. Brigid Aoibhe Sophia de Bhulbh"} From 67fa83e14f99252b9b2f9204165a3b334c27a353 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Wed, 28 Oct 2026 21:25:26 -0700 Subject: [PATCH 35/39] Fix the design-docs review's findings on the phase 3 and 4 design (#754) - rules/R2 gains an Except: a connective between two words of a part is set aside when the part is asked whether every word is particle vocabulary, so `Fritz Freiherr und von Bar` reads base 'Freiherr und von Bar'; today's base 'und' is a deviates: marker for #939. A connective ending the part stays a name word (`Juan de y` boundary). rules/R3's all-particles Except follows it, with its own marker. decisions/R2 records the case, dated 2026-10-28. - decisions/3-0-reevaluations records the render() signature break as waived, taking no 2.x warning, and updates the render-defaults and harness Standing lines. - The 3i counts are recounted at c8357e74 and dated; the Ministro reproduction names both titles added; Lim Mei Ling Jessica is scoped to FAMILY_FIRST. - rules_budget re-recorded (R2 and R3 grew by the new clause); corpus_rules gains the new example. Co-Authored-By: Claude Opus 5.5 --- docs/design/decisions/3-0-reevaluations.md | 11 ++++++----- .../R2-all-particles-family-base-divergence.md | 4 +++- docs/design/decisions/vocabulary-model.md | 8 ++++---- .../rules/R2-all-particles-family-base-divergence.md | 5 +++++ docs/design/rules/R3-initials-honor-the-fold.md | 3 ++- docs/design/vocabulary-model-rfc.md | 12 ++++++------ tests/v2/rules_budget.json | 4 ++-- tools/differential/corpus_rules.jsonl | 1 + 8 files changed, 29 insertions(+), 19 deletions(-) diff --git a/docs/design/decisions/3-0-reevaluations.md b/docs/design/decisions/3-0-reevaluations.md index be514212c..ca7929fe8 100644 --- a/docs/design/decisions/3-0-reevaluations.md +++ b/docs/design/decisions/3-0-reevaluations.md @@ -5,7 +5,7 @@ Promoted 2026-08-15 from session memory (Derek's 2026-07-30 ask; promotion appro ## Standing - The facade speaks v1's fields, `first` and `last`, over a core that speaks `given` and `family`, CJK names included; 3.0 can drop the aliasing — 2026-08-15. -- Render's default format is v1's, so an interpunct transcription renders space-joined (`str(HumanName("威廉·莎士比亚"))` is `'威廉 莎士比亚'`) and a caller reinstates the dot with a format of their own; 3.0's render defaults are a clean slate — 2026-08-15. +- Render's default format is v1's, so an interpunct transcription renders space-joined (`str(HumanName("威廉·莎士比亚"))` is `'威廉 莎士比亚'`) and a caller reinstates the dot with a format of their own, through 2.x; 3.0's default follows the order a name was read in and keeps its written separators, decided in decisions/vocabulary-model's #754 output-model entry, item 6 — 2026-08-15; 2026-10-28 #754 (Derek), THE RENDER SIGNATURE. - A `.pyi` stub typing `Policy.__init__` wide and its attributes narrow is a 3.0 candidate, deferred from #334 for want of a signature list with its own sync test — 2026-08-15. - The vocabulary DATA modules keep `nameparser.config` as their home in 3.0, so only the five shim names leave the package, and the five 2.2 alias messages (`prefixes.PREFIXES` and the rest) point at paths that stay — 2026-10-28 #357 (Derek). - `force_mixed_case_capitalization` is a `Constants` attribute and goes in 3.0 with it, the 2.8 construction warning being its removal warning — 2026-10-28 #925 (Derek), WARNINGS. @@ -13,7 +13,8 @@ Promoted 2026-08-15 from session memory (Derek's 2026-07-30 ask; promotion appro - Reading positionally where no script licenses family-first is the safe direction on its own, family-first being the marked case that needs affirmative evidence — 2026-08-15. - From 2.8, the last 2.x feature release, the uses 3.0 removes warn with a `DeprecationWarning` on every call at the caller's line naming code that runs: constructing `HumanName` or a `Constants`, loading either from a pickle, mutating the shared `CONSTANTS`, and reading `nameparser.config`'s three managers or `nameparser.util.lc`; importing `nameparser`, `nameparser.parser` or `HumanName` stays quiet, and so does reading the shared `CONSTANTS`, which no warning reaches — 2026-10-28 #925 (Derek), WARNINGS. - The 3.0 core API — `parse`, `Parser`, `Lexicon`, `Policy`, `Locale` and `ParsedName` — is frozen except for additions, so anything 3.0 later breaks there is a new decision needing its own 2.x warning — 2026-10-28 #925 (Derek), THE CORE API. -- `Lexicon`'s compiled-set fields are the first break decided under that freeze: 3.0 configures vocabulary by reading and keeps the sets internal; its 2.x warning is open — 2026-10-28 #754 (Derek), THE LEXICON SURFACE. +- `render(spec: str | None = None)` is a break decided under that freeze and waived, taking no 2.x warning, since no call that works today changes meaning — 2026-10-28 #754 (Derek), THE RENDER SIGNATURE. +- `Lexicon`'s compiled-set fields are the other break decided under that freeze: 3.0 configures vocabulary by reading and keeps the sets internal; its 2.x warning is open — 2026-10-28 #754 (Derek), THE LEXICON SURFACE. - The deprecation bridge's mechanics are the template for every remaining 2.x→3.0 shim: PEP 562 module `__getattr__` plus `__all__`, a warning per read location rather than per process, and the `TYPE_CHECKING` split — 2026-08-15. - A scoped config override through contextvars (`with nameparser.config.use(c):`) stays parked for 3.0, the shared `CONSTANTS` surviving 2.x as the one config channel that reaches parses inside code a caller does not own — 2026-07-06. - The 1.3.0-era legacy-pickle skip of `suffixes_prefixes_titles` rides in the shim's `__setstate__` and dies with it — 2026-08-16. @@ -21,11 +22,10 @@ Promoted 2026-08-15 from session memory (Derek's 2026-07-30 ask; promotion appro - Particle vocabulary is language judgement wearing a position heuristic, so a caller who knows a tradition should be able to opt into a tuned set, which needs packs to carry vocabulary overrides and not only additions; 3.0 work — 2026-08-17. - Which single-letter connectives read as initials differs by language, a per-word policy that `conjunctions_ambiguous` carries without packs (`e` and `i`); a pack can add a member, so an `nl` pack could mark `y`, and cannot remove one (the next line) — 2026-08-18; 2026-09-20 #397. - A locale pack's vocabulary is unioned onto the base and can remove nothing, so a caller removing a base entry, such as `e` from `conjunctions_ambiguous` for Portuguese, does it by hand; accepted for 2.x and part of 3.0's pack design — 2026-09-13, SHOULD A LOCALE PACK. -- The snapshot and `compare.py` read every corpus name under the default `Lexicon`, no corpus entry carrying a locale, so neither can show a pack's effect; the fix, a locale per corpus entry, is parked with the question above — 2026-09-13, THE DIFFERENTIAL HARNESS. - The shim raises on a `capitalization_exceptions` value that does not spell its key's letters and digits, as the core `Lexicon` does; only where the raise lands for a v1 caller, at the first parse, is shim-era — 2026-09-23 #459. - The shim's filter of v1 `Constants` entries dies with the shim, the core having no v1 reading to reproduce: it drops with one warning an entry v1 could never match, 1.4.0's two shipped `TITLES` typos silently, keeps a capitalized or edge-period key that v1 never matched and that matches now, an accepted widening, and drops a lone CJK full stop that 1.4.0 matched, its one departure from 1.4.0 (decisions/R4) — 2026-10-02 #541. -Not standing: the harness bullet's "at all", overtaken for the agreement records, which score the CLDR renderings under their language's pack since #673 (issue #672) and the native-script names since #693, both recorded in tools/differential/README.md and in no decisions entry — 2026-09-13, THE DIFFERENTIAL HARNESS; the parking lot's requests-Session-style `Parser(constants)` API, overtaken by 2.0's `Parser(lexicon=..., policy=...)` — 2026-07-06. +Not standing: the harness bullet, that the snapshot and `compare.py` cannot show a pack's effect, overtaken: the agreement records score the CLDR renderings under their language's pack since #673 (issue #672) and the native-script names since #693, both recorded in tools/differential/README.md, and phase 3's design has `snapshot.py`, `--local` included, take a language choice and print what it moves (decisions/vocabulary-model's #753 entry, item 7) — 2026-09-13, THE DIFFERENTIAL HARNESS; the parking lot's requests-Session-style `Parser(constants)` API, overtaken by 2.0's `Parser(lexicon=..., policy=...)` — 2026-07-06. Elsewhere: the explicit-comma line narrowed to a description for CJK script — decisions/cjk-comma-demotion's 2026-09-01 #469, SCOPES the FAMILY_COMMA bullet entry; the 1.4 gate the harness-baseline bullet expected 3.0 to dissolve, already retired, `compare.py --baseline 1.4.0` answering the question when asked — decisions/differential-ledger's 2026-10-13 #664 entries; the vocabulary model packs are to narrow, a strict pack dropping other languages' readings — decisions/vocabulary-model's 2026-10-15/16 entry; what the shim's filter keeps and drops — decisions/R4's 2026-10-02 #541 and 2026-10-02 #582 entries. @@ -76,4 +76,5 @@ Open: #753 (phase 3, language packs, where the three pack bullets land). - The shared-`CONSTANTS` mutation message drops the private `Constants` it offered as an escape, which 3.0 removes too, and names `Parser(lexicon=Lexicon.default().add(titles={'dean'}))`; the hook-override message (#280) adds that `HumanName` is removed in 3.0. Every warning is a `DeprecationWarning` rather than a `FutureWarning`, emitted on every call at the caller's line and never once per process (a process flag lets a dependency's first `HumanName` consume the warning meant for the caller, the reason the 2026-08-15 bridge template warns per read location, and turns this suite's `filterwarnings = error` into an order-dependent failure under `--dist worksteal`). A `DeprecationWarning` is hidden by default outside `__main__`, so the release log, README and docs/migrate.rst announce the removal loudly instead. Every message names code that type-checks and runs, pinned by tests/v2/test_deprecations.py. `force_mixed_case_capitalization`, a removal candidate since 2026-08-29, is a `Constants` attribute, so removing `Constants` removes it and the construction warning is its removal warning; that follows from the facade's removal rather than being decided apart from it. Phase 4's output model is announced in one line of docs/migrate.rst ("details with the 3.0 alphas"), nothing undecided listed. - 2026-10-28 #925 (Derek), THE CORE API — THE 3.0 CORE API IS FROZEN EXCEPT FOR ADDITIONS: `parse`, `Parser`, `Lexicon`, `Policy`, `Locale` and `ParsedName` keep their 2.x shape through 3.0, gaining fields, members and methods but losing and renaming none. Anything 3.0 later finds it must break there is a new decision, and it needs a 2.x release warning about it first, the bridge discipline in this file's preamble; Phase 3's draft, which assumes `Locale` and `Policy` keep their 2.x shape after the facade's removal, rests on this entry. -- 2026-10-28 #754 (Derek), THE LEXICON SURFACE — `Lexicon`'S COMPILED-SET FIELDS GO INTERNAL IN 3.0, THE FIRST BREAK DECIDED UNDER THE CORE API FREEZE ABOVE. A caller configures vocabulary by reading instead; the reasons and the weighing are decisions/vocabulary-model's 2026-10-28 #754 entry on the `Lexicon` surface. The freeze asks a break for a 2.x release warning first, and 2.8 was the last 2.x feature release, so how that warning ships (a 2.8.x patch, or the release notes alone) is open. +- 2026-10-28 #754 (Derek), THE RENDER SIGNATURE — `render(spec: str | None = None)` IS A BREAK DECIDED UNDER THE CORE API FREEZE ABOVE, AND ITS 2.x WARNING IS WAIVED. `None`, what `str()` passes, is the order-aware default, and a spec a caller writes renders as written; decisions/vocabulary-model's #754 output-model entry, item 6, holds the reasons. The freeze defines an addition and this is not one (introspecting the default gives `None`, and a subclass overriding `render(self, spec: str = ...)` becomes an incompatible override to a type checker), but no call that works today changes meaning, `render(None)` raising today, and 2.8 was the last 2.x feature release, so it ships in 3.0 with no 2.x warning. It also answers the 2026-08-15 bundle's (A) bullet on render's v1-inherited defaults. +- 2026-10-28 #754 (Derek), THE LEXICON SURFACE — `Lexicon`'S COMPILED-SET FIELDS GO INTERNAL IN 3.0, THE OTHER BREAK DECIDED UNDER THE CORE API FREEZE ABOVE, AND THE ONE WHOSE 2.x WARNING IS STILL TO SHIP. A caller configures vocabulary by reading instead; the reasons and the weighing are decisions/vocabulary-model's 2026-10-28 #754 entry on the `Lexicon` surface. The freeze asks a break for a 2.x release warning first, and 2.8 was the last 2.x feature release, so how that warning ships (a 2.8.x patch, or the release notes alone) is open. diff --git a/docs/design/decisions/R2-all-particles-family-base-divergence.md b/docs/design/decisions/R2-all-particles-family-base-divergence.md index 15f73135c..90d3d6ea3 100644 --- a/docs/design/decisions/R2-all-particles-family-base-divergence.md +++ b/docs/design/decisions/R2-all-particles-family-base-divergence.md @@ -8,11 +8,12 @@ - `initials()` reads the base, so the anchoring moves R3's output: `Anh Do` initials 'A. D.' and `Juan van der` 'J. v. d.' — 2026-08-16 (pre-merge coherence pass); 2026-08-18 #404/#385. - A particle run that joins nothing still chains whole into one part, so `Juan Smith van der` reads family 'van der' and middle 'Smith'; splitting it was declined, roles reassembling adjacent same-role pieces into one part and the base losing its full extent — 2026-08-18. - The base is the family without its working particles, wherever in the family they stand (`da Silva do Amaral de Souza, Miguel` → family_base 'Silva Amaral Souza', family_particles 'da do de'), a view the same under every convention and no culture's sort key — 2026-10-28 #754. +- A connective standing between two words of a part is set aside when the part is asked whether every word is particle vocabulary, so `Fritz Freiherr und von Bar` is all particles, base 'Freiherr und von Bar' and initials 'F. F. v. B.', where the parser reads base 'und', a tracked deviation; a connective ending the part is a name word, so `Juan de y` keeps base 'y' — 2026-10-28 #754 (Derek), A CONNECTIVE AMONG PARTICLES. - The unjoined mark is a statement about a whole part, and `initials()` honors it for every word in it, a connective included: under `Lexicon.default().add(particles={'y'})`, `Juan de y` reads family_base 'de y' and initials 'J. d. y.'; narrowing the readmission to the `particle` tag alone was tried and backed out — 2026-08-29 #461; 2026-09-20 #461. Elsewhere: where a connective initials, a part holding nothing else for it to join (`Jon Dough and` → 'J. D.', `Juan de y` → 'J. y.') — decisions/R3's 2026-09-20 #461 entry; case repair keeping that connective lowercase, its carve-out not gated on the mark (under `Lexicon.default().add(particles={'y'})`, `Juan de y` forced → family 'De y') — decisions/R4's 2026-08-29, the CONJUNCTION CARVE-OUT entry; case repair reading the unjoined mark for the whole part (`ANH DO` → 'Anh Do') — decisions/R4's 2026-08-29 #407 entry; the snapshot recording `initials()`, so a change to the base is visible there — decisions/differential-ledger's 2026-10-13 #664, a committed snapshot replaces the classified gate entry. -Open: #754 (whether a language pack supplies a convention's sort key or entry element from Phase 4's per-surname units, the RFC's Phase 4 Left open). +Open: #754 (whether a language pack supplies a convention's sort key or entry element from Phase 4's per-surname units, the RFC's Phase 4 Left open); #939 (a connective between particles reads as the base, the deviation rules/R2 and rules/R3 mark). ## Log @@ -40,6 +41,7 @@ Open: #754 (whether a language pack supplies a convention's sort key or entry el Measured 2026-10-28 on master `c8357e74`, counts and token shapes only, over the committed corpus's 6,533 distinct entries (`snapshot.corpus_entries()`, each parsed under its declared order) and `tse-donors`' 854,531 records (`local_corpora.read_records`, the filed name and the registry name each parsed in the default order). A family is read in render order (FOLDED_TAG tokens first), a working particle is a token tagged `particle` and not UNJOINED_TAG, and a family has an interior particle where a working particle stands after its first word that is neither a particle nor a conjunction; that family is where (1) and (2) differ. The three kinds of interior family are tested in this order, the first that fits counting: after a family comma, every interior particle standing after the first comma; with a leading run as well; with no leading run. In the default policy: committed, 24 against 684 with a leading run only (12 with no leading run, 8 whose interior particle stands after a family comma, 4 with a leading run as well); `tse-donors`, 12 filed names against 347,992 and 1 registry name against 392,134, since a Brazilian name's particles before its last word stand in `middle`. With `middle_as_family`: committed, 367 against 398 (262, 92 and 13); `tse-donors`' filed names, 268,591 against 79,926 (261,024, 5 and 7,562), the commonest family shape word-particle-word, 150,998 times. Recompute with those definitions; a change to the fold or to O6 moves them. Weighed against (2): with `middle_as_family`, the population the per-surname reading serves, it gives the 261,024 `tse-donors` names with no leading run a base equal to the whole family and an empty particle view, so `family_particles` stops naming the particles it is for; and it changes a frozen property's output for 24 committed entries in the default policy, where (1) changes no output. The facade whose `last_prefixes` it matches is removed in 3.0. Its one gain is a base that is a written name, which no convention's key needs either (below). The cost of (1), accepted: the base is not a name anyone writes ('Silva Amaral Souza'), nor any convention's sort key. Why the base is not a sort key, researched 2026-10-28. Portuguese and Brazilian practice files a name under its LAST surname, that surname's particle removed and set after the given names, the agnomens `Filho`, `Neto` and `Júnior` staying with the surname before them and an established compound surname kept whole: `SOUZA, Miguel da Silva do Amaral de`, `NOGUEIRA NETO, Wanderlino` (ABNT NBR 6023:2018, https://engenhariaeletrica.ufes.br/sites/engenhariaeletrica.ufes.br/files/field/anexo/abnt6023.2018.pdf; the ABNT Zotero style's comments, https://www.zotero.org/styles/universidade-federal-do-espirito-santo-abnt?source=1; USP's references manual, https://www.ip.usp.br/site/wp-content/uploads/2025/04/Manual-Referencias-ABNT-SabadiniSampaioNascimento.pdf; Cambridge University Library, https://www.lib.cam.ac.uk/node/3142; the UN cataloguing manual, https://www.un.org/Depts/dhl/unbisref_manual/nameguide/personal.htm; IFLA, Names of Persons: Portugal, 2009, https://ifla.org/files/assets/cataloguing/pubs/ifla_names_of_persons_portugal_2009.pdf). Spanish practice files under the FIRST surname, the second its secondary key (RAE, Ortografía de la lengua española, 2010, ch. VII, "Alfabetización de antropónimos", https://www.rae.es/ortografía/alfabetización-de-antropónimos, read 2026-10-28 in a browser, the page refusing a plain fetch): by its rule (a) a preposition, with or without an article, heading the FIRST surname is left out of the order and written after the given name (`Torre Ibarra, Ramón de la`); by rule (e) one heading the SECOND surname counts as part of the second key (`Gómez Casas`, `Gómez de la Iglesia`, `Gómez de Lara`, `Gómez del Castillo`, `Gómez Echagüe`, in that order); by rule (b) a leading article or `San`/`Santa` stays and is sorted on (`La Calle Ruiz, Carmen`); by rule (d) a `y` between the surnames counts (`Aguirre y Gómez-Pacheco`). The same page states the Portuguese convention from the other side, the paternal second surname being the axis. Neither convention's key is this base: Portuguese practice sorts `…da Silva de Souza do Amaral` before `…da Silva do Amaral de Souza` (Amaral before Souza) while both bases begin `Silva`, and Spanish practice keeps a second surname's particle (`Gómez de la Iglesia`) where the base drops it ('Gómez Iglesia'). The base dropping every working particle is nearest the Spanish first key alone for a preposition, rule (a) (`Torre Ibarra, Ramón de la` → base 'Torre Ibarra'), and departs from it on what rule (b) keeps: a leading article or `San` is a particle to the parse, so `La Calle Ruiz, Carmen` reads base 'Calle Ruiz' and `San Martín López, Juan` base 'Martín López'. So the base is the convention-neutral view, and a convention's key is built from the surnames by whoever knows the convention, the RFC's Phase 4 Left open. In the default policy, `Miguel da Silva do Amaral de Souza` already reads family 'de Souza', base 'Souza', which is the Portuguese entry element by coincidence: only `middle_as_family` gives the long family. +- 2026-10-28 #754 (Derek), A CONNECTIVE AMONG PARTICLES — R2 GAINS AN EXCEPT SETTING A JOINING CONNECTIVE ASIDE, AND THE PARSER'S READING IS A TRACKED DEVIATION (#939). The entry above asked R2's all-particles question of one trailing particle and left it as the code reads it; the design-docs review of that amendment found a second shape the code decides against R2's own Except ("anchor the base") and Accepted line ("A particle needs a base to attach to"): a connective between particles. `Fritz Freiherr und von Bar` reads family 'Freiherr und von Bar', family_base 'und', family_particles 'Freiherr von Bar', initials 'F. u.', while without `und` the family is all particles and its base is the whole of it. A conjunction alone became the base of a family of three particles, and the Then before the amendment ("without its leading particles") would have given 'und von Bar', so the code deviated, unmarked, before the amendment as well. Decided: a connective between two words of a part is set aside when the all-particles question is asked, as R3 sets the working particles aside when it asks whether a connective is joining, since a word doing a connective's work is no base; the particles then anchor the base, and the connective, joining them, contributes no initial (R3's Except amended with it: 'F. F. v. B.'). A connective ENDING the part stays out of the Except: it joins nothing, R3 already makes it a name word (`Juan de y` → base 'y', initials 'J. y.', decisions/R3's 2026-09-20 #461 entry), and setting it aside would make `de` the base and contradict that reading. Measured 2026-10-28 on `d97a910d`, counts and token shapes only, over the committed corpus's distinct entries (`snapshot.corpus_entries()`), each under its declared order: 3 families whose base holds only connectives in the default policy, 1 with the connective between particles (particle, connective, particle, particle), 1 ending in it (particle, connective) and 1 a lone connective, the last two outside the Except; the same 3 with `middle_as_family`. Recompute: a family whose `family_base` words are all family tokens tagged `conjunction`, split by whether the family's last token is one. Declined: diff --git a/docs/design/decisions/vocabulary-model.md b/docs/design/decisions/vocabulary-model.md index eeeedfd54..86f89f201 100644 --- a/docs/design/decisions/vocabulary-model.md +++ b/docs/design/decisions/vocabulary-model.md @@ -250,14 +250,14 @@ Open: #913 (the title words phase 2j left, read as titles though borne as names) 1. **A pack is a choice of languages carried by `Locale`.** `Locale` gains `languages`, a set of BCP 47 tags; `parser_for` unions the sets of the packs it is given, so the choice is order-independent while policy patches still fold in order; strict is an argument of the whole choice (`parser_for(locales.NE, locales.HI, strict=True)`), since a strict pack beside a preferring one has no meaning, reshaping decision 6's "strict variant". The shipped language packs are `Locale` constants with an empty lexicon fragment (`locales.NE`, `locales.PT_BR`); what a fragment holds once 3.0 configures vocabulary by reading is the 2026-10-28 #754 entry on the `Lexicon` surface. The dependency this rested on, `Locale` and `Policy` keeping their 2.x shape once the facade is gone, is answered by the 2026-10-28 3.0 freeze: both keep it, with additions. Weighed against: `Parser(languages=…)` beside `Locale`, two ways to choose one thing, and a `Policy` field, `Policy` being behavior and vocabulary being `Lexicon`'s. 2. **One mechanism.** The chosen languages select which readings the compiler compiles, the default being that compiler with every language and no choice. Turning on, prefer and strict are derived from the file each reading sits in, and nothing is written per pack beyond its language's readings: a chosen language's `default = false` readings join; prefer drops other languages' vocabulary readings of a word on which a chosen language holds one, never `mul.toml`'s (`Haddad, Abd`, suffix `Abd` today, reads given `Abd` with the en credential gone); strict drops other languages' name readings and the markers they derive. A clash a property records with a SHAPE reading is settled for the chosen language's own reading, decision 20 applied: `may_be_initial` does not hold on it, so a `pt` choice reads `ANA SILVA E SOUZA` family `SILVA E SOUZA` where the default reads middle `SILVA E` with a report. This answers decisions/3-0-reevaluations' 2026-09-13 bullet that a pack cannot remove `e`, and its 2026-08-17 bullet's "overrides rather than additions": a pack overrides by selection, never by a removal list. That removes base entries all the same, so mechanisms.md#LOCALE-PACKS-PURE-DATA's add-only contract is amended, in 3b's PR, which builds the selection (items 11 and 12), not by this decision. The shape-clash rule follows from decision 7, which decision 20 permits; it reaches every reading `may_be_initial` stands on (pt and it `e`, ca and pl `i`, the Italian one unjudged), and it forecloses an `nl` pack marking `y` (the 2026-08-18 bullet there), which waits with `fru` below. New beside decision 6: `mul.toml`'s exemption from prefer. Left open rather than built: a property a language's own reading should take only under its pack, Swedish `fru` and Icelandic `frú` as after-given salutations (rules/H8), refused in the default for the Cameroonian surname, the two readings in the inventory needing it. 3. **`default = false` is allowed on `given` and `family` readings, and turned on they derive markers like any name reading, never settling (decision 15 unchanged).** Measured on nepal-candidates' 126,981 Devanagari and romanized pairs, comparing every word's role across the two spellings: 123,440 read alike in the default; `given` readings on `श्री`, `guru` and `गुरु`, which make each a title only inside a title phrase (rules/H9), give 123,636 (198 fixed, 2 broken, both pairs whose romanization opens with a listed title); a `family` reading on `dc` as well, making the credential ambiguous, gives 123,639, the count still leaving `Ram Bahadur DC` a suffix; dropping `dc`'s credential, which settling by a name reading would do, gives 123,658, 19 more. Settling is not taken: the 19 pairs are the accepted price of decision 15, whose `do` case it protects. `bk`, `kc`, `gc` and `bc` read as the family in every shape tried (spoken order, a family comma, `FAMILY_FIRST`, dotted and Title case), so a reading would move nothing and none is added. Recompute: the readings moved into or out of `Lexicon` fields by hand, the pairs compared by `tuple(t.role for t in parser.parse(s).tokens)`. - 4. **Strict waits, out of the first packs, until name readings say whose names they are.** A name reading sits in the file whose source counted it: 113 of the 211 name readings are in en.toml, `do`'s Vietnamese Đỗ readings among them, vi.toml holding no `do`, so strict built on the files drops them under a `vi` choice, and simulated, `Minh Do` reads suffix `Do` and `Do Van Minh` family `Do Van Minh`, decision 15's harm. A census count belongs to no one language either. The candidates needing strict wait with it, `nl`'s prepositions among them, which need more than strict: turned on as possibly-given particles, `in` and `to` read `Park In Hwan` family `In Hwan` and `Nguyen To Nga` family `To Nga` (the marker acts at the head only), and check 11 accepts `may_be_given = false` in place of the Korean and Vietnamese readings. Recompute: count `given` and `family` readings per file with tools/vocab/build.py's `load`; simulate by removing `do` from `suffix_acronyms_ambiguous` and `particles_ambiguous`. Open: none of 3a–3h says whose names they are. Moving readings between files would not, a word being borne in several languages (`do` Vietnamese and Korean, `kim` Korean and Scandinavian), so the RFC points at an attribute on each name reading naming its bearers' languages, compiling to nothing until strict reads it, the first provenance anything would compute a reading from. It cannot be filled by script: 172 of the 211 count a US source and 21 none, per-country evidence is note prose in 40, and a box count says where a word was typed, not whose name it is (`do`'s ORCID family box is led by Brazil, 7,227 against Vietnam's 876, none of Brazil's `do` alone against Vietnam's 618; by share Vietnam leads, 1.90% to 0.81%; 2026-10-28, master `c8357e74`). The attribute names languages, so no country-to-language table is read at run time (item 6). A sub-phase of its own, 3i, after 3a, independent of the output model, moving no default parse; strict after it. + 4. **Strict waits, out of the first packs, until name readings say whose names they are.** A name reading sits in the file whose source counted it: 129 of the 227 name readings are in en.toml (2026-10-28, `c8357e74`), `do`'s Vietnamese Đỗ readings among them, vi.toml holding no `do`, so strict built on the files drops them under a `vi` choice, and simulated, `Minh Do` reads suffix `Do` and `Do Van Minh` family `Do Van Minh`, decision 15's harm. A census count belongs to no one language either. The candidates needing strict wait with it, `nl`'s prepositions among them, which need more than strict: turned on as possibly-given particles, `in` and `to` read `Park In Hwan` family `In Hwan` and `Nguyen To Nga` family `To Nga` (the marker acts at the head only), and check 11 accepts `may_be_given = false` in place of the Korean and Vietnamese readings. Recompute: count `given` and `family` readings per file with tools/vocab/build.py's `load`; simulate by removing `do` from `suffix_acronyms_ambiguous` and `particles_ambiguous`. Open: none of 3a–3h says whose names they are. Moving readings between files would not, a word being borne in several languages (`do` Vietnamese and Korean, `kim` Korean and Scandinavian), so the RFC points at an attribute on each name reading naming its bearers' languages, compiling to nothing until strict reads it, the first provenance anything would compute a reading from. It cannot be filled by script: 188 of the 227 count a source under a `us_` key and 21 none, per-country evidence is note prose, 40 of the notes naming nam_dict, and a box count says where a word was typed, not whose name it is (`do`'s ORCID family box is led by Brazil, 7,227 against Vietnam's 876, none of Brazil's `do` alone against Vietnam's 618; by share Vietnam leads, 1.90% to 0.81%; 2026-10-28, master `c8357e74`). The attribute names languages, so no country-to-language table is read at run time (item 6). A sub-phase of its own, 3i, after 3a, independent of the output model, moving no default parse; strict after it. 5. **The languages reach run time as a reading index.** A second generated module, one row per reading (text, tag, kind, behavioral properties), loaded for every caller with `Lexicon.default()`, since phase 4's `Reading.languages` (4a) is filled from it on every parse, rather than on first access to `reading.languages`, which would need a `Reading`, which holds no word, to carry one as a hidden key; the compiler moves from tools/vocab/build.py into the package so build and pack share one implementation, and a test holds that compiling the index with no choice reproduces `_compiled.py`. It is the per-word index item 13 of the 2026-10-23 #752 entry says `Token.reading` needs. A choice is applied as a delta against the default's compile, so it composes with a caller's `base=` lexicon and a `Locale`'s own fragment. Measured: 2,140 readings over 1,797 words, 106 in more than one file; the index about 93 KB of source, about 0.45 ms to import with its bytecode cached against about 20 ms for a cached `import nameparser` (2026-10-28, `c8357e74`; about 11 ms without the cache, the 2026-10-27 figure, taken under `PYTHONDONTWRITEBYTECODE`), once per process for every caller; the compile 9 ms, once per choice and only for a caller who makes one; loading the TOML 39 ms. Weighed against: per-language deltas compiled at build time, which do not compose (under prefer `{pt, en}` `do` keeps its particle and its credential, while Portuguese's delta drops the credential and English's the particle); every combination precompiled, combinatorial; the TOML read at run time, slower and either re-running the build's checks or skipping them. 6. **A chosen tag selects its file and its ancestors', never its descendants, and no macrolanguage table is read.** `pt-BR` selects `pt-BR.toml` and `pt.toml`; `pt` only `pt.toml`, so a caller in Portugal does not turn on the Brazilian agnomens (#843). The two files share no word today. Malay and Indonesian, which share `puan`, `putra` and `putri`, are chosen together by a caller who has both. The TOML names `prefer` and `strict` stay refused, the message saying a strength is the caller's choice and never data. 7. **A pack is measured three ways.** The default does not move (the committed snapshot, and the index invariant of item 5); a vocabulary pack's deviation surface, the words whose compiled readings its choice changes, is derived from the index, and tests/v2/test_locales.py's non-interference gate requires the default's reading for every corpus name without such a word, the policy and segmentation packs (JA, RU, TR_AZ, ZH) keeping their written `DEVIATES`; and `snapshot.py`, `--local` with `--base` included, takes a choice and prints what it moves by field, beside each pack's own ground truth (nepal-candidates' script pairs for `ne`, the CLDR agreement per pack, PSC's nationality slice). This answers decisions/3-0-reevaluations' 2026-09-13 bullet that the harness cannot see a pack. 8. **Third-party language files are deferred past phase 3.** A file joining the pool at run time would skip the checks that run across files (check 6's exclusions, check 11's particle statement), the ones that refuse a third-party `nl` `in` written with neither a `given` reading nor `may_be_given = false`. `Locale(lexicon=…)`, or its successor, stays the way to add vocabulary, that successor being configuration by reading (the 2026-10-28 #754 entry on the `Lexicon` surface). 9. **Every exclusion a pack turns on becomes a `default = false` reading,** the shape cs `st` already has, so that once the packs land an `excluded` key means wrong everywhere: the eight whose reason opens "pack-shaped" in 3c (cs `ml`, hr, sk and sl `st` and `ml`, fr `père`), and the rest in the PR of the pack that turns each on (`pan`, `roz`, `nic`, `shri`, `sig`, `jaun`, `bt`, `bay`, `sb`). Recompute: `grep -h 'pack-shaped' nameparser/vocab/*.toml | wc -l`. 10. **What stays out of phase 3.** ZH's surname list and the RU and TR_AZ endings are rule data for phase 4's `[rules]` (its 4f), and a Python pack gains `languages` only where a measurement shows what it moves; the candidates that wait on a position rule (Turkish `Bey` and `Hanım`, Hungarian `úr`, Thai `นาย`, French `M.`) are phase 4's; same-language clashes (`kumari`, `sayyid`) take phase 2j's route in the default, since no pack settles one. - 11. **The work order,** shape first and one PR each, measured on its own base: 3a the index and shared compiler with name readings allowed `default = false` (neutral); 3b the choice, tag selection, turning on, prefer, the shape clash, the derived deviation surface, `snapshot.py`'s choice and the amended contract (neutral); 3c the pack-shaped exclusions converted (neutral); then the packs, `ne` (3d), `hi` and `bn`'s Latin forms (3e), `cs`, `sk`, `sl` and `hr` (3f), `es` and `pt` with a connective that counts only inside a title run (3g), and `ga`/`gd` and `pt-BR` (3h); 3i, the bearers' languages of each name reading (item 4), after 3a, ahead of strict. 3g's connective is a rule with its own design, open, not decided with the work order: with `profesora` added and no rule, `Profesora de Historia Ana López` reads family `de Historia Ana López` (the rules/P1 fold) and `Ministro da Fazenda Carlos Souza` given `da`, while `Doctora de la Vega` must keep family `de la Vega`, the two separated by whether a name stands behind the phrase; it must reach only readings a pack turns on. + 11. **The work order,** shape first and one PR each, measured on its own base: 3a the index and shared compiler with name readings allowed `default = false` (neutral); 3b the choice, tag selection, turning on, prefer, the shape clash, the derived deviation surface, `snapshot.py`'s choice and the amended contract (neutral); 3c the pack-shaped exclusions converted (neutral); then the packs, `ne` (3d), `hi` and `bn`'s Latin forms (3e), `cs`, `sk`, `sl` and `hr` (3f), `es` and `pt` with a connective that counts only inside a title run (3g), and `ga`/`gd` and `pt-BR` (3h); 3i, the bearers' languages of each name reading (item 4), after 3a, ahead of strict. 3g's connective is a rule with its own design, open, not decided with the work order: with `profesora` and `ministro` added as titles and no rule, `Profesora de Historia Ana López` reads family `de Historia Ana López` (the rules/P1 fold) and `Ministro da Fazenda Carlos Souza` given `da`, while `Doctora de la Vega` must keep family `de la Vega`, the two separated by whether a name stands behind the phrase; it must reach only readings a pack turns on. 12. **Phase 3 is 3.0 work, opt-in, the default's parses unmoved, with phase 4's shape questions settled first** (the 2026-10-28 re-plan: 2.8 is the last 2.x feature release, and 3.0 ships phases 3 and 4 with the facade's removal). The output model decides what a pack can express (a patronymic field, a second surname, #326's credentials apart from generations, #704's `middle`). The output model was agreed on 2026-10-28 (the 2026-10-28 #754 entry). 3a and 3c are independent of it, and so are 3b's selection semantics, its API's dependency answered (item 1); 3f and 3h need 4b's `generation` view, 3g needs 4d's `surname2` and 4g's compound given name, and 4f, which compiles `[rules]` into a `Locale`'s policy patch, lands with or after 3b; `ne` needs none. One standing text is amended with 3b, mechanisms.md#LOCALE-PACKS-PURE-DATA's contract that a pack is add-only and declares its deviations in its docstring (a pack adds, and selects among the default's readings by language, a vocabulary pack's deviations derived); decisions/3-0-reevaluations' Standing line that removing base vocabulary through a pack is part of 3.0's pack design is carried out. - **2026-10-28 #754 (Derek) — PHASE 4'S OUTPUT MODEL IS AGREED: THE SEVEN ROLES STAY AND EVERY NEW FIELD IS A VIEW, MOST READ FROM `Token.reading`; A PATRONYMIC NAMES ITS WORDS WITHOUT EMPTYING `family`; THE RENDER DEFAULTS FOLLOW THE ORDER A NAME WAS READ IN.** Derek agreed every item in chat on 2026-10-28, items 6 and 4 and the `surnames` line of item 5 first, then the rest. The design overview is docs/design/vocabulary-model-rfc.md's Phase 4 section. Decision 8 (a patronymic field in 3.0) is kept and its stated value revised (item 3); decisions 9 and 14 are kept as written. The core API is frozen for 3.0 except for additions (decisions/3-0-reevaluations, 2026-10-28 #925, THE CORE API), and each item says whether it is one. The inputs are #754 and its comment of 2026-10-23, #326, #704 and #694. Measurements were taken 2026-10-28 on master `c8357e74` (Python 3.11.17) over the fielded local corpora (`orcid-researchers`, `psc-individuals`, `patentsview-inventors`), the committed fields files and the committed corpus's distinct entries (`snapshot.corpus_entries()`, each parsed under its declared order), with throwaway scripts that are not kept, so each figure is evidence for the decision and each sub-phase re-measures on its own base. Local-corpus figures are counts. Recompute the box counts by reading each source's records with `local_corpora.read_records`, counting its boxes by word count and word ending per country or nationality, and the role the parse gives the counted word with `parse()`; the ending tests are the RFC's Phase 4 table's, and other tests give counts a few percent apart. 1. **`Role` keeps its seven members, and every field asked for is a view.** A role partitions the tokens, and each field asked for (a second surname, a patronymic, generation and credentials, a second given name) subdivides a role that exists, so each is computed from the tokens, a property and a render key, except the second given name, which is the `middle` that exists (item 7), as `family_base` and `given_names` are (rules/R1, rules/R2). `as_dict()`, `comparison_key()` and the seven role properties are unchanged, and every view is an addition. Weighed against: a `Role` member for any of them, a break of the frozen `ParsedName` (`as_dict()` gains a key, `comparison_key()` a component, so a stored key stops matching, and the role its words leave returns less text than in 2.x), with no 2.x feature release left to warn in. @@ -265,8 +265,8 @@ Open: #913 (the title words phase 2j left, read as titles though borne as names) 3. **A `patronymic` view, filled by a word or a rule that says patronymic, with `family` and `middle` unchanged.** The default fills it from the vocabulary's `patronymic` particles (`bin`, `binti`, `ibn`, `bint`, `bar`, `bat` and the native-script forms) with the name they introduce (`Mahathir bin Mohamad` → family `bin Mohamad`, as today, and patronymic `bin Mohamad`); RU's and TR_AZ's rules fill it where a caller turns them on (`Ivan Ivanovich Ivanov` under RU → middle `Ivanovich` and patronymic `Ivanovich`); an Icelandic ending rule fills it under an `is` pack only, since the default cannot tell `-dóttir` and `-son` from Swedish surnames (ORCID's Swedish records end their family box in `-son` or `-dóttir` in 2,184 of 20,000). A matronymic, written the same way, fills it too. SUPERSEDES: the RFC's earlier statement of decision 8's value, that `family` can be empty for a name with a patronymic. Measured: filers put an Icelandic patronymic in the family box (ORCID 1,647 of 2,583 Icelandic records, PatentsView 535 of 722, PSC 101 of 125), and CLDR's `is` file's 3 Icelandic samples put it in `surname` or `surname-core` in all 9 of their renderings (its one foreign sample puts a `-dóttir` word in `surname2`), while Wikidata's `patronymic-is` and `patronymic-my` strata record 108 names with a patronymic and no family name, of which the parse puts the patronymic in `family` 107 times. An East Slavic patronymic stands in the given part: CLDR's 2 `ru` samples put it in `given2` in all 6 renderings; PatentsView's Russian given boxes end in one 8,423 times of 20,000, and the parse reads such a word as the middle 10,902 times of 10,962; 90 of the 112 Russian nationals in PSC who filled the middle-name box wrote one there. A `family` that keeps it and a `patronymic` that names it agree with both kinds of source. Weighed against: emptying `family` where the patronymic is recognized, which moves the field the fielded sources fill and changes `family` for every Icelandic and nasab name, a behavior change of a frozen property; and a neutral parent-names name for the view, which buys nothing the view can detect, since a matronymic and a patronymic are written alike. 4. **`generation` and `credentials` are views that partition `suffix` (#326)**, not roles, `generation` the tokens read as a generation or an agnomen, `credentials` the rest, the post-nominals that are neither (`Esq.`, the honorifics after a name) falling in `credentials`, CLDR's catch-all, with `Token.reading.kind` telling them apart for a caller who needs it. Measured: of the 1,054 committed corpus entries with a suffix, 33 hold a generation (or agnomen) and a listed credential together, 77 a generation beside any other suffix word, and 64 only listed post-nominals of other kinds (each suffix word classified by the RFC's Phase 4 recipe, joined tokens merged); of the 615 CLDR renderings with either field, the 75 whose suffix holds exactly CLDR's words split into CLDR's `generation` and `credentials` by the vocabulary's reading kind in all 75. Weighed against: two new `Role` members (item 1), and a third view for the honorifics, left open since CLDR has none. 5. **a `surname2` view, the family's second unit where rules/O6 read the family as two**, marked by a new stable tag where O6 takes its word, empty otherwise; `family` keeps both units as O6 makes it, and a family comma alone marks nothing (`Lloyd Webber, Andrew`). Measured: a two-word family box in ORCID's records from Spain 48.0%, Mexico 64.7%, Colombia 72.8% and Chile 50.5%, against 3.9% from the United States. Left open: a view for the first unit and its name, `surname` standing close to the `surnames` view that exists. Declined: redefining `surnames` as the two units, which would repeat `family`, the family holding both where O6 reads two, and lose what `surnames` does today, joining `middle` and `family` so that a double surname the default splits (`Gabriel García Márquez` → middle `García`) reads whole, the view rules/P2 names for that reading. - 6. **The render defaults follow the order the name was read in, a behavior change of the output rather than a break of the API.** `ParsedName` gains `order` (an addition, `ADDED_IN`, a 2.x pickle rendering as today, left out of comparison, which lets two equal names render differently, a cost accepted 2026-10-28: a 2.x pickle of `김민준` renders `'민준 김'` and a fresh parse `'김민준'`, and the two compare equal); the default `str()` and `render()` spec put the family first where `order` does, and join two words that stand one directly after the other, in the same order, in the original and the rendering with the kind of separator that stood between them there, normalized, decided 2026-10-28 (nothing; one ASCII space for any whitespace, the ideographic space U+3000 included (`山田　太郎` → `'山田 太郎'`, where today gives `'太郎 山田'`); an interpunct or a nakaguro the parse read as a separator alone, a spaced interpunct being a name word, rules/T3; one comma the parse reads as `, ` if ASCII, `، ` if Arabic and alone if full-width, since a full-width comma carries its own width in CJK text, a reason for the comma the rendering keeps and not for the whitespace it normalizes (the Japanese comma U+3001 is no comma to the parse, a parser question outside this design), so `John Smith ,Jr.` renders `'John Smith, Jr.'`; the written whitespace staying in `ParsedName.original` and the token spans), the spec's decoration between the two words' fields standing instead (the nickname's quotes, the `) ` after a maiden name), and any other pair or separator (a delimiter, a consumed marker word, two or more commas, a hyphen, a slash, a zero-width or directional mark) taking the spec's own joiner; `initials()` gives nothing for a word in a script with no initials. A name with a family comma or written given-first keeps today's word order and its separators by the same rule, a suffix comma alone leaving a family-first name family-first (`威廉·莎士比亚` keeps its dot; 20 of the 36 names whose interpunct `str()` drops are read given-first). A family comma is never kept, the rendering reversing its pair (`Smith, John` → `'John Smith'`), and one directly between the family and its suffix is, decided 2026-10-28 (`John Smith, Jr.` → `'John Smith, Jr.'`, today `'John Smith Jr.'`; 214 of the 1,003 committed corpus entries whose suffix follows the family write one there). Accepted with it: `Smith, John, Jr.`, whose suffix the rendering moves away from the given name before it, keeps `'John Smith Jr.'`, so two writings of one name render differently. Measured over the committed corpus's distinct entries, the counts reading the ASCII comma (the RFC gives the full-width and Arabic ones: 36 of the 728 hold a family comma the parse reads and keep today's word order): 728 without a comma read family-first, `str()` renders all 728 differently from their text (`김민준` → `'민준 김'`), 150 of them with the family's last word touching the given name's first, and it drops the U+00B7 interpunct from 36 of the 41 holding one. This carries out decisions/3-0-reevaluations' render-default clean slate. Decided with it on 2026-10-28 (the RFC's Phase 4 Render defaults): `render(spec: str | None = None)`, `None` the order-aware default `str()` passes and a caller's spec rendering exactly as written, so `render("{given} {family}")` keeps its 2.x meaning and a caller passing 2.x's default string keeps given-first output. Decided too: a family-first layout with the title first, the suffix last and the name roles in `order`'s sequence, each with its decoration, a family-comma name keeping a given-first `order`. The signature is a change, not an addition as the freeze defines one (no working call changes meaning, `render(None)` raising today, but the default's introspection and a subclass override typed `spec: str` do), Derek taking it as worth that. The title stays first though 50 of CLDR's 70 surnameFirst renderings with a title put it after the surname, because the default renders only an order the parse reads back, and under a family-first order the parse reads a bare title only at the front (`Wang Dr Xiaoming` → given `Dr`, `Wang Xiaoming Senator` → middle `Senator`); reading a title where CLDR writes it is a parser question, #938, and the default may follow it. Apart from that signature, no signature changes, but `str()`'s output changes for every name read family-first, every name whose words are written with nothing, an interpunct or a nakaguro between two words the rendering keeps side by side, and every name with one comma directly between its family and its suffix, and no 2.x release can warn about a `str()` result, so it is taken as an improvement for 3.0's release notes, not a break needing a warning. - 7. **`middle` keeps its meaning (#704)**, every word of the given part after the given name, the given name one unit unless a rule joins words into it, as rules/P5 already does (`Abdul Aziz Tariq` → given `Abdul Aziz`); a compound given name is a `[rules]` rule a caller or a pack turns on, CLDR's `given2` is `middle`, and `Kaur` and an English name beside a Chinese one stay in `middle`, an accepted cost. Measured: in PSC, whose form has a middle-name box, 2,499 of 1,040,708 forename boxes hold two or more words, and of 6,101 Spanish nationals 25 do while 2,621 fill the middle box; ORCID's and PatentsView's given box holds middles too, so it cannot separate the two. The `Maria do Carmo` shape, a particle inside a given box of three or more words with a family box filled, reads the given box's words into the family in 1,812 of 1,923 ORCID records and 1,165 of 1,219 PatentsView records from Brazil, Portugal, Spain, Mexico, Argentina, Colombia, Chile and Italy, which is that rule's case. Weighed against: #704's option (b), one given unit by default, which changes `middle` for every English name of three words. + 6. **The render defaults follow the order the name was read in, a behavior change of the output rather than a break of the API.** `ParsedName` gains `order` (an addition, `ADDED_IN`, a 2.x pickle rendering as today, left out of comparison, which lets two equal names render differently, a cost accepted 2026-10-28: a 2.x pickle of `김민준` renders `'민준 김'` and a fresh parse `'김민준'`, and the two compare equal); the default `str()` and `render()` spec put the family first where `order` does, and join two words that stand one directly after the other, in the same order, in the original and the rendering with the kind of separator that stood between them there, normalized, decided 2026-10-28 (nothing; one ASCII space for any whitespace, the ideographic space U+3000 included (`山田　太郎` → `'山田 太郎'`, where today gives `'太郎 山田'`); an interpunct or a nakaguro the parse read as a separator alone, a spaced interpunct being a name word, rules/T3; one comma the parse reads as `, ` if ASCII, `، ` if Arabic and alone if full-width, since a full-width comma carries its own width in CJK text, a reason for the comma the rendering keeps and not for the whitespace it normalizes (the Japanese comma U+3001 is no comma to the parse, a parser question outside this design), so `John Smith ,Jr.` renders `'John Smith, Jr.'`; the written whitespace staying in `ParsedName.original` and the token spans), the spec's decoration between the two words' fields standing instead (the nickname's quotes, the `) ` after a maiden name), and any other pair or separator (a delimiter, a consumed marker word, two or more commas, a hyphen, a slash, a zero-width or directional mark) taking the spec's own joiner; `initials()` gives nothing for a word in a script with no initials. A name with a family comma or written given-first keeps today's word order and its separators by the same rule, a suffix comma alone leaving a family-first name family-first (`威廉·莎士比亚` keeps its dot; 20 of the 36 names whose interpunct `str()` drops are read given-first). A family comma is never kept, the rendering reversing its pair (`Smith, John` → `'John Smith'`), and one directly between the family and its suffix is, decided 2026-10-28 (`John Smith, Jr.` → `'John Smith, Jr.'`, today `'John Smith Jr.'`; 214 of the 1,003 committed corpus entries whose suffix follows the family write one there). Accepted with it: `Smith, John, Jr.`, whose suffix the rendering moves away from the given name before it, keeps `'John Smith Jr.'`, so two writings of one name render differently. Measured over the committed corpus's distinct entries, the counts reading the ASCII comma (the RFC gives the full-width and Arabic ones: 36 of the 728 hold a family comma the parse reads and keep today's word order): 728 without a comma read family-first, `str()` renders all 728 differently from their text (`김민준` → `'민준 김'`), 150 of them with the family's last word touching the given name's first, and it drops the U+00B7 interpunct from 36 of the 41 holding one. This carries out decisions/3-0-reevaluations' render-default clean slate. Decided with it on 2026-10-28 (the RFC's Phase 4 Render defaults): `render(spec: str | None = None)`, `None` the order-aware default `str()` passes and a caller's spec rendering exactly as written, so `render("{given} {family}")` keeps its 2.x meaning and a caller passing 2.x's default string keeps given-first output. Decided too: a family-first layout with the title first, the suffix last and the name roles in `order`'s sequence, each with its decoration, a family-comma name keeping a given-first `order`. The signature is a change, not an addition as the freeze defines one (no working call changes meaning, `render(None)` raising today, but the default's introspection and a subclass override typed `spec: str` do), Derek taking it as worth that and waiving its 2.x warning (decisions/3-0-reevaluations, THE RENDER SIGNATURE). The title stays first though 50 of CLDR's 70 surnameFirst renderings with a title put it after the surname, because the default renders only an order the parse reads back, and under a family-first order the parse reads a bare title only at the front (`Wang Dr Xiaoming` → given `Dr`, `Wang Xiaoming Senator` → middle `Senator`); reading a title where CLDR writes it is a parser question, #938, and the default may follow it. Apart from that signature, no signature changes, but `str()`'s output changes for every name read family-first, every name whose words are written with nothing, an interpunct or a nakaguro between two words the rendering keeps side by side, and every name with one comma directly between its family and its suffix, and no 2.x release can warn about a `str()` result, so it is taken as an improvement for 3.0's release notes, not a break needing a warning. + 7. **`middle` keeps its meaning (#704)**, every word of the given part after the given name, the given name one unit unless a rule joins words into it, as rules/P5 already does (`Abdul Aziz Tariq` → given `Abdul Aziz`); a compound given name is a `[rules]` rule a caller or a pack turns on, CLDR's `given2` is `middle`, and `Kaur` stays in `middle`, as under FAMILY_FIRST does an English name after a Chinese one (`Lim Mei Ling Jessica` → middle `Ling Jessica`), an accepted cost. Measured: in PSC, whose form has a middle-name box, 2,499 of 1,040,708 forename boxes hold two or more words, and of 6,101 Spanish nationals 25 do while 2,621 fill the middle box; ORCID's and PatentsView's given box holds middles too, so it cannot separate the two. The `Maria do Carmo` shape, a particle inside a given box of three or more words with a family box filled, reads the given box's words into the family in 1,812 of 1,923 ORCID records and 1,165 of 1,219 PatentsView records from Brazil, Portugal, Spain, Mexico, Argentina, Colombia, Chile and Italy, which is that rule's case. Weighed against: #704's option (b), one given unit by default, which changes `middle` for every English name of three words. 8. **A language file's `[rules]` compiles into its shipped `Locale`'s policy patch**, each rule kind a `Policy` field (an addition), so RU and TR_AZ are rebuilt from their files and `Locale` gains no field for rules. With it, **phase 3's open API dependency is answered**: its design (the 2026-10-27 #753 entry, item 1) carries a choice of languages on `Locale.languages`, assuming `Locale` and `Policy` keep their 2.x shape after the facade's removal, and the 2026-10-28 freeze keeps them so. The rest of that design, strict as an argument of the whole choice included, stays phase 3's to decide; its RFC section records the dependency as answered. 9. **The work order**, one PR each, measured on its own base, the snapshot recording each new view where it is non-empty: 4a `Token.reading` and `ReadingKind.PATRONYMIC` (neutral; `languages` after 3a); 4b `generation` and `credentials` (neutral); 4c `patronymic` (neutral); 4d `surname2` (neutral); 4e the render defaults and `order`; 4f `[rules]` into the policy patch, with or after 3b; 4g the compound given name, after 4f; 4h the shape inventory per order (#469). Interleaved with phase 3 (its sub-phases the 2026-10-27 #753 entry's item 11): 4a beside 3a; 4b to 4d before the packs that need them, `cs` (3f) and `pt-BR` (3h) on 4b, `es` and `pt` (3g) on 4d and 4g; an `is` pack on 3b, 4c and 4f; 3i, the bearers' languages, on 3a alone. 10. **What stays out of the output model.** Gender (decision 9) waits, `Token.reading` able to carry it later as one more addition. Names of given names only are a rule's question, not a field's: ORCID's empty family box holds 68,006 of 1,831,533 records, 15.9% to 18.7% of those from India, Pakistan, Bangladesh and Indonesia, and the parse fills `family` in 51,367 of the 51,415 with two or more given-box words. #694's two-word surname after a given name (`Tariq Abdul Aziz`) is a parse rule for the default, outside phase 4. Phase 1's four reserved names said to be phase 4's (`separators`, `bare_only`, `leading_only`, `trailing_only`) have no sub-phase yet. diff --git a/docs/design/rules/R2-all-particles-family-base-divergence.md b/docs/design/rules/R2-all-particles-family-base-divergence.md index 69b307564..cbc6056f9 100644 --- a/docs/design/rules/R2-all-particles-family-base-divergence.md +++ b/docs/design/rules/R2-all-particles-family-base-divergence.md @@ -25,6 +25,11 @@ "Juan van der" → family_base="van der" "Juan van der" → family_particles="" ``` +- **A connective standing between two words of the part is set aside when the part is asked whether every word is particle vocabulary** (P3): it is joining them, and a word doing a connective's work is never the part's base, so particles joined by a connective are all particles and anchor the base. A connective ending the part joins nothing and is a name word (R3), so the particle before it still works as one. + ``` + "Fritz Freiherr und von Bar" → family_base="Freiherr und von Bar" deviates: #939 (today: family_base="und") + "Juan de y" → family_base="y" · boundary + ``` **Accepted:** diff --git a/docs/design/rules/R3-initials-honor-the-fold.md b/docs/design/rules/R3-initials-honor-the-fold.md index 2a7d7e05f..6fca989a7 100644 --- a/docs/design/rules/R3-initials-honor-the-fold.md +++ b/docs/design/rules/R3-initials-honor-the-fold.md @@ -37,9 +37,10 @@ **Except:** -- **A part whose every word is a particle initials those particles**: they are not acting as particles there but are the part's base (R2), and initial like any other name word, so a family that is all particles contributes its words rather than nothing. +- **A part whose every word is a particle initials those particles**: they are not acting as particles there but are the part's base (R2), and initial like any other name word, so a family that is all particles contributes its words rather than nothing. A connective standing between them is set aside in asking it, as R2 sets it aside, and joins them, so it contributes nothing. ``` "Juan van der" → initials="J. v. d." + "Fritz Freiherr und von Bar" → initials="F. F. v. B." deviates: #939 (today: initials="F. u.") ``` **Accepted:** diff --git a/docs/design/vocabulary-model-rfc.md b/docs/design/vocabulary-model-rfc.md index 5acb51855..0f1422885 100644 --- a/docs/design/vocabulary-model-rfc.md +++ b/docs/design/vocabulary-model-rfc.md @@ -213,9 +213,9 @@ Properties ride on their readings, so a choice that keeps a reading keeps its pr The 2 broken are pairs whose romanization opens with a listed title, which reads the `श्री` as a salutation (#870 counted `Mr` written for it twice). So the gate route delivers the `श्री` and `guru` fix without amending decision 15, and the 19 `dc` pairs it leaves are the price of keeping it. `bk`, `kc`, `gc` and `bc` need nothing: they read as the family in every shape tried (`Ram Bahadur BK`, `Ram Bahadur Kc`, `BK, Ram Bahadur`, `KC Sita` under `FAMILY_FIRST`), so a reading would move nothing. -**Strict cannot be built on today's files.** A name reading sits in the file whose source counted it, not the file of the language whose names carry it: 113 of the 211 name readings are in en.toml (2026-10-27), `do`'s Vietnamese Đỗ readings among them, and vi.toml holds no `do`. A strict `vi` choice built on the files would drop exactly those readings, and with them the gate: simulated, `Minh Do` reads suffix `Do` and `Do Van Minh` family `Do Van Minh`, the harm decision 15 is there to prevent. Nor does a census count belong to one language. So strict waits (decided 2026-10-28), out of the first packs, until name readings say whose names they are, and the candidates that need it wait with it. The `nl` prepositions show they would need more than strict: turned on as possibly-given particles, `in` and `to` already read `Park In Hwan` family `In Hwan` and `Nguyen To Nga` family `To Nga`, the marker acting at the head only, and check 11 lets their author write `may_be_given = false` instead of declaring the Korean and Vietnamese readings. +**Strict cannot be built on today's files.** A name reading sits in the file whose source counted it, not the file of the language whose names carry it: 129 of the 227 name readings are in en.toml (2026-10-28, `c8357e74`), `do`'s Vietnamese Đỗ readings among them, and vi.toml holds no `do`. A strict `vi` choice built on the files would drop exactly those readings, and with them the gate: simulated, `Minh Do` reads suffix `Do` and `Do Van Minh` family `Do Van Minh`, the harm decision 15 is there to prevent. Nor does a census count belong to one language. So strict waits (decided 2026-10-28), out of the first packs, until name readings say whose names they are, and the candidates that need it wait with it. The `nl` prepositions show they would need more than strict: turned on as possibly-given particles, `in` and `to` already read `Park In Hwan` family `In Hwan` and `Nguyen To Nga` family `To Nga`, the marker acting at the head only, and check 11 lets their author write `may_be_given = false` instead of declaring the Korean and Vietnamese readings. -**Open: nothing in the work order says whose names a name reading's are.** Strict waits on it, and none of 3a–3h does it. Moving readings between files would not: a census count belongs to no one language, and a word is borne in several, `do` as a Vietnamese and a Korean name, `kim` as a Korean surname and a Scandinavian given name. What it points at is an attribute on each name reading naming the languages of its bearers (`do`'s family reading `vi` and `ko`; the key's name still to choose), compiling to nothing until strict reads it. The counts to fill it exist, but none fills it mechanically. Of the 211 name readings, 172 count a US source and 21 none, and what per-country evidence they carry is prose in a `note`, as `do`'s nam_dict bands are (Korea 8, Vietnam 6, the Netherlands 1), in 40 of them by a text search. `name_usage.py --boxes` gives `orcid-researchers`' and `patentsview-inventors`' counts by box and country, and a box count says where a word was typed, not whose name it is: `do`'s family box in ORCID is led by Brazil, 7,227 holders against Vietnam's 876, and none of Brazil's is `do` alone (2,629 open with it, 4,598 hold it inside), the particle of a Brazilian surname, where Vietnam has 618 boxes of `do` alone and Korea 151. A share of each country's holders ranks them the other way (Vietnam 1.90%, Brazil 0.81%; PatentsView 1.78% and 0.64%), and `kim` reads cleanly either way, its family box 17.67% of ORCID's Korean holders and 13.30% of PatentsView's, its given box 0.93% of PatentsView's Danish inventors (2026-10-28, master `c8357e74`). So the fill is a person reading each of the 157 words with name readings, by share and by whole box, not a script. Two further consequences: the attribute names languages, filled from countries by that reader, since a country-to-language table read at run time is the macrolanguage table the tag matching below refuses; and it is the first provenance anything computes a reading from, where counts today are evidence nothing reads. Decided with it: a sub-phase of its own, 3i, after 3a, independent of the output model and moving no default parse, with strict and the candidates waiting on it after it. Recompute: count the name readings per file and per count key with `tomllib` over `nameparser/vocab/*.toml`; `name_usage.py do kim --no-wikidata --boxes`; the whole-box split by reading the cached ORCID extract's `family` and `country` keys. +**Open: nothing in the work order says whose names a name reading's are.** Strict waits on it, and none of 3a–3h does it. Moving readings between files would not: a census count belongs to no one language, and a word is borne in several, `do` as a Vietnamese and a Korean name, `kim` as a Korean surname and a Scandinavian given name. What it points at is an attribute on each name reading naming the languages of its bearers (`do`'s family reading `vi` and `ko`; the key's name still to choose), compiling to nothing until strict reads it. The counts to fill it exist, but none fills it mechanically. Of the 227 name readings (2026-10-28, `c8357e74`), 188 count a source under a `us_` key and 21 none, and what per-country evidence they carry is prose in a `note`, as `do`'s nam_dict bands are (Korea 8, Vietnam 6, the Netherlands 1), the notes of 40 naming nam_dict. `name_usage.py --boxes` gives `orcid-researchers`' and `patentsview-inventors`' counts by box and country, and a box count says where a word was typed, not whose name it is: `do`'s family box in ORCID is led by Brazil, 7,227 holders against Vietnam's 876, and none of Brazil's is `do` alone (2,629 open with it, 4,598 hold it inside), the particle of a Brazilian surname, where Vietnam has 618 boxes of `do` alone and Korea 151. A share of each country's holders ranks them the other way (Vietnam 1.90%, Brazil 0.81%; PatentsView 1.78% and 0.64%), and `kim` reads cleanly either way, its family box 17.67% of ORCID's Korean holders and 13.30% of PatentsView's, its given box 0.93% of PatentsView's Danish inventors (2026-10-28, master `c8357e74`). So the fill is a person reading each word with a name reading (165 on 2026-10-28, `c8357e74`), by share and by whole box, not a script. Two further consequences: the attribute names languages, filled from countries by that reader, since a country-to-language table read at run time is the macrolanguage table the tag matching below refuses; and it is the first provenance anything computes a reading from, where counts today are evidence nothing reads. Decided with it: a sub-phase of its own, 3i, after 3a, independent of the output model and moving no default parse, with strict and the candidates waiting on it after it. Recompute: count the `given` and `family` readings per file, per count key and by whether their `note` names nam_dict, with `tomllib` over `nameparser/vocab/*.toml`; `name_usage.py do kim --no-wikidata --boxes`; the whole-box split by reading the cached ORCID extract's `family` and `country` keys. ### Where the provenance lives at runtime @@ -266,7 +266,7 @@ Shape first, then the packs, each one PR measured on its own base, as phase 2's | 3h | `ga` and `gd` (`nic`, `mhic`, `mic`, `uí`), `pt-BR` (`F`, `N`, `sb`); `nl`'s prepositions wait with strict | 3b, 4b | | 3i | the languages of each name reading's bearers, an attribute compiling to nothing, filled word by word from per-country shares and whole-box counts (above); strict and its candidates after it | 3a (neutral) | -**Open:** 3g's connective needs a design of its own, not decided with the work order, because the pair that separates it is a count: `Profesora de Historia Ana López` should read title `Profesora de Historia`, while `Doctora de la Vega`, the same title, preposition and following word, has no name behind it and must keep family `de la Vega`. With `profesora` added and no rule, the first reads family `de Historia Ana López` (the rules/P1 fold), and `Ministro da Fazenda Carlos Souza` reads given `da`. The rule must reach only readings a pack turns on, so the default's parses stay where they are. +**Open:** 3g's connective needs a design of its own, not decided with the work order, because the pair that separates it is a count: `Profesora de Historia Ana López` should read title `Profesora de Historia`, while `Doctora de la Vega`, the same title, preposition and following word, has no name behind it and must keep family `de la Vega`. With `profesora` and `ministro` added as titles and no rule, the first reads family `de Historia Ana López` (the rules/P1 fold), and `Ministro da Fazenda Carlos Souza` reads given `da`. The rule must reach only readings a pack turns on, so the default's parses stay where they are. Languages with one to three candidates (`tr` `bay`, `it` `sig`, `lt` `jaun`, `ms` `bt`, `cy` `ab`/`ap`, `ber` `aït`) are a file each once 3b exists and need no sub-phase; `nb` and `sv`'s farm-name words wait for a measurement nobody has taken. @@ -319,7 +319,7 @@ What decides a view is the reading each token took, so `Token.reading` comes fir - **Generation and credentials** (#326): `generation`, the suffix tokens whose reading is a generation or an agnomen (`Jr.`, `III`, `Filho`), and `credentials`, the rest of the suffix (`PhD`, `Esq.`, the honorifics after a name), so the two partition `suffix` and `suffix` is unchanged. CLDR has only these two buckets; the post-nominals that are neither (`Esq.`, the CJK and Hebrew honorifics, `Rinpoche`) fall in `credentials`, CLDR's catch-all, and a caller who wants them apart reads `Token.reading.kind`, which says `post-nominal`. Measured 2026-10-28 on `c8357e74`, over the committed corpus's 6,533 distinct entries (`snapshot.corpus_entries()`) each parsed under its declared order: of the 1,054 with a suffix, 33 hold a generation (or agnomen) and a listed credential together, 77 a generation beside any other suffix word, and 64 only listed post-nominals of other kinds. Of the 615 CLDR renderings with a generation or credentials field, 75 parse with exactly CLDR's suffix words, and in all 75 a split by the vocabulary's reading kind gives CLDR's `generation` and `credentials` both; the other 540 differ in which words the suffix holds, a parse question the split cannot fix. Recompute: classify each suffix word (a token tagged `joined` merged into the one before it) by the kinds of its suffix readings in nameparser/vocab/, matched lowercased with its periods and spaces removed, skipping `default = false` readings, a generation or agnomen reading winning over a credential and a credential over the other kinds, an unlisted roman numeral `i` to `x` a generation and any other unlisted word unlisted; over the corpus entries above, and over `cldr_fields.jsonl` for CLDR. - **A patronymic** (decision 8): `patronymic`, the tokens whose reading is `PATRONYMIC`, in whichever role the parse put them, with `family` and `middle` unchanged. What fills it is a word or a rule that says patronymic, never a guess from position: the vocabulary's `patronymic` particles (`bin`, `binti`, `ibn`, `bint`, Hebrew `bar` and `bat` and the native-script forms) with the name after them, in the default, so `Mahathir bin Mohamad` reads family `bin Mohamad` as today and patronymic `bin Mohamad` beside it; the East Slavic and Turkic rules where RU or TR_AZ turns them on, so `Ivan Ivanovich Ivanov` under RU keeps middle `Ivanovich` and adds patronymic `Ivanovich`; and an Icelandic ending rule an `is` pack would turn on, the default unable to tell `Karlsdóttir` from the Swedish surname `Johansson`. The name `patronymic` is kept for a matronymic too, which is written the same way and cannot be told from one. **`family` is not emptied** by recognizing a patronymic, revising what this section said before: the people filling in the fielded corpora put the patronymic in the family box (below), as CLDR's samples put an Icelandic patronymic in `surname` or `surname-core` (CLDR's `is` file's 3 Icelandic samples, in all 9 of their renderings; its one foreign sample puts a `-dóttir` word in `surname2`) and a Russian one in `given2` (its 2 `ru` samples, in all 6 renderings), our `middle`, while Wikidata records an Icelandic name as a patronymic with no family name; a `family` that keeps it and a `patronymic` that names it satisfy both. A caller writing "Mr. {family}" reads `patronymic` to see that the family is one. - **A second surname**: `surname2`, the family's second name unit where a rule has read the family as two (rules/O6, which a caller or an `es` pack turns on), and empty otherwise, `family` keeping both units as O6 makes it today. O6 already knows which unit it took, so it marks where the second one starts with a new stable tag; a family comma alone marks nothing, since `Lloyd Webber, Andrew` is not two surnames, and a hyphenated family is one token and stays whole. Whether the first unit gets a view of its own, and its name (`surname` sits close to the `surnames` that exists), is a value question below. `surnames` keeps its meaning, `middle` and `family` joined, the reading of a double surname the default splits. -- **`middle`** (#704): keeps its meaning, every word of the given part after the given name, and the given name is one name unit unless a rule joins words into it, as rules/P5 already does (`Abdul Aziz Tariq` → given `Abdul Aziz`). A compound given name (`Maria do Carmo`, `María José`) is the `[rules]` rule above, turned on by a caller or a pack, moving its words into `given` only under that rule. CLDR's `given2` is `middle`, so no view is added. A Sikh `Kaur` and an English name beside a Chinese one (`Lim Mei Ling Jessica`) stay in `middle`, an accepted cost. +- **`middle`** (#704): keeps its meaning, every word of the given part after the given name, and the given name is one name unit unless a rule joins words into it, as rules/P5 already does (`Abdul Aziz Tariq` → given `Abdul Aziz`). A compound given name (`Maria do Carmo`, `María José`) is the `[rules]` rule above, turned on by a caller or a pack, moving its words into `given` only under that rule. CLDR's `given2` is `middle`, so no view is added. A Sikh `Kaur` stays in `middle`, and so, under FAMILY_FIRST, does an English name written after a Chinese one (`Lim Mei Ling Jessica` → given `Mei`, middle `Ling Jessica`), an accepted cost. **Measured in the fielded corpora** (2026-10-28, master `c8357e74`, counts only; recompute by counting each record's boxes by word count and word ending with `local_corpora.read_records`, grouped by the record's country or nationality, and the parse's role for the counted word with `parse()`. The ending tests, case-insensitive on the box's last word: an Icelandic patronymic a one-word family box ending `dóttir`, `dottir`, `sson` or `son` (the role count, `dóttir`, `dottir` or `sson`); an East Slavic patronymic `ovich`, `ovna`, `evich`, `evna`, `ichna` or their Cyrillic forms, in a given box of two or more words; nasab `bin`, `binti`, `bint`, `bte`, `bt`, `b.`, `ibn` or `ben` anywhere in the box. Other tests give other counts by a few percent): @@ -339,7 +339,7 @@ ORCID's and PatentsView's given box holds the middle names too, so a two-word gi - `ParsedName` gains `order`, the order the name was read under (its script's or the declared `name_order`, a family comma aside), an addition with `ADDED_IN` metadata whose missing value, in a 2.x pickle, renders as today, and left out of comparison as `Token.reading` is, so a loaded 2.x pickle still equals a fresh parse of its text. The cost, accepted 2026-10-28: two equal names can render differently, where today every rendering is derived from what `==` compares (a 2.x pickle of `김민준` renders `'민준 김'` and a fresh parse `'김민준'`, and a set keeps one of them); the views read from `Token.reading` share it. - `str()` and `render()`'s default spec put the family first where `order` does, and two words that stand one directly after the other, in the same order, in the original and in the rendering keep the kind of separator that stood between them in the original, written in a normalized form (decided 2026-10-28): nothing stays nothing; whitespace of any width or length becomes one ASCII space, the ideographic space U+3000 included (decided 2026-10-28: `山田　太郎` renders `'山田 太郎'`, the same ASCII space `str()` joins with today, where today's order gives `'太郎 山田'`, and a caller wanting native spacing passes a format spec of their own or reads `ParsedName.original`); an interpunct or a nakaguro the parse read as a separator stands alone, with no space around it (an interpunct written with spaces round it is not one, rules/T3 dividing at it only when flanked, so `威廉 · 莎士比亚` reads the dot as a name word and keeps rendering with its spaces, where `威廉·莎士比亚` keeps the dot unspaced; a spaced nakaguro is a separator, so `ジョン ・ スミス` renders `'ジョン・スミス'`); and one comma the parse reads as one becomes `, ` if ASCII and `، ` if Arabic (U+060C), no space before it and one after, and stands alone if full-width (U+FF0C), which carries its own width in CJK text, a reason that holds for the comma, a mark the rendering keeps, and not for whitespace, which the rendering normalizes, so `John Smith ,Jr.` and `John Smith, Jr.` render `'John Smith, Jr.'` and `John Smith， Jr.` renders `'John Smith，Jr.'`. The Japanese comma 、 (U+3001) is not one the parse reads as a comma, a parser question outside this design. Over that, the spec's decoration between the two fields the words belong to stands in place of the separator (the nickname's quotes, the `) ` after the maiden name, so `Jane Smith née Jones, PhD` keeps `'Jane Smith (Jones) PhD'`; 38 committed entries put a suffix directly after an undelimited maiden name); and any other pair, or any other separator (a delimiter, a marker word the parse consumed, two or more commas, a hyphen, a slash, a zero-width or directional mark), takes the spec's own joiner, as today (the written whitespace is not lost: `ParsedName.original` and the token spans keep it), so `John (Jack) Smith` keeps today's quoted nickname and `str(parse("김민준"))` is `'김민준'` and `str(parse("威廉·莎士比亚"))` keeps the dot. A name with a family comma or written given-first keeps today's word order, and its separators are kept by the same rule (a comma before a suffix alone does not make a family-first name given-first: `Wang Xiaoming, Jr.` under FAMILY_FIRST renders `'Wang Xiaoming, Jr.'`), so the given-first transcription `威廉·莎士比亚` keeps its dot too (20 of the 36 names whose interpunct `str()` drops are read given-first, 4 of them with a comma the parse reads (1 ASCII, 3 full-width)). A family comma is never kept, since the rendering reverses the pair it stands between (`Smith, John` → `'John Smith'`), while a comma directly between the family and its suffix is (`John Smith, Jr.` → `'John Smith, Jr.'`, where `str()` gives `'John Smith Jr.'` today): of the 1,003 committed corpus entries whose suffix starts after the family's last word ends, 214 have nothing between the two but one comma and whitespace (2026-10-28, `c8357e74`, each entry under its declared order; these counts read the ASCII comma, and 7 more write a full-width one there, while 36 of the 728 family-first names above hold a family comma the parse reads, 5 full-width and 31 Arabic (30 of the Arabic in `corpus_cldr.jsonl`), and so keep today's word order under the rule); 456 hold a comma somewhere between, most with a given name or title standing there too. Pairs the rendering separates, as `John` and `Jr.` in `Smith, John, Jr.`, take the spec's own joiner, as today, so `Smith, John, Jr.` keeps rendering `'John Smith Jr.'` while `John Smith, Jr.` becomes `'John Smith, Jr.'`: two writings of one name rendering differently, a consequence of following the text accepted with the choice. -- **How, decided 2026-10-28.** Today `str(name)` is `name.render()`, whose default is one fixed given-first spec, `'{title} {given} "{nickname}" {middle} {family} ({maiden}) {suffix}'`, which is why `김민준` renders `'민준 김'`; a default that reads `order` and the written separators cannot stay one string. So `render(spec: str | None = None)`: `None`, what `str()` passes, is the order-aware default, and a spec a caller writes renders exactly as written, its order and its literal joiners, with no separator rule applied, so `render("{given} {family}")` means in 3.0 what it meant in 2.x. That is a signature change, not an addition as the freeze defines one (decisions/3-0-reevaluations, THE CORE API), and Derek took it on 2026-10-28: no call that works today changes meaning, since `render(None)` raises `TypeError` today, but introspecting the default gives `None` rather than a spec, and a subclass overriding `render(self, spec: str = ...)` becomes an incompatible override to a type checker. The default's layout is picked by `order`, and a name with a family comma has a given-first `order` ("a family comma aside", above), so a given-first name and a family-comma name render by today's spec with the separator rule (`Smith, John` under FAMILY_FIRST still renders `'John Smith'`). For a family-first `order` the layout puts the title first and the suffix last, as the given-first spec does, and the three name roles between them in `order`'s own sequence, each keeping its decoration beside it: `'{title} {family} ({maiden}) {given} "{nickname}" {middle} {suffix}'` for FAMILY_FIRST and `'{title} {family} ({maiden}) {middle} {given} "{nickname}" {suffix}'` for FAMILY_FIRST_GIVEN_LAST, the separator rule then joining adjacent words (`王小明先生`, read family `王小明` and suffix `先生`, renders `'王小明先生'` where today gives `'王小明 先生'`; `Dr. Wang Xiaoming Jr.` under FAMILY_FIRST renders `'Dr. Wang Xiaoming Jr.'` where today gives `'Dr. Xiaoming Wang Jr.'`). CLDR's surnameFirst samples (`tools/differential/cldr_fields.jsonl`, release 48) agree on the core order and not on the rest: `ja` (`佐藤グレース恵子さん`) and `vi` (`ĐBQH Nguyễn Văn Anh`) put the second given name before the first, as FAMILY_FIRST_GIVEN_LAST does, and `ko` (`김민규헨리`) puts it after, as FAMILY_FIRST does; but most put the title after the family, not first: `pt` after it (`da Silva Sr. José Pedro Eduardo Jr., MP`), `ja` and `zh` last, and over every file's nativeFull surnameFirst rendering holding a title and a surname, 50 of 70 put the title after the surname (2026-10-28, positions of the field values in the text); `zh` also puts the generation first (`小彭德威东升议员先生`), and `vi` the credentials first. Keeping the title first, where the given-first default and the written name have it, goes against most of CLDR, but it is the one place the parse reads every title back from under a family-first order: `Senator Wang Xiaoming` reads title `Senator`, while `Wang Dr Xiaoming` reads given `Dr` and `Wang Xiaoming Senator` middle `Senator`, a bare title ending a name being a name word (rules/H5), only a period-marked one (`Wang Xiaoming Dr.`) reading as the title there (2026-10-28, under `Policy(name_order=FAMILY_FIRST)`). Derek decided on 2026-10-28 that the default renders only an order the parse reads back, so the title stays first; reading a title where CLDR writes it is the parser's question, #938, and the default may follow once the parse reads it there. A locale's own placement is the render patterns' job (below). The suffix last agrees with `ja` and `zh`, whose honorifics the parse reads as suffixes. For 3.0's notes: a caller who copied 2.x's default string and passes it keeps given-first, space-joined output; only a caller who passes nothing, `str()` included, gets the new default. `None` leaves both answers to how a pack's pattern is reached as open as they were: a spec string still means a spec, so `render("sorting")` still returns `'sorting'`, and a pattern is asked for by a new method or argument under (a) or through the parser under (b). +- **How, decided 2026-10-28.** Today `str(name)` is `name.render()`, whose default is one fixed given-first spec, `'{title} {given} "{nickname}" {middle} {family} ({maiden}) {suffix}'`, which is why `김민준` renders `'민준 김'`; a default that reads `order` and the written separators cannot stay one string. So `render(spec: str | None = None)`: `None`, what `str()` passes, is the order-aware default, and a spec a caller writes renders exactly as written, its order and its literal joiners, with no separator rule applied, so `render("{given} {family}")` means in 3.0 what it meant in 2.x. That is a signature change, not an addition as the freeze defines one (decisions/3-0-reevaluations, THE CORE API), and Derek took it on 2026-10-28, waiving the 2.x warning the freeze asks of a break (decisions/3-0-reevaluations, THE RENDER SIGNATURE): no call that works today changes meaning, since `render(None)` raises `TypeError` today, but introspecting the default gives `None` rather than a spec, and a subclass overriding `render(self, spec: str = ...)` becomes an incompatible override to a type checker. The default's layout is picked by `order`, and a name with a family comma has a given-first `order` ("a family comma aside", above), so a given-first name and a family-comma name render by today's spec with the separator rule (`Smith, John` under FAMILY_FIRST still renders `'John Smith'`). For a family-first `order` the layout puts the title first and the suffix last, as the given-first spec does, and the three name roles between them in `order`'s own sequence, each keeping its decoration beside it: `'{title} {family} ({maiden}) {given} "{nickname}" {middle} {suffix}'` for FAMILY_FIRST and `'{title} {family} ({maiden}) {middle} {given} "{nickname}" {suffix}'` for FAMILY_FIRST_GIVEN_LAST, the separator rule then joining adjacent words (`王小明先生`, read family `王小明` and suffix `先生`, renders `'王小明先生'` where today gives `'王小明 先生'`; `Dr. Wang Xiaoming Jr.` under FAMILY_FIRST renders `'Dr. Wang Xiaoming Jr.'` where today gives `'Dr. Xiaoming Wang Jr.'`). CLDR's surnameFirst samples (`tools/differential/cldr_fields.jsonl`, release 48) agree on the core order and not on the rest: `ja` (`佐藤グレース恵子さん`) and `vi` (`ĐBQH Nguyễn Văn Anh`) put the second given name before the first, as FAMILY_FIRST_GIVEN_LAST does, and `ko` (`김민규헨리`) puts it after, as FAMILY_FIRST does; but most put the title after the family, not first: `pt` after it (`da Silva Sr. José Pedro Eduardo Jr., MP`), `ja` and `zh` last, and over every file's nativeFull surnameFirst rendering holding a title and a surname, 50 of 70 put the title after the surname (2026-10-28, positions of the field values in the text); `zh` also puts the generation first (`小彭德威东升议员先生`), and `vi` the credentials first. Keeping the title first, where the given-first default and the written name have it, goes against most of CLDR, but it is the one place the parse reads every title back from under a family-first order: `Senator Wang Xiaoming` reads title `Senator`, while `Wang Dr Xiaoming` reads given `Dr` and `Wang Xiaoming Senator` middle `Senator`, a bare title ending a name being a name word (rules/H5), only a period-marked one (`Wang Xiaoming Dr.`) reading as the title there (2026-10-28, under `Policy(name_order=FAMILY_FIRST)`). Derek decided on 2026-10-28 that the default renders only an order the parse reads back, so the title stays first; reading a title where CLDR writes it is the parser's question, #938, and the default may follow once the parse reads it there. A locale's own placement is the render patterns' job (below). The suffix last agrees with `ja` and `zh`, whose honorifics the parse reads as suffixes. For 3.0's notes: a caller who copied 2.x's default string and passes it keeps given-first, space-joined output; only a caller who passes nothing, `str()` included, gets the new default. `None` leaves both answers to how a pack's pattern is reached as open as they were: a spec string still means a spec, so `render("sorting")` still returns `'sorting'`, and a pattern is asked for by a new method or argument under (a) or through the parser under (b). - `initials()` gives nothing for a word in a script with no initials (Han, Hangul, kana, `_policy._NO_INITIALS`), where today it gives the first character with a period (`'王.'`). Apart from that signature, these change no signature but change `str()`'s output for every name read family-first, every name whose words are written with nothing, an interpunct or a nakaguro between two words the rendering keeps side by side (not `김, 민준씨`, whose family comma moves `김` between `민준` and `씨`), and every name written with one comma directly between its family and its suffix, which no 2.x release can warn about; Derek took them as an improvement for 3.0's release notes rather than a break (item 6 of the entry). @@ -355,7 +355,7 @@ Apart from that signature, these change no signature but change `str()`'s output | new `Policy` fields for rules, and `Locale.languages` (phase 3) | additions | | `Role`, `as_dict()`, `comparison_key()`, the seven role properties | unchanged | | the default `str()`, `render()` spec and `initials()` output | behavior change, accepted for 3.0 | -| `render(spec: str \| None = None)` | a signature change, not an addition; decided 2026-10-28 | +| `render(spec: str \| None = None)` | a signature change, not an addition; decided 2026-10-28, its 2.x warning waived | | a `Role` member for any of these views | a break, declined | **Phase 3's open API dependency answered**: phase 3's design (above) carries a choice of languages on `Locale.languages` and notes it assumes `Locale` and `Policy` keep their 2.x shape once the facade is gone; the 2026-10-28 freeze answers that, both keeping it with additions. The rest of that item, strict as an argument of the whole choice included, is phase 3's to decide, and its section records the dependency as answered. With the `[rules]` proposal above, a shipped language pack is a `Locale` whose `languages` choose its vocabulary and whose `policy` carries its rules, so `Locale` needs no third field. diff --git a/tests/v2/rules_budget.json b/tests/v2/rules_budget.json index a9298b486..345b697ea 100644 --- a/tests/v2/rules_budget.json +++ b/tests/v2/rules_budget.json @@ -50,8 +50,8 @@ "P8": {"words": 515, "then_words": 450, "because_sentences": 2, "except": 0, "precedence": 0, "accepted": 2, "accepted_words": 309}, "P9": {"words": 405, "then_words": 265, "because_sentences": 1, "except": 2, "precedence": 0, "accepted": 0, "accepted_words": 0}, "R1": {"words": 176, "then_words": 145, "because_sentences": 1, "except": 0, "precedence": 0, "accepted": 1, "accepted_words": 146}, - "R2": {"words": 151, "then_words": 18, "because_sentences": 1, "except": 1, "precedence": 0, "accepted": 2, "accepted_words": 85}, - "R3": {"words": 483, "then_words": 407, "because_sentences": 2, "except": 1, "precedence": 0, "accepted": 3, "accepted_words": 409}, + "R2": {"words": 226, "then_words": 18, "because_sentences": 1, "except": 2, "precedence": 0, "accepted": 2, "accepted_words": 85}, + "R3": {"words": 506, "then_words": 407, "because_sentences": 2, "except": 1, "precedence": 0, "accepted": 3, "accepted_words": 409}, "R4": {"words": 991, "then_words": 729, "because_sentences": 1, "except": 4, "precedence": 0, "accepted": 4, "accepted_words": 600}, "R5": {"words": 250, "then_words": 154, "because_sentences": 2, "except": 0, "precedence": 0, "accepted": 0, "accepted_words": 0}, "S1": {"words": 306, "then_words": 26, "because_sentences": 1, "except": 2, "precedence": 3, "accepted": 0, "accepted_words": 0}, diff --git a/tools/differential/corpus_rules.jsonl b/tools/differential/corpus_rules.jsonl index beb2433f3..5fcf35049 100644 --- a/tools/differential/corpus_rules.jsonl +++ b/tools/differential/corpus_rules.jsonl @@ -213,6 +213,7 @@ "Freiherr von Berg MA Ma" "Freiherr von Berg, Ed" "Freiherr von Richthofen V" +"Fritz Freiherr und von Bar" "GRACE SMITH" "Gabriel García Márquez" "Gabriel García Márquez Jr." From 09b119ac5646e343bcbcd8c6d5417032b9937a06 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Wed, 28 Oct 2026 21:33:53 -0700 Subject: [PATCH 36/39] Carry the connective exception into R4 and narrow its wording (#939) - rules/R4's all-particles Except sets a connective between the part's words aside as R2 does, with `FRITZ FREIHERR UND VON BAR` marked deviates: #939 (today 'Fritz freiherr und von bar'). - rules/R2 and decisions/R2 say a joining connective anchors no base on its own but stays in the base view, instead of "is never the base", which its own example and R3's `Jon Dough and` contradict. - decisions/R2 names R4 beside R3, and its unjoined-mark Standing line is scoped to a connective ending the part. - rules_budget re-recorded; corpus_rules and the snapshot gain the new example, no recorded reading moved. Co-Authored-By: Claude Opus 5.5 --- .../decisions/R2-all-particles-family-base-divergence.md | 6 +++--- .../design/rules/R2-all-particles-family-base-divergence.md | 2 +- docs/design/rules/R4-case-repair-reads-the-unjoined-mark.md | 3 ++- tests/v2/rules_budget.json | 4 ++-- tools/differential/corpus_rules.jsonl | 1 + tools/differential/snapshot.jsonl | 1 + 6 files changed, 10 insertions(+), 7 deletions(-) diff --git a/docs/design/decisions/R2-all-particles-family-base-divergence.md b/docs/design/decisions/R2-all-particles-family-base-divergence.md index 90d3d6ea3..bfee2705d 100644 --- a/docs/design/decisions/R2-all-particles-family-base-divergence.md +++ b/docs/design/decisions/R2-all-particles-family-base-divergence.md @@ -8,8 +8,8 @@ - `initials()` reads the base, so the anchoring moves R3's output: `Anh Do` initials 'A. D.' and `Juan van der` 'J. v. d.' — 2026-08-16 (pre-merge coherence pass); 2026-08-18 #404/#385. - A particle run that joins nothing still chains whole into one part, so `Juan Smith van der` reads family 'van der' and middle 'Smith'; splitting it was declined, roles reassembling adjacent same-role pieces into one part and the base losing its full extent — 2026-08-18. - The base is the family without its working particles, wherever in the family they stand (`da Silva do Amaral de Souza, Miguel` → family_base 'Silva Amaral Souza', family_particles 'da do de'), a view the same under every convention and no culture's sort key — 2026-10-28 #754. -- A connective standing between two words of a part is set aside when the part is asked whether every word is particle vocabulary, so `Fritz Freiherr und von Bar` is all particles, base 'Freiherr und von Bar' and initials 'F. F. v. B.', where the parser reads base 'und', a tracked deviation; a connective ending the part is a name word, so `Juan de y` keeps base 'y' — 2026-10-28 #754 (Derek), A CONNECTIVE AMONG PARTICLES. -- The unjoined mark is a statement about a whole part, and `initials()` honors it for every word in it, a connective included: under `Lexicon.default().add(particles={'y'})`, `Juan de y` reads family_base 'de y' and initials 'J. d. y.'; narrowing the readmission to the `particle` tag alone was tried and backed out — 2026-08-29 #461; 2026-09-20 #461. +- A connective standing between two words of a part is set aside when the part is asked whether every word is particle vocabulary, so `Fritz Freiherr und von Bar` is all particles, base 'Freiherr und von Bar', initials 'F. F. v. B.' and case repair 'Fritz Freiherr und Von Bar', where the parser reads base 'und', a tracked deviation; a connective ending the part is a name word, so `Juan de y` keeps base 'y' — 2026-10-28 #754 (Derek), A CONNECTIVE AMONG PARTICLES. +- The unjoined mark is a statement about a whole part, and `initials()` honors it for every word in it, a connective ending the part included: under `Lexicon.default().add(particles={'y'})`, `Juan de y` reads family_base 'de y' and initials 'J. d. y.'; narrowing the readmission to the `particle` tag alone was tried and backed out — 2026-08-29 #461; 2026-09-20 #461. Elsewhere: where a connective initials, a part holding nothing else for it to join (`Jon Dough and` → 'J. D.', `Juan de y` → 'J. y.') — decisions/R3's 2026-09-20 #461 entry; case repair keeping that connective lowercase, its carve-out not gated on the mark (under `Lexicon.default().add(particles={'y'})`, `Juan de y` forced → family 'De y') — decisions/R4's 2026-08-29, the CONJUNCTION CARVE-OUT entry; case repair reading the unjoined mark for the whole part (`ANH DO` → 'Anh Do') — decisions/R4's 2026-08-29 #407 entry; the snapshot recording `initials()`, so a change to the base is visible there — decisions/differential-ledger's 2026-10-13 #664, a committed snapshot replaces the classified gate entry. @@ -41,7 +41,7 @@ Open: #754 (whether a language pack supplies a convention's sort key or entry el Measured 2026-10-28 on master `c8357e74`, counts and token shapes only, over the committed corpus's 6,533 distinct entries (`snapshot.corpus_entries()`, each parsed under its declared order) and `tse-donors`' 854,531 records (`local_corpora.read_records`, the filed name and the registry name each parsed in the default order). A family is read in render order (FOLDED_TAG tokens first), a working particle is a token tagged `particle` and not UNJOINED_TAG, and a family has an interior particle where a working particle stands after its first word that is neither a particle nor a conjunction; that family is where (1) and (2) differ. The three kinds of interior family are tested in this order, the first that fits counting: after a family comma, every interior particle standing after the first comma; with a leading run as well; with no leading run. In the default policy: committed, 24 against 684 with a leading run only (12 with no leading run, 8 whose interior particle stands after a family comma, 4 with a leading run as well); `tse-donors`, 12 filed names against 347,992 and 1 registry name against 392,134, since a Brazilian name's particles before its last word stand in `middle`. With `middle_as_family`: committed, 367 against 398 (262, 92 and 13); `tse-donors`' filed names, 268,591 against 79,926 (261,024, 5 and 7,562), the commonest family shape word-particle-word, 150,998 times. Recompute with those definitions; a change to the fold or to O6 moves them. Weighed against (2): with `middle_as_family`, the population the per-surname reading serves, it gives the 261,024 `tse-donors` names with no leading run a base equal to the whole family and an empty particle view, so `family_particles` stops naming the particles it is for; and it changes a frozen property's output for 24 committed entries in the default policy, where (1) changes no output. The facade whose `last_prefixes` it matches is removed in 3.0. Its one gain is a base that is a written name, which no convention's key needs either (below). The cost of (1), accepted: the base is not a name anyone writes ('Silva Amaral Souza'), nor any convention's sort key. Why the base is not a sort key, researched 2026-10-28. Portuguese and Brazilian practice files a name under its LAST surname, that surname's particle removed and set after the given names, the agnomens `Filho`, `Neto` and `Júnior` staying with the surname before them and an established compound surname kept whole: `SOUZA, Miguel da Silva do Amaral de`, `NOGUEIRA NETO, Wanderlino` (ABNT NBR 6023:2018, https://engenhariaeletrica.ufes.br/sites/engenhariaeletrica.ufes.br/files/field/anexo/abnt6023.2018.pdf; the ABNT Zotero style's comments, https://www.zotero.org/styles/universidade-federal-do-espirito-santo-abnt?source=1; USP's references manual, https://www.ip.usp.br/site/wp-content/uploads/2025/04/Manual-Referencias-ABNT-SabadiniSampaioNascimento.pdf; Cambridge University Library, https://www.lib.cam.ac.uk/node/3142; the UN cataloguing manual, https://www.un.org/Depts/dhl/unbisref_manual/nameguide/personal.htm; IFLA, Names of Persons: Portugal, 2009, https://ifla.org/files/assets/cataloguing/pubs/ifla_names_of_persons_portugal_2009.pdf). Spanish practice files under the FIRST surname, the second its secondary key (RAE, Ortografía de la lengua española, 2010, ch. VII, "Alfabetización de antropónimos", https://www.rae.es/ortografía/alfabetización-de-antropónimos, read 2026-10-28 in a browser, the page refusing a plain fetch): by its rule (a) a preposition, with or without an article, heading the FIRST surname is left out of the order and written after the given name (`Torre Ibarra, Ramón de la`); by rule (e) one heading the SECOND surname counts as part of the second key (`Gómez Casas`, `Gómez de la Iglesia`, `Gómez de Lara`, `Gómez del Castillo`, `Gómez Echagüe`, in that order); by rule (b) a leading article or `San`/`Santa` stays and is sorted on (`La Calle Ruiz, Carmen`); by rule (d) a `y` between the surnames counts (`Aguirre y Gómez-Pacheco`). The same page states the Portuguese convention from the other side, the paternal second surname being the axis. Neither convention's key is this base: Portuguese practice sorts `…da Silva de Souza do Amaral` before `…da Silva do Amaral de Souza` (Amaral before Souza) while both bases begin `Silva`, and Spanish practice keeps a second surname's particle (`Gómez de la Iglesia`) where the base drops it ('Gómez Iglesia'). The base dropping every working particle is nearest the Spanish first key alone for a preposition, rule (a) (`Torre Ibarra, Ramón de la` → base 'Torre Ibarra'), and departs from it on what rule (b) keeps: a leading article or `San` is a particle to the parse, so `La Calle Ruiz, Carmen` reads base 'Calle Ruiz' and `San Martín López, Juan` base 'Martín López'. So the base is the convention-neutral view, and a convention's key is built from the surnames by whoever knows the convention, the RFC's Phase 4 Left open. In the default policy, `Miguel da Silva do Amaral de Souza` already reads family 'de Souza', base 'Souza', which is the Portuguese entry element by coincidence: only `middle_as_family` gives the long family. -- 2026-10-28 #754 (Derek), A CONNECTIVE AMONG PARTICLES — R2 GAINS AN EXCEPT SETTING A JOINING CONNECTIVE ASIDE, AND THE PARSER'S READING IS A TRACKED DEVIATION (#939). The entry above asked R2's all-particles question of one trailing particle and left it as the code reads it; the design-docs review of that amendment found a second shape the code decides against R2's own Except ("anchor the base") and Accepted line ("A particle needs a base to attach to"): a connective between particles. `Fritz Freiherr und von Bar` reads family 'Freiherr und von Bar', family_base 'und', family_particles 'Freiherr von Bar', initials 'F. u.', while without `und` the family is all particles and its base is the whole of it. A conjunction alone became the base of a family of three particles, and the Then before the amendment ("without its leading particles") would have given 'und von Bar', so the code deviated, unmarked, before the amendment as well. Decided: a connective between two words of a part is set aside when the all-particles question is asked, as R3 sets the working particles aside when it asks whether a connective is joining, since a word doing a connective's work is no base; the particles then anchor the base, and the connective, joining them, contributes no initial (R3's Except amended with it: 'F. F. v. B.'). A connective ENDING the part stays out of the Except: it joins nothing, R3 already makes it a name word (`Juan de y` → base 'y', initials 'J. y.', decisions/R3's 2026-09-20 #461 entry), and setting it aside would make `de` the base and contradict that reading. Measured 2026-10-28 on `d97a910d`, counts and token shapes only, over the committed corpus's distinct entries (`snapshot.corpus_entries()`), each under its declared order: 3 families whose base holds only connectives in the default policy, 1 with the connective between particles (particle, connective, particle, particle), 1 ending in it (particle, connective) and 1 a lone connective, the last two outside the Except; the same 3 with `middle_as_family`. Recompute: a family whose `family_base` words are all family tokens tagged `conjunction`, split by whether the family's last token is one. +- 2026-10-28 #754 (Derek), A CONNECTIVE AMONG PARTICLES — R2 GAINS AN EXCEPT SETTING A JOINING CONNECTIVE ASIDE, AND THE PARSER'S READING IS A TRACKED DEVIATION (#939). The entry above asked R2's all-particles question of one trailing particle and left it as the code reads it; the design-docs review of that amendment found a second shape the code decides against R2's own Except ("anchor the base") and Accepted line ("A particle needs a base to attach to"): a connective between particles. `Fritz Freiherr und von Bar` reads family 'Freiherr und von Bar', family_base 'und', family_particles 'Freiherr von Bar', initials 'F. u.', while without `und` the family is all particles and its base is the whole of it. A conjunction alone became the base of a family of three particles, and the Then before the amendment ("without its leading particles") would have given 'und von Bar', so the code deviated, unmarked, before the amendment as well. Decided: a connective between two words of a part is set aside when the all-particles question is asked, as R3 sets the working particles aside when it asks whether a connective is joining, since a connective doing its work joins the words around it and anchors no base on its own, staying in the base view with them; the particles then anchor the base, and the connective, joining them, contributes no initial and keeps a connective's case (R3's Except amended with it, 'F. F. v. B.', and R4's, `FRITZ FREIHERR UND VON BAR` repairing to 'Fritz Freiherr und Von Bar' where today it gives 'Fritz freiherr und von bar'). A connective ENDING the part stays out of the Except: it joins nothing, R3 already makes it a name word (`Juan de y` → base 'y', initials 'J. y.', decisions/R3's 2026-09-20 #461 entry), and setting it aside would make `de` the base and contradict that reading. Measured 2026-10-28 on `d97a910d`, counts and token shapes only, over the committed corpus's distinct entries (`snapshot.corpus_entries()`), each under its declared order: 3 families whose base holds only connectives in the default policy, 1 with the connective between particles (particle, connective, particle, particle), 1 ending in it (particle, connective) and 1 a lone connective, the last two outside the Except; the same 3 with `middle_as_family`. Recompute: a family whose `family_base` words are all family tokens tagged `conjunction`, split by whether the family's last token is one. Declined: diff --git a/docs/design/rules/R2-all-particles-family-base-divergence.md b/docs/design/rules/R2-all-particles-family-base-divergence.md index cbc6056f9..bb85391f3 100644 --- a/docs/design/rules/R2-all-particles-family-base-divergence.md +++ b/docs/design/rules/R2-all-particles-family-base-divergence.md @@ -25,7 +25,7 @@ "Juan van der" → family_base="van der" "Juan van der" → family_particles="" ``` -- **A connective standing between two words of the part is set aside when the part is asked whether every word is particle vocabulary** (P3): it is joining them, and a word doing a connective's work is never the part's base, so particles joined by a connective are all particles and anchor the base. A connective ending the part joins nothing and is a name word (R3), so the particle before it still works as one. +- **A connective standing between two words of the part is set aside when the part is asked whether every word is particle vocabulary** (P3): it joins the words around it and anchors no base on its own, so it stays in the base view with them but does not stop the part from being all particles, and particles joined by a connective anchor the base. A connective ending the part joins nothing and is a name word (R3), so the particle before it still works as one. ``` "Fritz Freiherr und von Bar" → family_base="Freiherr und von Bar" deviates: #939 (today: family_base="und") "Juan de y" → family_base="y" · boundary diff --git a/docs/design/rules/R4-case-repair-reads-the-unjoined-mark.md b/docs/design/rules/R4-case-repair-reads-the-unjoined-mark.md index 150ed456e..4ddaa7708 100644 --- a/docs/design/rules/R4-case-repair-reads-the-unjoined-mark.md +++ b/docs/design/rules/R4-case-repair-reads-the-unjoined-mark.md @@ -104,10 +104,11 @@ **Except:** -- **A part whose every word is particle vocabulary** is repaired as ordinary name words, since none of them is doing a particle's work there (R2). +- **A part whose every word is particle vocabulary** is repaired as ordinary name words, since none of them is doing a particle's work there (R2). A connective standing between two of its words is set aside in asking it, as R2 sets it aside, and is repaired as the connective joining them that it is. ``` "ANH DO" → capitalized="Anh Do" "anh van do" → capitalized="Anh Van Do" + "FRITZ FREIHERR UND VON BAR" → capitalized="Fritz Freiherr und Von Bar" deviates: #939 (today: capitalized="Fritz freiherr und von bar") ``` - **A connective the parse read as the generation it also spells** is not a connective of this name at all, and is repaired as the generation it was read as. ``` diff --git a/tests/v2/rules_budget.json b/tests/v2/rules_budget.json index 345b697ea..d7c239606 100644 --- a/tests/v2/rules_budget.json +++ b/tests/v2/rules_budget.json @@ -50,9 +50,9 @@ "P8": {"words": 515, "then_words": 450, "because_sentences": 2, "except": 0, "precedence": 0, "accepted": 2, "accepted_words": 309}, "P9": {"words": 405, "then_words": 265, "because_sentences": 1, "except": 2, "precedence": 0, "accepted": 0, "accepted_words": 0}, "R1": {"words": 176, "then_words": 145, "because_sentences": 1, "except": 0, "precedence": 0, "accepted": 1, "accepted_words": 146}, - "R2": {"words": 226, "then_words": 18, "because_sentences": 1, "except": 2, "precedence": 0, "accepted": 2, "accepted_words": 85}, + "R2": {"words": 238, "then_words": 18, "because_sentences": 1, "except": 2, "precedence": 0, "accepted": 2, "accepted_words": 85}, "R3": {"words": 506, "then_words": 407, "because_sentences": 2, "except": 1, "precedence": 0, "accepted": 3, "accepted_words": 409}, - "R4": {"words": 991, "then_words": 729, "because_sentences": 1, "except": 4, "precedence": 0, "accepted": 4, "accepted_words": 600}, + "R4": {"words": 1021, "then_words": 729, "because_sentences": 1, "except": 4, "precedence": 0, "accepted": 4, "accepted_words": 600}, "R5": {"words": 250, "then_words": 154, "because_sentences": 2, "except": 0, "precedence": 0, "accepted": 0, "accepted_words": 0}, "S1": {"words": 306, "then_words": 26, "because_sentences": 1, "except": 2, "precedence": 3, "accepted": 0, "accepted_words": 0}, "S2": {"words": 657, "then_words": 119, "because_sentences": 1, "except": 0, "precedence": 7, "accepted": 3, "accepted_words": 222}, diff --git a/tools/differential/corpus_rules.jsonl b/tools/differential/corpus_rules.jsonl index 5fcf35049..38f351e9a 100644 --- a/tools/differential/corpus_rules.jsonl +++ b/tools/differential/corpus_rules.jsonl @@ -207,6 +207,7 @@ "Eric H. Holder, Jr., Secretary of State" "Esq. Smith" "É. y É. Smith" +"FRITZ FREIHERR UND VON BAR" "Field Marshal Bernard Montgomery" "Freiherr St John Smith" "Freiherr von Berg MA" diff --git a/tools/differential/snapshot.jsonl b/tools/differential/snapshot.jsonl index 38dc84d31..6efcee553 100644 --- a/tools/differential/snapshot.jsonl +++ b/tools/differential/snapshot.jsonl @@ -961,6 +961,7 @@ {"name": "Eyck, Jan van", "given": "Jan", "family": "van Eyck", "ambiguities": ["PARTICLE_OR_GIVEN"], "alternatives": ["PARTICLE_OR_GIVEN: name as middle (declared)"], "initials": "J. E."} {"name": "É. y É. Smith", "given": "É.", "middle": "y É.", "family": "Smith", "initials": "É. É. S."} {"name": "F. de Lange", "given": "F.", "family": "de Lange", "initials": "F. L."} +{"name": "FRITZ FREIHERR UND VON BAR", "given": "FRITZ", "family": "FREIHERR UND VON BAR", "initials": "F. U.", "capitalized": "Fritz freiherr und von bar"} {"name": "Fabbri Irene", "order": "FAMILY_FIRST", "given": "Irene", "family": "Fabbri", "initials": "I. F."} {"name": "Fabbri, Irene", "given": "Irene", "family": "Fabbri", "initials": "I. F."} {"name": "Fabián Pérez Ximeno", "given": "Fabián", "middle": "Pérez", "family": "Ximeno", "initials": "F. P. X."} From 2f76c163bf450b6c7873d320beb301485bd7be11 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Wed, 28 Oct 2026 21:55:57 -0700 Subject: [PATCH 37/39] Scope the connective exception to a word that is no particle (#939) - rules/R2, R3 and R4's connective set-aside reaches only a connective that is not also particle vocabulary; a word a caller lists as both keeps R2's readmission wherever it stands (the readings the #461 tests and INV2 pin). The default lists no word as both, so this is stated in prose with no example line. - decisions/R2's unjoined-mark Standing line again covers every connective and says how it relates to the set-aside; the case-repair value moves onto the all-capitals spelling; the Open line names R4. - decisions/R3 and decisions/R4 each gain a dated #939 pointer. - R4's new sentence is trimmed to the test; rules_budget re-recorded. Co-Authored-By: Claude Opus 5.5 --- .../decisions/R2-all-particles-family-base-divergence.md | 8 ++++---- docs/design/decisions/R3-initials-honor-the-fold.md | 1 + .../decisions/R4-case-repair-reads-the-unjoined-mark.md | 1 + .../rules/R2-all-particles-family-base-divergence.md | 2 +- docs/design/rules/R3-initials-honor-the-fold.md | 2 +- .../rules/R4-case-repair-reads-the-unjoined-mark.md | 2 +- tests/v2/rules_budget.json | 6 +++--- 7 files changed, 12 insertions(+), 10 deletions(-) diff --git a/docs/design/decisions/R2-all-particles-family-base-divergence.md b/docs/design/decisions/R2-all-particles-family-base-divergence.md index bfee2705d..c892897c5 100644 --- a/docs/design/decisions/R2-all-particles-family-base-divergence.md +++ b/docs/design/decisions/R2-all-particles-family-base-divergence.md @@ -8,12 +8,12 @@ - `initials()` reads the base, so the anchoring moves R3's output: `Anh Do` initials 'A. D.' and `Juan van der` 'J. v. d.' — 2026-08-16 (pre-merge coherence pass); 2026-08-18 #404/#385. - A particle run that joins nothing still chains whole into one part, so `Juan Smith van der` reads family 'van der' and middle 'Smith'; splitting it was declined, roles reassembling adjacent same-role pieces into one part and the base losing its full extent — 2026-08-18. - The base is the family without its working particles, wherever in the family they stand (`da Silva do Amaral de Souza, Miguel` → family_base 'Silva Amaral Souza', family_particles 'da do de'), a view the same under every convention and no culture's sort key — 2026-10-28 #754. -- A connective standing between two words of a part is set aside when the part is asked whether every word is particle vocabulary, so `Fritz Freiherr und von Bar` is all particles, base 'Freiherr und von Bar', initials 'F. F. v. B.' and case repair 'Fritz Freiherr und Von Bar', where the parser reads base 'und', a tracked deviation; a connective ending the part is a name word, so `Juan de y` keeps base 'y' — 2026-10-28 #754 (Derek), A CONNECTIVE AMONG PARTICLES. -- The unjoined mark is a statement about a whole part, and `initials()` honors it for every word in it, a connective ending the part included: under `Lexicon.default().add(particles={'y'})`, `Juan de y` reads family_base 'de y' and initials 'J. d. y.'; narrowing the readmission to the `particle` tag alone was tried and backed out — 2026-08-29 #461; 2026-09-20 #461. +- A connective that is not also particle vocabulary, standing between two words of a part, is set aside when the part is asked whether every word is particle vocabulary, so `Fritz Freiherr und von Bar` is all particles, base 'Freiherr und von Bar' and initials 'F. F. v. B.', and `FRITZ FREIHERR UND VON BAR` repairs to 'Fritz Freiherr und Von Bar', where the parser reads base 'und', a tracked deviation; a connective ending the part is a name word, so `Juan de y` keeps base 'y' — 2026-10-28 #754 (Derek), A CONNECTIVE AMONG PARTICLES. +- The unjoined mark is a statement about a whole part, and `initials()` honors it for every word in it, a connective included: under `Lexicon.default().add(particles={'y'})`, `Juan de y` reads family_base 'de y' and initials 'J. d. y.'; narrowing the readmission to the `particle` tag alone was tried and backed out. Such a word is listed as a particle too, so the line above's set-aside does not reach it, and it keeps this reading wherever it stands in the part (`Juan de y la` → 'J. d. y. l.') — 2026-08-29 #461; 2026-09-20 #461; 2026-10-28 #754 (Derek), A CONNECTIVE AMONG PARTICLES. Elsewhere: where a connective initials, a part holding nothing else for it to join (`Jon Dough and` → 'J. D.', `Juan de y` → 'J. y.') — decisions/R3's 2026-09-20 #461 entry; case repair keeping that connective lowercase, its carve-out not gated on the mark (under `Lexicon.default().add(particles={'y'})`, `Juan de y` forced → family 'De y') — decisions/R4's 2026-08-29, the CONJUNCTION CARVE-OUT entry; case repair reading the unjoined mark for the whole part (`ANH DO` → 'Anh Do') — decisions/R4's 2026-08-29 #407 entry; the snapshot recording `initials()`, so a change to the base is visible there — decisions/differential-ledger's 2026-10-13 #664, a committed snapshot replaces the classified gate entry. -Open: #754 (whether a language pack supplies a convention's sort key or entry element from Phase 4's per-surname units, the RFC's Phase 4 Left open); #939 (a connective between particles reads as the base, the deviation rules/R2 and rules/R3 mark). +Open: #754 (whether a language pack supplies a convention's sort key or entry element from Phase 4's per-surname units, the RFC's Phase 4 Left open); #939 (a connective between particles reads as the base, the deviation rules/R2, rules/R3 and rules/R4 mark). ## Log @@ -41,7 +41,7 @@ Open: #754 (whether a language pack supplies a convention's sort key or entry el Measured 2026-10-28 on master `c8357e74`, counts and token shapes only, over the committed corpus's 6,533 distinct entries (`snapshot.corpus_entries()`, each parsed under its declared order) and `tse-donors`' 854,531 records (`local_corpora.read_records`, the filed name and the registry name each parsed in the default order). A family is read in render order (FOLDED_TAG tokens first), a working particle is a token tagged `particle` and not UNJOINED_TAG, and a family has an interior particle where a working particle stands after its first word that is neither a particle nor a conjunction; that family is where (1) and (2) differ. The three kinds of interior family are tested in this order, the first that fits counting: after a family comma, every interior particle standing after the first comma; with a leading run as well; with no leading run. In the default policy: committed, 24 against 684 with a leading run only (12 with no leading run, 8 whose interior particle stands after a family comma, 4 with a leading run as well); `tse-donors`, 12 filed names against 347,992 and 1 registry name against 392,134, since a Brazilian name's particles before its last word stand in `middle`. With `middle_as_family`: committed, 367 against 398 (262, 92 and 13); `tse-donors`' filed names, 268,591 against 79,926 (261,024, 5 and 7,562), the commonest family shape word-particle-word, 150,998 times. Recompute with those definitions; a change to the fold or to O6 moves them. Weighed against (2): with `middle_as_family`, the population the per-surname reading serves, it gives the 261,024 `tse-donors` names with no leading run a base equal to the whole family and an empty particle view, so `family_particles` stops naming the particles it is for; and it changes a frozen property's output for 24 committed entries in the default policy, where (1) changes no output. The facade whose `last_prefixes` it matches is removed in 3.0. Its one gain is a base that is a written name, which no convention's key needs either (below). The cost of (1), accepted: the base is not a name anyone writes ('Silva Amaral Souza'), nor any convention's sort key. Why the base is not a sort key, researched 2026-10-28. Portuguese and Brazilian practice files a name under its LAST surname, that surname's particle removed and set after the given names, the agnomens `Filho`, `Neto` and `Júnior` staying with the surname before them and an established compound surname kept whole: `SOUZA, Miguel da Silva do Amaral de`, `NOGUEIRA NETO, Wanderlino` (ABNT NBR 6023:2018, https://engenhariaeletrica.ufes.br/sites/engenhariaeletrica.ufes.br/files/field/anexo/abnt6023.2018.pdf; the ABNT Zotero style's comments, https://www.zotero.org/styles/universidade-federal-do-espirito-santo-abnt?source=1; USP's references manual, https://www.ip.usp.br/site/wp-content/uploads/2025/04/Manual-Referencias-ABNT-SabadiniSampaioNascimento.pdf; Cambridge University Library, https://www.lib.cam.ac.uk/node/3142; the UN cataloguing manual, https://www.un.org/Depts/dhl/unbisref_manual/nameguide/personal.htm; IFLA, Names of Persons: Portugal, 2009, https://ifla.org/files/assets/cataloguing/pubs/ifla_names_of_persons_portugal_2009.pdf). Spanish practice files under the FIRST surname, the second its secondary key (RAE, Ortografía de la lengua española, 2010, ch. VII, "Alfabetización de antropónimos", https://www.rae.es/ortografía/alfabetización-de-antropónimos, read 2026-10-28 in a browser, the page refusing a plain fetch): by its rule (a) a preposition, with or without an article, heading the FIRST surname is left out of the order and written after the given name (`Torre Ibarra, Ramón de la`); by rule (e) one heading the SECOND surname counts as part of the second key (`Gómez Casas`, `Gómez de la Iglesia`, `Gómez de Lara`, `Gómez del Castillo`, `Gómez Echagüe`, in that order); by rule (b) a leading article or `San`/`Santa` stays and is sorted on (`La Calle Ruiz, Carmen`); by rule (d) a `y` between the surnames counts (`Aguirre y Gómez-Pacheco`). The same page states the Portuguese convention from the other side, the paternal second surname being the axis. Neither convention's key is this base: Portuguese practice sorts `…da Silva de Souza do Amaral` before `…da Silva do Amaral de Souza` (Amaral before Souza) while both bases begin `Silva`, and Spanish practice keeps a second surname's particle (`Gómez de la Iglesia`) where the base drops it ('Gómez Iglesia'). The base dropping every working particle is nearest the Spanish first key alone for a preposition, rule (a) (`Torre Ibarra, Ramón de la` → base 'Torre Ibarra'), and departs from it on what rule (b) keeps: a leading article or `San` is a particle to the parse, so `La Calle Ruiz, Carmen` reads base 'Calle Ruiz' and `San Martín López, Juan` base 'Martín López'. So the base is the convention-neutral view, and a convention's key is built from the surnames by whoever knows the convention, the RFC's Phase 4 Left open. In the default policy, `Miguel da Silva do Amaral de Souza` already reads family 'de Souza', base 'Souza', which is the Portuguese entry element by coincidence: only `middle_as_family` gives the long family. -- 2026-10-28 #754 (Derek), A CONNECTIVE AMONG PARTICLES — R2 GAINS AN EXCEPT SETTING A JOINING CONNECTIVE ASIDE, AND THE PARSER'S READING IS A TRACKED DEVIATION (#939). The entry above asked R2's all-particles question of one trailing particle and left it as the code reads it; the design-docs review of that amendment found a second shape the code decides against R2's own Except ("anchor the base") and Accepted line ("A particle needs a base to attach to"): a connective between particles. `Fritz Freiherr und von Bar` reads family 'Freiherr und von Bar', family_base 'und', family_particles 'Freiherr von Bar', initials 'F. u.', while without `und` the family is all particles and its base is the whole of it. A conjunction alone became the base of a family of three particles, and the Then before the amendment ("without its leading particles") would have given 'und von Bar', so the code deviated, unmarked, before the amendment as well. Decided: a connective between two words of a part is set aside when the all-particles question is asked, as R3 sets the working particles aside when it asks whether a connective is joining, since a connective doing its work joins the words around it and anchors no base on its own, staying in the base view with them; the particles then anchor the base, and the connective, joining them, contributes no initial and keeps a connective's case (R3's Except amended with it, 'F. F. v. B.', and R4's, `FRITZ FREIHERR UND VON BAR` repairing to 'Fritz Freiherr und Von Bar' where today it gives 'Fritz freiherr und von bar'). A connective ENDING the part stays out of the Except: it joins nothing, R3 already makes it a name word (`Juan de y` → base 'y', initials 'J. y.', decisions/R3's 2026-09-20 #461 entry), and setting it aside would make `de` the base and contradict that reading. Measured 2026-10-28 on `d97a910d`, counts and token shapes only, over the committed corpus's distinct entries (`snapshot.corpus_entries()`), each under its declared order: 3 families whose base holds only connectives in the default policy, 1 with the connective between particles (particle, connective, particle, particle), 1 ending in it (particle, connective) and 1 a lone connective, the last two outside the Except; the same 3 with `middle_as_family`. Recompute: a family whose `family_base` words are all family tokens tagged `conjunction`, split by whether the family's last token is one. +- 2026-10-28 #754 (Derek), A CONNECTIVE AMONG PARTICLES — R2 GAINS AN EXCEPT SETTING A JOINING CONNECTIVE ASIDE, AND THE PARSER'S READING IS A TRACKED DEVIATION (#939). The entry above asked R2's all-particles question of one trailing particle and left it as the code reads it; the design-docs review of that amendment found a second shape the code decides against R2's own Except ("anchor the base") and Accepted line ("A particle needs a base to attach to"): a connective between particles. `Fritz Freiherr und von Bar` reads family 'Freiherr und von Bar', family_base 'und', family_particles 'Freiherr von Bar', initials 'F. u.', while without `und` the family is all particles and its base is the whole of it. A conjunction alone became the base of a family of three particles, and the Then before the amendment ("without its leading particles") would have given 'und von Bar', so the code deviated, unmarked, before the amendment as well. Decided: a connective that is not also particle vocabulary, between two words of a part, is set aside when the all-particles question is asked, as R3 sets the working particles aside when it asks whether a connective is joining, since a connective doing its work joins the words around it and anchors no base on its own, staying in the base view with them; the particles then anchor the base, and the connective, joining them, contributes no initial and keeps a connective's case (R3's Except amended with it, 'F. F. v. B.', and R4's, `FRITZ FREIHERR UND VON BAR` repairing to 'Fritz Freiherr und Von Bar' where today it gives 'Fritz freiherr und von bar'). A connective ENDING the part stays out of the Except: it joins nothing, R3 already makes it a name word (`Juan de y` → base 'y', initials 'J. y.', decisions/R3's 2026-09-20 #461 entry), and setting it aside would make `de` the base and contradict that reading. A word a caller lists as both a connective and a particle stays out too: the question asks it as a particle, so it keeps the readmission of the 2026-09-20 #461 entry wherever it stands, under `Lexicon.default().add(particles={'y'})` `Juan de y la` initialing 'J. d. y. l.', `Juan van y der` 'J. v. y. d.' and `Anh y Van` 'A. y. V.', the readings test_render's and test_parser's #461 tests and test_properties' INV2 pin; the default vocabulary lists no word as both, so no example line reaches that shape. Measured 2026-10-28 on `d97a910d`, counts and token shapes only, over the committed corpus's distinct entries (`snapshot.corpus_entries()`), each under its declared order: 3 families whose base holds only connectives in the default policy, 1 with the connective between particles (particle, connective, particle, particle), 1 ending in it (particle, connective) and 1 a lone connective, the last two outside the Except; the same 3 with `middle_as_family`. Recompute: a family whose `family_base` words are all family tokens tagged `conjunction`, split by whether the family's last token is one. Declined: diff --git a/docs/design/decisions/R3-initials-honor-the-fold.md b/docs/design/decisions/R3-initials-honor-the-fold.md index 63eb3474d..a5e845a45 100644 --- a/docs/design/decisions/R3-initials-honor-the-fold.md +++ b/docs/design/decisions/R3-initials-honor-the-fold.md @@ -57,3 +57,4 @@ R3's earlier history is under `decisions/R2`, which this entry does not repeat: - 2026-10-19 #761 (skeleton migration) — the rule text no longer carries its argument and history, which moved here or were already here. WHY THE ORDER CLAUSE IS STATED IN R3 and not left to R1 (R1's subject is every field, and `initials()` is a view, not a field) is the 2026-08-30 #408 entry's "WHY R3 GAINED A SENTENCE"; the rule keeps the clause itself, and that the field settles order and nothing else. THE FACADE VIEW'S MECHANISM — its connective question falling back through `_render._reads_as_conjunction`, the helper case repair calls, and its particle question being `_is_particle`'s live vocabulary lookup — is the 2026-09-13 #528 entry; the rule keeps what the view does (reads the parse's connective reading where a token backs the word, looks every word up for the particle question) and the two contrasts it states, `de la vega` spliced and `Ph. D., John`. THE PARITY-BREAK HISTORY moved: before #461 the given group's connective initial was pinned by tests and stated by no rule ("pinned-but-undocumented"), #461 specified it by the same sentence as the middle and family groups, and that one sentence is what reduced the four candidate answers the 2026-08-29 bullet under `decisions/R2` listed for "John and Jane Smith" to one — the 2026-09-20 #461 entry above holds the argument and its measurements; the rule keeps the parity break and restoration as its last Accepted bullet. Nothing about the behavior moved. - 2026-10-19 (Derek), #761 — A PARTICLE IS A PARTICLE BY POSITION. "Particles contribute nothing" read as contradicting `Nguyen, Van Le` → "V. L. N.", where the vocabulary's `van` initials. It does not: whether a word is a particle is settled by where it stands in the name, not by membership in the set, and `Van` there is a given name. A Then bullet now says so, carrying `Nguyen, Van Le` and `Anh Do`. No reading moved. +- 2026-10-28 #939 — A CONNECTIVE BETWEEN PARTICLES: rules/R3's all-particles Except sets aside a connective that is not also particle vocabulary, standing between the part's words, as rules/R2 does, so `Fritz Freiherr und von Bar` should initial 'F. F. v. B.' where it gives 'F. u.', a deviation #939 tracks; decided and measured in decisions/R2's 2026-10-28 #754 (Derek), A CONNECTIVE AMONG PARTICLES entry. diff --git a/docs/design/decisions/R4-case-repair-reads-the-unjoined-mark.md b/docs/design/decisions/R4-case-repair-reads-the-unjoined-mark.md index 52cbc4025..68efa35d5 100644 --- a/docs/design/decisions/R4-case-repair-reads-the-unjoined-mark.md +++ b/docs/design/decisions/R4-case-repair-reads-the-unjoined-mark.md @@ -114,3 +114,4 @@ Accepted costs, deferred to the rescoped #459 rather than relitigated here: the SUPERSEDES: 2026-08-29, WHY CASE IS NOT A COMPARED SURFACE; and in 2026-08-29, THE VERIFICATION LIMIT, its claims that no ledger run at any baseline can see a change that moves only case and that nothing is owed for one, and its 2026-09-02 amendment's conclusion that no `capitalized()` column is owed yet. `compare.py --baseline 2.1.0` now shows that entry's own deviation (`ANH DO` → 'Anh Do', where 2.1.0 gave 'Anh do'); a 1.4.0 baseline still cannot. - 2026-10-22 #838 — CAPITALS INSIDE A WORD. The fall-through was `str.capitalize()` on the whole `_WORD` match, which holds a word's interior full stops and any modifier letter (category Lm, which `\w` matches) that opens it, so a capital not standing first was lowered: `j.r.r. tolkien` → 'J.r.r. Tolkien', `AHMAD IBN-ʿALĪ` → 'Ahmad ibn-ʿalī', and under force `A.B. Vajpayee` → 'A.b. Vajpayee'. 1.4.0 through 2.5.0 give the old readings (wheels, measured 2026-10-22, `HumanName(n).capitalize(force=True)`), so this is a fix, not a regression. The fix title-cases each full-stop chunk from its first character outside Lm. TWO ANSWERS. Every chunk is a word of its own rather than every letter a capital, which gives `St.John`, not `ST.JOHN`, and matches the hyphen split: a chunk takes the Mac/Mc convention as a hyphen part does (`J.MCDONALD SMITH` → 'J.McDonald Smith', where 2.5.0 gave 'J.mcdonald'; `mcdonald.jones` → 'McDonald.Jones'), which the first version missed, the Mac branch then running before the split and keeping everything after the first full stop as written (review of #839). Only Lm is passed over: an apostrophe of any spelling (`'`, U+2018, U+2019) is not a word character, so `'abd` was already 'Abd'; a digit is not passed over, or `3rd` would become '3Rd'. THE COST, ACCEPTED: an unlisted abbreviation whose own ending follows its full stop reads as two words. The snapshot showed it on the CLDR fictional `Dut.ra Prof.ra` ('Dut.Ra Prof.Ra' under force), and with it go the one-case Italian salutations (`SIG.RA` → 'Sig.Ra', `DOTT.SSA` → 'Dott.Ssa') that `str.capitalize()` had always repaired correctly, and the Latvian and Norwegian degrees (`dr.philol.` → 'Dr.Philol.', `st.repr.` → 'St.Repr.', three CLDR-only rows), which now read as their spaced spellings do (`Dr. med. univ.` forced → 'Dr. Med. Univ.', decisions/R5's title-repair evidence). A NARROWING WAS BUILT AND DECLINED: two or more letters after the last full stop, closed by none, kept lowercase as an abbreviation's written ending. It kept `Sig.ra` but gave `st.john` → 'St.john', `lt.gov` → 'Lt.gov' and forced `J.Smith` → 'J.smith', and still missed `dr.philol.`, whose ending is closed; a shape cannot tell an ending from a joined name or word, so the remedy is a listed casing, a `cased` mask asked before this clause. DECIDED (Derek, 2026-10-22): the `St.John` reading, with the Italian abbreviated salutations to be listed as titles with masks before 2.6.0 ships. Unlisted joined credentials are a vocabulary question as well (`jd.cpa` → 'Jd.Cpa'). BONUS: 'ʼn' upper-cases to the Lm apostrophe plus 'N', so 'ʼNa' is now a fixpoint, where the second pass gave 'ʼna'; capitalized()'s docstring no longer lists it. SNAPSHOT, 2026-10-22: the moved entries are listed in this change's diff of snapshot.jsonl, each read. Pinned by rules/R4's examples (each fix row fails on the parent) and test_a_word_takes_its_capital_past_a_caseless_lead. - 2026-10-22 #840 — THE REMEDY FOR #838's ITALIAN COST SHIPS: the abbreviated salutations and letter-heading honorifics whose ending follows their full stop are titles with `cased` masks (decisions/salutation-titles), so `SIG.RA MARIA ROSSI` repairs to 'Sig.ra Maria Rossi' again, as 1.4.0 through 2.5.0 gave it, and `dott.ssa` to 'Dott.ssa'. The mask is asked before the chunk split, the precedence the 2026-10-22 #838 entry relied on. The accepted cost stands for an unlisted ending (the Sardinian `DUT.RA` → 'Dut.Ra', CLDR's `Dut.ra Prof.ra`). A spaced `Sig. ra` is out of reach: it joins as a title run, `ra` being a listed title, and forced repair writes 'Sig. Ra'. +- 2026-10-28 #939 — A CONNECTIVE BETWEEN PARTICLES: rules/R4's all-particles Except sets aside a connective that is not also particle vocabulary, standing between the part's words, as rules/R2 does, so `FRITZ FREIHERR UND VON BAR` should repair to 'Fritz Freiherr und Von Bar' where it gives 'Fritz freiherr und von bar', a deviation #939 tracks; decided and measured in decisions/R2's 2026-10-28 #754 (Derek), A CONNECTIVE AMONG PARTICLES entry. diff --git a/docs/design/rules/R2-all-particles-family-base-divergence.md b/docs/design/rules/R2-all-particles-family-base-divergence.md index bb85391f3..2ed917c17 100644 --- a/docs/design/rules/R2-all-particles-family-base-divergence.md +++ b/docs/design/rules/R2-all-particles-family-base-divergence.md @@ -25,7 +25,7 @@ "Juan van der" → family_base="van der" "Juan van der" → family_particles="" ``` -- **A connective standing between two words of the part is set aside when the part is asked whether every word is particle vocabulary** (P3): it joins the words around it and anchors no base on its own, so it stays in the base view with them but does not stop the part from being all particles, and particles joined by a connective anchor the base. A connective ending the part joins nothing and is a name word (R3), so the particle before it still works as one. +- **A connective that is not also particle vocabulary, standing between two words of the part, is set aside when the part is asked whether every word is particle vocabulary** (P3): it joins the words around it and anchors no base on its own, so it stays in the base view with them but does not stop the part from being all particles, and particles joined by a connective anchor the base. A word listed as both a connective and a particle is asked the question as a particle, wherever it stands. A connective ending the part joins nothing and is a name word (R3), so the particle before it still works as one. ``` "Fritz Freiherr und von Bar" → family_base="Freiherr und von Bar" deviates: #939 (today: family_base="und") "Juan de y" → family_base="y" · boundary diff --git a/docs/design/rules/R3-initials-honor-the-fold.md b/docs/design/rules/R3-initials-honor-the-fold.md index 6fca989a7..51bbc9cde 100644 --- a/docs/design/rules/R3-initials-honor-the-fold.md +++ b/docs/design/rules/R3-initials-honor-the-fold.md @@ -37,7 +37,7 @@ **Except:** -- **A part whose every word is a particle initials those particles**: they are not acting as particles there but are the part's base (R2), and initial like any other name word, so a family that is all particles contributes its words rather than nothing. A connective standing between them is set aside in asking it, as R2 sets it aside, and joins them, so it contributes nothing. +- **A part whose every word is a particle initials those particles**: they are not acting as particles there but are the part's base (R2), and initial like any other name word, so a family that is all particles contributes its words rather than nothing. A connective that is not also particle vocabulary, standing between them, is set aside in asking it, as R2 sets it aside, and joins them, so it contributes nothing; a word listed as both is one of the part's particles and initials with them. ``` "Juan van der" → initials="J. v. d." "Fritz Freiherr und von Bar" → initials="F. F. v. B." deviates: #939 (today: initials="F. u.") diff --git a/docs/design/rules/R4-case-repair-reads-the-unjoined-mark.md b/docs/design/rules/R4-case-repair-reads-the-unjoined-mark.md index 4ddaa7708..c7025f91f 100644 --- a/docs/design/rules/R4-case-repair-reads-the-unjoined-mark.md +++ b/docs/design/rules/R4-case-repair-reads-the-unjoined-mark.md @@ -104,7 +104,7 @@ **Except:** -- **A part whose every word is particle vocabulary** is repaired as ordinary name words, since none of them is doing a particle's work there (R2). A connective standing between two of its words is set aside in asking it, as R2 sets it aside, and is repaired as the connective joining them that it is. +- **A part whose every word is particle vocabulary** is repaired as ordinary name words, since none of them is doing a particle's work there (R2). A connective that is not also particle vocabulary, standing between two of its words, is set aside in asking it, as R2 sets it aside. ``` "ANH DO" → capitalized="Anh Do" "anh van do" → capitalized="Anh Van Do" diff --git a/tests/v2/rules_budget.json b/tests/v2/rules_budget.json index d7c239606..d6cfc0766 100644 --- a/tests/v2/rules_budget.json +++ b/tests/v2/rules_budget.json @@ -50,9 +50,9 @@ "P8": {"words": 515, "then_words": 450, "because_sentences": 2, "except": 0, "precedence": 0, "accepted": 2, "accepted_words": 309}, "P9": {"words": 405, "then_words": 265, "because_sentences": 1, "except": 2, "precedence": 0, "accepted": 0, "accepted_words": 0}, "R1": {"words": 176, "then_words": 145, "because_sentences": 1, "except": 0, "precedence": 0, "accepted": 1, "accepted_words": 146}, - "R2": {"words": 238, "then_words": 18, "because_sentences": 1, "except": 2, "precedence": 0, "accepted": 2, "accepted_words": 85}, - "R3": {"words": 506, "then_words": 407, "because_sentences": 2, "except": 1, "precedence": 0, "accepted": 3, "accepted_words": 409}, - "R4": {"words": 1021, "then_words": 729, "because_sentences": 1, "except": 4, "precedence": 0, "accepted": 4, "accepted_words": 600}, + "R2": {"words": 264, "then_words": 18, "because_sentences": 1, "except": 2, "precedence": 0, "accepted": 2, "accepted_words": 85}, + "R3": {"words": 527, "then_words": 407, "because_sentences": 2, "except": 1, "precedence": 0, "accepted": 3, "accepted_words": 409}, + "R4": {"words": 1016, "then_words": 729, "because_sentences": 1, "except": 4, "precedence": 0, "accepted": 4, "accepted_words": 600}, "R5": {"words": 250, "then_words": 154, "because_sentences": 2, "except": 0, "precedence": 0, "accepted": 0, "accepted_words": 0}, "S1": {"words": 306, "then_words": 26, "because_sentences": 1, "except": 2, "precedence": 3, "accepted": 0, "accepted_words": 0}, "S2": {"words": 657, "then_words": 119, "because_sentences": 1, "except": 0, "precedence": 7, "accepted": 3, "accepted_words": 222}, From cbaaa015e2c1ac6cb8163fb6d7b6eff9762df4ec Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Wed, 28 Oct 2026 22:22:11 -0700 Subject: [PATCH 38/39] Pin the shared word between particles and correct its attribution (#939) - tests/v2/test_render.py gains a test pinning a word listed as both a particle and a conjunction where it stands between particles (`Juan de y la` base 'de y la', initials 'J. d. y. l.'; `Fritz Freiherr y von Bar` 'F. F. y. v. B.'), guarding R2's scoping of its connective set-aside. A mutation setting aside every interior connective of an all-particle part fails it, while the neighboring #461 test still passes. - decisions/R2 names the test pinning each reading; the interior ones were attributed to tests that never reach them. Co-Authored-By: Claude Opus 5.5 --- ...R2-all-particles-family-base-divergence.md | 2 +- tests/v2/test_render.py | 19 +++++++++++++++++++ 2 files changed, 20 insertions(+), 1 deletion(-) diff --git a/docs/design/decisions/R2-all-particles-family-base-divergence.md b/docs/design/decisions/R2-all-particles-family-base-divergence.md index c892897c5..b9696d87c 100644 --- a/docs/design/decisions/R2-all-particles-family-base-divergence.md +++ b/docs/design/decisions/R2-all-particles-family-base-divergence.md @@ -41,7 +41,7 @@ Open: #754 (whether a language pack supplies a convention's sort key or entry el Measured 2026-10-28 on master `c8357e74`, counts and token shapes only, over the committed corpus's 6,533 distinct entries (`snapshot.corpus_entries()`, each parsed under its declared order) and `tse-donors`' 854,531 records (`local_corpora.read_records`, the filed name and the registry name each parsed in the default order). A family is read in render order (FOLDED_TAG tokens first), a working particle is a token tagged `particle` and not UNJOINED_TAG, and a family has an interior particle where a working particle stands after its first word that is neither a particle nor a conjunction; that family is where (1) and (2) differ. The three kinds of interior family are tested in this order, the first that fits counting: after a family comma, every interior particle standing after the first comma; with a leading run as well; with no leading run. In the default policy: committed, 24 against 684 with a leading run only (12 with no leading run, 8 whose interior particle stands after a family comma, 4 with a leading run as well); `tse-donors`, 12 filed names against 347,992 and 1 registry name against 392,134, since a Brazilian name's particles before its last word stand in `middle`. With `middle_as_family`: committed, 367 against 398 (262, 92 and 13); `tse-donors`' filed names, 268,591 against 79,926 (261,024, 5 and 7,562), the commonest family shape word-particle-word, 150,998 times. Recompute with those definitions; a change to the fold or to O6 moves them. Weighed against (2): with `middle_as_family`, the population the per-surname reading serves, it gives the 261,024 `tse-donors` names with no leading run a base equal to the whole family and an empty particle view, so `family_particles` stops naming the particles it is for; and it changes a frozen property's output for 24 committed entries in the default policy, where (1) changes no output. The facade whose `last_prefixes` it matches is removed in 3.0. Its one gain is a base that is a written name, which no convention's key needs either (below). The cost of (1), accepted: the base is not a name anyone writes ('Silva Amaral Souza'), nor any convention's sort key. Why the base is not a sort key, researched 2026-10-28. Portuguese and Brazilian practice files a name under its LAST surname, that surname's particle removed and set after the given names, the agnomens `Filho`, `Neto` and `Júnior` staying with the surname before them and an established compound surname kept whole: `SOUZA, Miguel da Silva do Amaral de`, `NOGUEIRA NETO, Wanderlino` (ABNT NBR 6023:2018, https://engenhariaeletrica.ufes.br/sites/engenhariaeletrica.ufes.br/files/field/anexo/abnt6023.2018.pdf; the ABNT Zotero style's comments, https://www.zotero.org/styles/universidade-federal-do-espirito-santo-abnt?source=1; USP's references manual, https://www.ip.usp.br/site/wp-content/uploads/2025/04/Manual-Referencias-ABNT-SabadiniSampaioNascimento.pdf; Cambridge University Library, https://www.lib.cam.ac.uk/node/3142; the UN cataloguing manual, https://www.un.org/Depts/dhl/unbisref_manual/nameguide/personal.htm; IFLA, Names of Persons: Portugal, 2009, https://ifla.org/files/assets/cataloguing/pubs/ifla_names_of_persons_portugal_2009.pdf). Spanish practice files under the FIRST surname, the second its secondary key (RAE, Ortografía de la lengua española, 2010, ch. VII, "Alfabetización de antropónimos", https://www.rae.es/ortografía/alfabetización-de-antropónimos, read 2026-10-28 in a browser, the page refusing a plain fetch): by its rule (a) a preposition, with or without an article, heading the FIRST surname is left out of the order and written after the given name (`Torre Ibarra, Ramón de la`); by rule (e) one heading the SECOND surname counts as part of the second key (`Gómez Casas`, `Gómez de la Iglesia`, `Gómez de Lara`, `Gómez del Castillo`, `Gómez Echagüe`, in that order); by rule (b) a leading article or `San`/`Santa` stays and is sorted on (`La Calle Ruiz, Carmen`); by rule (d) a `y` between the surnames counts (`Aguirre y Gómez-Pacheco`). The same page states the Portuguese convention from the other side, the paternal second surname being the axis. Neither convention's key is this base: Portuguese practice sorts `…da Silva de Souza do Amaral` before `…da Silva do Amaral de Souza` (Amaral before Souza) while both bases begin `Silva`, and Spanish practice keeps a second surname's particle (`Gómez de la Iglesia`) where the base drops it ('Gómez Iglesia'). The base dropping every working particle is nearest the Spanish first key alone for a preposition, rule (a) (`Torre Ibarra, Ramón de la` → base 'Torre Ibarra'), and departs from it on what rule (b) keeps: a leading article or `San` is a particle to the parse, so `La Calle Ruiz, Carmen` reads base 'Calle Ruiz' and `San Martín López, Juan` base 'Martín López'. So the base is the convention-neutral view, and a convention's key is built from the surnames by whoever knows the convention, the RFC's Phase 4 Left open. In the default policy, `Miguel da Silva do Amaral de Souza` already reads family 'de Souza', base 'Souza', which is the Portuguese entry element by coincidence: only `middle_as_family` gives the long family. -- 2026-10-28 #754 (Derek), A CONNECTIVE AMONG PARTICLES — R2 GAINS AN EXCEPT SETTING A JOINING CONNECTIVE ASIDE, AND THE PARSER'S READING IS A TRACKED DEVIATION (#939). The entry above asked R2's all-particles question of one trailing particle and left it as the code reads it; the design-docs review of that amendment found a second shape the code decides against R2's own Except ("anchor the base") and Accepted line ("A particle needs a base to attach to"): a connective between particles. `Fritz Freiherr und von Bar` reads family 'Freiherr und von Bar', family_base 'und', family_particles 'Freiherr von Bar', initials 'F. u.', while without `und` the family is all particles and its base is the whole of it. A conjunction alone became the base of a family of three particles, and the Then before the amendment ("without its leading particles") would have given 'und von Bar', so the code deviated, unmarked, before the amendment as well. Decided: a connective that is not also particle vocabulary, between two words of a part, is set aside when the all-particles question is asked, as R3 sets the working particles aside when it asks whether a connective is joining, since a connective doing its work joins the words around it and anchors no base on its own, staying in the base view with them; the particles then anchor the base, and the connective, joining them, contributes no initial and keeps a connective's case (R3's Except amended with it, 'F. F. v. B.', and R4's, `FRITZ FREIHERR UND VON BAR` repairing to 'Fritz Freiherr und Von Bar' where today it gives 'Fritz freiherr und von bar'). A connective ENDING the part stays out of the Except: it joins nothing, R3 already makes it a name word (`Juan de y` → base 'y', initials 'J. y.', decisions/R3's 2026-09-20 #461 entry), and setting it aside would make `de` the base and contradict that reading. A word a caller lists as both a connective and a particle stays out too: the question asks it as a particle, so it keeps the readmission of the 2026-09-20 #461 entry wherever it stands, under `Lexicon.default().add(particles={'y'})` `Juan de y la` initialing 'J. d. y. l.', `Juan van y der` 'J. v. y. d.' and `Anh y Van` 'A. y. V.', the readings test_render's and test_parser's #461 tests and test_properties' INV2 pin; the default vocabulary lists no word as both, so no example line reaches that shape. Measured 2026-10-28 on `d97a910d`, counts and token shapes only, over the committed corpus's distinct entries (`snapshot.corpus_entries()`), each under its declared order: 3 families whose base holds only connectives in the default policy, 1 with the connective between particles (particle, connective, particle, particle), 1 ending in it (particle, connective) and 1 a lone connective, the last two outside the Except; the same 3 with `middle_as_family`. Recompute: a family whose `family_base` words are all family tokens tagged `conjunction`, split by whether the family's last token is one. +- 2026-10-28 #754 (Derek), A CONNECTIVE AMONG PARTICLES — R2 GAINS AN EXCEPT SETTING A JOINING CONNECTIVE ASIDE, AND THE PARSER'S READING IS A TRACKED DEVIATION (#939). The entry above asked R2's all-particles question of one trailing particle and left it as the code reads it; the design-docs review of that amendment found a second shape the code decides against R2's own Except ("anchor the base") and Accepted line ("A particle needs a base to attach to"): a connective between particles. `Fritz Freiherr und von Bar` reads family 'Freiherr und von Bar', family_base 'und', family_particles 'Freiherr von Bar', initials 'F. u.', while without `und` the family is all particles and its base is the whole of it. A conjunction alone became the base of a family of three particles, and the Then before the amendment ("without its leading particles") would have given 'und von Bar', so the code deviated, unmarked, before the amendment as well. Decided: a connective that is not also particle vocabulary, between two words of a part, is set aside when the all-particles question is asked, as R3 sets the working particles aside when it asks whether a connective is joining, since a connective doing its work joins the words around it and anchors no base on its own, staying in the base view with them; the particles then anchor the base, and the connective, joining them, contributes no initial and keeps a connective's case (R3's Except amended with it, 'F. F. v. B.', and R4's, `FRITZ FREIHERR UND VON BAR` repairing to 'Fritz Freiherr und Von Bar' where today it gives 'Fritz freiherr und von bar'). A connective ENDING the part stays out of the Except: it joins nothing, R3 already makes it a name word (`Juan de y` → base 'y', initials 'J. y.', decisions/R3's 2026-09-20 #461 entry), and setting it aside would make `de` the base and contradict that reading. A word a caller lists as both a connective and a particle stays out too: the question asks it as a particle, so it keeps the readmission of the 2026-09-20 #461 entry wherever it stands, under `Lexicon.default().add(particles={'y'})` `Juan de y la` reading base 'de y la' and initials 'J. d. y. l.' and `Fritz Freiherr y von Bar` 'F. F. y. v. B.', pinned by test_render's `test_initials_readmits_a_shared_word_between_particles`, beside the end and start of a part, `Juan de y` 'J. d. y.' and `Anh y Van` 'A. y. V.', pinned by its `test_initials_readmits_a_conjunction_in_a_particle_part`; the default vocabulary lists no word as both, so no example line reaches that shape. Measured 2026-10-28 on `d97a910d`, counts and token shapes only, over the committed corpus's distinct entries (`snapshot.corpus_entries()`), each under its declared order: 3 families whose base holds only connectives in the default policy, 1 with the connective between particles (particle, connective, particle, particle), 1 ending in it (particle, connective) and 1 a lone connective, the last two outside the Except; the same 3 with `middle_as_family`. Recompute: a family whose `family_base` words are all family tokens tagged `conjunction`, split by whether the family's last token is one. Declined: diff --git a/tests/v2/test_render.py b/tests/v2/test_render.py index 258f62782..9f4c787dd 100644 --- a/tests/v2/test_render.py +++ b/tests/v2/test_render.py @@ -243,6 +243,25 @@ def test_initials_readmits_a_conjunction_in_a_particle_part() -> None: assert parse("Juan y").initials() == "J. y." +def test_initials_readmits_a_shared_word_between_particles() -> None: + """rules/R2's scoping of its connective set-aside (#939). + + R2 sets aside a connective standing between the words of a part + when it asks whether every word is particle vocabulary, but only a + connective that is not also particle vocabulary: a word listed as + both is asked the question as a particle WHEREVER it stands, so it + keeps the readmission the test above pins at the end of a part and + at its start. These are the interior rows, which nothing else + reaches -- INV2's connective grid never puts the shared word + between particles. + """ + p = Parser(lexicon=Lexicon.default().add(particles={"y"})) + de_y_la = p.parse("Juan de y la") + assert de_y_la.family_base == "de y la" + assert de_y_la.initials() == "J. d. y. l." + assert p.parse("Fritz Freiherr y von Bar").initials() == "F. F. y. v. B." + + def test_repair_keeps_a_lone_connective_lowercase_where_it_initials( ) -> None: """rules/R4's own reading, and the view split it accepts. From 41255b82985c727e4bd474f2bd6b296de34e2126 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Thu, 29 Oct 2026 12:05:27 -0700 Subject: [PATCH 39/39] Record the Lexicon surface break's 2.x warning as waived (#754) decided 2026-10-29: like the render() signature, the Lexicon surface break takes no 2.x warning. Unlike it, the break does change calls that work today, but 2.8 was the last 2.x feature release and no deprecation path ships for the compiled-set fields, so 3.0's release notes carry the change. decisions/3-0-reevaluations gains THE LEXICON WARNING entry and its Standing line; decisions/vocabulary-model and the RFC no longer list the warning as open. Co-Authored-By: Claude Opus 5.5 --- docs/design/decisions/3-0-reevaluations.md | 5 +++-- docs/design/decisions/vocabulary-model.md | 2 +- docs/design/vocabulary-model-rfc.md | 2 +- 3 files changed, 5 insertions(+), 4 deletions(-) diff --git a/docs/design/decisions/3-0-reevaluations.md b/docs/design/decisions/3-0-reevaluations.md index d21aa3059..094c5c8a2 100644 --- a/docs/design/decisions/3-0-reevaluations.md +++ b/docs/design/decisions/3-0-reevaluations.md @@ -13,7 +13,7 @@ Promoted 2026-08-15 from session memory (Derek's 2026-07-30 ask; promotion appro - From 2.8, the last 2.x feature release, the uses 3.0 removes warn with a `DeprecationWarning` on every call at the caller's line naming code that runs: constructing `HumanName` or a `Constants`, loading either from a pickle, mutating the shared `CONSTANTS`, and reading `nameparser.config`'s three managers or `nameparser.util.lc`; importing `nameparser`, `nameparser.parser` or `HumanName` stays quiet, and so does reading the shared `CONSTANTS`, which no warning reaches — 2026-10-28 #925 (Derek), WARNINGS.; 3.0 removes those uses and their warnings with them — 2026-10-28 #357, REMOVAL. - The 3.0 core API — `parse`, `Parser`, `Lexicon`, `Policy`, `Locale` and `ParsedName` — is frozen except for additions, so anything 3.0 later breaks there is a new decision needing its own 2.x warning — 2026-10-28 #925 (Derek), THE CORE API. - `render(spec: str | None = None)` is a break decided under that freeze and waived, taking no 2.x warning, since no call that works today changes meaning — 2026-10-28 #754 (Derek), THE RENDER SIGNATURE. -- `Lexicon`'s compiled-set fields are the other break decided under that freeze: 3.0 configures vocabulary by reading and keeps the sets internal; its 2.x warning is open — 2026-10-28 #754 (Derek), THE LEXICON SURFACE. +- `Lexicon`'s compiled-set fields are the other break decided under that freeze: 3.0 configures vocabulary by reading and keeps the sets internal; its 2.x warning is waived too, no 2.x release being left to carry one, so the change is announced in 3.0's release notes alone — 2026-10-28 #754 (Derek), THE LEXICON SURFACE; 2026-10-29 #754 (Derek), THE LEXICON WARNING. - 3.0 keeps no shared configuration, so the contextvars override once parked for it (`with nameparser.config.use(c):`) is not built: it existed to scope the shared `CONSTANTS`, and a parse's configuration is now the `Parser` it runs on — 2026-10-28 #357, REMOVAL. - An explicit family comma beats script for Latin text, a doctrine inherited from v1 that stands on its own; for CJK script it is a description, not a promise (Elsewhere, below) — 2026-08-15. - Particle vocabulary is language judgement wearing a position heuristic, so a caller who knows a tradition should be able to opt into a tuned set, which needs packs to carry vocabulary overrides and not only additions; 3.0 work — 2026-08-17. @@ -73,7 +73,8 @@ Open: #753 (phase 3, language packs, where the three pack bullets land). Every warning is a `DeprecationWarning` rather than a `FutureWarning`, emitted on every call at the caller's line and never once per process (a process flag lets a dependency's first `HumanName` consume the warning meant for the caller, the reason the 2026-08-15 bridge template warns per read location, and turns this suite's `filterwarnings = error` into an order-dependent failure under `--dist worksteal`). A `DeprecationWarning` is hidden by default outside `__main__`, so the release log, README and docs/migrate.rst announce the removal loudly instead. Every message names code that type-checks and runs, pinned by tests/v2/test_deprecations.py. `force_mixed_case_capitalization`, a removal candidate since 2026-08-29, is a `Constants` attribute, so removing `Constants` removes it and the construction warning is its removal warning; that follows from the facade's removal rather than being decided apart from it. Phase 4's output model is announced in one line of docs/migrate.rst ("details with the 3.0 alphas"), nothing undecided listed. - 2026-10-28 #925 (Derek), THE CORE API — THE 3.0 CORE API IS FROZEN EXCEPT FOR ADDITIONS: `parse`, `Parser`, `Lexicon`, `Policy`, `Locale` and `ParsedName` keep their 2.x shape through 3.0, gaining fields, members and methods but losing and renaming none. Anything 3.0 later finds it must break there is a new decision, and it needs a 2.x release warning about it first, the bridge discipline in this file's preamble; Phase 3's draft, which assumes `Locale` and `Policy` keep their 2.x shape after the facade's removal, rests on this entry. - 2026-10-28 #754 (Derek), THE RENDER SIGNATURE — `render(spec: str | None = None)` IS A BREAK DECIDED UNDER THE CORE API FREEZE ABOVE, AND ITS 2.x WARNING IS WAIVED. `None`, what `str()` passes, is the order-aware default, and a spec a caller writes renders as written; decisions/vocabulary-model's #754 output-model entry, item 6, holds the reasons. The freeze defines an addition and this is not one (introspecting the default gives `None`, and a subclass overriding `render(self, spec: str = ...)` becomes an incompatible override to a type checker), but no call that works today changes meaning, `render(None)` raising today, and 2.8 was the last 2.x feature release, so it ships in 3.0 with no 2.x warning. It also answers the 2026-08-15 bundle's (A) bullet on render's v1-inherited defaults. -- 2026-10-28 #754 (Derek), THE LEXICON SURFACE — `Lexicon`'S COMPILED-SET FIELDS GO INTERNAL IN 3.0, THE OTHER BREAK DECIDED UNDER THE CORE API FREEZE ABOVE, AND THE ONE WHOSE 2.x WARNING IS STILL TO SHIP. A caller configures vocabulary by reading instead; the reasons and the weighing are decisions/vocabulary-model's 2026-10-28 #754 entry on the `Lexicon` surface. The freeze asks a break for a 2.x release warning first, and 2.8 was the last 2.x feature release, so how that warning ships (a 2.8.x patch, or the release notes alone) is open. +- 2026-10-28 #754 (Derek), THE LEXICON SURFACE (its open 2.x warning superseded by 2026-10-29 #754 (Derek), THE LEXICON WARNING, below) — `Lexicon`'S COMPILED-SET FIELDS GO INTERNAL IN 3.0, THE OTHER BREAK DECIDED UNDER THE CORE API FREEZE ABOVE, AND THE ONE WHOSE 2.x WARNING IS STILL TO SHIP. A caller configures vocabulary by reading instead; the reasons and the weighing are decisions/vocabulary-model's 2026-10-28 #754 entry on the `Lexicon` surface. The freeze asks a break for a 2.x release warning first, and 2.8 was the last 2.x feature release, so how that warning ships (a 2.8.x patch, or the release notes alone) is open. +- 2026-10-29 #754 (Derek), THE LEXICON WARNING — THE `Lexicon` SURFACE BREAK'S 2.x WARNING IS WAIVED, as the render signature's is. Unlike that break, this one does change calls that work today: a caller who reads a compiled-set field or configures one by name (`Lexicon.default().add(titles={'dean'})`, `dataclasses.replace(lexicon, titles=...)`) has to move to configuring by reading. But 2.8 was the last 2.x feature release, so no 2.x release is left to carry a warning, and no deprecation path ships for the compiled-set fields: they change in 3.0, and its release notes and migration guide carry the change. - 2026-10-28 #357, REMOVAL — THE V1 COMPATIBILITY LAYER IS REMOVED for 3.0, the removal 2.8 warned of (2026-10-28 #925 (Derek), WARNINGS; #935). Gone: `HumanName` (`nameparser/_facade.py`) and `nameparser.parser`; `nameparser.config`'s `Constants`, `CONSTANTS`, `SetManager`, `TupleManager` and `RegexTupleManager` (`nameparser/_config_shim.py`); `nameparser.util`; the five 1.x word-list names 2.2 retired, with their alias bridge (`config/_deprecated.py`) and the `prefixes.py` and `bound_first_names.py` modules; `config/_titles_2_6_0.py`; the pickle-migration shims; and the 2.8 deprecation warnings. Stayed: the vocabulary data modules in `nameparser.config` (2026-10-28 #357 (Derek)), whose `__init__` is now a docstring, and the core API, unchanged (2026-10-28 #925 (Derek), THE CORE API). SUPERSEDES: the 2026-07-06 parking lot's contextvars override, which existed to scope the shared `CONSTANTS` and has nothing left to scope, a parse's configuration being the `Parser` it runs on, so it is not built; and the 2026-10-28 #357 (Derek) entry's statement that what 3.0 removes from the package is the five shim names, with its statement that the five alias messages stand: the removal also takes the five 1.x word-list names 2.2 retired, their alias bridge and messages, and the `prefixes.py` and `bound_first_names.py` modules. Carried out rather than decided, each plan done as written: the 2026-08-15 bundle's facade aliasing (every surface speaks `given` and `family`) and its deprecation-bridge template, which has no 2.x→3.0 shim left to serve; the 2026-08-16 legacy-pickle skip; `force_mixed_case_capitalization`, gone with `Constants` (2026-10-28 #925 (Derek), WARNINGS), so rules/R5's override has the one route `capitalized(force=True)`, the 2026-08-29 bullet's "Without the shim there is one route" (decisions/R5); and the "dies with the shim" of the #459, #541 and #752 (B) bullets — the raise's landing at a v1 caller's first parse, the filter of v1 `Constants` entries and the 2.6.0 title list. The core `Lexicon` raises on a mismatched mask as before. diff --git a/docs/design/decisions/vocabulary-model.md b/docs/design/decisions/vocabulary-model.md index 86f89f201..1b982c7ae 100644 --- a/docs/design/decisions/vocabulary-model.md +++ b/docs/design/decisions/vocabulary-model.md @@ -271,4 +271,4 @@ Open: #913 (the title words phase 2j left, read as titles though borne as names) 9. **The work order**, one PR each, measured on its own base, the snapshot recording each new view where it is non-empty: 4a `Token.reading` and `ReadingKind.PATRONYMIC` (neutral; `languages` after 3a); 4b `generation` and `credentials` (neutral); 4c `patronymic` (neutral); 4d `surname2` (neutral); 4e the render defaults and `order`; 4f `[rules]` into the policy patch, with or after 3b; 4g the compound given name, after 4f; 4h the shape inventory per order (#469). Interleaved with phase 3 (its sub-phases the 2026-10-27 #753 entry's item 11): 4a beside 3a; 4b to 4d before the packs that need them, `cs` (3f) and `pt-BR` (3h) on 4b, `es` and `pt` (3g) on 4d and 4g; an `is` pack on 3b, 4c and 4f; 3i, the bearers' languages, on 3a alone. 10. **What stays out of the output model.** Gender (decision 9) waits, `Token.reading` able to carry it later as one more addition. Names of given names only are a rule's question, not a field's: ORCID's empty family box holds 68,006 of 1,831,533 records, 15.9% to 18.7% of those from India, Pakistan, Bangladesh and Indonesia, and the parse fills `family` in 51,367 of the 51,415 with two or more given-box words. #694's two-word surname after a given name (`Tariq Abdul Aziz`) is a parse rule for the default, outside phase 4. Phase 1's four reserved names said to be phase 4's (`separators`, `bare_only`, `leading_only`, `trailing_only`) have no sub-phase yet. - **2026-10-28 #754, render patterns — A `Locale` MAY CARRY CLDR-SHAPED RENDER PATTERNS (A GIVEN-FIRST AND A SURNAME-FIRST DISPLAY PATTERN AND A SORTING PATTERN), A SHAPE AGREED NOW FOR A FEATURE THAT COMES LATER AS AN ADDITION.** The RFC's Phase 4 section, "Render patterns a pack may carry", holds the shape: the pack field, a table of Phase 4's views against CLDR's person-name fields, and the gap it shows, CLDR's `surname-prefix` and `surname-core` belonging to the first surname where `family_particles` and `family_base` read the whole family, which the per-surname units close. CLDR is a starting point, not the authority: its `pt` sorting pattern files under the first surname and leaves `surname2` out, which is not ABNT's last-surname entry (decisions/R2). Proposed there, not decided: a casefolded key method beside `comparison_key()`, collation left to the caller; and, open for a decision, whether a pattern is reached through a convention recorded on the name or through the parser. -- **2026-10-28 #754, the `Lexicon` surface (Derek) — A CALLER CONFIGURES VOCABULARY BY READING, AND THE COMPILED SETS BECOME INTERNAL.** Derek decided in chat on 2026-10-28, choosing it over keeping a public field per compiled set and over keeping 2.0.0's eleven while hiding the eleven added since. A caller states what a word may be in the language files' terms (a `given` reading on `guru`, no credential reading on `dc`) and `Lexicon` derives the sets and gates as the build does. The reasons: the set mechanics are hard to keep straight even for the maintainer, a reading is what a caller can know about a word, and it is the form a pack contribution takes. Today the same configuration needs rules/H9's set algebra, `Lexicon.default().remove(titles={'guru'}).add(titles_ambiguous={'guru'})` (`Guru Prasad Sharma` → given `Guru`) and `remove(suffix_acronyms={'dc'})` (`Ram Bahadur DC` → family `DC`), while `add(suffix_acronyms_ambiguous={'dc'})` leaves suffix `DC` (2026-10-28, run on this branch's base). A caller's own removal may settle a clash: decision 15 binds the default and the packs, not a caller's data. `dataclasses.fields(Lexicon)` gives 22 public fields on 2026-10-28, eleven since 2.0.0's eleven; the RFC's Phase 4 "Left open" holds the weighing. A break of the frozen `Lexicon`, recorded in decisions/3-0-reevaluations. Open: the method names, whether a caller's own language file can join the pool (phase 3's item 8 defers third-party files), and the 2.x warning the freeze asks for, 2.8 being the last 2.x feature release. +- **2026-10-28 #754, the `Lexicon` surface (Derek) — A CALLER CONFIGURES VOCABULARY BY READING, AND THE COMPILED SETS BECOME INTERNAL.** Derek decided in chat on 2026-10-28, choosing it over keeping a public field per compiled set and over keeping 2.0.0's eleven while hiding the eleven added since. A caller states what a word may be in the language files' terms (a `given` reading on `guru`, no credential reading on `dc`) and `Lexicon` derives the sets and gates as the build does. The reasons: the set mechanics are hard to keep straight even for the maintainer, a reading is what a caller can know about a word, and it is the form a pack contribution takes. Today the same configuration needs rules/H9's set algebra, `Lexicon.default().remove(titles={'guru'}).add(titles_ambiguous={'guru'})` (`Guru Prasad Sharma` → given `Guru`) and `remove(suffix_acronyms={'dc'})` (`Ram Bahadur DC` → family `DC`), while `add(suffix_acronyms_ambiguous={'dc'})` leaves suffix `DC` (2026-10-28, run on this branch's base). A caller's own removal may settle a clash: decision 15 binds the default and the packs, not a caller's data. `dataclasses.fields(Lexicon)` gives 22 public fields on 2026-10-28, eleven since 2.0.0's eleven; the RFC's Phase 4 "Left open" holds the weighing. A break of the frozen `Lexicon`, recorded in decisions/3-0-reevaluations. Open: the method names, and whether a caller's own language file can join the pool (phase 3's item 8 defers third-party files). The 2.x warning the freeze asks for was waived on 2026-10-29, 2.8 being the last 2.x feature release (decisions/3-0-reevaluations, THE LEXICON WARNING). diff --git a/docs/design/vocabulary-model-rfc.md b/docs/design/vocabulary-model-rfc.md index 66dbcde92..0115eae3c 100644 --- a/docs/design/vocabulary-model-rfc.md +++ b/docs/design/vocabulary-model-rfc.md @@ -400,7 +400,7 @@ Each sub-phase is one PR, measured on its own base as phase 2's and phase 3's ar A shape question, decided 2026-10-28: -- **Whether `Lexicon` keeps a public field for each compiled set**, or answers word, kind and property queries from phase 3's reading index and keeps its sets internal. `dataclasses.fields(Lexicon)` gives 22 public fields on 2026-10-28, 11 of them added since 2.0.0's 11 (`surnames`, `honorific_tails`, `conjunctions_ambiguous`, `after_given_titles`, `suffix_words_ambiguous`, `agnomens`, `given_name_abbreviations`, `titles_ambiguous`, `title_modifiers`, `title_styles`, `title_phrases`), each a new parse behavior's set, and a rule keyed on a title kind or on one language's reading of a kind (phase 3's design, "a property a language's own reading should take only under its pack") would add more. Keeping the fields: each is frozen API, shown in the bounded repr (`Lexicon(default + titles: +1)`), and pickled, where `Lexicon`'s own `__setstate__`, unlike the `ADDED_IN` guard the other types share, refuses any pickle whose fields differ, so every new field breaks every older `Lexicon` pickle, and with it every pickled `Parser` and locale pack, until that guard learns the marker. Turning them internal: a break that has to land in 3.0, where a query added beside the fields is an addition that can come later; a caller who configures vocabulary by the field today (`Lexicon.add(titles=...)`, `dataclasses.replace`) needs another way in, for words that carry no kind and no language; and phase 3's reading index, which a `Lexicon` answering queries would read, is already loaded for every caller with `Lexicon.default()` (about 0.45 ms with its bytecode cached, by phase 3's timing above), so answering from it adds no load. The parser need not pay either way: a frozenset test per word costs no frame and an internal cache keeps that path, while a query method called per word would add one per token, the cost the frame budget counts as scaling with the input. **Derek decided on 2026-10-28 that a caller configures vocabulary by reading and the compiled sets become internal** (decisions/vocabulary-model's 2026-10-28 #754 entry on the `Lexicon` surface). A caller states what a word may be in the language files' terms, a `given` reading on `guru`, no credential reading on `dc`, and `Lexicon` derives the sets and gates the build derives, so making `Guru` a possible given name needs no knowledge of rules/H9's sets, where today it is `remove(titles={'guru'}).add(titles_ambiguous={'guru'})`, and `add(given_name_titles=...)` or `add(suffix_acronyms_ambiguous=...)`, the plausible wrong guesses, do something else. A reading is also what a caller can know about a word and what a pack contribution states. Removing a reading from a caller's own lexicon may settle a clash; decision 15 binds the default and the packs, not a caller's choice about their own data. It breaks the frozen `Lexicon` (decisions/3-0-reevaluations); the method names, whether a caller's own language file can join the pool later (phase 3 defers it), and the 2.x warning the freeze asks for are open. +- **Whether `Lexicon` keeps a public field for each compiled set**, or answers word, kind and property queries from phase 3's reading index and keeps its sets internal. `dataclasses.fields(Lexicon)` gives 22 public fields on 2026-10-28, 11 of them added since 2.0.0's 11 (`surnames`, `honorific_tails`, `conjunctions_ambiguous`, `after_given_titles`, `suffix_words_ambiguous`, `agnomens`, `given_name_abbreviations`, `titles_ambiguous`, `title_modifiers`, `title_styles`, `title_phrases`), each a new parse behavior's set, and a rule keyed on a title kind or on one language's reading of a kind (phase 3's design, "a property a language's own reading should take only under its pack") would add more. Keeping the fields: each is frozen API, shown in the bounded repr (`Lexicon(default + titles: +1)`), and pickled, where `Lexicon`'s own `__setstate__`, unlike the `ADDED_IN` guard the other types share, refuses any pickle whose fields differ, so every new field breaks every older `Lexicon` pickle, and with it every pickled `Parser` and locale pack, until that guard learns the marker. Turning them internal: a break that has to land in 3.0, where a query added beside the fields is an addition that can come later; a caller who configures vocabulary by the field today (`Lexicon.add(titles=...)`, `dataclasses.replace`) needs another way in, for words that carry no kind and no language; and phase 3's reading index, which a `Lexicon` answering queries would read, is already loaded for every caller with `Lexicon.default()` (about 0.45 ms with its bytecode cached, by phase 3's timing above), so answering from it adds no load. The parser need not pay either way: a frozenset test per word costs no frame and an internal cache keeps that path, while a query method called per word would add one per token, the cost the frame budget counts as scaling with the input. **Derek decided on 2026-10-28 that a caller configures vocabulary by reading and the compiled sets become internal** (decisions/vocabulary-model's 2026-10-28 #754 entry on the `Lexicon` surface). A caller states what a word may be in the language files' terms, a `given` reading on `guru`, no credential reading on `dc`, and `Lexicon` derives the sets and gates the build derives, so making `Guru` a possible given name needs no knowledge of rules/H9's sets, where today it is `remove(titles={'guru'}).add(titles_ambiguous={'guru'})`, and `add(given_name_titles=...)` or `add(suffix_acronyms_ambiguous=...)`, the plausible wrong guesses, do something else. A reading is also what a caller can know about a word and what a pack contribution states. Removing a reading from a caller's own lexicon may settle a clash; decision 15 binds the default and the packs, not a caller's choice about their own data. It breaks the frozen `Lexicon` (decisions/3-0-reevaluations); the method names and whether a caller's own language file can join the pool later (phase 3 defers it) are open, and the 2.x warning the freeze asks for was waived on 2026-10-29, no 2.x feature release being left to carry it (decisions/3-0-reevaluations, THE LEXICON WARNING). Value questions inside the shapes above: