Join an elided particle or Arabic article written after the given name to the family (#700)

AGENTS.md

@@ -253,7 +253,7 @@ Add a dedicated `copy.deepcopy()` round-trip test for it too (see `test_regexes_
 
 **Unknown-key attribute access on `TupleManager`/`RegexTupleManager` warns (1.4, #256) — 2.0 note: the warning became a hard `AttributeError` naming the known keys (shim `TupleManager`); the paragraph below describes the deleted v1 machinery and applies only when reading v1 history.** — a key not currently in the dict emits `DeprecationWarning` naming the miss and the known keys (`_warn_unknown_key`), before falling back to the same `None`/`EMPTY_REGEX` default as before; `.get()` stays silent. Dunder probes (`__deepcopy__`) still raise `AttributeError` outright, and single-underscore probes (`_repr_html_`, IPython's `_ipython_canary_method_should_not_exist_`, etc.) are excluded from the warning too — no real config key starts with `_`, so both guards just keep protocol/introspection probes from misfiring as "typo" warnings. This means internal parser code that reads `self.C.regexes.<key>` unconditionally (e.g. `squash_bidi`'s `bidi`) now warns if a caller's custom `regexes` dict omits that key — a previously-silent partial-override pattern is on the same deprecation path as an actual typo.
 
-**Adding a word to a config set** — first check the *other* sets for the same word (grep `nameparser/config/` or intersect the sets in a `python3 -c`). Real overlaps exist: `do`/`st`/`mc` ∈ `PARTICLES` ∩ `TITLES`/`SUFFIX_ACRONYMS`; `abd` = "ABD" ∈ `SUFFIX_ACRONYMS`; `abu` ∈ `PARTICLES` ∩ `BOUND_GIVEN_NAMES` (position-dependent: leading token → given-name join, mid-name → family-name join). Usually position-dependent and harmless, but can force a guard or an exclusion (the `last_base` all-particles guard; dropping `abd` from `BOUND_GIVEN_NAMES`). Then check the word against real names, not memory: `uv run python tools/name_usage.py <word>` counts its bearers as a given name and as a family name (US Census surnames, SSA given names, Wikidata with countries of citizenship). Pass a name as it is written (`McDonald`): Wikidata labels match exactly, and no rule rebuilds an internal capital from `mcdonald`. It reports and does not decide — read the counts against the position the set's claim acts on. And check the predicates that test a member's SPELLING rather than its membership: they encode what the set's members happened to look like when they were written. `_pieces.starts_a_credential_run` implemented rules.md#S2's "Latin" as `isascii()`, exact until #661 added `mlađi`, which would have read `mlađi Jones` differently from `stariji Jones`; the design-docs review caught it, after a first draft had credited the test to the wrong clause of S2 and proposed deleting it on a mutation that moved nothing because nothing pinned it. (#679 then dropped the script test on purpose, after measuring what it protected.) Before adding the first member of a new script, or the first accented one, grep the set's consumers for `isascii`, `isalpha` and script tests, and read the rule each one implements.
+**Adding a word to a config set** — first check the *other* sets for the same word (grep `nameparser/config/` or intersect the sets in a `python3 -c`). Real overlaps exist: `do`/`st`/`mc` ∈ `PARTICLES` ∩ `TITLES`/`SUFFIX_ACRONYMS`; `abd` = "ABD" ∈ `SUFFIX_ACRONYMS`; `abu` ∈ `PARTICLES` ∩ `BOUND_GIVEN_NAMES` (position-dependent: leading token → given-name join, mid-name → family-name join). Usually position-dependent and harmless, but can force a guard or an exclusion (the `last_base` all-particles guard; dropping `abd` from `BOUND_GIVEN_NAMES`). Then check the word against real names, not memory: `uv run python tools/name_usage.py <word>` counts its bearers as a given name and as a family name (US Census surnames, SSA given names, Wikidata with countries of citizenship). Pass a name as it is written (`McDonald`): Wikidata labels match exactly, and no rule rebuilds an internal capital from `mcdonald`. It reports and does not decide — read the counts against the position the set's claim acts on. And check the predicates that test a member's SPELLING rather than its membership: they encode what the set's members happened to look like when they were written. `_pieces.starts_a_credential_run` implemented rules.md#S2's "Latin" as `isascii()`, exact until #661 added `mlađi`, which would have read `mlađi Jones` differently from `stariji Jones`; the design-docs review caught it, after a first draft had credited the test to the wrong clause of S2 and proposed deleting it on a mutation that moved nothing because nothing pinned it. (#679 then dropped the script test on purpose, after measuring what it protected.) Before adding the first member of a new script, or the first accented one, grep the set's consumers for `isascii`, `isalpha` and script tests, and read the rule each one implements. The same goes for the first member holding PUNCTUATION: #700 added `d'` and `az-`, the parse read them right and every snapshot move was an intended one, and `capitalized()` still wrote `D' Ivoi`, because case repair splits a token at its apostrophe or hyphen and asks the vocabulary of the stem. `capitalized()` is invisible to the snapshot (`initials()` is recorded there), so parse such a member's names through `capitalized()` and `capitalized(force=True)` in each case before calling it done.
 
 **Before adding a short/common word to `PARTICLES` globally**, test it mid-string against realistic 3-token names, not just check for English-word collisions: Korean/Vietnamese given names put a short syllable in the middle slot (`Park In Hwan`, `Nguyen To Nga`), and Western names put a bare initial there (`John V. Smith`). A word that looks safe ("nobody is named 'to'") can still swallow a real middle name/initial into the last name once it's a global prefix — confirmed regressions for `to`/`in`/`an`/`ten`/`then` and bare `v` this way (PR #191).
 

docs/design/decisions.md

@@ -172,6 +172,11 @@ Decided 2026-08-19 (was Open: [#380](https://github.com/derek73/python-nameparse
   NOT DECIDED HERE: the mixed-case `Doe, Jane PhD vd Ma` (middle 'vd Ma', silent), the issue's fourth bullet. Its reading is unchanged; whether the family-comma path should report the chain's fork there is a separate question. Open: #578 (decided the same day, the S2 section's #578 entry: silent, as it reads). Nor the positional read of a part before a comma followed by no name word, where the case still splits the word: `Smith vd Ma, Prof.` reads given 'Smith', family 'vd Ma', and `SMITH VD MA, PROF.` family 'SMITH', suffix 'VD MA', on this tree and on `e56dd296` alike — that part reads as the comma-less name does, and the comma-less name splits the same way.
   BLAST RADIUS, measured 2026-10-02 with `e56dd296` (the pre-change master) as the comparator. The gate's output is byte-identical at all five baselines apart from the tree path: no corpus name holds these shapes, the rules.md examples aside. Over a generated grid, 5,676 of 409,498 parses move, every one in one of two classes — 3,534 where a family gains the words behind its particle, 2,142 where `vd` or `mc` leaves the family for the credential run — and no list of ambiguity kinds changes; 1,310 of the movers are written in mixed case. Recompute: from the pool `vd mc de van do Ma MA PhD Jr John Berg V la Prof.`, build every `<1 or 2 words>, <exactly 2 words>`, `<3 words>, <1 word>`, `Smith <the same 1-3 words>, John`, `Doe, Jane <2-4 words>` and `Smith, <2-4 words>`; deduplicate, then parse each as written, `.upper()` and `.lower()`, keyed by the resulting text; keep the entries whose `as_dict()` or sorted ambiguity kinds differ between the trees, and class each by whether the new family holds every old family word and more, or the old family lost only `vd`/`mc` while the suffix kept every word. Cost: the given-slot test was QUADRATIC as first written, rescanning the particle run once per member — the second review measured 29,670 → 165,591 frames on `SMITH, VD ` + `DE `×k + `MA `×k at k = 256 against `e56dd296`, where an earlier wall-clock check at k = 100 and 400 had read 3.18× and passed for linear. The run's end is now kept once per title set: 1.90–1.99× per doubling of k from 16 to 256, and `_REREAD_SHAPES["lead"]` in tests/v2/test_benchmark.py guards it (generator frames at 8 units against 32: 3.38× kept, 7.53× rescanning, bound 6.0). The frame-count reference band did not move.
 - 2026-10-06 #613 — THE COMMA SITE IS DECIDED IN THE PASS THAT READS THE GIVEN PART. #611 found P6 showing the decisive sign smaller: assign held back from #602's run exactly the particle tail post_rules would attach a stage later, by asking `particle_tail` over roles that meant "not yet read" there and "suffix" in post_rules. The attachment and its third exception (`_inside_a_credential_run`) moved into assign, right after the given-part walk, and `particle_tail` is asked ONCE, at the moment the walk reaches #602's run: the pieces in front of the run hold the walk's roles and the run's words none, the state in which its answer equals the one over the finished roles (argued piece by piece in its docstring). post_rules ran P1's leading fold before P6, so assign asks P1's condition itself (`is_lone_never_given_particle` on the given piece), which is what keeps `Smith, de Mesnil van` the fold's. H1 and M4, the other rules ahead of the old site, are gated on there being no family, which a family comma always has. The no-comma site (#467) stays in post_rules, since it runs over the order P1's family-first redistribution leaves. NOTHING MOVES, measured 2026-10-06 against the commit before it: the P6 grid (103,786 distinct texts: each of the heads `Berg, Jan`, `Berg, Jan Piet`, `BERG, JAN`, `Berg, Dr. Jan`, `Berg, de`, `Berg, van`, `Berg, abdul`, `Smith, de Mesnil`, `de la, Jan` and `Nguyen, Thi`, followed by every sequence of zero to three of `van de der vd VD mc do DO Do PhD Jr. MA Ma Prof. V la Piet`, with and without a trailing `, Jr.` -- under the default policy, `lenient_comma_suffixes=False`, a family-first order and `middle_as_family`; comparator every token's role and folded or unjoined tag, every ambiguity's kind, detail and token IN ORDER, and `initials()`) and the #613 harness's 80,423 texts are identical, the gate's output byte-identical at all five baselines but for the baseline path, and the suite unchanged. ONE SHAPE DID MOVE, and the PR review found it on a grid with an EMPTY part before the comma, a head no measurement above held: there no family is named, H1 and M4 are not gated off, and they had made the given word the family before the old site attached -- `, Mr. Jones vd` read title 'Mr.', family 'Jones', suffix 'vd', and assign attaching first read given 'Jones', family 'vd' (177 texts on that grid). Reading P6 strictly instead (no family named, no attachment) moved 1,302 texts against master, `, Jan van` among them, so the attachment behind an empty family part stays in post_rules, after H1 and M4, through the same helper; that grid now differs from b547c247 only in C1's decided classes. Frames drop on every family-comma name, post_rules' block having paid a second `particle_tail` and a generator-driven search for the given word on every one, tail or not, where assign now attaches only behind a non-empty tail: `Smith, John` 169 → 160, `Smith, John, Jr.` 213 → 204, `Doe, Jane nee Smith` 302 → 293, a no-comma name unchanged.
+- 2026-10-17 #700 — THE ELIDED PARTICLE AND THE HYPHEN-ATTACHED ARABIC ARTICLE ARE PARTICLE VOCABULARY, NEVER-GIVEN, AND THE RULE IS UNCHANGED. Library catalogs file `Paul d'Ivoi` as `Ivoi, Paul d'` and `Ḥusnī az-Zaʿīm` as `Zaʿīm, Ḥusnī az-`, the particle split off with its mark still on it; both read as a middle name (1.4.0 and 2.4 alike) because neither written form was in `PARTICLES`. P6 asks only the particle tag, so the fix is the vocabulary alone: no code moved. Three decisions, each taken by Derek on the #700 session's options. ELIDED SET: `d'`, `l'`, `dell'`, each also in its U+2019 spelling, and `de’` and `’t` beside the existing `de'` and `'t` (`’t` added in the PR review, a gap the test review found), since nothing folds the typographic apostrophe to the ASCII one; particles.py asserts at import that every elided entry has both spellings. ARTICLES: a CLOSED list — `al-`, `ul-`, the article assimilated to each of the fourteen sun letters in ASCII and DMG spellings (`ar-`, `az-`, `aš-`, `ash-`, `aṭ-`, …; lām's being `al-` itself), and the Egyptian `el-` with its assimilations in ASCII spellings only (`es-`, `esh-`, `ed-`, …; no `eš-` or `eṣ-`) — all never-given, where the bare `al` stays ambiguous (Al Gore): written apart with its hyphen, the word is the article and nothing else. RENDERING: the attached family joins with a space like every particle, `d' Ivoi` and `al- Rashid`.
+  Declined: a shape rule (a vowel, consonants, a hyphen) in place of the list. It would need a new predicate and the bare-word collision check the issue named, for a population the list already covers; the corpora attest seven article spellings (`al- az- aš- ar- es- as- aṭ-`), all in it — recompute: every whitespace- or comma-bounded token of `tools/differential/corpus*.jsonl` matching `\w{1,4}-` at a name's end.
+  Not decided here: the GLUED forms (`Paul d'Ivoi`, `Harun al-Rashid`) stay one word with no particle split, out of #700's scope by its own statement. Hyphen-minus only: U+2010 and the non-breaking hyphen are not listed. The Scottish Gaelic `a’` (`Mac a’ Ghobhainn`, in corpus_cldr.jsonl) is mid-name and not an index-form particle, and was not added. Nor are the elisions no corpus holds (`dall'`, `nell'`, `degl'`): the list is closed, so such a spelling reads as a middle name until it is listed.
+  RENDERING FOLLOWED THE VOCABULARY, found in review: R4's `capitalized()` split a token at its mark and asked the particle test of the stem, so `IVOI, PAUL D'` repaired to `Paul D' Ivoi` — 30 of the 32 first-listed spellings, all but `al-` and `de’`, whose stems are particles themselves. The whole token is now asked the particle arm's question first, and lowercased only where that arm would lowercase it; wherever the arm declines (an all-particle part, the given or suffix role, a recorded casing) the token goes to the per-word walk as before — a first version sent every whole-token particle down the arm's other clauses instead, which moved `smith, john, ed-` from 'ED-' to 'Ed-' and a configured `van-der` from 'Van-Der' to 'Van-der' (the fix commit's own review). The one shipped word outside #700 it reaches is `'t`, whose stem is `t`: `van 't Hoff` repairs to 'van 't Hoff', where 1.4.0 through 2.4.2 wrote 'van 'T Hoff' — R4's reading, and the right Dutch one. A casing configured for a stem (`capitalization_exceptions` `al` → `AL`) no longer reaches the marked spelling, which is now a particle in its own right: `al-` is lowercased. The snapshot cannot see a render view (decisions.md#R4); measured instead by `capitalized()` and `capitalized(force=True)` against `00dd2759`, over every name in `de6c045d`'s `tools/differential/corpus*.jsonl` as written, lowercased and uppercased (28 names move, every move lowercasing one token ending in `'` or `-`), and over every particle in eight positions under all three orders (adding `'T` → `'t`, which no corpus name held until R4's `van 't hoff` example joined corpus_rules.jsonl). The boundary `Ivoi, d'` was taken out of rules.md#P6: P6 does not attach there, and the reading it gets instead (family `Ivoi d'`, as `Jong, de` has read since 2.0) is the never-given vocabulary's family-comma reading, which P6's statement does not describe — a P6 example would have pinned another rule's output. It moves from given `d'` (1.4.0 through 2.4.2, the word being no particle there), and so does every new entry standing alone after a comma (`Smith, al-`); it stays pinned as a case row.
+  Measured 2026-10-17 against the parent commit `b4a004c3`: 39 corpus names move, every one the same shape — a trailing `d'` or article leaves the middle name and joins the family (38 in corpus_gnd.jsonl, `Orval, Pierre d'` in corpus_conventions.jsonl), and no ambiguity kind changes. Recompute: `uv run python tools/differential/snapshot.py --since b4a004c3`. That is evidence about the corpus as much as the change: these are the comma-bearing names ending in one of the new forms, and GND's index forms are where they occur.
 
 ### P7 — a one-letter particle against the initial it spells
 

docs/design/rules.md

@@ -949,6 +949,18 @@ P6. Rationale: a particle ending the name has nothing to link
       "doe, jane phd vd ma"       →  suffix="phd vd ma"
       "DOE, JANE PHD VD"          →  family="VD DOE"  · boundary
       "DOE, JANE MA VD PHD"       →  suffix="MA VD PHD"
+    A particle an index form writes apart with its elision or its
+    hyphen still on it is a particle like any other here, where the
+    vocabulary lists that spelling — the elided "d'" and the Arabic
+    article in its plain and assimilated spellings, "al-" among them
+    — and of
+    the never-given kind, since written apart the word is the article
+    and nothing else. It joins with a space like every particle;
+    written glued to the family ("d'Ivoi") it is one word and this
+    rule never sees it.
+      "Ivoi, Paul d'"             →  family="d' Ivoi"
+      "Rashid, Harun al-"         →  family="al- Rashid"
+      "Zaʿīm, Ḥusnī az-"          →  family="az- Zaʿīm"
     Without a comma, a declared family-first order has named the
     family in the same way and the attachment fires there too — but
     only where the run ENDS the name and stands in a MIDDLE — the one
@@ -2862,6 +2874,9 @@ R4. Rationale: case repair is a display concern, applied only on
       "ANH DO"                    →  capitalized="Anh Do"
       "anh van do"                →  capitalized="Anh Van Do"
       "SEÁN Ó MURCHÚ"             →  capitalized="Seán Ó Murchú"
+      "IVOI, PAUL D'"             →  capitalized="Paul d' Ivoi"
+      "zaʿīm, ḥusnī az-"          →  capitalized="Ḥusnī az- Zaʿīm"
+      "jacobus henricus van 't hoff"  →  capitalized="Jacobus Henricus van 't Hoff"
       "ina binti navalamar"       →  capitalized="Ina binti Navalamar"  · boundary
       "john smith phd"            →  capitalized="John Smith PhD"
       "john smith ph.d."          →  capitalized="John Smith Ph.D."