Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -278,7 +278,7 @@ Add a dedicated `copy.deepcopy()` round-trip test for it too (see `test_regexes_

**Adding a flag-gated post-parse transform** (reorder/adjust) — add a `Constants` boolean (default `False`), implement a `handle_*()` method, and call it in `post_process()` after `handle_firstnames()` and before `handle_capitalization()`, gated on the flag. Default-off keeps existing parses byte-for-byte unchanged. Two shipped examples: `patronymic_name_order` gates both `handle_east_slavic_patronymic_name_order()` (#85) and `handle_turkic_patronymic_name_order()` (#185) — one flag driving two independent handlers, added in the same `post_process()` slot; `middle_name_as_last` gates `handle_middle_name_as_last()` (#133), which folds `middle_list` into `last_list`.

**Validating a new parsing rule** — before implementing, simulate it in a throwaway script against `TEST_NAMES` (plus a few target-language examples) to catch regressions/false-positives early. E.g. this surfaced the `last_base` `do`/`st`/`mc` empties and the patronymic `"David Michael Abramovich"` false-reorder. After the change, `tests/v2/test_snapshot.py` lists every corpus name it moved, grouped by changed field -- read them, and re-record with `uv run python tools/differential/snapshot.py --record` once every move is intended, so the PR diff of `snapshot.jsonl` shows the reviewer the same list. `uv run python tools/differential/cldr_agreement.py` reports how the tree's agreement with CLDR's stated name fields moved; `tests/v2/test_cldr_agreement.py` fails on any move, and `tools/differential/cldr_agreement.json` doubles as a measured list of names the parser and CLDR read differently. The same command and test score the GND corpus's real names (`gnd_fields.jsonl`, record `gnd_agreement.json`) by the same rules, and `--record` rewrites every record, printing per source what it accepts. A locale-pack change reads the second view the same command prints, per pack (`cldr_agreement_locales.json`, #672, and `wikidata_native_agreement_locales.json` for real Han and kana names); its `JA` rows run only with the `ja` extra installed. A change to Han, Hangul or kana order or to Korean segmentation reads `wikidata_native_agreement.json` too: the only real names in those scripts the corpora hold. It holds no honorific, so a change to the honorific peel moves nothing there. A change to comma or credential handling also reads the local corpus of hand-typed names (`snapshot.py --local ctgov-officials`, recorded first against the base checkout with `--tree` (a base read once from the same fetch is copied rather than parsed again; `--sample N` reads a fixed sample while iterating); tools/differential/README.md's "Local corpora"): the one population of names nobody chose for a test or a catalog, so it shows how often each shape is typed, and the one no CI run sees. A change to title handling reads `ukec-individuals` the same way: UK donors' names in spoken order and mixed case, in 8,219 of whose 15,214 `parse()` read a title on 2026-10-22. A change to particles or generational words reads `snapshot.py --local tse-donors` the same way, Brazilian names in spoken order dense with both. A change to initials, to a title word outside English, or to how a script's combining marks are read reads `snapshot.py --local nepal-candidates` the same way: Nepal's local election candidates, each a romanized name (surnames written as initials, `B.K.`, `K.C.`; honorific titles such as `Shekh` in front, and `Md.`, which its period makes a title by shape) beside the same name in Devanagari, written with vowel signs and the virama, combining marks the letter-shape tests do not count as letters (decisions/nfd-shapes), and with the joiners ZWJ and ZWNJ. It is the only large population of South Asian names the corpora hold, and its corpus reads both scripts. A change to non-Latin scripts or to Arabic-script names reads `snapshot.py --local jrc-names` the same way, and `local_corpora.py groups jrc-names`: the spellings of one person across scripts and transliterations, harvested from news text, with 17,074 Arabic-script spellings where the committed corpora hold 315 distinct names with an Arabic letter, 96 of them CLDR's fictional renderings (2026-10-24), and under a licence that keeps even a reading of one of its names in the cache. A change to which field a word lands in also reads `local_agreement.py psc-individuals`, recorded the same way: real names with the four boxes their filers typed them into (UK companies' people with significant control), scored field by field and by nationality. It is the only real fielded population outside the committed fields files, and no CI run sees it either.
**Validating a new parsing rule** — before implementing, simulate it in a throwaway script against `TEST_NAMES` (plus a few target-language examples) to catch regressions/false-positives early. E.g. this surfaced the `last_base` `do`/`st`/`mc` empties and the patronymic `"David Michael Abramovich"` false-reorder. After the change, `tests/v2/test_snapshot.py` lists every corpus name it moved, grouped by changed field -- read them, and re-record with `uv run python tools/differential/snapshot.py --record` once every move is intended, so the PR diff of `snapshot.jsonl` shows the reviewer the same list. `uv run python tools/differential/cldr_agreement.py` reports how the tree's agreement with CLDR's stated name fields moved; `tests/v2/test_cldr_agreement.py` fails on any move, and `tools/differential/cldr_agreement.json` doubles as a measured list of names the parser and CLDR read differently. The same command and test score the GND corpus's real names (`gnd_fields.jsonl`, record `gnd_agreement.json`) by the same rules, and `--record` rewrites every record, printing per source what it accepts. A locale-pack change reads the second view the same command prints, per pack (`cldr_agreement_locales.json`, #672, and `wikidata_native_agreement_locales.json` for real Han and kana names); its `JA` rows run only with the `ja` extra installed. A change to Han, Hangul or kana order or to Korean segmentation reads `wikidata_native_agreement.json` too: the only real names in those scripts the corpora hold. It holds no honorific, so a change to the honorific peel moves nothing there. A change to comma or credential handling also reads the local corpus of hand-typed names (`snapshot.py --local ctgov-officials`, recorded first against the base checkout with `--tree` (a base read once from the same fetch is copied rather than parsed again; `--sample N` reads a fixed sample while iterating; the recorded `snapshot.jsonl` and psc's `agreement.jsonl` are one file per source in the shared cache, so a session measuring in another worktree overwrites your baseline: record it again right before each check, as #752's phase 2f found when ctgov reported a sibling branch's 1,334 joined-credential readings as its own moves); tools/differential/README.md's "Local corpora"): the one population of names nobody chose for a test or a catalog, so it shows how often each shape is typed, and the one no CI run sees. A change to title handling reads `ukec-individuals` the same way: UK donors' names in spoken order and mixed case, in 8,219 of whose 15,214 `parse()` read a title on 2026-10-22. A change to particles or generational words reads `snapshot.py --local tse-donors` the same way, Brazilian names in spoken order dense with both. A change to initials, to a title word outside English, or to how a script's combining marks are read reads `snapshot.py --local nepal-candidates` the same way: Nepal's local election candidates, each a romanized name (surnames written as initials, `B.K.`, `K.C.`; honorific titles such as `Shekh` in front, and `Md.`, which its period makes a title by shape) beside the same name in Devanagari, written with vowel signs and the virama, combining marks the letter-shape tests do not count as letters (decisions/nfd-shapes), and with the joiners ZWJ and ZWNJ. It is the only large population of South Asian names the corpora hold, and its corpus reads both scripts. A change to non-Latin scripts or to Arabic-script names reads `snapshot.py --local jrc-names` the same way, and `local_corpora.py groups jrc-names`: the spellings of one person across scripts and transliterations, harvested from news text, with 17,074 Arabic-script spellings where the committed corpora hold 315 distinct names with an Arabic letter, 96 of them CLDR's fictional renderings (2026-10-24), and under a licence that keeps even a reading of one of its names in the cache. A change to which field a word lands in also reads `local_agreement.py psc-individuals`, recorded the same way: real names with the four boxes their filers typed them into (UK companies' people with significant control), scored field by field and by nationality. It is the only real fielded population outside the committed fields files, and no CI run sees it either.

## Gotchas

Expand Down
6 changes: 4 additions & 2 deletions docs/design/decisions/C1-suffix-comma-decision.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,11 +10,11 @@
- The CJK stage asks the comma decision itself rather than a copy of it — 2026-10-06 #613, the CJK radar tier.
- An empty part before the comma is read by the decision, not left to the listing walk — 2026-10-06 #613, the PR review.
- A bound caps word that carries the name's case contrast keeps its capitals' marks, so case repair leaves it as written (rules/R4) — 2026-10-06 #613, the PR review.
- A generation ending the name before the comma meets paired initials after it this way: dotted, they stay the given name and the generation the suffix (C5); undotted as two capitals, they are the credential (C12); a roman numeral S10 reads does not yet stay the suffix, tracked by #774 — 2026-10-19 #761.
- A generation ending the name before the comma meets paired initials after it this way: dotted, they stay the given name and the generation the suffix (C5); undotted as two capitals, they are the credential (C12); a roman numeral S10 reads stays the suffix too — 2026-10-19 #761; 2026-10-25 #774.

Not standing: 2026-10-04 #611's "the rethinks are filed, not done" — #613 rethought C1 (folding P6 in) and #614 S2, both shipped in 2.4.0, which overtook its "neither for 2.4".

Open: #774 — a roman numeral S10 reads by shape is lost into the family before dotted paired initials (`John Smith V, M.J.`).
Open: none in this file.

## Log

Expand All @@ -34,3 +34,5 @@ Open: #774 — a roman numeral S10 reads by shape is lost into the family before
SUPERSEDES: decisions/C4's 2026-10-01 #562 bullet's mechanism and its accepted reports (its pair report and the settled-run test); decisions/C13's 2026-10-04 #603 bullet's "segment decides" wording; and decisions/C3's #575 bullet's account of WHY segment builds its unit facts from vocabulary -- `_comma._whole_name` counts over classify's tags, and `name_word_count`, which reads the vocabulary, is asked by decide and by script_segment, the agreement sweep `test_surname_unit_tags_agree_with_classify` keeping the vocabulary view equal to the tags.
- 2026-10-18 #728 — SPLIT. rules/C1 held what are now C1 and C3–C11, and this file their history; each entry moved to the file of the rule it decided, and the #429, #436/#437 and #511 entries, which decide R1's entry pass, to decisions/R1. The entries kept here moved nothing but the pointers the split broke and the measurement paragraphs, each cut to one line naming its population, counts, baseline and PR. The #611 entry stays here, an open question on #728.
- 2026-10-19 (Derek), #761 — A GENERATION BEFORE THE COMMA AGAINST PAIRED INITIALS AFTER IT: DOTTED, THE INITIALS ARE THE GIVEN NAME AND THE GENERATION STAYS THE SUFFIX; UNDOTTED, THEY ARE THE CREDENTIAL. rules/C12 (a suffix word ending the name before the comma makes the comma a suffix comma) and rules/C5 (paired initials after the comma are the given name) both claim `John Smith Jr., M.J.`, and no rule text said which wins; the C section's move to the skeleton format, which gives a hub Precedence bullets, asked. Measured on 732a2716: `John Smith Jr., M.J.` reads given 'M.J.', family 'John Smith', suffix 'Jr.', and `John Smith Jr., MJ` reads given 'John', family 'Smith', suffix 'Jr., MJ', each reporting `suffix-or-name`. Decided: that reading is the rule, for every generation C12 reads, written in rules/C1's Precedence with both as examples and in C5's and C12's Precedence where they meet. It is the reading C12's Accepted clause (`Lloret Palol II, JA`) and its `Smith Jones Jr., J.A.` boundary already gave behind a two-word surname. The review found a roman numeral S10 reads by shape breaking it today: `John Smith V, M.J.` gives given 'M.J.', family 'John Smith V' and no suffix, while `John Smith V, MJ` reads suffix 'V, MJ'. So does every numeral S10 reads (`John Smith VI, M.J.`, `I`, `IX`, `X` alike). Derek ruled that a bug, so C1's Precedence carries the intended reading as two `deviates:` examples, V for the one-letter count and VI for the shape-only path, tracked by #774. The same pass collected into C1's Precedence the order among C4, C5 and C6 that their own texts already state (C4 counts name words first and asks written case second, C5 is the exception to that count, C6 gates when an unlisted all-caps word joins the class), deciding nothing new there; and the strict switch left C1 for rules/C14 (decisions/C14).
- 2026-10-25 #774 — A NUMERAL ENDING THE FAMILY PART IS READ AS THE COMMA-LESS NAME READS IT. The family part's walk took a word into the suffix by the suffix-piece test alone, so a listed generation stayed the suffix (`John Smith III, M.J.` → suffix 'III') while `V`, which the initial veto refuses, and `VI`, which no list holds, went into the family (`John Smith V, M.J.` → family 'John Smith V'), the split rules/C1's #761 Precedence example tracked as a `deviates:` row. The walk now asks the numeral's predicate (decisions/vocabulary-model's 2026-10-25 2f entry) of a single-word piece shaped as a roman numeral with nothing but suffix pieces behind it: the generation where it reads as one, a one-letter numeral needing two pieces of the part in front of it, as S10's count asks of the comma-less name's pieces, and reported where a shape reads it, as the comma-less name reports. `John Smith V, M.J.` and `John Smith VI, M.J.` keep the numeral as the suffix, `Smith VI, John` reads suffix 'VI', and `Smith V, John` keeps family 'Smith V' (one word in front), silent, the comma having named the family. The two `deviates:` markers come off.
ACCEPTED, found by the PR's reviews: a part holding a connective surname reads its one-letter numeral as its comma-less twin does only where the two join the connective alike. The part before the comma joins `Vega y Lopez` into one piece where the comma-less `Vega y Lopez V` leaves it unjoined, so `Vega y Lopez V, Mary` keeps family 'Vega y Lopez V' where the twin reads suffix 'V'. Counting a join's words instead fixed that row and broke the longer joins, which both spellings join (`Smith and Jones V, Mary`, `Ortega y Gasset y Lopez V, Mary`, family 'V' comma-less and suffix 'V' with the count), so the piece count stands. Also as on master: `St Smith V, Mary` (a title-particle is no title run before a family comma, rules/H6) and `Jane nee Smith VI, Mary` (the clause keeps the numeral) differ from their twins.
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@

Elsewhere: paired initials after a generation, dotted the given name and undotted the credential (`John Smith Jr., M.J.` → given 'M.J.'; `John Smith Jr., MJ` → suffix 'Jr., MJ') — decisions/C1's 2026-10-19 (Derek), #761 entry; `mbchb`, whose addition #656 declined, is listed (`Smith, MBChB` → suffix 'MBChB') — decisions/suffix-acronym-collisions' 2026-10-21 (#797) entry.

Open: #774 — a roman numeral S10 reads is lost into the family before dotted paired initials (`John Smith V, M.J.`), decisions/C1's Open line.
Open: none in this file.

## Log

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@

Elsewhere: paired initials are the exception to the ambiguous class's count, read as the given name after the comma (`García Márquez, G.J.` → given 'G.J.', family 'García Márquez'), and what in front of them makes them the credential — decisions/S9's 2026-09-30 (Derek), #563 entry; behind a one-word particle surname (`De La Cruz, M.J. K.L.` → given 'M.J.', middle 'K.L.', the second pair ending the given part as decisions/S3's 2026-10-24 #864 entry reads it) — decisions/C3's 2026-10-01 (Derek), #575 entry; a generation ending the name before the comma keeps dotted paired initials the given name and makes two undotted capitals the credential (`John Smith Jr., M.J.` → given 'M.J.', suffix 'Jr.'; `John Smith Jr., MJ` → suffix 'Jr., MJ') — decisions/C1's 2026-10-19 (Derek), #761 entry.

Open: #774 — a roman numeral S10 reads is lost into the family before dotted paired initials (`John Smith V, M.J.` → family 'John Smith V'), decisions/C1's Open line.
Open: none in this file.

## Log

Expand Down
Loading
Loading