Add the reading index and share the vocabulary compiler (#753, 3a)

docs/design/decisions/vocabulary-model.md

@@ -10,7 +10,8 @@ The vocabulary was a set of base sets plus marker subsets (`particles` and `part
 - The default loads every language and keeps working for names of any language. Choosing a language is to narrow it in phase 3 (#753): a pack is to settle only the clashes where its language has a vocabulary reading, a strict pack to drop other languages' name-level readings too, and a pack's preferred reading to be able to outrank one a shape rule derives — 2026-10-15/16.
 - `e`, `y` and `i` stay in the default; choosing a language is to settle their one-case fork, which no shipped pack does yet — 2026-10-15/16.
 - A patronymic field is to come in 3.0, as a view that names its words without emptying `family`, and the particle kinds already draw the `clan`/`patronymic` line for it: a patronymic is followed by a parent's name, a clan prefix by a family name the lineage shares — 2026-10-15/16; 2026-10-19 #749; 2026-10-28 #754 (Derek).
-- A language pack is a choice of languages on `Locale.languages` selecting which readings compile, a tag choosing its own file and its ancestors', and a reading index loaded for every caller carries the languages to run time; strict waits for 3i, and third-party language files are deferred — 2026-10-27 #753 (Derek).
+- A language pack is a choice of languages on `Locale.languages` selecting which readings compile, a tag choosing its own file and its ancestors', and a reading index, loaded on its first lookup, carries the languages to run time; strict waits for 3i, and third-party language files are deferred — 2026-10-27 #753 (Derek); 2026-10-29 #753, phase 3a (Derek).
+- A name reading written `default = false` derives no marker in the default and counts for no check guarding the default's compile; a particle whose word's only `given` readings are off is refused until a pack needs one — 2026-10-29 #753, phase 3a.
 - A caller configures vocabulary by reading, `Lexicon` deriving the sets and gates the build derives, and the compiled sets become internal in 3.0 — 2026-10-28 #754, the `Lexicon` surface (Derek).
 - A `Locale` may carry CLDR-shaped render patterns, display and sorting, a later addition whose shape is agreed; how a call reaches one is open — 2026-10-28 #754, render patterns.
 - 3.0's output model keeps the seven roles and adds every new field as a view, read from `Token.reading`, the reading each token took, or for `surname2` from rules/O6's tag: `generation` and `credentials` partitioning `suffix`, `patronymic`, and `surname2` where rules/O6 read two family names; `middle` and `surnames` keep their meaning, a compound given name being a rule a caller or pack turns on; the default `str()` renders a name in the order it was read in with its written separators in a normalized form, through `render(spec: str | None = None)`, `None` being that default; a language file's `[rules]` compile into its `Locale`'s policy patch — 2026-10-28 #754 (Derek).
@@ -272,3 +273,12 @@ Open: #913 (the title words phase 2j left, read as titles though borne as names)
   10. **What stays out of the output model.** Gender (decision 9) waits, `Token.reading` able to carry it later as one more addition. Names of given names only are a rule's question, not a field's: ORCID's empty family box holds 68,006 of 1,831,533 records, 15.9% to 18.7% of those from India, Pakistan, Bangladesh and Indonesia, and the parse fills `family` in 51,367 of the 51,415 with two or more given-box words. #694's two-word surname after a given name (`Tariq Abdul Aziz`) is a parse rule for the default, outside phase 4. Phase 1's four reserved names said to be phase 4's (`separators`, `bare_only`, `leading_only`, `trailing_only`) have no sub-phase yet.
 - **2026-10-28 #754, render patterns — A `Locale` MAY CARRY CLDR-SHAPED RENDER PATTERNS (A GIVEN-FIRST AND A SURNAME-FIRST DISPLAY PATTERN AND A SORTING PATTERN), A SHAPE AGREED NOW FOR A FEATURE THAT COMES LATER AS AN ADDITION.** The RFC's Phase 4 section, "Render patterns a pack may carry", holds the shape: the pack field, a table of Phase 4's views against CLDR's person-name fields, and the gap it shows, CLDR's `surname-prefix` and `surname-core` belonging to the first surname where `family_particles` and `family_base` read the whole family, which the per-surname units close. CLDR is a starting point, not the authority: its `pt` sorting pattern files under the first surname and leaves `surname2` out, which is not ABNT's last-surname entry (decisions/R2). Proposed there, not decided: a casefolded key method beside `comparison_key()`, collation left to the caller; and, open for a decision, whether a pattern is reached through a convention recorded on the name or through the parser.
 - **2026-10-28 #754, the `Lexicon` surface (Derek) — A CALLER CONFIGURES VOCABULARY BY READING, AND THE COMPILED SETS BECOME INTERNAL.** Derek decided in chat on 2026-10-28, choosing it over keeping a public field per compiled set and over keeping 2.0.0's eleven while hiding the eleven added since. A caller states what a word may be in the language files' terms (a `given` reading on `guru`, no credential reading on `dc`) and `Lexicon` derives the sets and gates as the build does. The reasons: the set mechanics are hard to keep straight even for the maintainer, a reading is what a caller can know about a word, and it is the form a pack contribution takes. Today the same configuration needs rules/H9's set algebra, `Lexicon.default().remove(titles={'guru'}).add(titles_ambiguous={'guru'})` (`Guru Prasad Sharma` → given `Guru`) and `remove(suffix_acronyms={'dc'})` (`Ram Bahadur DC` → family `DC`), while `add(suffix_acronyms_ambiguous={'dc'})` leaves suffix `DC` (2026-10-28, run on this branch's base). A caller's own removal may settle a clash: decision 15 binds the default and the packs, not a caller's data. `dataclasses.fields(Lexicon)` gives 22 public fields on 2026-10-28, eleven since 2.0.0's eleven; the RFC's Phase 4 "Left open" holds the weighing. A break of the frozen `Lexicon`, recorded in decisions/3-0-reevaluations. Open: the method names, and whether a caller's own language file can join the pool (phase 3's item 8 defers third-party files). The 2.x warning the freeze asks for was waived on 2026-10-29, 2.8 being the last 2.x feature release (decisions/3-0-reevaluations, THE LEXICON WARNING).
+
+- **2026-10-29 #753, phase 3a — THE READING INDEX AND THE SHARED COMPILER LAND, AND A NAME READING MAY BE WRITTEN `default = false`; NO PARSE MOVES.** Carries out items 3 and 5 of the 2026-10-27 #753 entry, and 3a of its item 11.
+  - **What landed.** tools/vocab/build.py writes a second generated module, nameparser/config/_index.py, beside _compiled.py: for each word, one row per reading in every file, on by default or not, `(tag, kind, properties)`, the properties those the compiler reads (each category's behavioral ones, and `same_as` on a `given` reading); counts, notes, exclusions and the stated `may_be_given` stay out, being checked at build time and compiled to nothing. The compiler moved into nameparser/_reading_index.py (`compile_readings`, with `CATEGORY`, `BEHAVIORAL` and `GATED_WORDS`), which the build loads by path as it loads _lexicon.py; the build keeps the checks and compiles _compiled.py from the index rows, so the module the parser reads is the index compiled. tests/v2/test_reading_index.py holds the no-choice invariant: `compile_readings(index())`, rendered as the build renders it, is _compiled.py byte for byte. The run-time reader is `readings_of(text)`, every `IndexReading` (text, tag, kind, props) of a word, for `Reading.languages`, which 4a (#942) adds but leaves unfilled and a follow-up after 3a and 4a both merge fills; `IndexReading.kind` is the vocabulary's kind (`credential`, `preposition`), and mapping it to a `ReadingKind` is that follow-up's.
+  - **A name reading written `default = false` derives no marker in the default** (item 3, the half before a pack turns it on): the word's other readings compile as they would without it. So it counts for none of the checks guarding the default's compile: an agnomen on by default needs a name reading on by default (check 11, else it reads ungated), and a bound given name that is also a particle, which needs a `given` reading on by default (else Lexicon raises at import), is refused by check 11's particle half below before check 9 reads it. A name reading off beside a suffix word kind no gate reads is still refused, gating nothing when on either.
+  - **Left open: a particle whose word's only `given` readings are off is refused.** Check 11 asks such a particle to state `may_be_given = false` exactly where its word has no `given` reading. The default compiles it never-given, a pack turning the given reading on compiles it possibly given, and which of the two the stated property describes is a choice no 3.0 pack needs yet (`nl`'s prepositions wait with strict). The build refuses the shape, naming both ways out, rather than pick a side; the first pack that needs it decides, with its measurement.
+  - **The index is written grouped by word, and an `IndexReading` is built only for a word looked up.** On a flat draft of this change, one row per line, grouping the rows at run time took about 0.56 ms; building every row's object takes about 1.4 ms (`index()`, which only a compile over the whole pool calls); each against the import's own cost, so the build writes one line per word and the lookup table is a single `dict()` call. A change to a word's readings is still a one-line diff.
+  - **The index loads on the first `readings_of` call, not inside `Lexicon.default()` (decided 2026-10-29, Derek).** SUPERSEDES: item 5's "loaded for every caller with `Lexicon.default()`". Proposed by 3a and chosen over loading it with the default lexicon. What it buys depends on the state of `Reading.languages`. Today, and until the follow-up that fills it after 3a and 4a (#942) both merge, no parse reads the index, so no caller pays the import, parsing or not, where loading it with the default lexicon would cost every parsing caller the import for nothing. Once the follow-up fills `Reading.languages`, the first parse will load it, so every parsing caller will pay it once, as item 5 measured, and only a caller who imports nameparser without parsing is spared. Wherever the follow-up calls `readings_of`, the import moves with the call.
+  - **Measured** 2026-10-29 on `9210f221` plus this change, Python 3.11.17, medians of fresh processes with bytecode cached as an installed wheel has it: importing the index takes 0.56 to 0.68 ms across three runs on a shared machine, about 3% of `import nameparser`'s 18 to 24 ms in the same runs, above the 0.45 ms item 5 measured on a flat draft of 2,140 rows; the first lookup after it, 0.07 ms; compiling the whole index, 2.1 ms; uncached, the import takes about 13 to 18 ms (13 after `import nameparser` in this session, 17 to 19 in review's runs, a figure no wheel install pays). The index holds 2,157 readings over 1,798 words, 106 of them in more than one file, and 3 readings written `default = false`; it is about 95 KB of source. Recompute: `uv run python tools/vocab/build.py`, then with `PYTHONDONTWRITEBYTECODE` unset, time `import nameparser.config._index` after `import nameparser` in fresh processes (the first run writes the bytecode), and count with `from nameparser.config._index import READINGS`.
+  - Not touched: `Lexicon` and every other pickled type gain no field, so no pickle layout moves, and 3a forces no public name; the configure-by-reading methods stay open.

docs/design/mechanisms.md

@@ -230,9 +230,9 @@ This catalog converts discovery into a one-time cost — it does not remove disc
 
 **Contract statement.** Compile the source into a committed, generated module that the runtime imports, and test that the committed module is byte-identical to a fresh compile, failing with the command that refreshes it; run the compiler's checks over the shipped source in the same test, with warnings as errors, and give each check a mutation it must refuse.
 
-**How it works.** The parser imports nameparser/config/_compiled.py, frozenset literals one entry per line so a one-word change is a one-line diff; tools/vocab/build.py writes it from nameparser/vocab/*.toml; tests/v2/test_vocab.py compiles in memory and compares. The source can grow descriptive data (notes, counts, expansions) that compiles to nothing and costs the runtime nothing. Cost: a data change touches two files and needs the tool run, which the staleness test turns from a silent drift into a CI failure carrying the fix.
+**How it works.** The parser imports nameparser/config/_compiled.py, frozenset literals one entry per line so a one-word change is a one-line diff; tools/vocab/build.py writes it from nameparser/vocab/*.toml, and beside it a second committed artifact, the reading index nameparser/config/_index.py, both from the same in-memory rows, which the shared compiler in nameparser/_reading_index.py compiles into the first (#753, phase 3a); tests/v2/test_vocab.py and tests/v2/test_reading_index.py compile in memory and compare each. The source can grow descriptive data (notes, counts, expansions) that compiles to nothing and costs the runtime nothing. Cost: a data change touches three files (the source and the two generated modules) and needs the tool run, which the staleness test turns from a silent drift into a CI failure carrying the fix.
 
-**Lives in.** tools/vocab/build.py, tests/v2/test_vocab.py (decisions/vocabulary-model).
+**Lives in.** tools/vocab/build.py, nameparser/_reading_index.py, tests/v2/test_vocab.py, tests/v2/test_reading_index.py (decisions/vocabulary-model).
 
 **Reach for it when.** A data file the runtime would otherwise parse, or a hand-kept Python copy of data that has a better home — and not for a measured expectation, which is RECORDED-ROSTERS.