diff --git a/AGENTS.md b/AGENTS.md index b6f82813..329539b7 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -256,7 +256,7 @@ The 2.0 rewrite landed as underscore-private modules beside the v1 code, which 2 **Adding a curated sub-set of an existing vocabulary set** (a marker over a base set, as `AFTER_GIVEN_TITLES` is over `TITLES`) — it is a behavioral PROPERTY of a reading, not a second word list: a row in `nameparser/_reading_index.py`'s `BEHAVIORAL` table, a branch in its `compile_readings` adding the words whose reading carries it and a key in its `Compiled` TypedDict, a constant of the same name in `tools/vocab/build.py`'s `CONSTANTS` roster (an import-time assert there holds the two to one list), and the `NAME = _compiled.NAME` assignment in the config module. A property the compiler reads is kept in the reading index by `indexed()`, which reads `BEHAVIORAL`, so a new one reaches a choice of languages too. The subset relation then holds by construction, since the property cannot exist without its reading, so it needs no check. A marker that records a clash with a NAME is the other shape: derived from the word's name reading beside the clashing one (`SUFFIX_ACRONYMS_AMBIGUOUS`, `SUFFIX_WORDS_AMBIGUOUS`, and the complement `NON_GIVEN_NAME_PARTICLES` works from), not written as a property, so it too holds by construction (decisions/vocabulary-model, 2026-10-25). A relation construction does not give (`∩ == ∅` with a set the sub-set cannot logically overlap; a sub-set member that is also a title is silently inert, title handling consuming it first) goes in `check_pool`, with a mutation in `tests/v2/test_vocab.py`'s table. Do **not** forbid `titles ∩ particles` — that overlap is intentional (`freiherr`, `st`). A caller's `Lexicon.add()` to a sub-set does **not** add the word to its base, so a caller adding a brand-new word adds it to both. Two traps a gating set met (#752 phase 2i). Where a caller listing a word in the BASE set has always meant "read it as this everywhere" (the 1.4 recipe `c.titles.add("dean")`), the gated words compile OUT of the base rather than into both, or the gate overrides the caller's listing: `TITLES_AMBIGUOUS` is not a subset of `TITLES` for that reason, and a word in both reads as a title. And `_VOCAB_FIELDS` is read whole by two rosters in `_pipeline/_vocab.py`, the capitals shape's "unlisted" (`in_any_wordlist`) and the comma contrast's "claimed as non-name" (`_NON_NAME_FIELDS`), so a new field joins both by default; a field whose words may be names leaves both, which the grid found as `John Smith, DEAN` reversed where C6 had read a credential; #789's `given_name_abbreviations` met it again (`John Smith, JAS` and `Jas SMITH, XYZ` reversed), found by review because its grid held only the dotted spellings the set was for -- a new field's grid holds the spellings it must NOT move too. -**Validating a new parsing rule** — before implementing, simulate it in a throwaway script against `TEST_NAMES` (plus a few target-language examples) to catch regressions/false-positives early. E.g. this surfaced the `last_base` `do`/`st`/`mc` empties and the patronymic `"David Michael Abramovich"` false-reorder. After the change, `tests/v2/test_snapshot.py` lists every corpus name it moved, grouped by changed field -- read them, and re-record with `uv run python tools/differential/snapshot.py --record` once every move is intended, so the PR diff of `snapshot.jsonl` shows the reviewer the same list. `uv run python tools/differential/cldr_agreement.py` reports how the tree's agreement with CLDR's stated name fields moved; `tests/v2/test_cldr_agreement.py` fails on any move, and `tools/differential/cldr_agreement.json` doubles as a measured list of names the parser and CLDR read differently. The same command and test score the GND corpus's real names (`gnd_fields.jsonl`, record `gnd_agreement.json`) by the same rules, and `--record` rewrites every record, printing per source what it accepts. A locale-pack change reads the second view the same command prints, per pack (`cldr_agreement_locales.json`, #672, and `wikidata_native_agreement_locales.json` for real Han and kana names); its `JA` rows run only with the `ja` extra installed. A change to Han, Hangul or kana order or to Korean segmentation reads `wikidata_native_agreement.json` too: the only real names in those scripts the corpora hold. It holds no honorific, so a change to the honorific peel moves nothing there. A change to comma or credential handling also reads the local corpus of hand-typed names (`snapshot.py --local ctgov-officials --base `, which reads the base from a baseline kept for its package's files and never the shared snapshot beside the corpus (a base read once from the same fetch is copied rather than parsed again; `--sample N` reads a fixed sample while iterating; the shared snapshot beside the corpus is one file per source for every worktree, and #752's phase 2f found ctgov reporting a sibling branch's 1,334 joined-credential readings as its own moves against it); tools/differential/README.md's "Local corpora"): the one population of names nobody chose for a test or a catalog, so it shows how often each shape is typed, and the one no CI run sees. A change to title handling reads `ukec-individuals` the same way: UK donors' names in spoken order and mixed case, in 8,219 of whose 15,214 `parse()` read a title on 2026-10-22. A change to English titles, honorifics or post-nominals also reads `snapshot.py --local govuk-people` the same way: GOV.UK's display names of ministers, officials and officers, the title run and the post-nominals as an office writes them (`The Rt Hon`, `Sir`, `Baroness`, ranks; `KC`, `CBE`, `KCB`), in 2,093 of whose 6,021 `parse()` read one or both on 2026-10-30. A change to particles or generational words reads `snapshot.py --local tse-donors` the same way, Brazilian names in spoken order dense with both. A change to initials, to a title word outside English, or to how a script's combining marks are read reads `snapshot.py --local nepal-candidates` the same way: Nepal's local election candidates, each a romanized name (surnames written as initials, `B.K.`, `K.C.`; honorific titles such as `Guru` in front, `Shekh`, a title there until #752's phase 2j made it one only inside a title phrase, and `Md.`, which its period makes a title by shape) beside the same name in Devanagari, written with vowel signs and the virama, combining marks the letter-shape tests do not count as letters (decisions/nfd-shapes), and with the joiners ZWJ and ZWNJ. It is the only large population of South Asian names the corpora hold, and its corpus reads both scripts. A change to non-Latin scripts or to Arabic-script names reads `snapshot.py --local jrc-names` the same way, and `local_corpora.py groups jrc-names`: the spellings of one person across scripts and transliterations, harvested from news text, with 17,074 Arabic-script spellings where the committed corpora hold 315 distinct names with an Arabic letter, 96 of them CLDR's fictional renderings (2026-10-24), and under a licence that keeps even a reading of one of its names in the cache. A change to which field a word lands in also reads `local_agreement.py psc-individuals --base ` the same way: real names with the four boxes their filers typed them into (UK companies' people with significant control), scored field by field and by nationality; and `local_agreement.py orcid-researchers --base `, ORCID holders' own two boxes (given names, family name) from every country, scored by the country they work or live in -- a population each person split for themselves, academic but worldwide, the one to read for a change a language pack makes; and `local_agreement.py patentsview-inventors --base `, US patents' inventors in the two boxes their filings give, split by a professional filer, with how many people filed under each name in each country, the one to read for how common a name's word is among a country's US-patent inventors. These are the only real fielded populations outside the committed fields files, and no CI run sees them either. +**Validating a new parsing rule** — before implementing, simulate it in a throwaway script against `TEST_NAMES` (plus a few target-language examples) to catch regressions/false-positives early. E.g. this surfaced the `last_base` `do`/`st`/`mc` empties and the patronymic `"David Michael Abramovich"` false-reorder. After the change, `tests/v2/test_snapshot.py` lists every corpus name it moved, grouped by changed field -- read them, and re-record with `uv run python tools/differential/snapshot.py --record` once every move is intended, so the PR diff of `snapshot.jsonl` shows the reviewer the same list. `uv run python tools/differential/cldr_agreement.py` reports how the tree's agreement with CLDR's stated name fields moved; `tests/v2/test_cldr_agreement.py` fails on any move, and `tools/differential/cldr_agreement.json` doubles as a measured list of names the parser and CLDR read differently. The same command and test score the GND corpus's real names (`gnd_fields.jsonl`, record `gnd_agreement.json`) by the same rules, and `--record` rewrites every record, printing per source what it accepts. A locale-pack change reads the second view the same command prints, per pack (`cldr_agreement_locales.json`, #672, and `wikidata_native_agreement_locales.json` for real Han and kana names); its `JA` rows run only with the `ja` extra installed. A change to Han, Hangul or kana order or to Korean segmentation reads `wikidata_native_agreement.json` too: the only real names in those scripts the corpora hold. It holds no honorific, so a change to the honorific peel moves nothing there. A change to comma or credential handling also reads the local corpus of hand-typed names (`snapshot.py --local ctgov-officials --base `, which reads the base from a baseline kept for its package's files and never the shared snapshot beside the corpus (a base read once from the same fetch is copied rather than parsed again; `--sample N` reads a fixed sample while iterating; the shared snapshot beside the corpus is one file per source for every worktree, and #752's phase 2f found ctgov reporting a sibling branch's 1,334 joined-credential readings as its own moves against it); tools/differential/README.md's "Local corpora"): the one population of names nobody chose for a test or a catalog, so it shows how often each shape is typed, and the one no CI run sees. A change to title handling reads `ukec-individuals` the same way: UK donors' names in spoken order and mixed case, in 8,219 of whose 15,214 `parse()` read a title on 2026-10-22. A change to English titles, honorifics or post-nominals also reads `snapshot.py --local govuk-people` the same way: GOV.UK's display names of ministers, officials and officers, the title run and the post-nominals as an office writes them (`The Rt Hon`, `Sir`, `Baroness`, ranks; `KC`, `CBE`, `KCB`), in 2,093 of whose 6,021 `parse()` read one or both on 2026-10-30. A change to particles or generational words reads `snapshot.py --local tse-donors` the same way, Brazilian names in spoken order dense with both. A change to initials, to a title word outside English, or to how a script's combining marks are read reads `snapshot.py --local nepal-candidates` the same way: Nepal's local election candidates, each a romanized name (surnames written as initials, `B.K.`, `K.C.`; honorific titles such as `Guru` in front, `Shekh`, a title there until #752's phase 2j made it one only inside a title phrase, and `Md.`, which its period makes a title by shape) beside the same name in Devanagari, written with vowel signs and the virama, combining marks the letter-shape tests do not count as letters (decisions/nfd-shapes), and with the joiners ZWJ and ZWNJ. It is the only large population of South Asian names the corpora hold, and its corpus reads both scripts. A change to non-Latin scripts or to Arabic-script names reads `snapshot.py --local jrc-names` the same way, and `local_corpora.py groups jrc-names`: the spellings of one person across scripts and transliterations, harvested from news text, with 17,074 Arabic-script spellings where the committed corpora hold 315 distinct names with an Arabic letter, 96 of them CLDR's fictional renderings (2026-10-24), and under a licence that keeps even a reading of one of its names in the cache. A change to which field a word lands in also reads `local_agreement.py psc-individuals --base ` the same way: real names with the four boxes their filers typed them into (UK companies' people with significant control), scored field by field and by nationality; and `local_agreement.py orcid-researchers --base `, ORCID holders' own two boxes (given names, family name) from every country, scored by the country they work or live in -- a population each person split for themselves, academic but worldwide, the one to read for a change a language pack makes; and `local_agreement.py patentsview-inventors --base `, US patents' inventors in the two boxes their filings give, split by a professional filer, with how many people filed under each name in each country, the one to read for how common a name's word is among a country's US-patent inventors. A change to titles or post-nominals, or to the `cs` pack, reads `local_agreement.py cz-candidates --base `, and with `--locale cs` too: Czech election candidates' four boxes (the title before the name, given names, surname, the title after it), scored by election, the only population whose titles are stated apart from the name, 36% of its 170,181 records titled (2026-10-30), nearly all academic degrees; its names are composed from the boxes (`Ing. Jan Novák, Ph.D.`), so it says nothing about how a writer spaces or punctuates them. These are the only real fielded populations outside the committed fields files, and no CI run sees them either. ## Gotchas diff --git a/tests/v2/test_local_agreement.py b/tests/v2/test_local_agreement.py index 0d9484b5..4b8d96bd 100644 --- a/tests/v2/test_local_agreement.py +++ b/tests/v2/test_local_agreement.py @@ -177,7 +177,7 @@ def test_a_record_of_another_fetch_is_refused( agreement.main([_PSC]) path = _own_cache / "local" / _PSC / agreement.RECORD header = json.loads(path.read_text().splitlines()[0]) - assert set(header) == {"raw_sha256", "scorer", "data_date", "tree", "entries"} + assert set(header) == {"raw_sha256", "scorer", "data_date", "tree", "entries", "locale"} def test_a_run_with_no_record_names_the_command_that_writes_one( @@ -478,7 +478,7 @@ def test_a_score_read_in_workers_is_the_score_read_in_process( def test_a_score_in_workers_refuses_a_parser_it_cannot_send_them() -> None: from nameparser import Lexicon configured = Parser(lexicon=Lexicon.default().add(titles={"partner"})) - with pytest.raises(ValueError, match="default Parser"): + with pytest.raises(ValueError, match=r"parser_of\(''\)"): agreement.score(list(_ALL), _PSC_FIELDS, configured, jobs=2) agreement.score(list(_ALL), _PSC_FIELDS, configured) # the control: in process @@ -544,3 +544,130 @@ def test_a_fielded_source_naming_an_unknown_role_or_no_family_is_refused( fields: dict[str, str]) -> None: with pytest.raises(ValueError, match="each box names roles of"): agreement.Fielded(fields, group="nationality") + + +# -- a source with a box after the name (cz-candidates) -------------------------- + +_VOLBY = corpora.VOLBY +_VOLBY_FIELDS = agreement.FIELDED[_VOLBY] + + +def _candidate(before: str, given: str, family: str, after: str = "", + election: str = "kv2022") -> dict[str, str]: + boxes = {"titulpred": before, "jmeno": given, "prijmeni": family, "titulza": after} + return {"name": importlib.import_module("local_volby").compose(boxes), + "election": election, **{k: v for k, v in boxes.items() if v}} + + +def _volby_read(record: dict[str, str], fielded: object = _VOLBY_FIELDS + ) -> tuple[dict[str, str], bool, bool]: + """The record's reading, as _read gives it, under `fielded`.""" + reading = agreement.read(record, fielded, Parser(), _LISTED) + return reading.got, reading.set_aside, reading.by_suffix + + +def test_a_suffix_is_compared_with_the_box_after_the_name_not_the_surname() -> None: + assert _volby_read(_candidate("Ing.", "Jan", "Novák", "Ph.D.")) == ({}, False, False) + # The control: read as a source with no suffix box, the parser's + # suffix is appended to the family, which the surname box then misses. + joined = agreement.Fielded({"titulpred": "title", "jmeno": "given+middle", + "prijmeni": "family"}, group="election") + assert _volby_read(_candidate("Ing.", "Jan", "Novák", "Ph.D."), + joined)[0] == {"family": "Novák Ph.D."} + + +def test_a_title_box_the_parse_misses_is_scored_where_the_source_sets_nothing_aside() -> None: + # Bc. is a title by its shape (rules/H2) and listed in no vocabulary, + # so the title box holds an unlisted word; the parse reads the + # diploma after the name as a second title. + missed = _candidate("Bc.", "Jan", "Novák", "DiS.") + assert _volby_read(missed)[:2] == ({"title": "Bc. DiS.", "suffix": ""}, False) + # The control: a source that sets aside would excuse the parser's miss. + setting = agreement.Fielded(dict(_VOLBY_FIELDS.fields), group="election") + assert _volby_read(missed, setting)[1] + + +def test_the_report_of_a_source_with_a_suffix_box_counts_by_election( + monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr(agreement, "GROUP_FLOOR", 1) + rows = [_candidate("Ing.", "Jan", "Novák"), _candidate("", "Eva", "Malá", "DiS."), + _candidate("Mgr.", "Bo", "Dvořák", election="ps2025")] + result = agreement.score(rows, _VOLBY_FIELDS, Parser()) + assert result.scored == {"kv2022": 2, "ps2025": 1} + assert result.misses == {"kv2022": {"title": 1, "suffix": 1}} + lines = agreement.report(result, _VOLBY_FIELDS, 5) + assert lines[:2] == ["scored 3; none set aside", "every field agrees: 2 of 3 (66.7%)"] + assert not any("set-aside" in line for line in lines) + + +def test_a_locale_scores_its_pack_s_parser_against_a_record_of_its_own( + monkeypatch: pytest.MonkeyPatch, _own_cache: Path, + capsys: pytest.CaptureFixture[str]) -> None: + from nameparser import locales, parser_for + assert agreement.parser_of("cs") == parser_for(locales.CS) + assert agreement.parser_of("") == Parser() + # JUDr. et Mgr. is a run of two titles only where JUDr. is listed, + # as locales.CS lists it: the two parsers read it apart. + rows = [_candidate("JUDr. et Mgr.", "Jan", "Novák"), _candidate("Ing.", "Eva", "Malá")] + plant_corpus(monkeypatch, corpora, _VOLBY, rows, "kv2022 2026-03-28", "candidates", + {"elections": "kv2022 ps2025", "read_version": 1}) + assert agreement.main([_VOLBY, "--record"]) == 0 + assert agreement.main([_VOLBY, "--record", "--locale", "CS"]) == 0 + out = capsys.readouterr().out + assert "under parser_for(locales.CS)" in out + folder = _own_cache / "local" / _VOLBY + default, cs = (agreement.read_record(folder / name, corpora.load(_VOLBY), locale) + for name, locale in (("agreement.jsonl", ""), ("agreement.cs.jsonl", "cs"))) + assert len(default) == 1 and cs == {} + assert agreement.main([_VOLBY, "--locale", "cs"]) == 0 # its own record: no moves + with pytest.raises(SystemExit, match="scored under the locale 'cs', not '\\(default\\)'"): + agreement.read_record(folder / "agreement.cs.jsonl", corpora.load(_VOLBY)) + loaded = corpora.load(_VOLBY) + assert agreement.kept_path(_VOLBY, loaded, "d" * 64, "cs") != agreement.kept_path( + _VOLBY, loaded, "d" * 64) + + +def test_a_score_in_workers_reads_with_the_locale_s_parser( + monkeypatch: pytest.MonkeyPatch) -> None: + rows = [_candidate("JUDr. et Mgr.", "Jan", "Novák"), _candidate("Ing.", "Eva", "Malá")] + cs = agreement.parser_of("cs") + serial = agreement.score(rows, _VOLBY_FIELDS, cs) + monkeypatch.setattr(importlib.import_module("snapshot"), "PARALLEL_FLOOR", 1) + assert agreement.score(rows, _VOLBY_FIELDS, cs, jobs=2, locale="cs") == serial + assert serial != agreement.score(rows, _VOLBY_FIELDS, Parser()) # the control + with pytest.raises(ValueError, match="parser_of\\('cs'\\)"): + agreement.score(rows, _VOLBY_FIELDS, Parser(), jobs=2, locale="cs") + + +def test_an_unknown_locale_is_refused_naming_the_packs( + monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str]) -> None: + plant_corpus(monkeypatch, corpora, _VOLBY, [_candidate("Ing.", "Eva", "Malá")], + "kv2022 2026-03-28", "candidates", + {"elections": "kv2022 ps2025", "read_version": 1}) + # With no record and with a base too: the locale is refused first, not + # as "no record" or as the base's child failing on it. + for extra in (["--record"], [], ["--base", str(agreement.HERE.parents[1])]): + with pytest.raises(SystemExit) as raised: + agreement.main([_VOLBY, *extra, "--locale", "xx"]) + assert raised.value.code == 2, extra + assert "no such pack; the packs are" in capsys.readouterr().err, extra + + +def test_a_base_check_under_a_locale_scores_the_base_under_it( + monkeypatch: pytest.MonkeyPatch, _own_cache: Path, tmp_path: Path, + capsys: pytest.CaptureFixture[str]) -> None: + rows = [_candidate("JUDr. et Mgr.", "Jan", "Novák"), _candidate("Ing.", "Eva", "Malá")] + plant_corpus(monkeypatch, corpora, _VOLBY, rows, "kv2022 2026-03-28", "candidates", + {"elections": "kv2022 ps2025", "read_version": 1}) + copy = checkout_copy(tmp_path / "copy") + assert agreement.main([_VOLBY, "--base", str(copy), "--locale", "cs"]) == 0 + assert "matches the base" in capsys.readouterr().out + [kept] = (_own_cache / "local" / _VOLBY / agreement.KEPT).glob("*.jsonl") + assert kept.name.endswith("-cs.jsonl") + # The control: the default's base reads the JUDr. run apart, so its + # record, kept beside the locale's, is not the locale's. + assert agreement.main([_VOLBY, "--base", str(copy)]) == 0 + records = sorted((_own_cache / "local" / _VOLBY / agreement.KEPT).glob("*.jsonl")) + assert len(records) == 2 + assert {len(agreement.read_record(r, corpora.load(_VOLBY), "cs" if r == kept else "")) + for r in records} == {0, 1} diff --git a/tests/v2/test_local_corpora.py b/tests/v2/test_local_corpora.py index 20880504..4224e714 100644 --- a/tests/v2/test_local_corpora.py +++ b/tests/v2/test_local_corpora.py @@ -49,10 +49,10 @@ # the fetches read, so a patch reaches the code only when it is set on # the module holding the name (local_core.RETRIES, local_fec.FEC_MAX_BAD). # A second load by path (load_tool) would be a copy nothing reads. -core, ctgov, fec, tse, tw, ukec, psc, jrc, nepal, orcid, patentsview = ( +core, ctgov, fec, tse, tw, ukec, psc, jrc, nepal, orcid, patentsview, volby = ( importlib.import_module(f"local_{key}") for key in ("core", "ctgov", "fec", "tse", "tw", "ukec", "psc", "jrc", "nepal", "orcid", - "patentsview")) + "patentsview", "volby")) _SOURCE = "ctgov-officials" _DATE = "2026-10-21T09:00:05" @@ -3347,11 +3347,12 @@ def test_no_module_imports_a_sibling_constant() -> None: # module. A `from local_x import CONSTANT` would bind a second name a # patch on local_x cannot reach, and a test could pass with its guard # off. The modules are SOURCES' own, so a new source is covered. PSC, - # JRC, ORCID and PV are sources' names, keys of SOURCES, and no fetch's setting. + # JRC, ORCID, PV and VOLBY are sources' names, keys of SOURCES, and no + # fetch's setting. import ast sources = {source.fetch.__module__ for source in local.SOURCES.values()} assert sources == {m.__name__ for m in (ctgov, fec, tse, tw, ukec, psc, jrc, nepal, orcid, - patentsview)} | {"local_govuk"} + patentsview, volby)} | {"local_govuk"} siblings = {"local_corpora", "local_core", *sources} imported = {(path.stem, alias.name) for path in (Path(str(local.__file__)).parent / f"{s}.py" @@ -3360,7 +3361,8 @@ def test_no_module_imports_a_sibling_constant() -> None: if isinstance(node, ast.ImportFrom) and node.module in siblings for alias in node.names if alias.name.lstrip("_").isupper()} assert imported == {("local_corpora", "PSC"), ("local_corpora", "JRC"), - ("local_corpora", "ORCID"), ("local_corpora", "PV")} + ("local_corpora", "ORCID"), ("local_corpora", "PV"), + ("local_corpora", "VOLBY")} def test_the_fold_lists_everything_its_functions_call_and_read() -> None: diff --git a/tests/v2/test_local_volby.py b/tests/v2/test_local_volby.py new file mode 100644 index 00000000..2bcde930 --- /dev/null +++ b/tests/v2/test_local_volby.py @@ -0,0 +1,296 @@ +"""Offline checks on tools/differential/local_volby.py, the cz-candidates +local corpus of the Czech election candidate registers. + +Every name here is invented: the registers' names are real people's, +so they stay in the cache. What these pin is how a record's name is +composed from its boxes, which register zip is read off an election's +index page, and the fetch's refusals: an answer that is not a zip, a +register missing a box's column, cut inside its last row, holding +another number of candidates than its pin, with too many rows skipped +or too few titled, and a corpus of other elections or another version +of the composition. The network is a stub answering by URL. +""" +import csv +import importlib +import io +import time +import urllib.error +import urllib.request +import zipfile + +import pytest + +from ._differential_fixtures import _Response, load_tool, own_cache + +local = load_tool("local_corpora") +# The module the fetch reads its constants from, by name: a patch set on +# a second load by path would reach nothing (test_local_corpora.py). +volby = importlib.import_module("local_volby") + +_VOLBY = volby.VOLBY + + +@pytest.fixture(autouse=True) +def _own_cache(monkeypatch: pytest.MonkeyPatch, tmp_path_factory: pytest.TempPathFactory) -> None: + own_cache(monkeypatch, tmp_path_factory) + + +#: The municipal register's layout, cut to the columns read and two +#: beside them, and the Chamber register's, in another column order: +#: the boxes are read by name. +_KV_HEADER = ["DATUMVOLEB", "OKRES", "JMENO", "PRIJMENI", "TITULPRED", "TITULZA", "VEK"] +_PS_HEADER = ["VOLKRAJ", "PORCISLO", "JMENO", "PRIJMENI", "TITULZA", "TITULPRED", "VEK"] + + +def _kv(given: str, family: str, before: str = "", after: str = "") -> list[str]: + return ["20220923", "7204", given, family, before, after, "50"] + + +def _ps(given: str, family: str, before: str = "", after: str = "") -> list[str]: + return ["1", "1", given, family, after, before, "40"] + + +#: A titled candidate standing twice under the same boxes (one padded), a +#: candidate with a title after the name only, one with both, and one +#: untitled: 5 rows, 4 records. +_KV_ROWS = [_kv("Jan", "Novák", "Ing."), _kv(" Jan ", "Novák ", "Ing."), + _kv("Eva", "Malá", after="DiS."), _kv("Petr Pavel", "Veselý", "doc. MUDr.", "CSc."), + _kv("Ida", "Krátká")] +_PS_ROWS = [_ps("Jan", "Novák", "Ing."), _ps("Bo", "Dvořák Malá", "Mgr. et Mgr.")] + + +def _register(header: list[str], rows: list[list[str]], *, end: str = "\r\n") -> bytes: + out = io.StringIO(newline="") + writer = csv.writer(out, lineterminator="\r\n", quoting=csv.QUOTE_NONNUMERIC) + writer.writerow(header) + writer.writerows(rows) + text = out.getvalue() + return (text[:-2] + end if end != "\r\n" else text).encode() + + +def _zip(member: str, body: bytes) -> bytes: + out = io.BytesIO() + with zipfile.ZipFile(out, "w") as archive: + archive.writestr(member, body) + archive.writestr(member.replace(".csv", ".json"), b"{}") + return out.getvalue() + + +def _index(*files: str) -> bytes: + links = "".join(f'CSV\n' for f in files) + return f"{links}popis".encode() + + +_KV_ZIP = "KV2022reg20260328_csv.zip" +_PS_ZIP = "PS2025reg20251005_csv.zip" + + +def _serve(monkeypatch: pytest.MonkeyPatch, *, kv: bytes | None = None, ps: bytes | None = None, + kv_index: bytes | None = None) -> list[str]: + """Answer each election's index page and register zip, the fixture's + unless given; the URLs asked. The pins are the fixture's counts.""" + monkeypatch.setattr(volby, "VOLBY_ELECTIONS", { + "kv2022": ("csv_od/kvrk.csv", len(_KV_ROWS)), "ps2025": ("csv_od/psrk.csv", len(_PS_ROWS))}) + answers = { + volby.index_url("kv2022"): kv_index or _index(_KV_ZIP.replace("_csv", "_xml"), _KV_ZIP), + volby.index_url("ps2025"): _index(_PS_ZIP), + f"{volby.VOLBY_SITE}kv2022/{_KV_ZIP}": + kv if kv is not None else _zip("csv_od/kvrk.csv", _register(_KV_HEADER, _KV_ROWS)), + f"{volby.VOLBY_SITE}ps2025/{_PS_ZIP}": + ps if ps is not None else _zip("csv_od/psrk.csv", _register(_PS_HEADER, _PS_ROWS)), + } + asked: list[str] = [] + + def urlopen(request: urllib.request.Request, timeout: int) -> _Response: + asked.append(request.full_url) + if request.full_url not in answers: + raise urllib.error.HTTPError(request.full_url, 404, "Not Found", {}, None) # type: ignore[arg-type] + return _Response(answers[request.full_url]) + monkeypatch.setattr(urllib.request, "urlopen", urlopen) + monkeypatch.setattr(time, "sleep", lambda seconds: None) + return asked + + +def test_a_fetch_keeps_each_candidate_s_boxes_once_with_the_composed_name( + monkeypatch: pytest.MonkeyPatch) -> None: + asked = _serve(monkeypatch) + corpus = local.fetch(_VOLBY) + assert asked == [volby.index_url("kv2022"), f"{volby.VOLBY_SITE}kv2022/{_KV_ZIP}", + volby.index_url("ps2025"), f"{volby.VOLBY_SITE}ps2025/{_PS_ZIP}"] + assert corpus.records == [ + {"name": "Eva Malá, DiS.", "election": "kv2022", "jmeno": "Eva", "prijmeni": "Malá", + "titulza": "DiS."}, + {"name": "Ida Krátká", "election": "kv2022", "jmeno": "Ida", "prijmeni": "Krátká"}, + {"name": "Ing. Jan Novák", "election": "kv2022", "jmeno": "Jan", "prijmeni": "Novák", + "titulpred": "Ing."}, + {"name": "doc. MUDr. Petr Pavel Veselý, CSc.", "election": "kv2022", + "jmeno": "Petr Pavel", "prijmeni": "Veselý", "titulpred": "doc. MUDr.", + "titulza": "CSc."}, + {"name": "Ing. Jan Novák", "election": "ps2025", "jmeno": "Jan", "prijmeni": "Novák", + "titulpred": "Ing."}, + {"name": "Mgr. et Mgr. Bo Dvořák Malá", "election": "ps2025", "jmeno": "Bo", + "prijmeni": "Dvořák Malá", "titulpred": "Mgr. et Mgr."}] + assert corpus.meta["data_date"] == "kv2022 2026-03-28, ps2025 2025-10-05" + assert corpus.meta["detail"] == { + "elections": "kv2022 ps2025", "read_version": volby.VOLBY_READ_VERSION, + "kv2022": {"zip": _KV_ZIP, "candidates": 5, "titled": 4, "records": 4, + "records_with_a_comma_inside_a_box": 0}, + "ps2025": {"zip": _PS_ZIP, "candidates": 2, "titled": 2, "records": 2, + "records_with_a_comma_inside_a_box": 0}} + # One composed name in two elections is one corpus name. + assert corpus.meta["names"] == 5 + assert corpus.header().endswith("6 records (local corpus cz-candidates)") + + +@pytest.mark.parametrize(("boxes", "name"), [ + ({"jmeno": "Jan", "prijmeni": "Novák"}, "Jan Novák"), + ({"titulpred": "Ing.", "jmeno": "Jan", "prijmeni": "Novák", "titulza": "Ph.D."}, + "Ing. Jan Novák, Ph.D."), + ({"jmeno": "Jan", "prijmeni": "Novák", "titulza": "Ph.D., MBA"}, "Jan Novák, Ph.D., MBA"), + ({"titulpred": "Bc.", "jmeno": "Jan", "prijmeni": "Novák", "titulza": ""}, "Bc. Jan Novák"), +]) +def test_a_name_is_composed_in_the_czech_convention(boxes: dict[str, str], name: str) -> None: + assert volby.compose(boxes) == name + + +def test_the_newest_register_the_index_links_is_read(monkeypatch: pytest.MonkeyPatch) -> None: + # ps2021's index links a revision beside the first register; a stamp + # older than the newest is a superseded file. + older = "KV2022reg20221010_csv.zip" + _serve(monkeypatch, kv_index=_index(older, _KV_ZIP, "KV2022reg20260328_xlsx.zip")) + assert local.fetch(_VOLBY).meta["detail"]["kv2022"]["zip"] == _KV_ZIP + # The older stamp's zip would be asked for, and answered 404, were + # it the one picked: the control. + assert max(volby._zip_link("kv2022").findall(_index(older).decode()))[0] == older + + +def test_a_register_named_with_underscores_is_found(monkeypatch: pytest.MonkeyPatch) -> None: + # kv2018's index names it KV2018_reg_20230224_csv.zip. + page = _index("KV2018_reg_20230224_xml.zip", "KV2018_reg_20230224_csv.zip").decode() + assert volby._zip_link("kv2018").findall(page) == [("KV2018_reg_20230224_csv.zip", + "20230224")] + + +def test_an_index_linking_no_register_is_refused(monkeypatch: pytest.MonkeyPatch) -> None: + _serve(monkeypatch, kv_index=_index("KV2022regPopis.pdf")) + with pytest.raises(ValueError, match="links no register CSV zip"): + local.fetch(_VOLBY) + assert local.load(_VOLBY) is None + + +def test_an_answer_that_is_not_a_zip_is_retried_then_refused( + monkeypatch: pytest.MonkeyPatch) -> None: + # A guessed stamp answers an HTML page under a .zip name; read as a + # body cut short, it is asked for again, and then refused. + asked = _serve(monkeypatch, kv=b"404 Not Found") + with pytest.raises(ValueError, match="not a zip"): + local.fetch(_VOLBY) + core = importlib.import_module("local_core") + assert asked.count(f"{volby.VOLBY_SITE}kv2022/{_KV_ZIP}") == core.RETRIES + + +@pytest.mark.parametrize("missing", list(volby.VOLBY_BOXES)) +def test_a_register_missing_a_box_s_column_is_refused( + monkeypatch: pytest.MonkeyPatch, missing: str) -> None: + header = [c if c != missing else f"{c}_X" for c in _KV_HEADER] + _serve(monkeypatch, kv=_zip("csv_od/kvrk.csv", _register(header, _KV_ROWS))) + with pytest.raises(ValueError, match=f"no column {missing}: its layout has changed"): + local.fetch(_VOLBY) + + +def test_a_register_without_its_member_is_refused(monkeypatch: pytest.MonkeyPatch) -> None: + _serve(monkeypatch, kv=_zip("csv/kvrk.csv", _register(_KV_HEADER, _KV_ROWS))) + with pytest.raises(ValueError, match="holds no csv_od/kvrk.csv"): + local.fetch(_VOLBY) + + +#: The fixture with its last candidate's surname box holding a line +#: break, as a free-text box can: a cut just after the break ends the +#: body with one, so only the short last row tells it from a whole one. +_BROKEN = _register(_KV_HEADER, [*_KV_ROWS[:-1], _kv("Ida", "Krátká\r\nDlouhá", "Ing.")]) + + +@pytest.mark.parametrize("body", [ + _BROKEN[:_BROKEN.index("Dlouhá".encode())], + _register(_KV_HEADER, _KV_ROWS)[:-2], +], ids=["at a line break inside a quoted field", "before the last line break"]) +def test_a_register_cut_short_is_refused(monkeypatch: pytest.MonkeyPatch, body: bytes) -> None: + rows = list(csv.reader(io.StringIO(body.decode(), newline=""))) + assert len(rows) == len(_KV_ROWS) + 1 # the cut passes the row count + _serve(monkeypatch, kv=_zip("csv_od/kvrk.csv", body)) + with pytest.raises(ValueError, match="last row is cut short"): + local.fetch(_VOLBY) + # The control: the whole body, its box's line break read as a space. + _serve(monkeypatch, kv=_zip("csv_od/kvrk.csv", _BROKEN)) + assert {"jmeno": "Ida", "prijmeni": "Krátká Dlouhá"}.items() <= next( + r for r in local.fetch(_VOLBY).records if r["jmeno"] == "Ida").items() + + +def test_a_register_holding_another_number_of_candidates_is_refused( + monkeypatch: pytest.MonkeyPatch) -> None: + _serve(monkeypatch, kv=_zip("csv_od/kvrk.csv", _register(_KV_HEADER, _KV_ROWS[:-1]))) + with pytest.raises(ValueError, match="holds 4 candidates where VOLBY_ELECTIONS pins 5"): + local.fetch(_VOLBY) + + +@pytest.mark.parametrize(("row", "match"), [ + (_kv("Jan", "Novák") + ["extra"], "not the header's width"), + (_kv("", "Novák", "Ing."), "no given name or surname"), + (_kv("Jan", " ", "Ing."), "no given name or surname"), +], ids=["a row too wide", "no given name", "a blank surname"]) +def test_a_register_skipping_past_its_share_is_refused( + monkeypatch: pytest.MonkeyPatch, row: list[str], match: str) -> None: + rows = [*_KV_ROWS[:-1], row] + _serve(monkeypatch, kv=_zip("csv_od/kvrk.csv", _register(_KV_HEADER, rows))) + with pytest.raises(ValueError, match=f"1 of 5 of the kv2022 register's rows are.*{match}"): + local.fetch(_VOLBY) + # The control: under a share one row may pass, it is counted, not kept. + monkeypatch.setattr(volby, "VOLBY_MAX_BAD", 1 / 5) + detail = local.fetch(_VOLBY).meta["detail"]["kv2022"] + assert detail["records"] == 3 and len(detail) == 6 + + +def test_a_register_with_its_titles_emptied_is_refused(monkeypatch: pytest.MonkeyPatch) -> None: + rows = [_kv("Jan", "Novák")] * 4 + [_kv("Eva", "Malá")] + _serve(monkeypatch, kv=_zip("csv_od/kvrk.csv", _register(_KV_HEADER, rows))) + with pytest.raises(ValueError, match="0 of the kv2022 register's 5 candidates filed a title"): + local.fetch(_VOLBY) + # The control: one titled candidate in five is at the floor. + rows[-1] = _kv("Eva", "Malá", "Ing.") + _serve(monkeypatch, kv=_zip("csv_od/kvrk.csv", _register(_KV_HEADER, rows))) + assert local.fetch(_VOLBY).records + + +def test_a_register_not_utf8_is_refused(monkeypatch: pytest.MonkeyPatch) -> None: + # The csv/ folder's copy is in windows-1250, which csv_od/'s is not. + body = _register(_KV_HEADER, _KV_ROWS).decode().encode("cp1250") + _serve(monkeypatch, kv=_zip("csv_od/kvrk.csv", body)) + with pytest.raises(ValueError, match="is not UTF-8"): + local.fetch(_VOLBY) + + +@pytest.mark.parametrize(("constant", "value", "match"), [ + ("VOLBY_ELECTIONS", {"kv2022": ("csv_od/kvrk.csv", 5)}, + "holds the elections kv2022 ps2025.*reads kv2022"), + ("VOLBY_READ_VERSION", 99, "composed by version 1 of compose.*version 99"), +]) +def test_a_corpus_of_other_elections_or_another_composition_is_refused_on_load( + monkeypatch: pytest.MonkeyPatch, constant: str, value: object, match: str) -> None: + _serve(monkeypatch) + local.fetch(_VOLBY) + assert local.load(_VOLBY) is not None # the control + monkeypatch.setattr(volby, constant, value) + with pytest.raises(ValueError, match=f"{match}.*fetch {_VOLBY} --refresh"): + local.load(_VOLBY) + + +def test_a_comma_typed_inside_a_box_is_kept_and_counted(monkeypatch: pytest.MonkeyPatch) -> None: + # The composed name then holds a comma shape nobody wrote; the record + # is kept as filed, since the score reads the boxes, and counted. + rows = [*_KV_ROWS[:-2], _kv("Ida", "Krátká, roz. Dlouhá"), _kv("Bo", "Malý", "Ing., Bc.")] + _serve(monkeypatch, kv=_zip("csv_od/kvrk.csv", _register(_KV_HEADER, rows))) + corpus = local.fetch(_VOLBY) + assert {"Ida Krátká, roz. Dlouhá", "Ing., Bc. Bo Malý"} <= {r["name"] for r in corpus.records} + assert corpus.meta["detail"]["kv2022"]["records_with_a_comma_inside_a_box"] == 2 + # A comma in the box after the name is the convention's own, not counted. + assert corpus.meta["detail"]["ps2025"]["records_with_a_comma_inside_a_box"] == 0 diff --git a/tools/differential/README.md b/tools/differential/README.md index 29eaf57a..d7be0780 100644 --- a/tools/differential/README.md +++ b/tools/differential/README.md @@ -263,7 +263,7 @@ uv run python tools/differential/build_shapes_corpus.py --coverage A local corpus holds real names that are not committed: fetched into the cache by `local_corpora.py`, read by `snapshot.py --local` and by a row of `tools/name_usage.py`, scored by `local_agreement.py` where its records carry their fields, and seen by no reviewer and no CI run (the SURVEY job in mechanisms.md#NAME-DATA-JOBS, and WATCH and SCORE outside CI). What reaches the repository is what a contributor cites from it, counts and shapes written with made-up names. -`local_corpora.py` holds the `SOURCES` registry, the cache's files, the views and the command line; each source's fetch is a module of its own (`local_ctgov.py`, `local_fec.py`, `local_tse.py`, `local_tw.py`, `local_ukec.py`, `local_govuk.py`, `local_psc.py`, `local_jrc.py`, `local_nepal.py`, `local_orcid.py`, `local_patentsview.py`) whose docstring says what the source is and what its terms allow, built on the retry policy, share gate and downloads in `local_core.py`. A new source is a module and a `SOURCES` row. A source whose records hold a value its own code derives, rather than one the source gave (`jrc-names`' script label), stores a read version in `meta.json`'s detail and refuses on load a corpus read by another (`JRC_READ_VERSION`, bumped by hand with any change to how a record is built; `psc-individuals` does the same for its cached extracts, with `PSC_EXTRACT_VERSION` in each extract's file name): the sha256 checks prove the files are the ones fetched, not that today's code would write them. A constant such as `TSE_MAX_BAD` or `RETRIES` lives in one module, which every reader reads it through, so a test patches it there; no other module `from`-imports it, so a patch set elsewhere raises rather than missing the code (`test_no_module_imports_a_sibling_constant` holds this). +`local_corpora.py` holds the `SOURCES` registry, the cache's files, the views and the command line; each source's fetch is a module of its own (`local_ctgov.py`, `local_fec.py`, `local_tse.py`, `local_tw.py`, `local_ukec.py`, `local_govuk.py`, `local_psc.py`, `local_jrc.py`, `local_nepal.py`, `local_orcid.py`, `local_patentsview.py`, `local_volby.py`) whose docstring says what the source is and what its terms allow, built on the retry policy, share gate and downloads in `local_core.py`. A new source is a module and a `SOURCES` row. A source whose records hold a value its own code derives, rather than one the source gave (`jrc-names`' script label), stores a read version in `meta.json`'s detail and refuses on load a corpus read by another (`JRC_READ_VERSION`, bumped by hand with any change to how a record is built; `psc-individuals` does the same for its cached extracts, with `PSC_EXTRACT_VERSION` in each extract's file name): the sha256 checks prove the files are the ones fetched, not that today's code would write them. A constant such as `TSE_MAX_BAD` or `RETRIES` lives in one module, which every reader reads it through, so a test patches it there; no other module `from`-imports it, so a patch set elsewhere raises rather than missing the code (`test_no_module_imports_a_sibling_constant` holds this). ``` uv run python tools/differential/local_corpora.py fetch ctgov-officials @@ -303,6 +303,7 @@ Any snapshot reading of 5,000 entries or more, committed or local, parses in a w | `patentsview-inventors` | the inventor of every patent the USPTO granted from 1976 to 2025, from PatentsView's granted-patent tables on the USPTO Open Data Portal (release to 2025-12-31): each inventor's name in the filing's two boxes (first name, last name) and the country of the inventor's address. A record is a distinct name in a country, with the number of people who filed under it (`people`, counted by PatentsView's disambiguated inventor id, so a prolific inventor weighs once): 24,037,380 inventor rows, one per inventor per patent, held 5,319,206 such names on 2026-10-28, 99.3% of rows with a country. The extract keeps every name with its count, the corpus at most 20,000 names a country (657,991 records: 202 countries, 23 of them at the cap, and the records with no country as one more group of 20,000, counted in that total). Split by a professional filer, the applicant's attorney, rather than the bearer: `parse()` agreed with 94.7% of the records, 99.8% in Japan, 99.5% in China and 99.0% in the US, so it reads beside `orcid-researchers` as how a name is written against how its bearer types it. A comma stands in the last box of 92,503 of the extract's names, against 129 in the first, before suffixes and legal roles typed after a deceased inventor's name: the text after it is exactly `Jr.` in 55,459 names, `III` in 14,227, `deceased` in 4,635, `heir` in 514, `legal representative` in 393, `executrix` in 328 and `administrator` in 288 (2026-10-28). CC BY 4.0, credit PatentsView; the portal needs an API key | names in any script but Latin, since a filing is romanized; titles and credentials, which a filing has no place for; names outside patenting, which is mostly engineering and science, and its countries' shares are of patents filed in the US, not of their populations | | `jrc-names` | the person entities of the EU Joint Research Centre's JRC-Names, each with the spellings of its name harvested from news text in many languages and scripts: a record is one distinct spelling, with its entity and its script. The file (`entities.gzip`, 14.8 MB) is pinned by sha256 and dated by its Last-Modified, 2024-03-15. Of its 1,317,586 spellings kept, 1,203,634 are Latin, 66,673 Cyrillic, 17,074 Arabic (7,833 persons, 5,347 of them spelled in Latin too), 13,039 Katakana and 8,016 Han, and 218,822 persons have two or more spellings, 25,415 of them in two scripts or more (a spelling whose letters are in two scripts, or none, not counted as one). The population for spelling and transliteration variants of one name, read by the `groups` view, and for Arabic-script names. Licensed under the JRC-Names EULA, personal and non-transferable, with no redistribution or derivative works, so its views' output and its snapshot stay in the cache like every local corpus's, and here the licence, not only the names being real people's, is why | titles and credentials, since the names are harvested from running text (`parse()` read a title in 3,011 of its 1,315,051 names and a suffix in 1,940, 2026-10-24); the ASCII comma, which none holds (1,510 Arabic spellings hold an Arabic comma, U+060C, which the parser reads as one); the Indic scripts (Devanagari 827 spellings, Bengali 419, the others fewer); people the news did not name | | `nepal-candidates` | every candidate in Nepal's local elections of 2079 BS (2022), from the Election Commission's results site (result.election.gov.np), one file per local body: a record per distinct pair of the Commission's romanization (`name`, the one `parse()` reads) and the name in Devanagari as filed (`devanagari`). The corpus and the word index hold both, so a snapshot and a `name_usage.py` lookup read both scripts; `tally`, which counts what follows a comma, reads the romanized name. Of 126,981 pairs fetched 2026-10-24 (152,960 candidates in 753 files, 26 of whose romanizations were a placeholder such as `- - -` and are dropped), the romanized names carry surnames written as initials (`B.K.`, `K.C.`, `G.C.`), honorific words that are titles in front (`Guru`; `Shekh` until #752's phase 2j made it a title only in a title phrase) and `Md.`, Muhammad's abbreviation, which its period makes a title by shape; the Devanagari ones carry vowel signs and the virama, combining marks the letter-shape tests do not count as letters (decisions/nfd-shapes), and the joiners ZWJ and ZWNJ. The population for romanized South Asian names and for Devanagari. The site states no licence, which the credit line says | case and the comma, since the Commission's romanization is one consistent hand; names a person typed themselves; credentials and generational words; the parliament and provincial candidates, whose files carry Devanagari only; names outside Nepal | +| `cz-candidates` | every candidate on the registers the Czech Statistical Office (ČSÚ) publishes on volby.gov.cz for the 2022 municipal elections (`kv2022`) and the 2025 Chamber of Deputies election (`ps2025`): a record is a distinct candidate's four name boxes in one election, the title before the name (`TITULPRED`), the given names (`JMENO`), the surname (`PRIJMENI`) and the title after it (`TITULZA`), with the election; 165,736 records of kv2022's 197,506 candidates and 4,445 of ps2025's 4,475 on 2026-10-30. A record's name is COMPOSED, not written by anyone: the boxes joined in the Czech convention, `TITULPRED JMENO PRIJMENI, TITULZA` (`Ing. Jan Novák, Ph.D.`). 61,320 of its 170,181 records carry a title (58,667 before the name, 7,687 after it), nearly all academic degrees (`Ing.`, `Mgr.`, `Bc.`, `MUDr.`; `DiS.`, `Ph.D.`, `MBA`): the population for titles and post-nominals the filer kept apart from the name. Scored by `local_agreement.py` by election, under the default parser and under `--locale cs`. ČSÚ licenses it under CC BY 4.0, asking that derived data be marked as derived, which the composed names are and the credit line says | case, spacing and punctuation of a writer's own, since the string is composed; the family-comma shape; courtesy titles and offices; names outside Czechia; the elections not in `VOLBY_ELECTIONS` (the index lists kv2002 to kv2026 and ps2002 to ps2025) | A live source has no fixed snapshot to pin, so the cached copy is the pin: `meta.json` holds the source's data date and the sha256 of `raw.jsonl` and `corpus.jsonl`, checked on every load, and every report, the `name_usage.py` row included, carries the source and that date. Quote a count with the date, and expect a later fetch to differ. A local snapshot records which fetch it was read from, and `snapshot.py --local` refuses to compare it with a corpus fetched since, which would report the names the source gained and lost as moves: after a `fetch --refresh`, record again from the base checkout. ClinicalTrials.gov's terms ask anyone redistributing its data to credit it, show NLM's processing date and say what was changed; keeping the names in the cache and citing counts with that header meets them. The FEC's bar on soliciting and commercial use is in the credit line of every `fec-individuals` report, so a count quoted from it carries the bar with it. A record there is a distinct name, not a contribution or a contributor: one donor gives many times (34,264,742 rows held 1,893,964 names in the file dated 2026-10-18), and counting contributions would weight a shape by how often its bearers give. `tools/name_usage.py` looks a word up in each corpus's word index, cached as `wordindex.json` beside it on first use and keyed to the records' sha256, to the code that cuts and folds a word and to the record keys it reads (`Source.corpus_keys`; a record whose two names write one word counts once), so a refetch, a change to the fold or a change to which of a record's names are read rebuilds it; a single-word lookup it answers parses neither the records nor the names. Measured 2026-10-22 with both corpora fetched, a single-word `name_usage.py --no-wikidata` run took 2.5s with their indexes cached (12.5s before the cache, 14.8s on the run that builds it). A spaced word (`Ph. D.`) still scans every name, which took 8.6s, nearly all of it the FEC corpus's. For `orcid-researchers` and `patentsview-inventors`, whose extracts keep every record's boxes and country uncapped, `name_usage.py --boxes` adds a row by box and country, read from a second index, `boxindex.json.gz` beside the corpus (`cached_box_index`): per word and country the people whose given box or family box writes it, each country shown as a share of its people, so a word can be weighed in the country a language pack is for (`--country CC`). It is keyed to the extract's sha256, taken from the bytes as they are read, and to a fingerprint of the reader and the fold, so a refetch rebuilds it. Building it took 2 min 19 s for ORCID's 24,990,909 records (3,780,309 words in 6,660,756 word-and-country cells, 41 MB cached) on 2026-10-28. With both indexes cached a lookup took 10.4 s with `--boxes` against 5.3 s without, the difference nearly all reading the two indexes (the 369 MB extract hashes in a fraction of a second once the system has it cached), which is why the rows are asked for rather than always built; an extract that cannot be read fails its own row, naming the refresh. `fetch --refresh` re-downloads the FEC's zip rather than re-reading the cached one, replacing it only once the new one is whole, and a corpus fetched for another cycle than `FEC_CYCLE` is refused on load. The `fec-individuals` fetch refuses a file whose lines stop matching the 21 columns it reads, or whose names stop holding a comma, rather than read it as a smaller or a different corpus. A `ukec-individuals` record is a distinct name too, not a donation or a `DonorId`: one name often holds several IDs, which seem to come with reports rather than with people (15,214 names held 24,473 on 2026-10-22, the IDs counted in `meta.json`'s detail; 2,663 names held several at one `RegulatedEntityId` alone, counted from the export and not kept, though some of those may be two people of one name), so a count of IDs would weight a name by how often it was reported. Its fetch reads the register's total from the JSON search under the same filter before and after the export, and refuses an export holding another number of rows, or a register that changed in between: the export ignores `rows=`, so the total is what tells a cut or capped export from the whole register. It refuses an export whose last row is cut short (Python's `csv` returns a body cut inside a row as a short last row, which the row count would otherwise take for a whole one), and a header other than the 29 columns `UKEC_COLUMNS` names, and counts into `meta.json`'s detail the rows it skips: rows not 29 fields, rows from a donor that is not an individual (the filter asks the server to leave those out), and individuals with no name. Past `UKEC_MAX_BAD` of either of the first two, or `UKEC_MAX_NAMELESS` of the third (4 of 42,355 rows were nameless on 2026-10-22), it refuses the export. The data date is when the first search answered, in UTC: the register states no date of its own, so the counts above, dated by the day they were fetched (2026-10-22), carry the data date 2026-10-23T03:13:19Z in their header. `tw-donors` keeps one zip per report under `reports/`, whole, its masked ID, address and phone columns included, as the FEC's zip is kept: the platform's download answers a report that does not exist and a fault alike, with a 500 and an incident number, so the fetch reads which reports exist from the platform's listing and refuses a listed report it cannot download. It reads each file's columns by name, since a 2020 report lacks a column the newest ones write, and refuses a column it does not know, a category it does not know, a row whose serial column names another report (the zip is not the one asked for), a listing that changes during the fetch, and a report past `TW_MAX_REPORT_BAD` of whose rows it reads past. It reads back the rows a stray quote took into one field (169 rows in 12 records of three reports on 2026-10-22, 8 of them two rows joined by two stray quotes), counts donor cells that hold no name (匿名, a dash) or a masked one, and caches a downloaded zip only once its rows are read, so a refused zip is asked for again. A zip the platform rebuilt under the name it had is not seen: delete `reports/` to download every report again. A layout check written from one surveyed file meets every other file only at fetch time: `tw-donors`' first three fetches were each refused partway, by a report with no income writing one row of dashes, by a 2020 report with 24 columns where the surveyed one had 25, and by a row with no category. Each refusal cost every download again until the source cached its zips, so a new source that downloads many files caches each one before its checks run. `tse-donors` caches nothing but the corpus: each election's national member (`receitas_candidatos__BRASIL.csv`) is read from the TSE's CDN in HTTP Range requests, 209 MB for the two elections where their zips are 1.82 GB (2026-10-22), because the TSE regenerates the zips every few days and a cached one would be stale within the week. Every range request carries `If-Range` with the ETag the HEAD gave if it is strong, else its `Last-Modified`, and every part must state the `Content-Range` asked for, so a zip replaced during the read is refused rather than read across two versions; a HEAD that states no length, neither a strong ETag nor a `Last-Modified`, or `Accept-Ranges` other than `bytes` is refused before the read. A fetch took 69s on 2026-10-22. It refuses a zip without the national member, a header that is not 60 columns or lacks its three columns, a `DS_ORIGEM_RECEITA` value it does not know (a renamed category, or the file no longer in Latin-1), malformed rows or lines folded into a field past `TSE_MAX_BAD`, an election with too few rows from individuals (`TSE_MIN_INDIVIDUALS`: a member cut to its header, or the donations moved under another category) too many without a name or registry name (`TSE_MAX_MISSING`, counting a placeholder such as `#NULO` or `***` as none: a masked column) or with a number in one (`TSE_MAX_NUMBERED`: a numeric column moved under a name), too few distinct filed or registry names (`TSE_MIN_DISTINCT`: a column masked with one word), and a corpus fetched for other elections than `TSE_ELECTIONS`. The CPF in each row is unmasked in the file, and like every other column it is read past, never stored or printed; a pair either of whose names holds more than `TSE_MAX_DIGITS` digits is dropped and counted, since campaigns sometimes typed a number, a CPF among them, into the name field. A record is a pair, so the counts in its reports (`tally`, the `name_usage.py` row) are of pairs, and a filed name standing beside two registry names counts twice. @@ -316,7 +317,11 @@ A `govuk-people` record is a distinct name, as a `ukec-individuals` one is, and `nepal-candidates` reads each local body's file through the site's own JSON handler, which asks for the site's double-submit check: the `CsrfToken` cookie a GET of the site sets, echoed in an `X-CSRF-Token` header, with the `Referer` its pages send. The site refuses a User-Agent holding `python` or `curl` with a 403 (2026-10-24), so this source sends a browser's form naming the tool (`NEPAL_USER_AGENT`) where the others send the repository's URL. A 403 mid-fetch is what a lapsed token gets, and renews the session once before it is raised; a 404 counts the file as missing, its id kept in `meta.json`'s detail, and with the empty files (a list of no candidates, as a site that blanked its files would answer) past `NEPAL_MAX_MISSING` of the `NEPAL_BODIES` bodies refuses the fetch, the empty ones' ids kept too. The election is over and its files final, so the candidates are pinned as the bodies are: a fetch reading every file must read exactly `NEPAL_CANDIDATES`, so a site that trimmed its files (to the winners, or to one post) is refused however well formed what is left; with files missing or empty the pin cannot be exact, and a shortfall of up to `NEPAL_MAX_SHORT` of it is accepted, so a few candidates trimmed elsewhere are not seen then. Every listed file answered on 2026-10-24: the 18 a survey on 2026-10-23 saw refused, among them ids 5551 to 5554, answered when asked again. A file is JSON, so a body cut short does not parse, and is retried as one cut in transit rather than read as a body with fewer candidates. The fetch refuses a lookup listing other than `NEPAL_BODIES` bodies (the election is over, so its files are final), a file that is not a list of candidates, too many candidates with no romanized or no Devanagari name (`NEPAL_MAX_NAMELESS`) or with a name in the wrong script (`NEPAL_MAX_WRONG_SCRIPT`: two columns swapped, or a romanization dropped), and too few distinct names in either column (`NEPAL_MIN_DISTINCT`). A record is a distinct pair, as `tse-donors`' are, not a candidate row: a body lists a candidate once per post, and a common name recurs across bodies. The `pairs` view refuses it, its records carrying no reference name: that view compares the letters of two spellings, and a romanization shares none with its Devanagari, so the Devanagari is kept under a key of its own. `meta.json` records the keys the corpus was built from (`corpus_keys`), and a corpus written under other keys is refused on load. A fetch took 28 minutes on 2026-10-24 at `NEPAL_DELAY` (0.3 s) a request. -`local_agreement.py` does the SCORE job for a local corpus. It parses each filed display name and compares the parser's roles with the boxes the source states (title, given, middle and family for `psc-individuals`; given and middle joined against the given-names box, then family, for `orcid-researchers`), folding case, periods, commas and spacing. A suffix the parser peels is compared inside the surname box: the display name joins the surname last, so a trailing suffix was typed into it whenever it is filled, with or without a comma (`Smith, Jr.`); the fold read the comma as a miss until 2026-10-28, which moved 83 `psc-individuals` records and 561 `orcid-researchers` records to agreeing and none away. Where a source has a title box, a title the parser reads otherwise, holding a word the vocabulary lists as neither a title nor a connective (a job typed as a title), is set aside as filer error rather than scored, and the report lists the commonest such titles as leads for the vocabulary. The report gives agreement overall and by the group the source states (nationality, country), misses per field, and the commonest disagreement shapes, as counts, word counts and the commonest set-aside titles; `--list N` prints moved records, names and all, for reading locally. Its record, `agreement.jsonl` beside the corpus, holds every record that did not wholly agree, so a re-run reports what moved. As with the local snapshot, check it from the branch against a base you name (`--base `, whose record is kept under `agreements/` beside the corpus, keyed by the fetch, the base's `nameparser/` files, the interpreter's minor version and the scorer, the ten most recent kept; the scorer's fingerprint reads only its own source's `FIELDED` entry, so adding a source leaves the others' records standing, which the whole table, hashed until `patentsview-inventors` was added on 2026-10-28, did not), or, alone, record the shared one from the base checkout (`--record --tree `) and run it from the branch; it refuses a record from another fetch, or one written by another version of the scorer, whose own change would otherwise read as the parser's moves. No CI run sees it. +`cz-candidates` reads each election's index page (`https://volby.gov.cz/opendata//_opendata.htm`) for the register's CSV zip, whose file name carries a date stamp (`KV2022reg20260328_csv.zip`, `KV2018_reg_20230224_csv.zip`): a guessed stamp answers an HTML 404 page, so the stamp is read off the page, the newest where it links several (ps2021's links a revision), and an answer that is not a zip is retried as a body cut short, then refused. It reads the UTF-8 copy in `csv_od/` (the zip's `csv/` holds one in windows-1250) by column name, since the registers' other columns differ by election, and refuses a register missing a box's column, one whose last row is cut short (the CSV gotcha in AGENTS.md), one holding another number of candidates than `VOLBY_ELECTIONS` pins (a register is not final when its election is held: kv2022's carries the by-elections held through 2024 and was re-issued with a stamp of 2026-03-28, so the pin, not the election's date, is what refuses a changed register), past `VOLBY_MAX_BAD` of rows not the header's width or with no given name or surname, and under `VOLBY_MIN_TITLED` of candidates titled (a title column emptied or moved). A box is kept with its runs of spaces read as one, and a comma typed inside a box before the last (`Dopitová, roz. Lišková` in the surname box, `Ing., Bc.` in the title box: 21 records on 2026-10-30) is kept as filed and counted in `meta.json`'s detail (`records_with_a_comma_inside_a_box`), an accepted cost: the composed name then holds a comma shape nobody wrote, but the score reads the boxes, which say what was meant; the age, occupation, municipality and party columns are read past, and nothing is cached but the corpus. A record is a distinct set of boxes in one election, so a candidate who stood twice in one election under the same boxes, or two people who share them, is one record, which is why the corpus counts records, not candidates: 31,770 of kv2022's rows fold into another. Adding an election is a row in `VOLBY_ELECTIONS`, measured first. `meta.json` holds the elections and `VOLBY_READ_VERSION`, the version of `compose`, and a corpus of other elections, or composed by another version, is refused on load. The data date names each register's stamp. A fetch took 25s on 2026-10-30 (13 MB of zips), and a score 3.4s with a worker per CPU. + +Its score reads the title before the name against `TITULPRED`, the given and middle names against `JMENO`, the family against `PRIJMENI` and the suffix against `TITULZA`. It is the one source with a box after the name, so no suffix is compared inside the surname box, and the one that sets nothing aside: its title box holds the degrees the law asks filed there, and the default vocabulary lists few of them (`Bc.` and `Ing.` read as titles by their shape alone, rules/H2), so the set-aside took 1,059 records on 2026-10-30, each the parser's own miss (`Bc. Jan Novák, DiS.` read with the title `Bc. DiS.`; the split by box is in `Fielded`'s docstring). On that fetch every field agreed for 96.96% of the 170,181 records under the default and 96.97% under `cs`, 92.9% and 93.0% of the 61,320 titled ones; the box before the name agreed for 97.7% of the 58,667 filled, the box after it for 56.0% of the 7,687 filled, and the family for 99.0% of all. The misses are, by far, `DiS.` (a diploma written after the name): of the 3,248 records whose box after the name holds it (`re.search(r"\bdis\b", box.replace(".", "").casefold())`), that box disagrees in 3,240, 3,232 of them because the parse reads `DiS.` as a title (its title holding `dis` by the same test) and 8 because it reads the whole name before the comma as the family (`DiS. et DiS.` six times), and a two-word surname read as a middle name and the family, 1,483 under the default and 1,315 under `cs` (with `Policy(two_family_names=True)` the family box's misses fell from 1,683 to 825 under the default and 986 under `cs`, and the given box's from 1,828 and 1,644 to 743 and 732); then a connective between two titles the shape alone reads (`Ing. et Ing.`, `Bc. et Bc.`) read as given names, 164 and 153, where `Mgr. et Mgr.`, whose `Mgr.` is listed, reads as a title in 218 of 223. Under `cs` a generational word typed into the surname box (`Novák ml.`, 177; `st.`, 57) reads as the suffix it is, which the score counts against the empty box after the name: there the box, not the parse, is out of place. + +`local_agreement.py` does the SCORE job for a local corpus. It parses each filed display name and compares the parser's roles with the boxes the source states (title, given, middle and family for `psc-individuals`; given and middle joined against the given-names box, then family, for `orcid-researchers`; title, given and middle joined, family and suffix for `cz-candidates`), folding case, periods, commas and spacing. Where the source has no box after the name, a suffix the parser peels is compared inside the surname box: the display name joins the surname last, so a trailing suffix was typed into it whenever it is filled, with or without a comma (`Smith, Jr.`); the fold read the comma as a miss until 2026-10-28, which moved 83 `psc-individuals` records and 561 `orcid-researchers` records to agreeing and none away. Where a source has a title box, and save `cz-candidates` (above), a title the parser reads otherwise, holding a word the vocabulary lists as neither a title nor a connective (a job typed as a title), is set aside as filer error rather than scored, and the report lists the commonest such titles as leads for the vocabulary. The report gives agreement overall and by the group the source states (nationality, country), misses per field, and the commonest disagreement shapes, as counts, word counts and the commonest set-aside titles; `--list N` prints moved records, names and all, for reading locally. Its record, `agreement.jsonl` beside the corpus, holds every record that did not wholly agree, so a re-run reports what moved. As with the local snapshot, check it from the branch against a base you name (`--base `, whose record is kept under `agreements/` beside the corpus, keyed by the fetch, the base's `nameparser/` files, the interpreter's minor version and the scorer, the ten most recent kept; the scorer's fingerprint reads only its own source's `FIELDED` entry, so adding a source leaves the others' records standing, which the whole table, hashed until `patentsview-inventors` was added on 2026-10-28, did not), or, alone, record the shared one from the base checkout (`--record --tree `) and run it from the branch; it refuses a record from another fetch, or one written by another version of the scorer, whose own change would otherwise read as the parser's moves. `--locale cs` scores the parser of a locale pack, `parser_for(locales.CS)`, against a record of its own (`agreement.cs.jsonl`, and kept records keyed by the locale too), so a pack's score and the default's never stand for each other; `measure.py` runs the default's alone. No CI run sees it. `pairs` reads a paired source (`tse-donors`, whose records carry the registry's name beside the filed one) as two spellings of one person, and sorts each pair by how the filed name departs from the reference: the first of its shapes that fits, from case, accents (case unchanged) and case and accents together, through words left out, shortened or misspelled, to few words in common. A word left out or added counts once per pair, and is listed only when ten pairs share it. `--agreement` also parses both spellings of every pair with a reference name that differs, and counts the pairs whose two readings end the family name in the same word (two readings with no family name share none). It lists the accents-only pairs read with two surnames by the two spellings' last words, the same ten-pair floor applied. @@ -352,7 +357,7 @@ Measured 2026-10-27 on a 12-CPU machine (load average 3 to 15 from other session uv run python tools/differential/measure.py --base ../master-checkout ``` -The steps: the committed snapshot (`snapshot.py`, the check `tests/v2/test_snapshot.py` makes), `cldr_agreement.py`, `snapshot.py --local --base` for every fetched local corpus, `local_agreement.py SOURCE --base` for every fetched source whose records carry their fields (`psc-individuals`, `orcid-researchers`, `patentsview-inventors`), and `local_corpora.py alike --base` for every fetched source that pairs spellings. Only the local steps read `--base`: the first two compare the tree with the records committed on its branch, so on a branch that has re-recorded `snapshot.jsonl` the committed snapshot says no moves and the moves are in the PR's diff of that file, and the summary labels each step with what it compared against. A local corpus that is not fetched gets no step, the summary names it, and a run in which no step read `--base` exits 1. Each step's whole output goes to a file of its own under `--out`, by default a folder named for the time under the cache's `measure/`, outside the repository, since a local snapshot lists its moves by name. The summary, `summary.txt` beside them and printed, gives each step's status (no moves, moves, or failed with its exit code), its time, and its unindented lines (the first few and the last, where the verdicts are, up to `SUMMARY_LINES`, and any line saying part of a check was not measured), never the indented ones where the tools list names; a crash's traceback is indented below a line naming only its type, since its message may hold a name. A step that fails makes the run exit 1; one that finds moves does not. Each tool refuses with `SystemExit("message")`, which exits 1 as a move does, so measure runs every step through its own `--step` wrapper, which turns a message exit or an uncaught exception into exit 2: a refused base or a damaged corpus is reported as a failure, never as moves. +The steps: the committed snapshot (`snapshot.py`, the check `tests/v2/test_snapshot.py` makes), `cldr_agreement.py`, `snapshot.py --local --base` for every fetched local corpus, `local_agreement.py SOURCE --base` for every fetched source whose records carry their fields (`psc-individuals`, `orcid-researchers`, `patentsview-inventors`, `cz-candidates`), and `local_corpora.py alike --base` for every fetched source that pairs spellings. Only the local steps read `--base`: the first two compare the tree with the records committed on its branch, so on a branch that has re-recorded `snapshot.jsonl` the committed snapshot says no moves and the moves are in the PR's diff of that file, and the summary labels each step with what it compared against. A local corpus that is not fetched gets no step, the summary names it, and a run in which no step read `--base` exits 1. Each step's whole output goes to a file of its own under `--out`, by default a folder named for the time under the cache's `measure/`, outside the repository, since a local snapshot lists its moves by name. The summary, `summary.txt` beside them and printed, gives each step's status (no moves, moves, or failed with its exit code), its time, and its unindented lines (the first few and the last, where the verdicts are, up to `SUMMARY_LINES`, and any line saying part of a check was not measured), never the indented ones where the tools list names; a crash's traceback is indented below a line naming only its type, since its message may hold a name. A step that fails makes the run exit 1; one that finds moves does not. Each tool refuses with `SystemExit("message")`, which exits 1 as a move does, so measure runs every step through its own `--step` wrapper, which turns a message exit or an uncaught exception into exit 2: a refused base or a damaged corpus is reported as a failure, never as moves. For a vocabulary word whose reading acts on a name position, `tools/vocab/word_report.py` gives the per-word view before and alongside this: the word in a grid of shapes, its incidence in every fetched local corpus, and its `name_usage.py` rows (`nameparser/vocab/README.md`, "Adding a word"). diff --git a/tools/differential/local_agreement.py b/tools/differential/local_agreement.py index 8c85fde7..f52bc7de 100644 --- a/tools/differential/local_agreement.py +++ b/tools/differential/local_agreement.py @@ -6,7 +6,8 @@ and each field is compared with the box the filer typed it into (`psc-individuals`' title, forename, middle name and surname boxes; `orcid-researchers`' given-names box, holding the middle names too, and -family-name box, typed by the holder). What +family-name box, typed by the holder; `cz-candidates`' title before the +name, given names, surname and title after it). What cldr_agreement.py does for the committed fields files, over names that are never committed -- so no CI run and no reviewer sees this score, and its record lives in the cache beside the corpus, where a re-run @@ -26,6 +27,11 @@ uv run python tools/differential/local_agreement.py psc-individuals --record --tree ../master-checkout uv run python tools/differential/local_agreement.py psc-individuals +--locale cs scores the parser of a locale pack (parser_for(locales.CS)) +instead of the default one, against a record of its own +(agreement.cs.jsonl, and kept records keyed by the locale too), so the +default's record and a pack's never stand for each other. + The report prints counts and shapes, never a record: agreement overall and by the group a source states (a nationality, a country), the misses per field, the commonest disagreement shapes, a shape being each disagreeing @@ -44,10 +50,12 @@ fold only keeps a box's case, periods, commas or spacing from reading as a miss: a suffix typed into the surname box after a comma (`Smith, Jr.`) is the parser's family and suffix rejoined. -- There is no suffix box. A suffix the parser peels (`Jr`, `OBE`) stood - last in the display name, so it was typed into the last filled box, - normally the surname: the parser's family is compared with its suffix appended, - and the report counts the agreements that needed it. +- Where a source has no suffix box, a suffix the parser peels (`Jr`, + `OBE`) stood last in the display name, so it was typed into the last + filled box, normally the surname: the parser's family is compared + with its suffix appended, and the report counts the agreements that + needed it. Where it has one (`cz-candidates`' title after the name), + the suffix is compared with that box alone. - Where a source has a title box, a filed title the parser does not read as the title, holding a word the vocabulary lists in neither its titles nor its connectives, is set aside rather than scored: a job (`Managing Partner`) or a name @@ -103,10 +111,22 @@ class Fielded: values joined by a space in that order: ORCID's given-names box holds the middle names too. A source with no title box sets nothing aside. The group is title-cased for the report unless `group_cased` says - it is written as it should print (a country code).""" + it is written as it should print (a country code). `sets_aside` + turns off the title set-aside (read) for a source whose title box + holds what the law asks filed there: the Czech register's holds the + candidate's degrees, most of which the default vocabulary does not + list (`Bc.`, `Ing.`). The set-aside took 1,059 of its records on + 2026-10-30, every one a degree or a military rank the candidate + filed and none a job: 756 whose box after the name holds `dis` + (re.search(r"\bdis\b", box.replace(".", "").casefold()); `Bc. Jan + Novák, DiS.` read with title `Bc. DiS.`), 179 of the rest whose + title box holds `et` (re.search(r"\bet\b", box); `Ing. et Ing.` + read as given names; 182 by that test alone, 3 holding both), and + 124 others.""" fields: dict[str, str] group: str group_cased: bool = False + sets_aside: bool = True def __post_init__(self) -> None: # A misspelled role would find no box (_box), and a misspelled @@ -125,7 +145,10 @@ def __post_init__(self) -> None: group_cased=True), local_corpora.PV: Fielded( {"given": "given+middle", "family": "family"}, group="country", - group_cased=True)} + group_cased=True), + local_corpora.VOLBY: Fielded( + {"titulpred": "title", "jmeno": "given+middle", "prijmeni": "family", + "titulza": "suffix"}, group="election", group_cased=True, sets_aside=False)} #: The fewest records sharing a set-aside title for the report to print #: it: a word typed into the title box by this many filers is no one #: person's name. @@ -168,7 +191,7 @@ def read(record: dict[str, str], fielded: Fielded, parser: Parser, got = {role: " ".join(v for r in role.split("+") if (v := getattr(name, r))) for role in fielded.fields.values()} by_suffix = False - if name.suffix: + if name.suffix and not _box(fielded, "suffix"): family = fielded.fields[_box(fielded, "family")] with_suffix = f"{got[family]} {name.suffix}".strip() by_suffix = fold(with_suffix) == fold(record.get(_box(fielded, "family"), "")) @@ -176,7 +199,7 @@ def read(record: dict[str, str], fielded: Fielded, parser: Parser, wrong = {role: got[role] for box, role in fielded.fields.items() if fold(record.get(box, "")) != fold(got[role])} title = record.get(_box(fielded, "title"), "") - if title and "title" in wrong and unlisted(title, listed): + if fielded.sets_aside and title and "title" in wrong and unlisted(title, listed): return Reading(wrong, set_aside=True) return Reading(wrong, by_suffix=by_suffix and not wrong) @@ -224,19 +247,21 @@ class Score: def score(rows: Iterator[dict[str, str]] | list[dict[str, str]], fielded: Fielded, - parser: Parser, *, jobs: int = 1) -> Score: + parser: Parser, *, jobs: int = 1, locale: str = "") -> Score: """Every record read (read) and counted. With `jobs` above 1 the records are read in that many worker processes (snapshot.parallel), - each with a Parser() of its own, so `parser` must be the default - one: a configured parser cannot be sent to a worker by its name.""" + each building the parser of its own that `locale` names (parser_of), + so `parser` must be that one: a configured parser cannot be sent to + a worker by its name.""" rows = list(rows) - if jobs > 1 and parser != type(parser)(): - raise ValueError("score() reads in workers with the default Parser(); " - "pass jobs=1 to score with another parser") + if jobs > 1 and parser != parser_of(locale): + raise ValueError(f"score() reads in workers with parser_of({locale!r}); " + f"pass jobs=1 to score with another parser") if jobs > 1: from snapshot import parallel source = next(s for s, f in FIELDED.items() if f == fielded) - readings = parallel(_read_chunk, rows, jobs, source, module="local_agreement") + readings = parallel(_read_chunk, rows, jobs, source, locale, + module="local_agreement") else: listed = _title_words(parser.lexicon) readings = [read(record, fielded, parser, listed) for record in rows] @@ -258,15 +283,26 @@ def score(rows: Iterator[dict[str, str]] | list[dict[str, str]], fielded: Fielde return result -def _read_chunk(rows: list[dict[str, str]], source: str) -> tuple[str, list[Reading]]: +def _read_chunk(rows: list[dict[str, str]], source: str, locale: str = "" + ) -> tuple[str, list[Reading]]: """The nameparser this process imported, and its reading of every record of `source`: score()'s unit of work in a worker.""" import nameparser - parser = nameparser.Parser() + parser = parser_of(locale) listed = _title_words(parser.lexicon) return nameparser.__file__, [read(r, FIELDED[source], parser, listed) for r in rows] +def parser_of(locale: str) -> Parser: + """The parser a score reads with: Parser() for no locale, else + parser_for the pack `locale` names (`cs` for locales.CS).""" + import nameparser + import nameparser.locales + if not locale: + return nameparser.Parser() + return nameparser.parser_for(getattr(nameparser.locales, locale.upper())) + + def _pct(n: int, of: int) -> str: return f"{100 * n / of:5.1f}%" if of else " n/a" @@ -276,9 +312,11 @@ def report(result: Score, fielded: Fielded, top: int) -> list[str]: total, agreed = sum(result.scored.values()), sum(result.agreed.values()) misses: Counter[str] = sum(result.misses.values(), Counter()) out = [f"scored {total:,}; set aside {sum(result.set_aside.values()):,} " - f"(a title box the parser read otherwise, holding an unlisted word)", - f"every field agrees: {agreed:,} of {total:,} ({_pct(agreed, total).strip()}); " - f"{result.by_suffix:,} of them with the parser's suffix in the surname box", + f"(a title box the parser read otherwise, holding an unlisted word)" + if fielded.sets_aside else f"scored {total:,}; none set aside", + f"every field agrees: {agreed:,} of {total:,} ({_pct(agreed, total).strip()})" + + ("" if _box(fielded, "suffix") else + f"; {result.by_suffix:,} of them with the parser's suffix in the surname box"), "misses per field: " + ", ".join(f"{r} {misses[r]:,}" for r in roles), "", f"by {fielded.group} (at least {GROUP_FLOOR:,} records), misses per field:", f" {fielded.group:<24} {'records':>10} {'agree':>7} " @@ -291,33 +329,39 @@ def report(result: Score, fielded: Fielded, top: int) -> list[str]: + " ".join(f"{row[r]:>8,}" for r in roles)) out += ["", f"the {top} commonest disagreement shapes (field: words filed->parsed):"] out += [f" {n:>9,} {s}" for s, n in result.shapes.most_common(top)] - out += ["", f"set-aside title boxes written by at least {TITLE_FLOOR} records:"] - out += [f" {n:>9,} {t}" for t, n in result.set_aside.most_common() - if n >= TITLE_FLOOR][:top] + if fielded.sets_aside: + out += ["", f"set-aside title boxes written by at least {TITLE_FLOOR} records:"] + out += [f" {n:>9,} {t}" for t, n in result.set_aside.most_common() + if n >= TITLE_FLOOR][:top] return out -def record_path(source: str) -> Path: - return local_corpora.directory(source, create=False) / RECORD +def record_path(source: str, locale: str = "") -> Path: + """The shared record, one per locale: agreement.jsonl for the + default parser, agreement.cs.jsonl for locales.CS.""" + name = RECORD.replace(".jsonl", f".{locale}.jsonl") if locale else RECORD + return local_corpora.directory(source, create=False) / name def write_record(path: Path, local: local_corpora.Local, entries: dict[str, dict[str, Any]], - tree: str) -> None: + tree: str, locale: str = "") -> None: header = {"raw_sha256": local.meta["sha256"], "scorer": scorer_fingerprint(local.source), "data_date": local.meta["data_date"], "tree": tree, - "entries": len(entries)} + "entries": len(entries), "locale": locale} lines = [corpus_build.jsonl_line(header)] lines += [corpus_build.jsonl_line({"key": k, **v}) for k, v in sorted(entries.items())] corpus_build.write_atomic(path, ("\n".join(lines) + "\n").encode()) -def read_record(path: Path, local: local_corpora.Local) -> dict[str, dict[str, Any]]: +def read_record(path: Path, local: local_corpora.Local, locale: str = "" + ) -> dict[str, dict[str, Any]]: """The recorded entries, or SystemExit naming the record command where there is no record, one that is not a record this tool wrote, one recorded from another fetch of the corpus, which would report - the records the source gained and lost as moves, or one recorded by + the records the source gained and lost as moves, one recorded by another version of the scorer, which would report its own change as - the parser's.""" + the parser's, or one scored under another locale than `locale` + (a record written before the header named one was the default's).""" again = f"Record it again from the base checkout: {_record_command(local.source)}" if not path.exists(): raise SystemExit(f"no record at {path}: record one first, from the base " @@ -345,13 +389,16 @@ def read_record(path: Path, local: local_corpora.Local) -> dict[str, dict[str, A if recorded.get("scorer") != scorer_fingerprint(local.source): raise SystemExit(f"{path} was recorded by another version of this scorer, " f"so its moves would be the scorer's. {again}") + if (held := recorded.get("locale", "")) != locale: + raise SystemExit(f"{path} was scored under the locale {held or '(default)'!r}, " + f"not {locale or '(default)'!r}. {again}") return entries #: What decides a record's reading, besides the parser: scorer_fingerprint #: hashes them, as local_corpora.fold_fingerprint hashes the word fold. _SCORER_CODE = ("fold", "_title_words", "unlisted", "read", "_box", "Reading", - "key", "score", "_read_chunk") + "key", "score", "_read_chunk", "parser_of") def scorer_fingerprint(source: str) -> str: @@ -375,42 +422,44 @@ class by its methods), of corpus_build.jsonl_line, which writes an KEPT = "agreements" -def kept_path(source: str, local: local_corpora.Local, digest: str) -> Path: +def kept_path(source: str, local: local_corpora.Local, digest: str, + locale: str = "") -> Path: """Where a record of `local`'s fetch scored by a package of `digest` is kept: keyed by everything a record depends on besides the parser - (the fetch, the interpreter's minor version, the scorer), so a key - that matches is a record of this base.""" + (the fetch, the interpreter's minor version, the scorer, the + locale), so a key that matches is a record of this base.""" py = f"py{sys.version_info[0]}{sys.version_info[1]}" return (record_path(source).parent / KEPT / f"{local.meta['sha256'][:12]}-{digest[:16]}-{py}" - f"-{scorer_fingerprint(source)[:12]}.jsonl") + f"-{scorer_fingerprint(source)[:12]}{f'-{locale}' if locale else ''}.jsonl") -def _score_base(root: Path, source: str, out: Path, jobs: int = 1) -> None: +def _score_base(root: Path, source: str, out: Path, jobs: int = 1, + locale: str = "") -> None: """`root` scored into `out` in a child process (one process imports one nameparser), which refuses a package other than `root`'s.""" import snapshot command = [sys.executable, str(HERE / "local_agreement.py"), source, "--record", "--tree", str(root), "--out", str(out), - "--jobs", str(jobs)] + "--jobs", str(jobs), *(["--locale", locale] if locale else [])] done = subprocess.run(command, env=snapshot.child_env(), capture_output=True, text=True) if done.returncode: raise SystemExit(f"scoring the base {root} failed:\n{done.stdout}{done.stderr}") -def base_entries(root: Path, local: local_corpora.Local, jobs: int = 1 - ) -> dict[str, dict[str, Any]]: +def base_entries(root: Path, local: local_corpora.Local, jobs: int = 1, + locale: str = "") -> dict[str, dict[str, Any]]: """The base checkout's record, kept for its package's digest or scored into one now, and a line saying which base was read. Never the shared record: a sibling session's would pass for this base.""" import snapshot digest = snapshot._source_digest(root) - path = kept_path(local.source, local, digest) + path = kept_path(local.source, local, digest, locale) how = f"reused the kept record {path.name}" if path.exists(): try: - entries = read_record(path, local) + entries = read_record(path, local, locale) except SystemExit as exc: print(f"deleted the kept record {path.name}: {exc}; scoring the base\n") path.unlink(missing_ok=True) @@ -426,8 +475,8 @@ def base_entries(root: Path, local: local_corpora.Local, jobs: int = 1 # the child: a record published under the key it was scored for. at = snapshot.scratch(path.parent) try: - _score_base(root, local.source, at / path.name, jobs) - entries = read_record(at / path.name, local) + _score_base(root, local.source, at / path.name, jobs, locale) + entries = read_record(at / path.name, local, locale) if snapshot._source_digest(root) != digest: raise SystemExit(f"{root / 'nameparser'} changed while it was scored, " f"so its record may be half old and half new; run again") @@ -494,7 +543,11 @@ def main(argv: list[str] | None = None) -> int: help="print up to N moved records of each kind, names and all") ap.add_argument("--jobs", type=int, metavar="N", default=os.cpu_count() or 1, help="score in N worker processes (default: every CPU)") + ap.add_argument("--locale", metavar="CODE", default="", + help="score the parser of this locale pack (cs: locales.CS) " + "instead of the default, against a record of its own") args = ap.parse_args(argv) + locale = args.locale.lower() if args.base and (args.record or args.out): ap.error("--base compares and records nothing but its kept record; " "drop --record and --out") @@ -502,26 +555,35 @@ def main(argv: list[str] | None = None) -> int: ap.error("--out names where --record writes") local = local_corpora.require(args.source) root = Path(args.tree).resolve() if args.tree else HERE.parents[1] - if args.base: - from snapshot import base_root - old = base_entries(base_root(args.base, root), local, args.jobs) - elif not args.record: - old = read_record(record_path(args.source), local) if args.tree: sys.path.insert(0, str(root)) import nameparser + import nameparser.locales from snapshot import check_source at = check_source(nameparser.__file__, root) + # Before any record is read or base scored, whose refusals would + # otherwise stand in for this one ("no record", "scoring the base failed"). + if locale and locale not in nameparser.locales.available(): + ap.error(f"--locale {args.locale}: no such pack; the packs are " + + ", ".join(nameparser.locales.available())) + if args.base: + from snapshot import base_root + old = base_entries(base_root(args.base, root), local, args.jobs, locale) + elif not args.record: + old = read_record(record_path(args.source, locale), local, locale) fielded = FIELDED[args.source] print(local.header()) + if locale: + print(f"under parser_for(locales.{locale.upper()})") try: - result = score(records(local), fielded, nameparser.Parser(), jobs=args.jobs) + result = score(records(local), fielded, parser_of(locale), jobs=args.jobs, + locale=locale) except local_corpora.CorpusDamaged as exc: raise SystemExit(str(exc)) from None print("\n".join(report(result, fielded, args.top))) if args.record: - out = args.out or record_path(args.source) - write_record(out, local, result.entries, str(at.parent)) + out = args.out or record_path(args.source, locale) + write_record(out, local, result.entries, str(at.parent), locale) print(f"\nrecorded {len(result.entries):,} records that did not wholly " f"agree to {out}") return 0 diff --git a/tools/differential/local_corpora.py b/tools/differential/local_corpora.py index fe2eb8b7..5707a1d8 100644 --- a/tools/differential/local_corpora.py +++ b/tools/differential/local_corpora.py @@ -45,7 +45,9 @@ record's two boxes joined, with the boxes, the credit name and the country with where it came from; one distinct PatentsView inventor name in a country, its two boxes joined, with the boxes, the country - and how many people filed under it. The counts are counted from here. + and how many people filed under it; one distinct Czech election + candidate's four boxes composed into a name, with the boxes and the + election. The counts are counted from here. - corpus.jsonl -- the distinct names, in the corpus line format snapshot.py and compare.py --corpus read: each record's "name", and for nepal-candidates its Devanagari name too, and for @@ -84,7 +86,8 @@ local_govuk.py (govuk-people), local_psc.py (psc-individuals), local_jrc.py (jrc-names), local_nepal.py (nepal-candidates), local_orcid.py -(orcid-researchers) and local_patentsview.py (patentsview-inventors). +(orcid-researchers), local_patentsview.py (patentsview-inventors) and +local_volby.py (cz-candidates). """ from __future__ import annotations @@ -135,6 +138,7 @@ from local_tse import check_tse, fetch_tse # noqa: E402 from local_tw import fetch_tw # noqa: E402 from local_ukec import fetch_ukec # noqa: E402 +from local_volby import VOLBY, check_volby, fetch_volby # noqa: E402 RAW, CORPUS, META, SNAPSHOT = "raw.jsonl", "corpus.jsonl", "meta.json", "snapshot.jsonl" INDEX = "wordindex.json" @@ -257,6 +261,12 @@ def index(self) -> WordIndex: "Open Data Portal (CC BY 4.0; credit PatentsView)", unit="names", fetch=fetch_pv, renew=renew_pv, check=check_pv, extract=pv_extract, boxes=pv_boxes), + VOLBY: Source( + credit="ČSÚ (Czech Statistical Office) election candidate registers, " + "volby.gov.cz (CC BY 4.0, csu.gov.cz/podminky_pro_vyuzivani_a_dalsi_" + "zverejnovani_statistickych_udaju_csu; each name composed from " + "the filed boxes, so derived data)", + unit="records", fetch=fetch_volby, check=check_volby), } diff --git a/tools/differential/local_volby.py b/tools/differential/local_volby.py new file mode 100644 index 00000000..67bcb644 --- /dev/null +++ b/tools/differential/local_volby.py @@ -0,0 +1,244 @@ +"""The cz-candidates local corpus: the Czech election candidate registers. + +cz-candidates -- every candidate on the registers the Czech Statistical +Office (ČSÚ) publishes for an election on volby.gov.cz, one CSV per +election, each candidate's name in the four boxes it was filed in: the +title before the name (TITULPRED), the given names (JMENO), the +surname (PRIJMENI) and the title after it (TITULZA). A local corpus +whose names carry their fields (local_agreement.FIELDED), so +local_agreement.py scores the parser against them, by election. It is +the corpora's one large population of titled names with the title +stated: of the 197,506 candidates of the 2022 municipal elections, +30.9% filed a title before the name and 3.8% one after it (2026-10-30), +nearly all academic degrees (Ing., Mgr., Bc., MUDr.; Ph.D., DiS., +MBA), with no courtesy titles or offices. + +A record's name is COMPOSED, not written by anyone: the boxes joined in +the Czech convention, `TITULPRED JMENO PRIJMENI, TITULZA` (`Ing. Jan +Novák, Ph.D.`), so it holds no case, spacing or punctuation of its own +beyond what the boxes hold. A comma a filer typed inside a box before +the last (`Dopitová, roz. Lišková` in the surname box, `Ing., Bc.` in +the title box: 21 records on 2026-10-30) makes a comma shape nobody +wrote in the composed name; such a record is kept, as filed, and +counted (records_with_a_comma_inside_a_box), an accepted cost: the +score compares the parse with the boxes, which say what was meant, so +dropping it would only hide the miss. The register's terms +(https://csu.gov.cz/podminky_pro_vyuzivani_a_dalsi_zverejnovani_statistickych_udaju_csu, +which the open-data index links, read 2026-10-30) license ČSÚ's +statistical information under CC BY 4.0, and ask that whoever spreads +the data state the licence, best by a link to that page, and mark +modified or derived data as such, never as unchanged official +statistics; the composed names are derived, which the credit line +says. The names are personal data, published by law, and are cached, +never committed. +""" +from __future__ import annotations + +import csv +import io +import re +import sys +import urllib.parse +import zipfile +from collections import Counter +from typing import Any + +from local_core import Fetched, _refuse_past, _retried + + +VOLBY = "cz-candidates" +VOLBY_SITE = "https://volby.gov.cz/opendata/" +#: Each election read, by the code volby.gov.cz files it under: the +#: register's member in the zip, and the candidates it held on +#: 2026-10-30. A register is not final when its election is held: the +#: kv2022 one carries the by-elections held through 2024 and was +#: re-issued with a stamp of 2026-03-28. The count pin is what protects: +#: a register holding another number (a revision, or a register trimmed +#: to the elected) is refused until it is measured and the pin moved by +#: hand. The index lists kv2002 to kv2026 and +#: ps2002 to ps2025 too; adding one is a row here, measured first. +VOLBY_ELECTIONS: dict[str, tuple[str, int]] = { + "kv2022": ("csv_od/kvrk.csv", 197_506), + "ps2025": ("csv_od/psrk.csv", 4_475), +} +#: The four name boxes, by column, and the record key each is kept +#: under, in the order the composed name joins them (compose). +VOLBY_BOXES = {"TITULPRED": "titulpred", "JMENO": "jmeno", "PRIJMENI": "prijmeni", + "TITULZA": "titulza"} +#: How a record's name is built from its boxes (compose), in meta.json's +#: detail: a change to compose bumps it, and a corpus composed by +#: another version is refused on load (check_volby), since the sha256 +#: checks prove the files are the ones fetched, not that today's code +#: would compose them so. +VOLBY_READ_VERSION = 1 +#: The share of an election's rows a fetch skips before refusing the +#: register: rows that are not the header's width, and candidates with +#: no given name or no surname (none of either in 202,000 rows on +#: 2026-10-30). +VOLBY_MAX_BAD = 1 / 10_000 +#: The least share of an election's candidates filing a title before or +#: after the name: 32.2% of kv2022 and 45.6% of ps2025 did on +#: 2026-10-30. A title column emptied or renamed would otherwise read +#: as a register of untitled names, the shape this corpus is for. +VOLBY_MIN_TITLED = 1 / 5 + + +def _zip_link(election: str) -> re.Pattern[str]: + """The register's CSV zip as the election's index page links it: + named by the election and a date stamp (KV2022reg20260328_csv.zip, + KV2018_reg_20230224_csv.zip), which is read off the page, since a + guessed stamp answers an HTML 404 page.""" + return re.compile(rf'href="({re.escape(election)}_?reg_?(\d{{8}})_csv\.zip)"', + re.IGNORECASE) + + +def index_url(election: str) -> str: + return f"{VOLBY_SITE}{election}/{election}_opendata.htm" + + +def _register_zip(election: str) -> tuple[str, str]: + """The newest register zip the index page links, and its stamp as a + date: ps2021's lists a revision beside the first register.""" + url = index_url(election) + page = _retried(url, lambda body: body, 0).decode("utf-8", "replace") + links = _zip_link(election).findall(page) + if not links: + raise ValueError(f"{url} links no register CSV zip: the page has changed") + file, stamp = max(links, key=lambda link: link[1]) + return urllib.parse.urljoin(url, file), f"{stamp[:4]}-{stamp[4:6]}-{stamp[6:]}" + + +def _a_zip(body: bytes) -> bytes: + """`body` where it is a zip, else ValueError, which _retried takes + for a body cut short in transit.""" + if not zipfile.is_zipfile(io.BytesIO(body)): + raise ValueError(f"the answer is not a zip ({body[:15]!r}...)") + return body + + +def compose(boxes: dict[str, str]) -> str: + """A record's name: its boxes joined in the Czech convention, + `TITULPRED JMENO PRIJMENI, TITULZA`, an empty box left out with its + separator.""" + name = " ".join(v for k in ("titulpred", "jmeno", "prijmeni") if (v := boxes.get(k))) + return f"{name}, {boxes['titulza']}" if boxes.get("titulza") else name + + +def _volby_rows(text: str, election: str) -> tuple[set[str], dict[str, int]]: + """One election's register read down to what the corpus keeps: each + distinct candidate's four boxes, stripped, keyed by the composed + name, and the counts. Columns are read by name, since the elections' + registers differ in every other column, and a register missing a + box's column, or whose last row is cut short, is refused.""" + rows = list(csv.reader(io.StringIO(text, newline=""))) + header = rows[0] if rows else [] + missing = [c for c in VOLBY_BOXES if c not in header] + if missing: + raise ValueError(f"the {election} register has no column {', '.join(missing)}: " + f"its layout has changed under VOLBY_BOXES") + # csv returns a body cut inside its last row as a short last row + # (AGENTS.md's Gotchas): a cut only shortens a row, so a long last + # row is left to the malformed share. + if not text.endswith("\n") or len(rows) < 2 or len(rows[-1]) < len(header): + raise ValueError(f"the {election} register's last row is cut short or " + f"blank; fetch again") + at = {key: header.index(column) for column, key in VOLBY_BOXES.items()} + counts: Counter[str] = Counter() + kept: set[str] = set() + for row in rows[1:]: + counts["candidates"] += 1 + if len(row) != len(header): + counts["rows_not_the_header_s_width"] += 1 + continue + boxes = {key: " ".join(row[i].split()) for key, i in at.items()} + if not boxes["jmeno"] or not boxes["prijmeni"]: + counts["candidates_without_a_given_name_or_surname"] += 1 + continue + counts["titled"] += bool(boxes["titulpred"] or boxes["titulza"]) + kept.add("\t".join(boxes[k] for k in VOLBY_BOXES.values())) + return kept, dict(counts) + + +def _volby_read(election: str, body: bytes) -> tuple[list[dict[str, str]], dict[str, int]]: + """One election's records from its zip, or ValueError where the + register is not the one this reads: no member, not UTF-8, another + number of candidates than VOLBY_ELECTIONS pins, past VOLBY_MAX_BAD + of its rows skipped, or under VOLBY_MIN_TITLED of them titled.""" + member, pinned = VOLBY_ELECTIONS[election] + try: + with zipfile.ZipFile(io.BytesIO(body)) as archive: + if member not in archive.namelist(): + raise ValueError(f"the {election} zip holds no {member}") + raw = archive.read(member) + except zipfile.BadZipFile as exc: + raise ValueError(f"the {election} zip cannot be read ({exc}); fetch again" + ) from None + try: + text = raw.decode("utf-8-sig") + except UnicodeDecodeError as exc: + raise ValueError(f"the {election} register is not UTF-8 ({exc})") from None + try: + kept, counts = _volby_rows(text, election) + except csv.Error as exc: + raise ValueError(f"the {election} register is not CSV this reads ({exc})" + ) from None + candidates = counts["candidates"] + if candidates != pinned: + raise ValueError(f"the {election} register holds {candidates:,} candidates " + f"where VOLBY_ELECTIONS pins {pinned:,}: a changed register; " + f"measure it and move the pin") + are = f"of the {election} register's rows are" + _refuse_past([(counts.get("rows_not_the_header_s_width", 0), candidates, VOLBY_MAX_BAD, + f"{are} not the header's width"), + (counts.get("candidates_without_a_given_name_or_surname", 0), candidates, + VOLBY_MAX_BAD, f"{are} a candidate with no given name or surname")], + "the register is not the one this reads") + if counts.get("titled", 0) < candidates * VOLBY_MIN_TITLED: + raise ValueError(f"{counts.get('titled', 0):,} of the {election} register's " + f"{candidates:,} candidates filed a title: a title column " + f"emptied or moved") + records = [] + commas = 0 + for line in sorted(kept): + boxes = dict(zip(VOLBY_BOXES.values(), line.split("\t"), strict=True)) + commas += any("," in boxes[k] for k in ("titulpred", "jmeno", "prijmeni")) + records.append({"name": compose(boxes), "election": election, + **{k: v for k, v in boxes.items() if v}}) + return records, {**counts, "records": len(records), + "records_with_a_comma_inside_a_box": commas} + + +def fetch_volby() -> Fetched: + """The distinct candidates of every election in VOLBY_ELECTIONS, one + record per election and distinct four boxes: the composed name + (compose), the election and the boxes filled. A candidate who stood + twice in one election under the same boxes, or two who share them, + are one record; the age, occupation, municipality and party columns + are read past. The data date names each election's register stamp, + the date the zip's file name carries.""" + records: list[dict[str, str]] = [] + detail: dict[str, Any] = {"elections": " ".join(VOLBY_ELECTIONS), + "read_version": VOLBY_READ_VERSION} + stamps = [] + for election in VOLBY_ELECTIONS: + url, stamp = _register_zip(election) + print(f"downloading {url} ...", file=sys.stderr) + body = _retried(url, _a_zip, 0, timeout=300) + read, counts = _volby_read(election, body) + records += read + stamps.append(f"{election} {stamp}") + detail[election] = {"zip": url.rsplit("/", 1)[1], **counts} + return Fetched(records, ", ".join(stamps), detail) + + +def check_volby(meta: dict[str, Any]) -> str: + """Why a corpus fetched for other elections than VOLBY_ELECTIONS, + or composed by another VOLBY_READ_VERSION, is refused, or "".""" + detail = meta["detail"] + if detail.get("elections") != " ".join(VOLBY_ELECTIONS): + return (f"the corpus holds the elections {detail.get('elections')}, and this " + f"tool reads {' '.join(VOLBY_ELECTIONS)}") + if detail.get("read_version") != VOLBY_READ_VERSION: + return (f"the corpus was composed by version {detail.get('read_version')} of " + f"compose, and this tool is version {VOLBY_READ_VERSION}") + return ""