diff --git a/AGENTS.md b/AGENTS.md index d2bf7ae33..0af9caf01 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -267,7 +267,7 @@ The 2.0 rewrite lands as underscore-private modules alongside the v1 code. These **Adding a curated sub-set of an existing vocabulary set** (a marker over a base set, as `AFTER_GIVEN_TITLES` is over `TITLES`) — it is a behavioral PROPERTY of a reading, not a second word list: a row in `tools/vocab/build.py`'s `BEHAVIORAL` table, a branch in `compile_pool` adding the words whose reading carries it, a constant in its `CONSTANTS` roster and a key of the same name in its `Compiled` TypedDict (an import-time assert holds the two to one list), and the `NAME = _compiled.NAME` assignment in the config module. The subset relation then holds by construction, since the property cannot exist without its reading, so it needs no check. A marker that records a clash with a NAME is the other shape: derived from the word's name reading beside the clashing one (`SUFFIX_ACRONYMS_AMBIGUOUS`, `SUFFIX_WORDS_AMBIGUOUS`, and the complement `NON_GIVEN_NAME_PARTICLES` works from), not written as a property, so it too holds by construction (decisions/vocabulary-model, 2026-10-25). A relation construction does not give (`∩ == ∅` with a set the sub-set cannot logically overlap; a sub-set member that is also a title is silently inert, title handling consuming it first) goes in `check_pool`, with a mutation in `tests/v2/test_vocab.py`'s table. Do **not** forbid `titles ∩ particles` — that overlap is intentional (`freiherr`, `st`). On the v1 side a sub-set new in 2.x, such as `honorific_tails`, `conjunctions_ambiguous`, `after_given_titles` or `suffix_words_ambiguous`, gets no `Constants` attribute and rides in `_snapshot()` as an intersection with its base, the shim roster under the 2.0 conventions listing each (the v1 ATTRIBUTE names did not move in #293, only the module constants did). A runtime `.add()` to the sub-set's `SetManager` does **not** propagate to the parent's (same as `first_name_titles`→`titles`), so a caller adding a brand-new word adds it to both. Two traps a gating set met (#752 phase 2i). Where a caller listing a word in the BASE set has always meant "read it as this everywhere" (the 1.4 recipe `c.titles.add("dean")`), the gated words compile OUT of the base rather than into both, or the gate overrides the caller's listing: `TITLES_AMBIGUOUS` is not a subset of `TITLES` for that reason, and a word in both reads as a title. And `_VOCAB_FIELDS` is read whole by two rosters in `_pipeline/_vocab.py`, the capitals shape's "unlisted" (`in_any_wordlist`) and the comma contrast's "claimed as non-name" (`_NON_NAME_FIELDS`), so a new field joins both by default; a field whose words may be names leaves both, which the grid found as `John Smith, DEAN` reversed where C6 had read a credential; #789's `given_name_abbreviations` met it again (`John Smith, JAS` and `Jas SMITH, XYZ` reversed), found by review because its grid held only the dotted spellings the set was for -- a new field's grid holds the spellings it must NOT move too. -**Validating a new parsing rule** — before implementing, simulate it in a throwaway script against `TEST_NAMES` (plus a few target-language examples) to catch regressions/false-positives early. E.g. this surfaced the `last_base` `do`/`st`/`mc` empties and the patronymic `"David Michael Abramovich"` false-reorder. After the change, `tests/v2/test_snapshot.py` lists every corpus name it moved, grouped by changed field -- read them, and re-record with `uv run python tools/differential/snapshot.py --record` once every move is intended, so the PR diff of `snapshot.jsonl` shows the reviewer the same list. `uv run python tools/differential/cldr_agreement.py` reports how the tree's agreement with CLDR's stated name fields moved; `tests/v2/test_cldr_agreement.py` fails on any move, and `tools/differential/cldr_agreement.json` doubles as a measured list of names the parser and CLDR read differently. The same command and test score the GND corpus's real names (`gnd_fields.jsonl`, record `gnd_agreement.json`) by the same rules, and `--record` rewrites every record, printing per source what it accepts. A locale-pack change reads the second view the same command prints, per pack (`cldr_agreement_locales.json`, #672, and `wikidata_native_agreement_locales.json` for real Han and kana names); its `JA` rows run only with the `ja` extra installed. A change to Han, Hangul or kana order or to Korean segmentation reads `wikidata_native_agreement.json` too: the only real names in those scripts the corpora hold. It holds no honorific, so a change to the honorific peel moves nothing there. A change to comma or credential handling also reads the local corpus of hand-typed names (`snapshot.py --local ctgov-officials --base `, which reads the base from a baseline kept for its package's files and never the shared snapshot beside the corpus (a base read once from the same fetch is copied rather than parsed again; `--sample N` reads a fixed sample while iterating; the shared snapshot beside the corpus is one file per source for every worktree, and #752's phase 2f found ctgov reporting a sibling branch's 1,334 joined-credential readings as its own moves against it); tools/differential/README.md's "Local corpora"): the one population of names nobody chose for a test or a catalog, so it shows how often each shape is typed, and the one no CI run sees. A change to title handling reads `ukec-individuals` the same way: UK donors' names in spoken order and mixed case, in 8,219 of whose 15,214 `parse()` read a title on 2026-10-22. A change to particles or generational words reads `snapshot.py --local tse-donors` the same way, Brazilian names in spoken order dense with both. A change to initials, to a title word outside English, or to how a script's combining marks are read reads `snapshot.py --local nepal-candidates` the same way: Nepal's local election candidates, each a romanized name (surnames written as initials, `B.K.`, `K.C.`; honorific titles such as `Guru` in front, `Shekh`, a title there until #752's phase 2j made it one only inside a title phrase, and `Md.`, which its period makes a title by shape) beside the same name in Devanagari, written with vowel signs and the virama, combining marks the letter-shape tests do not count as letters (decisions/nfd-shapes), and with the joiners ZWJ and ZWNJ. It is the only large population of South Asian names the corpora hold, and its corpus reads both scripts. A change to non-Latin scripts or to Arabic-script names reads `snapshot.py --local jrc-names` the same way, and `local_corpora.py groups jrc-names`: the spellings of one person across scripts and transliterations, harvested from news text, with 17,074 Arabic-script spellings where the committed corpora hold 315 distinct names with an Arabic letter, 96 of them CLDR's fictional renderings (2026-10-24), and under a licence that keeps even a reading of one of its names in the cache. A change to which field a word lands in also reads `local_agreement.py psc-individuals --base ` the same way: real names with the four boxes their filers typed them into (UK companies' people with significant control), scored field by field and by nationality. It is the only real fielded population outside the committed fields files, and no CI run sees it either. +**Validating a new parsing rule** — before implementing, simulate it in a throwaway script against `TEST_NAMES` (plus a few target-language examples) to catch regressions/false-positives early. E.g. this surfaced the `last_base` `do`/`st`/`mc` empties and the patronymic `"David Michael Abramovich"` false-reorder. After the change, `tests/v2/test_snapshot.py` lists every corpus name it moved, grouped by changed field -- read them, and re-record with `uv run python tools/differential/snapshot.py --record` once every move is intended, so the PR diff of `snapshot.jsonl` shows the reviewer the same list. `uv run python tools/differential/cldr_agreement.py` reports how the tree's agreement with CLDR's stated name fields moved; `tests/v2/test_cldr_agreement.py` fails on any move, and `tools/differential/cldr_agreement.json` doubles as a measured list of names the parser and CLDR read differently. The same command and test score the GND corpus's real names (`gnd_fields.jsonl`, record `gnd_agreement.json`) by the same rules, and `--record` rewrites every record, printing per source what it accepts. A locale-pack change reads the second view the same command prints, per pack (`cldr_agreement_locales.json`, #672, and `wikidata_native_agreement_locales.json` for real Han and kana names); its `JA` rows run only with the `ja` extra installed. A change to Han, Hangul or kana order or to Korean segmentation reads `wikidata_native_agreement.json` too: the only real names in those scripts the corpora hold. It holds no honorific, so a change to the honorific peel moves nothing there. A change to comma or credential handling also reads the local corpus of hand-typed names (`snapshot.py --local ctgov-officials --base `, which reads the base from a baseline kept for its package's files and never the shared snapshot beside the corpus (a base read once from the same fetch is copied rather than parsed again; `--sample N` reads a fixed sample while iterating; the shared snapshot beside the corpus is one file per source for every worktree, and #752's phase 2f found ctgov reporting a sibling branch's 1,334 joined-credential readings as its own moves against it); tools/differential/README.md's "Local corpora"): the one population of names nobody chose for a test or a catalog, so it shows how often each shape is typed, and the one no CI run sees. A change to title handling reads `ukec-individuals` the same way: UK donors' names in spoken order and mixed case, in 8,219 of whose 15,214 `parse()` read a title on 2026-10-22. A change to particles or generational words reads `snapshot.py --local tse-donors` the same way, Brazilian names in spoken order dense with both. A change to initials, to a title word outside English, or to how a script's combining marks are read reads `snapshot.py --local nepal-candidates` the same way: Nepal's local election candidates, each a romanized name (surnames written as initials, `B.K.`, `K.C.`; honorific titles such as `Guru` in front, `Shekh`, a title there until #752's phase 2j made it one only inside a title phrase, and `Md.`, which its period makes a title by shape) beside the same name in Devanagari, written with vowel signs and the virama, combining marks the letter-shape tests do not count as letters (decisions/nfd-shapes), and with the joiners ZWJ and ZWNJ. It is the only large population of South Asian names the corpora hold, and its corpus reads both scripts. A change to non-Latin scripts or to Arabic-script names reads `snapshot.py --local jrc-names` the same way, and `local_corpora.py groups jrc-names`: the spellings of one person across scripts and transliterations, harvested from news text, with 17,074 Arabic-script spellings where the committed corpora hold 315 distinct names with an Arabic letter, 96 of them CLDR's fictional renderings (2026-10-24), and under a licence that keeps even a reading of one of its names in the cache. A change to which field a word lands in also reads `local_agreement.py psc-individuals --base ` the same way: real names with the four boxes their filers typed them into (UK companies' people with significant control), scored field by field and by nationality; and `local_agreement.py orcid-researchers --base `, ORCID holders' own two boxes (given names, family name) from every country, scored by the country they work or live in -- a population each person split for themselves, academic but worldwide, the one to read for a change a language pack makes. These are the only real fielded populations outside the committed fields files, and no CI run sees them either. ## Gotchas @@ -305,7 +305,7 @@ The 2.0 rewrite lands as underscore-private modules alongside the v1 code. These **Perf regressions are caught by the scaling tests, not the absolute-time ones** — the `_thousand_names` tests use constant-size, delimiter-free input and are structurally blind to a complexity regression. A new inner loop, a policy-gated stage or a shape that needs a prefix gets a row in `tests/v2/test_benchmark.py`: its module docstring says which table, and the comment beside each table carries the measurements behind its rule. Coverage is collected on ONE interpreter, 3.14's (`COVERAGE_ON` in `.github/workflows/python-package.yml`, whose comment says why). -**A JSONL file written with `json.dumps(..., ensure_ascii=False)` is cut by `str.splitlines()`** — `json.dumps` leaves U+2028, U+2029 and U+0085 unescaped, and `splitlines()` splits at all three, so one name pasted from a document holding a line separator turns its line into two halves of invalid JSON, and the file into one no reader can load. #801's review found it in the local corpora (a single official among 536,981 would have made the cache unloadable for good), and the review of that fix found it again one file over, in `snapshot.py`'s `dump`/`load`, which the fix had not touched. So it is fixed at the shared source now: `corpus_build.jsonl_line` escapes the three and `corpus_build.jsonl_lines` cuts at `"\n"` only, and `corpus_build`'s corpus and fields writers, `snapshot.py`, `cldr_agreement.py`, `local_corpora.py`, `local_psc.py`'s extracts and `local_agreement.py` go through them. A new JSONL writer or reader in tools/ goes through the same two. Still outside them: the builders that write their own lines (`build_cjk_corpus.py`, `_conventions`, `_shapes`, `_rules`, `_issues`), `compare.py`'s worker stream, and the tests' corpus reads (`_differential_fixtures.py`, `test_corpora.py`); none of the 16 committed JSONL files held one on 2026-10-21. +**A JSONL file written with `json.dumps(..., ensure_ascii=False)` is cut by `str.splitlines()`** — `json.dumps` leaves U+2028, U+2029 and U+0085 unescaped, and `splitlines()` splits at all three, so one name pasted from a document holding a line separator turns its line into two halves of invalid JSON, and the file into one no reader can load. #801's review found it in the local corpora (a single official among 536,981 would have made the cache unloadable for good), and the review of that fix found it again one file over, in `snapshot.py`'s `dump`/`load`, which the fix had not touched. So it is fixed at the shared source now: `corpus_build.jsonl_line` escapes the three and `corpus_build.jsonl_lines` cuts at `"\n"` only, and `corpus_build`'s corpus and fields writers, `snapshot.py`, `cldr_agreement.py`, `local_corpora.py`, `local_psc.py`'s extracts and `local_agreement.py` go through them. A new JSONL writer or reader in tools/ goes through the same two. Still outside them: the builders that write their own lines (`build_cjk_corpus.py`, `_conventions`, `_shapes`, `_rules`, `_issues`), `compare.py`'s worker stream, the tests' corpus reads (`_differential_fixtures.py`, `test_corpora.py`), and `local_orcid`'s extract, streamed through `gzip.open(..., newline="\n")` rather than read whole for its 25M lines, which cuts at `"\n"` only as `jsonl_lines` does; none of the 16 committed JSONL files held one on 2026-10-21. **A row count does not prove a CSV whole: Python's `csv` returns a body cut inside a row as a short last row, without raising** -- even inside an open quote (`'a,b\r\n"x,y'` gives `[['a', 'b'], ['x,y']]`). So a fetch that checks its row count against the source's stated total, and tolerates a few malformed rows, passes a cut body one name short. #842's review found it in `ukec-individuals` (two reviewers independently); the fix requires the body to end with a line break and its last row not to be short (a cut only shortens a row; a long last row is left to the malformed share), with a test for each arm, since a quoted field holding a line break lets a cut end at one. A new CSV source (a `tools/differential/local_.py` module) takes the same check (`local_ukec._ukec_rows`'s last-row test), and its skip shares are tested at their shipped values, not only patched open for a small fixture. diff --git a/tests/v2/test_local_agreement.py b/tests/v2/test_local_agreement.py index fdb95f205..8ad0ddd97 100644 --- a/tests/v2/test_local_agreement.py +++ b/tests/v2/test_local_agreement.py @@ -468,3 +468,57 @@ def test_a_score_whose_worker_read_another_nameparser_aborts( monkeypatch.setattr(nameparser, "__file__", "/elsewhere/nameparser/__init__.py") with pytest.raises(SystemExit, match="a worker imported nameparser from"): agreement.score(list(_ALL), _PSC_FIELDS, Parser(), jobs=2) + + +# -- a source whose given box holds the middle names (orcid-researchers) ----- + +_ORCID_FIELDS = agreement.FIELDED[corpora.ORCID] + + +def _held(given: str, family: str, country: str = "NZ") -> dict[str, str]: + return {"name": f"{given} {family}".strip(), "given": given, "family": family, + "country": country} + + +def test_a_given_box_holding_the_middle_names_is_compared_with_both() -> None: + # The parser reads Marie as a middle name; the box holds both words. + agrees = agreement.read(_held("Ann Marie", "Lee"), _ORCID_FIELDS, Parser(), _LISTED) + assert (agrees.got, agrees.set_aside) == ({}, False) + # Off: the box compared with the given name alone misses. + alone = agreement.Fielded({"given": "given", "family": "family"}, group="country") + assert agreement.read(_held("Ann Marie", "Lee"), alone, Parser(), _LISTED).got == { + "given": "Ann"} + + +def test_a_source_with_no_title_box_scores_a_read_title_as_a_miss_not_filer_error() -> None: + reading = agreement.read(_held("Dr Ann", "Lee"), _ORCID_FIELDS, Parser(), _LISTED) + assert (reading.got, reading.set_aside) == ({"given+middle": "Ann"}, False) + + +def test_a_peeled_suffix_is_compared_inside_the_family_box_of_a_joined_source() -> None: + reading = agreement.read(_held("Bob", "Roe Jr"), _ORCID_FIELDS, Parser(), _LISTED) + assert (reading.got, reading.by_suffix) == ({}, True) + + +def test_a_country_code_group_prints_as_written(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr(agreement, "GROUP_FLOOR", 1) + result = agreement.score([_held("Ann", "Lee"), _held("Bo Yu", "Wu Li", "CN")], + _ORCID_FIELDS, Parser()) + assert result.scored == {"NZ": 1, "CN": 1} + assert result.misses == {"CN": {"given+middle": 1, "family": 1}} + assert result.shapes == {"given+middle 2->3, family 2->1": 1} + + +def test_the_roles_a_box_may_name_are_the_parsers() -> None: + from nameparser import Role + assert agreement.ROLES == {role.value for role in Role} + + +@pytest.mark.parametrize("fields", [ + {"forename": "given", "surname": "famly"}, + {"forename": "given+midle", "surname": "family"}, + {"forename": "given"}]) +def test_a_fielded_source_naming_an_unknown_role_or_no_family_is_refused( + fields: dict[str, str]) -> None: + with pytest.raises(ValueError, match="each box names roles of"): + agreement.Fielded(fields, group="nationality") diff --git a/tests/v2/test_local_corpora.py b/tests/v2/test_local_corpora.py index a31ab9fee..d5125351d 100644 --- a/tests/v2/test_local_corpora.py +++ b/tests/v2/test_local_corpora.py @@ -49,9 +49,9 @@ # the fetches read, so a patch reaches the code only when it is set on # the module holding the name (local_core.RETRIES, local_fec.FEC_MAX_BAD). # A second load by path (load_tool) would be a copy nothing reads. -core, ctgov, fec, tse, tw, ukec, psc, jrc, nepal = ( +core, ctgov, fec, tse, tw, ukec, psc, jrc, nepal, orcid = ( importlib.import_module(f"local_{key}") - for key in ("core", "ctgov", "fec", "tse", "tw", "ukec", "psc", "jrc", "nepal")) + for key in ("core", "ctgov", "fec", "tse", "tw", "ukec", "psc", "jrc", "nepal", "orcid")) _SOURCE = "ctgov-officials" _DATE = "2026-10-21T09:00:05" @@ -2669,7 +2669,8 @@ def test_a_second_corpus_key_the_records_do_not_fill_is_refused( assert local.load(_NEPAL) is None -@pytest.mark.parametrize("source", sorted(set(local.SOURCES) - {_NEPAL})) +@pytest.mark.parametrize("source", sorted( + s for s, row in local.SOURCES.items() if row.corpus_keys == ("name",))) def test_every_other_source_s_corpus_and_index_read_the_name_alone( monkeypatch: pytest.MonkeyPatch, source: str) -> None: # A record's other keys (psc's boxes, tse's registry name) are no @@ -3344,11 +3345,11 @@ def test_no_module_imports_a_sibling_constant() -> None: # tse.TSE_MAX_BAD), and every other reader reads it through that # module. A `from local_x import CONSTANT` would bind a second name a # patch on local_x cannot reach, and a test could pass with its guard - # off. The modules are SOURCES' own, so a new source is covered. PSC and - # JRC are sources' names, keys of SOURCES, and no fetch's setting. + # off. The modules are SOURCES' own, so a new source is covered. PSC, + # JRC and ORCID are sources' names, keys of SOURCES, and no fetch's setting. import ast sources = {source.fetch.__module__ for source in local.SOURCES.values()} - assert sources == {m.__name__ for m in (ctgov, fec, tse, tw, ukec, psc, jrc, nepal)} + assert sources == {m.__name__ for m in (ctgov, fec, tse, tw, ukec, psc, jrc, nepal, orcid)} siblings = {"local_corpora", "local_core", *sources} imported = {(path.stem, alias.name) for path in (Path(str(local.__file__)).parent / f"{s}.py" @@ -3356,7 +3357,8 @@ def test_no_module_imports_a_sibling_constant() -> None: for node in ast.walk(ast.parse(path.read_text(encoding="utf-8"))) if isinstance(node, ast.ImportFrom) and node.module in siblings for alias in node.names if alias.name.lstrip("_").isupper()} - assert imported == {("local_corpora", "PSC"), ("local_corpora", "JRC")} + assert imported == {("local_corpora", "PSC"), ("local_corpora", "JRC"), + ("local_corpora", "ORCID")} def test_the_fold_lists_everything_its_functions_call_and_read() -> None: diff --git a/tests/v2/test_local_orcid.py b/tests/v2/test_local_orcid.py new file mode 100644 index 000000000..fb4a4684f --- /dev/null +++ b/tests/v2/test_local_orcid.py @@ -0,0 +1,561 @@ +"""Offline checks on tools/differential/local_orcid.py, the +orcid-researchers local corpus. + +Every name here is invented, and no reading of a real ORCID record may +reach the repository: the file is CC0, but its names are living +people's, which is why the corpus is local. What these pin is what a +summary gives (the boxes, the country's order of preference, the +refusals), the stream's resuming and its refusal of a file that is not +the pinned one, what the fetch keeps and counts, and the per-country +cap. +""" +import email.message +import gzip +import hashlib +import importlib +import io +import json +import tarfile +import urllib.request +from pathlib import Path + +import pytest + +from ._differential_fixtures import load_tool, own_cache + +local = load_tool("local_corpora") +# The module the fetch reads its constants from, by name: a patch set on +# a second load by path would reach nothing (test_local_corpora.py). +orcid = importlib.import_module("local_orcid") + +_ORCID = "orcid-researchers" +_MODIFIED = "Fri, 17 Oct 2025 13:21:49 GMT" +_NS = " ".join(f'xmlns:{p}="http://www.orcid.org/ns/{p}"' for p in ( + "record", "person", "personal-details", "other-name", "address", "common", + "activities", "employment", "education", "preferences")) + + +@pytest.fixture(autouse=True) +def _own_cache(monkeypatch: pytest.MonkeyPatch, tmp_path_factory: pytest.TempPathFactory) -> Path: + return own_cache(monkeypatch, tmp_path_factory) + + +def _org(kind: str, country: str) -> str: + return (f"" + f"<{kind}:{kind}-summary>Uni" + f"X{country}" + f"" + f"") + + +def _xml(given: str | None = "Ann", family: str = "", credit: str = "", + other: tuple[str, ...] = (), address: str = "", employment: str = "", + education: str = "", locale: str = "en") -> bytes: + """A summary as the file writes one; `given` None leaves out the + whole name, as a record whose name is private does.""" + name = "" if given is None else ( + "" + + (f"{given}" if given else "") + + (f"{family}" if family else "") + + (f"{credit}" if credit else "") + + "") + others = "".join(f"{o}" + f"" for o in other) + home = (f"{address}" + f"" if address else "") + return (f'' + f"{locale}" + f"{name}" + f"{others}{home}" + f"" + + (_org("education", education) if education else "") + + (_org("employment", employment) if employment else "") + + "").encode() + + +def test_a_summary_gives_its_boxes_other_names_and_the_holders_own_country() -> None: + body = _xml("Ann Marie", "Lee", "A. M. Lee", ("Анна Ли",), address="nz", + employment="FR", education="DE", locale="ru") + assert orcid._summary(body) == ("record", { + "given": "Ann Marie", "family": "Lee", "credit": "A. M. Lee", "other": ["Анна Ли"], + "country": "NZ", "country_from": "address", "locale": "ru"}) + + +@pytest.mark.parametrize(("where", "country_from"), [ + ({"employment": "FR", "education": "DE"}, "employment"), + ({"education": "DE"}, "education")]) +def test_without_an_address_the_country_is_the_first_organizations( + where: dict[str, str], country_from: str) -> None: + # An organization's own address is {.../common}address: it never + # reads as the holder's. + _, record = orcid._summary(_xml(employment=where.get("employment", ""), + education=where.get("education", ""))) + assert record is not None + assert (record["country"], record["country_from"]) == ( + where[country_from], country_from) + + +def test_a_summary_with_no_country_keeps_its_name_without_one() -> None: + assert orcid._summary(_xml("Bo", "Wu", locale="")) == ("record", {"given": "Bo", "family": "Wu"}) + + +@pytest.mark.parametrize(("body", "outcome"), [ + (_xml(None), "name_private"), + (_xml(""), "name_empty"), + (b"]>&a;', "dtd_refused")]) +def test_a_summary_without_a_public_name_is_counted_and_gives_no_record( + body: bytes, outcome: str) -> None: + assert orcid._summary(body) == (outcome, None) + + +# -- the stream -------------------------------------------------------------- + +class _Response: + """A ranged response serving `body` from `start`, cut after `cut` + bytes with a reset connection where `cut` is set.""" + + def __init__(self, body: bytes, start: int, *, cut: int | None = None, + size: int | None = None, modified: str = _MODIFIED) -> None: + self.status = 206 + self.headers = email.message.Message() + total = len(body) if size is None else size + self.headers["Content-Range"] = f"bytes {start}-{total - 1}/{total}" + self.headers["Last-Modified"] = modified + self._body = io.BytesIO(body[start:]) + self._cut = cut + + def end_at(self, n: int) -> None: + """End the body cleanly after `n` bytes, as a server closing early does.""" + self._body = io.BytesIO(self._body.read(n)) + + def readinto(self, buffer: memoryview) -> int: + if self._cut is not None and self._body.tell() >= self._cut: + raise ConnectionResetError("reset by peer") + limit = len(buffer) if self._cut is None else min(len(buffer), self._cut - self._body.tell()) + return self._body.readinto(memoryview(buffer)[:limit]) + + def close(self) -> None: + pass + + +def _summaries_in(body: bytes) -> int: + """The summaries a served tar.gz holds, or 0 for a body that is not one.""" + try: + with tarfile.open(fileobj=io.BytesIO(body), mode="r:gz") as tar: + return sum(m.isfile() and bool(orcid._SUMMARY.fullmatch(m.name)) for m in tar) + except (tarfile.TarError, OSError, EOFError): + return 0 + + +def _serve(monkeypatch: pytest.MonkeyPatch, body: bytes, *, + cuts: tuple[int, ...] = (), size: int | None = None, + modified: str = _MODIFIED, status: int = 206, + eofs: tuple[int, ...] = ()) -> list[str]: + """`body` as the pinned file, its size, MD5 and count of summaries + pinned, each connection cut after the next of `cuts` bytes with a + reset, or ending cleanly after the next of `eofs`; the Ranges asked.""" + monkeypatch.setattr(orcid, "ORCID_SIZE", len(body)) + monkeypatch.setattr(orcid, "ORCID_MD5", hashlib.md5(body).hexdigest()) + monkeypatch.setattr(orcid, "ORCID_SUMMARIES", _summaries_in(body)) + monkeypatch.setattr(importlib.import_module("local_core").time, "sleep", lambda _s: None) + asked: list[str] = [] + pending, ends = list(cuts), list(eofs) + + def urlopen(request: urllib.request.Request, timeout: int) -> _Response: + assert request.full_url == orcid.ORCID_URL + ranged = request.get_header("Range") + assert ranged is not None + asked.append(ranged) + start = int(ranged.removeprefix("bytes=").removesuffix("-")) + response = _Response(body, start, cut=pending.pop(0) if pending else None, size=size, + modified=modified) + response.status = status + if ends: + response.end_at(ends.pop(0)) + return response + + monkeypatch.setattr(orcid.urllib.request, "urlopen", urlopen) + return asked + + +def test_a_stream_cut_mid_body_resumes_from_the_byte_it_reached( + monkeypatch: pytest.MonkeyPatch) -> None: + body = bytes(range(256)) * 40 + asked = _serve(monkeypatch, body, cuts=(1000, 2500)) + stream = orcid._RangeStream(orcid.ORCID_URL, orcid.ORCID_SIZE, orcid.ORCID_MD5) + assert io.BufferedReader(stream, 700).read() == body + stream.finish() + assert asked == ["bytes=0-", "bytes=1000-", "bytes=3500-"] + + +def test_a_stream_refuses_a_file_replaced_mid_read(monkeypatch: pytest.MonkeyPatch) -> None: + body = bytes(range(256)) * 8 + _serve(monkeypatch, body, cuts=(100,)) + stream = orcid._RangeStream(orcid.ORCID_URL, orcid.ORCID_SIZE, orcid.ORCID_MD5) + assert stream.read(100) + monkeypatch.setattr(orcid.urllib.request, "urlopen", lambda request, timeout: _Response( + body, 100, modified="Sat, 01 Nov 2025 00:00:00 GMT")) + with pytest.raises(ValueError, match="not the pinned one"): + stream.read(100) + + +def test_a_stream_refuses_a_file_of_another_size(monkeypatch: pytest.MonkeyPatch) -> None: + body = b"x" * 100 + _serve(monkeypatch, body, size=101) + with pytest.raises(ValueError, match="not the pinned one"): + orcid._RangeStream(orcid.ORCID_URL, orcid.ORCID_SIZE, orcid.ORCID_MD5).read(10) + + +def test_a_stream_whose_bytes_are_not_the_pinned_md5_is_refused_at_the_end( + monkeypatch: pytest.MonkeyPatch) -> None: + body = b"x" * 100 + _serve(monkeypatch, body) + stream = orcid._RangeStream(orcid.ORCID_URL, orcid.ORCID_SIZE, "0" * 32) + assert stream.read() == body + with pytest.raises(ValueError, match="is pinned; fetch it again"): + stream.finish() + + +# -- the fetch --------------------------------------------------------------- + +def _tar(*members: tuple[str, bytes]) -> bytes: + raw = io.BytesIO() + with tarfile.open(fileobj=raw, mode="w") as tar: + for name, body in members: + info = tarfile.TarInfo(name) + info.size = len(body) + tar.addfile(info, io.BytesIO(body)) + return gzip.compress(raw.getvalue()) + + +def _member(digits: str, body: bytes) -> tuple[str, bytes]: + return f"ORCID_2025_10_summaries/{digits[-3:]}/0000-0002-1825-{digits}.xml", body + + +_FILE = [_member("0097", _xml("Ann", "Lee", "Lee, Ann", address="NZ")), + _member("1097", _xml("Ann", "Lee", "Lee, Ann", address="NZ")), + _member("2097", _xml("Bo", "Wu", employment="CN")), + _member("3097", _xml("Cy")), + _member("409X", _xml(None))] + + +def test_an_orcid_fetch_keeps_each_record_once_and_counts_the_rest( + monkeypatch: pytest.MonkeyPatch) -> None: + _serve(monkeypatch, _tar(*_FILE)) + corpus = local.fetch(_ORCID) + assert corpus.records == [ + {"credit": "Lee, Ann", "country": "NZ", "country_from": "address", + "family": "Lee", "given": "Ann", "name": "Ann Lee"}, + {"country": "CN", "country_from": "employment", "family": "Wu", "given": "Bo", + "name": "Bo Wu"}, + {"given": "Cy", "name": "Cy"}] + assert corpus.meta["data_date"] == "2025-10-17T13:21:49" + assert corpus.meta["detail"] == { + "kept": orcid._kept(), "summaries": 5, "record": 4, "name_private": 1, + "countries": 2, "countries_at_the_cap": 0} + # The corpus reads the credit name beside the joined boxes. + corpus_lines = (local.directory(_ORCID) / local.CORPUS).read_text(encoding="utf-8") + assert '"Lee, Ann"' in corpus_lines and '"Ann Lee"' in corpus_lines + + +def test_the_extract_keeps_every_record_with_its_other_names_and_locale( + monkeypatch: pytest.MonkeyPatch) -> None: + _serve(monkeypatch, _tar(_member("0097", _xml("Ann", "Lee", "A. Lee", other=("Анна",), + address="NZ")))) + local.fetch(_ORCID) + with gzip.open(orcid.extract_path(), "rt", encoding="utf-8") as lines: + header, *records = lines.read().splitlines() + assert '"file": "ORCID_2025_10_summaries.tar.gz"' in header + assert records == [ + '{"country": "NZ", "country_from": "address", "credit": "A. Lee", "family": "Lee",' + ' "given": "Ann", "locale": "en", "other": ["Анна"]}'] + + +def test_workers_extract_what_one_process_does(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr(orcid, "_BATCH", 2) + body = _tar(*_FILE) + extracts = [] + for jobs in (1, 2): + _serve(monkeypatch, body) + orcid.renew_orcid(jobs=jobs) + extracts.append(gzip.decompress(orcid.extract_path().read_bytes())) + assert extracts[0] == extracts[1] + + +def test_the_extract_holds_no_member_it_has_read(monkeypatch: pytest.MonkeyPatch) -> None: + held: list[int] = [] + next_member = tarfile.TarFile.next + + def counting(self: tarfile.TarFile) -> tarfile.TarInfo | None: + held.append(len(self.members)) # type: ignore[attr-defined] # undeclared in typeshed + return next_member(self) + + _serve(monkeypatch, _tar(*_FILE)) # counts the tar's summaries, before the patch + monkeypatch.setattr(tarfile.TarFile, "next", counting) + orcid.renew_orcid(jobs=1) + # Without clearing, the list grows by a member a read: [0, 1, 2, ...]. + assert len(held) > 3 and max(held) <= 1 + + +def test_a_fetch_that_fails_its_md5_keeps_the_extract_it_had( + monkeypatch: pytest.MonkeyPatch) -> None: + _serve(monkeypatch, _tar(*_FILE)) + orcid.renew_orcid(jobs=1) + before = orcid.extract_path().read_bytes() + _serve(monkeypatch, _tar(*_FILE[:2])) + monkeypatch.setattr(orcid, "ORCID_MD5", "0" * 32) + with pytest.raises(ValueError, match="is pinned"): + orcid.renew_orcid(jobs=1) + assert orcid.extract_path().read_bytes() == before + assert sorted(p.name for p in orcid.extract_path().parent.iterdir()) == [ + orcid.extract_path().name] + + +def test_a_file_past_the_share_of_unreadable_summaries_is_refused( + monkeypatch: pytest.MonkeyPatch) -> None: + _serve(monkeypatch, _tar(*_FILE, _member("5097", b" None: + stray = ("ORCID_2025_10_summaries/README.txt", b"not a summary") + _serve(monkeypatch, _tar(*_FILE, stray)) + with pytest.raises(ValueError, match="1 of 6 files in the tar are not summaries"): + local.fetch(_ORCID) + monkeypatch.setattr(orcid, "ORCID_MAX_BAD", 1 / 5) + assert local.fetch(_ORCID).meta["detail"]["members_not_summaries"] == 1 + + +def test_a_corpus_kept_under_another_cap_is_refused_on_load( + monkeypatch: pytest.MonkeyPatch) -> None: + _serve(monkeypatch, _tar(*_FILE)) + local.fetch(_ORCID) + assert local.load(_ORCID) is not None + monkeypatch.setattr(orcid, "ORCID_CAP", 7) + with pytest.raises(local.CorpusDamaged, match="at most 7 records a country"): + local.load(_ORCID) + + +# -- the cap ----------------------------------------------------------------- + +def _people(country: str, n: int) -> list[dict[str, str]]: + return [{"name": f"Ann Lee{i}", "country": country} for i in range(n)] + + +def test_the_cap_samples_a_large_country_and_keeps_a_small_one_whole() -> None: + records = _people("NZ", 30) + _people("FR", 3) + [{"name": "Cy"}] * 2 + kept = orcid.orcid_kept(iter(records), 10) + by_country = [r.get("country", "") for r in kept] + assert (by_country.count("NZ"), by_country.count("FR"), by_country.count("")) == (10, 3, 1) + + +def test_the_cap_keeps_the_same_records_whatever_the_order_or_the_rest() -> None: + nz = _people("NZ", 40) + kept = orcid.orcid_kept(iter(nz), 10) + shuffled = orcid.orcid_kept(iter(nz[::-1] + _people("FR", 50) + nz[:5]), 10) + assert sorted(map(str, kept)) == sorted(map(str, (r for r in shuffled if r["country"] == "NZ"))) + + +# -- what a summary gives, further ------------------------------------------- + +def test_the_first_organization_naming_a_country_gives_it() -> None: + body = _xml(employment="FR").replace( + b"", + b"" + b"" + b"DE" + b"" + b"") + assert b"DE" in body + _, record = orcid._summary(body) + assert record is not None and record["country"] == "FR" + + +@pytest.mark.parametrize(("body", "outcome", "record"), [ + (_xml("", credit="A. Lee"), "name_empty", None), + (_xml("", "Lee"), "record", {"family": "Lee", "locale": "en"})]) +def test_a_name_is_its_two_boxes_and_a_credit_name_alone_is_none( + body: bytes, outcome: str, record: dict[str, str] | None) -> None: + assert orcid._summary(body) == (outcome, record) + + +# -- the stream, further ----------------------------------------------------- + +def test_a_stream_refuses_an_answer_that_is_not_a_range(monkeypatch: pytest.MonkeyPatch) -> None: + # A server ignoring Range answers 200 with the body from byte 0, which + # on a resume would splice the file's start into the stream. + _serve(monkeypatch, b"x" * 100, status=200) + with pytest.raises(ValueError, match="answered 200"): + orcid._RangeStream(orcid.ORCID_URL, orcid.ORCID_SIZE, orcid.ORCID_MD5).read(10) + + +def test_a_stream_refuses_a_file_that_states_no_last_modified( + monkeypatch: pytest.MonkeyPatch) -> None: + # Without it the replaced-file check would have nothing to compare. + _serve(monkeypatch, b"x" * 100, modified="") + with pytest.raises(ValueError, match="last modified not stated"): + orcid._RangeStream(orcid.ORCID_URL, orcid.ORCID_SIZE, orcid.ORCID_MD5).read(10) + + +def test_a_body_that_ends_early_without_an_error_resumes(monkeypatch: pytest.MonkeyPatch) -> None: + body = bytes(range(256)) * 8 + asked = _serve(monkeypatch, body, eofs=(1000,)) + stream = orcid._RangeStream(orcid.ORCID_URL, orcid.ORCID_SIZE, orcid.ORCID_MD5) + assert io.BufferedReader(stream, 700).read() == body + stream.finish() + assert asked == ["bytes=0-", "bytes=1000-"] + + +def test_a_stream_rides_out_more_failures_than_a_small_request_would( + monkeypatch: pytest.MonkeyPatch) -> None: + body = bytes(range(256)) * 4 + failures = orcid.ORCID_RETRIES - 1 + assert failures > importlib.import_module("local_core").RETRIES + asked = _serve(monkeypatch, body, cuts=(0,) * failures) + sleeps: list[float] = [] + monkeypatch.setattr(importlib.import_module("local_core").time, "sleep", sleeps.append) + stream = orcid._RangeStream(orcid.ORCID_URL, orcid.ORCID_SIZE, orcid.ORCID_MD5) + assert stream.read() == body + assert len(asked) == failures + 1 + assert sleeps == [min(2 ** n, 60) for n in range(failures)] + + +def test_a_gzip_whose_crc_fails_after_the_tar_ends_is_refused( + monkeypatch: pytest.MonkeyPatch) -> None: + body = bytearray(_tar(*_FILE)) + body[-8] ^= 0xFF # the trailer's CRC32, read only past the tar's end + _serve(monkeypatch, bytes(body)) + with pytest.raises(gzip.BadGzipFile, match="CRC check failed"): + orcid.renew_orcid(jobs=1) + assert not orcid.extract_path().parent.exists() or not any( + orcid.extract_path().parent.iterdir()) + + +# -- the reader's checks ----------------------------------------------------- + +def _write_extract(header: dict[str, object] | bytes, lines: tuple[str, ...] = ()) -> Path: + """An extract written by hand: `header` and `lines`, or bytes as they stand.""" + path = orcid.extract_path() + path.parent.mkdir(parents=True, exist_ok=True) + if isinstance(header, bytes): + path.write_bytes(header) + else: + text = json.dumps(header) + "\n" + "".join(f"{line}\n" for line in lines) + path.write_bytes(gzip.compress(text.encode())) + return path + + +_HEAD = {"file": orcid.ORCID_FILE, "data_date": "2025-10-17T13:21:49", "counts": {"record": 0}} + + +@pytest.mark.parametrize("header", [ + {**_HEAD, "file": "ORCID_2024_10_summaries.tar.gz"}, + {**_HEAD, "counts": [0]}, + {**_HEAD, "counts": {"record": "0"}}, + {k: v for k, v in _HEAD.items() if k != "data_date"}, + {**_HEAD, "data_date": 2025}, + gzip.compress(b"", mtime=0), # a gzip header holds its time: one id for every worker + b"not gzip"], ids=["another file", "counts not a dict", "a count not an int", + "no data date", "a data date not text", "empty", "not gzip"]) +def test_a_file_that_is_not_an_extract_is_refused_naming_the_refresh( + header: dict[str, object] | bytes) -> None: + with pytest.raises(ValueError, match="not an extract this tool wrote.*--refresh"): + orcid._orcid_read(_write_extract(header)) + + +def test_a_tar_read_only_in_part_is_refused_and_the_extract_kept( + monkeypatch: pytest.MonkeyPatch) -> None: + # tarfile ends a stream at a damaged header without raising, and the + # bytes still pass their MD5: only the pinned count shows it. + _serve(monkeypatch, _tar(*_FILE)) + orcid.renew_orcid(jobs=1) + before = orcid.extract_path().read_bytes() + monkeypatch.setattr(orcid, "ORCID_SUMMARIES", len(_FILE) + 1) + with pytest.raises(ValueError, match="the tar was read in part"): + orcid.renew_orcid(jobs=1) + assert orcid.extract_path().read_bytes() == before + assert sorted(p.name for p in orcid.extract_path().parent.iterdir()) == [ + orcid.extract_path().name] + + +def test_an_extract_missing_records_its_header_counts_is_refused( + monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr(orcid, "ORCID_SUMMARIES", 2) + path = _write_extract({**_HEAD, "counts": {"record": 2}}, + ('{"country": "NZ", "family": "Lee", "given": "Ann"}',)) + with pytest.raises(ValueError, match="holds 1 records .* counts 2 records"): + orcid._orcid_read(path) + + +@pytest.mark.parametrize(("extra", "says"), [ + (_member("5097", b"'), + "1 of 6 summaries declare a DTD or an entity"), + (("ORCID_2025_10_summaries/README.txt", b"x"), "1 of 6 files in the tar are not summaries"), + (_member("5097", _xml("Di", "Ng", address="nzl")), + "1 of 4 countries are not two capital letters")]) +def test_a_file_past_a_share_of_what_it_reads_past_is_refused_before_it_is_kept( + monkeypatch: pytest.MonkeyPatch, extra: tuple[str, bytes], says: str) -> None: + _serve(monkeypatch, _tar(*_FILE)) + orcid.renew_orcid(jobs=1) + before = orcid.extract_path().read_bytes() + _serve(monkeypatch, _tar(*_FILE, extra)) + with pytest.raises(ValueError, match=says): + orcid.renew_orcid(jobs=1) + assert orcid.extract_path().read_bytes() == before + + +@pytest.mark.parametrize("bad", [1, 2]) +def test_the_share_of_unreadable_summaries_is_held_at_its_shipped_value( + monkeypatch: pytest.MonkeyPatch, bad: int) -> None: + assert orcid.ORCID_MAX_BAD == 1 / 10_000 + monkeypatch.setattr(orcid, "ORCID_SUMMARIES", 10_000) + good = 10_000 - bad + lines = tuple(f'{{"country": "NZ", "family": "Lee{i}", "given": "Ann"}}' for i in range(good)) + path = _write_extract({**_HEAD, "counts": {"record": good, "not_xml": bad}}, lines) + if bad == 1: + assert orcid._orcid_read(path).detail["not_xml"] == 1 + else: + with pytest.raises(ValueError, match="2 of 10,000 summaries are not XML"): + orcid._orcid_read(path) + + +@pytest.mark.parametrize(("members", "says"), [ + # The name moved: every summary reads as private. + ([_member(f"{n}097", _xml(None)) for n in range(4)] + _FILE[:1], + "only 1 of 5 summaries give a record"), + # The family box was renamed: no record holds one. + ([_member(f"{n}097", _xml(f"Ann{n}", address="NZ")) for n in range(4)], + "only 0 of 4 records hold a family name"), + # The address and organization namespaces moved: no record has a country. + ([_member(f"{n}097", _xml(f"Ann{n}", "Lee")) for n in range(4)], + "only 0 of 4 records hold a country")]) +def test_a_file_whose_layout_moved_under_the_reader_is_refused( + monkeypatch: pytest.MonkeyPatch, members: list[tuple[str, bytes]], says: str) -> None: + _serve(monkeypatch, _tar(*members)) + with pytest.raises(ValueError, match=says): + local.fetch(_ORCID) + + +def test_the_countries_at_the_cap_leave_out_the_records_with_none( + monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr(orcid, "ORCID_CAP", 1) + _serve(monkeypatch, _tar(*_FILE)) + detail = local.fetch(_ORCID).meta["detail"] + assert (detail["countries"], detail["countries_at_the_cap"]) == (2, 2) + + +def test_an_extract_of_another_version_is_streamed_again(monkeypatch: pytest.MonkeyPatch) -> None: + _serve(monkeypatch, _tar(*_FILE)) + orcid.fetch_orcid() + asked = _serve(monkeypatch, _tar(*_FILE)) + orcid.fetch_orcid() + assert asked == [] # the extract cached for this version is read + monkeypatch.setattr(orcid, "ORCID_EXTRACT_VERSION", orcid.ORCID_EXTRACT_VERSION + 1) + orcid.fetch_orcid() + assert asked == ["bytes=0-"] diff --git a/tests/v2/test_measure.py b/tests/v2/test_measure.py index 074a6d98e..5662e0ec9 100644 --- a/tests/v2/test_measure.py +++ b/tests/v2/test_measure.py @@ -33,12 +33,14 @@ def _fetch(cache: Path, *sources: str) -> None: def test_the_plan_runs_each_check_for_the_corpora_fetched(_own_cache: Path, tmp_path: Path) -> None: - _fetch(_own_cache, "ctgov-officials", corpora.PSC, corpora.JRC, "tse-donors") + _fetch(_own_cache, "ctgov-officials", corpora.PSC, corpora.JRC, corpora.ORCID, + "tse-donors") steps = measure.plan(tmp_path, 3) assert [s.label for s in steps] == [ "committed snapshot", "CLDR and GND agreement", "local snapshot ctgov-officials", f"local snapshot {corpora.JRC}", - f"local snapshot {corpora.PSC}", "local snapshot tse-donors", + f"local snapshot {corpora.ORCID}", f"local snapshot {corpora.PSC}", + "local snapshot tse-donors", f"agreement {corpora.ORCID}", f"agreement {corpora.PSC}", f"alike {corpora.JRC}", "alike tse-donors"] for step in steps: # Every step through the wrapper: unwrapped, a refusal reads as moves. @@ -50,7 +52,7 @@ def test_the_plan_runs_each_check_for_the_corpora_fetched(_own_cache: Path, assert step.against == measure.AGAINST_RECORD assert steps[0].command[-2:] == ["--jobs", "3"] assert measure.unmeasured() == sorted(set(corpora.SOURCES) - { - "ctgov-officials", corpora.PSC, corpora.JRC, "tse-donors"}) + "ctgov-officials", corpora.PSC, corpora.JRC, corpora.ORCID, "tse-donors"}) def test_the_plan_with_nothing_fetched_runs_the_committed_checks(tmp_path: Path) -> None: diff --git a/tools/differential/README.md b/tools/differential/README.md index ec4bd5adb..da235cb09 100644 --- a/tools/differential/README.md +++ b/tools/differential/README.md @@ -248,7 +248,7 @@ uv run python tools/differential/build_shapes_corpus.py --coverage A local corpus holds real names that are not committed: fetched into the cache by `local_corpora.py`, read by `snapshot.py --local` and by a row of `tools/name_usage.py`, scored by `local_agreement.py` where its records carry their fields, and seen by no reviewer and no CI run (the SURVEY job in mechanisms.md#NAME-DATA-JOBS, and WATCH and SCORE outside CI). What reaches the repository is what a contributor cites from it, counts and shapes written with made-up names. -`local_corpora.py` holds the `SOURCES` registry, the cache's files, the views and the command line; each source's fetch is a module of its own (`local_ctgov.py`, `local_fec.py`, `local_tse.py`, `local_tw.py`, `local_ukec.py`, `local_psc.py`, `local_jrc.py`, `local_nepal.py`) whose docstring says what the source is and what its terms allow, built on the retry policy, share gate and downloads in `local_core.py`. A new source is a module and a `SOURCES` row. A source whose records hold a value its own code derives, rather than one the source gave (`jrc-names`' script label), stores a read version in `meta.json`'s detail and refuses on load a corpus read by another (`JRC_READ_VERSION`, bumped by hand with any change to how a record is built; `psc-individuals` does the same for its cached extracts, with `PSC_EXTRACT_VERSION` in each extract's file name): the sha256 checks prove the files are the ones fetched, not that today's code would write them. A constant such as `TSE_MAX_BAD` or `RETRIES` lives in one module, which every reader reads it through, so a test patches it there; no other module `from`-imports it, so a patch set elsewhere raises rather than missing the code (`test_no_module_imports_a_sibling_constant` holds this). +`local_corpora.py` holds the `SOURCES` registry, the cache's files, the views and the command line; each source's fetch is a module of its own (`local_ctgov.py`, `local_fec.py`, `local_tse.py`, `local_tw.py`, `local_ukec.py`, `local_psc.py`, `local_jrc.py`, `local_nepal.py`, `local_orcid.py`) whose docstring says what the source is and what its terms allow, built on the retry policy, share gate and downloads in `local_core.py`. A new source is a module and a `SOURCES` row. A source whose records hold a value its own code derives, rather than one the source gave (`jrc-names`' script label), stores a read version in `meta.json`'s detail and refuses on load a corpus read by another (`JRC_READ_VERSION`, bumped by hand with any change to how a record is built; `psc-individuals` does the same for its cached extracts, with `PSC_EXTRACT_VERSION` in each extract's file name): the sha256 checks prove the files are the ones fetched, not that today's code would write them. A constant such as `TSE_MAX_BAD` or `RETRIES` lives in one module, which every reader reads it through, so a test patches it there; no other module `from`-imports it, so a patch set elsewhere raises rather than missing the code (`test_no_module_imports_a_sibling_constant` holds this). ``` uv run python tools/differential/local_corpora.py fetch ctgov-officials @@ -282,7 +282,8 @@ Any snapshot reading of 5,000 entries or more, committed or local, parses in a w | `tse-donors` | every individual donor to a candidate in Brazil's 2022 general and 2024 municipal elections (the TSE's candidates' accounts, rows from `Recursos de pessoas físicas`), a record per distinct pair of the name as the campaign filed it (`name`) and the federal tax registry's name for the same donor (`registry`): a typed form with its reference form beside it. Of 854,531 records fetched 2026-10-22 (848,351 distinct filed names), 205,925 differ from their registry name -- an agnomen dropped or abbreviated (`JR` filed for the registry's `JUNIOR` in 1,615, `JÚNIOR` for `JUNIOR` in 975), a word left out. Nearly all in capitals (845,498 filed names), in spoken order with no comma (184 records hold one), and full of particles and agnomens (`DE`, `DA`, `DOS`; `FILHO` ends 10,955 distinct filed names, `JUNIOR` 13,784, `NETO` 8,410). Published by the TSE under CC BY | case, mostly; the family-comma shape; credentials and titles, which a donation form has no place for; donors to parties rather than candidates; names outside Brazil; names in any script but Latin | | `tw-donors` | the distinct names of every individual donor in every campaign finance report Taiwan's Control Yuan publishes on its political donation platform (ardata.cy.gov.tw, whose zips data.gov.tw also lists): 454 reports of 64 elections on 2026-10-22, presidential, legislative and local: every report from 2018's elections on, and the residual reports of 2014's and 2016's. Each report's zip is cached under the name the platform lists it by, which carries the date it was built, so a refresh downloads only the reports published or rebuilt since. Names as a candidate's staff typed them into a filing, published in full where the ID, address and phone beside them are masked. Of the 308,667 names dated 2026-10-05, 296,820 were three Han characters, 5,449 four and 5,260 two; the population for unspaced Han names written family first, and for the married double surname (冠夫姓) a four-character name often is. Under the Open Government Data License, version 1.0, with the platform's own bar on using the data for profit, credit investigation or another improper purpose | names in spoken order; Latin script (80 of the names dated 2026-10-05 held a Latin letter); the comma (3 of them held a fullwidth one, none an ASCII one); case; names outside Taiwan; donors whose names the report does not publish, such as anonymous gifts; Japanese kanji names, which the Han script cannot tell from Chinese | | `ukec-individuals` | the distinct names of the individual donors in the UK Electoral Commission's register of donations (to parties, their members, third parties, permitted participants and regulated donees), from the search's CSV export under the site's own default filter, fetched whole on every fetch (9.5 MB on 2026-10-22, accepted dates from 2001 to 2026). The default leaves out donations reported only before a poll (`prePoll=false`: 374 rows and 21 names on 2026-10-22). A name as the donee's return gave it, mostly in its spoken order and mixed case, with the donor's title: of its 15,214 names on 2026-10-22, `parse()` read a title in 8,219 (its `title` field non-empty) (`Mr`, `Ms`, `Mrs`, `Lord`, `Dr`, `Sir`; 32 a run of two or more, `The Rt Hon`) and a suffix in 320 (`MP`, `OBE`, `CBE`, `QC`). The population for titled British names. The Commission publishes its documents under the Open Government Licence v3.0, but the register states no reuse terms of its own, which the credit line says | the family-comma shape (2 names hold a comma) and case (9 are in capitals); small donors, since a donation is reported only above a threshold that depends on the donee; names outside the UK; names in any script but Latin | -| `psc-individuals` | the people with significant control of UK companies, and the beneficial owners of overseas entities, from Companies House's daily snapshot (33 zipped parts of JSON lines, 2.44 GB on 2026-10-22): each person's display name, the four boxes the filer typed it into (title, forename, middle name, surname) and the stated nationality, nothing else. A record is a distinct filed name, kept one in eight by a hash of the record (1,040,708 of 8,322,939 on 2026-10-22), a size the machinery can hold in memory. The one local corpus whose names carry their fields, scored by `local_agreement.py` | names nobody filed into boxes, so almost no comma (838 of 1,040,708 on 2026-10-22), no credential run and almost no suffix; a display name that is not its boxes joined, which the fetch drops (2 of 14,025,738 individuals on 2026-10-22) | +| `psc-individuals` | the people with significant control of UK companies, and the beneficial owners of overseas entities, from Companies House's daily snapshot (33 zipped parts of JSON lines, 2.44 GB on 2026-10-22): each person's display name, the four boxes the filer typed it into (title, forename, middle name, surname) and the stated nationality, nothing else. A record is a distinct filed name, kept one in eight by a hash of the record (1,040,708 of 8,322,939 on 2026-10-22), a size the machinery can hold in memory. A local corpus whose names carry their fields, scored by `local_agreement.py`, as `orcid-researchers` is | names nobody filed into boxes, so almost no comma (838 of 1,040,708 on 2026-10-22), no credential run and almost no suffix; a display name that is not its boxes joined, which the fetch drops (2 of 14,025,738 individuals on 2026-10-22) | +| `orcid-researchers` | the public names of ORCID iD holders, researchers worldwide, from ORCID's Public Data File 2025 (`ORCID_2025_10_summaries.tar.gz`, 46.3 GB, data as of 2025-10-17), read as a stream and never stored: each line of its extract is the two boxes the holder typed their own name into (given names, family name), the credit name they publish under where public, other names, and a country, taken from their public address, else the first employment that names one, else the first education that does; a corpus record keeps the boxes, the credit name and the country. The fetch caches an extract of every summary with a public name (24,990,909 of 26,078,951 on 2026-10-27, 1,086,356 private and 1,686 empty), 369 MB, and the corpus keeps at most `ORCID_CAP` (20,000) records a country, the lowest by a hash of the record, so a change to the cap re-reads the extract rather than the 46 GB file: 1,831,533 records: 250 countries, 65 of them at the cap, and the records with no country (14,293,215 of the extract, 57%) as one more group of 20,000, counted in that total. A record's name is its boxes joined, given first; the credit name is a second corpus key (125,960 kept records hold one). Scored by `local_agreement.py` by country, the boxes as the holder split them. CC0, with ORCID asking for credit and respect for the privacy of the people in it | frequency, since the cap flattens every large country to 20,000; people outside research; name order, which ORCID does not record, so a holder who typed a family-first name into the given box reads as a disagreement; names with no family name and names whose family name is private, which look alike (1,308,200 of the extract hold no family name); titles, credentials and suffixes, which the form has no box for | | `jrc-names` | the person entities of the EU Joint Research Centre's JRC-Names, each with the spellings of its name harvested from news text in many languages and scripts: a record is one distinct spelling, with its entity and its script. The file (`entities.gzip`, 14.8 MB) is pinned by sha256 and dated by its Last-Modified, 2024-03-15. Of its 1,317,586 spellings kept, 1,203,634 are Latin, 66,673 Cyrillic, 17,074 Arabic (7,833 persons, 5,347 of them spelled in Latin too), 13,039 Katakana and 8,016 Han, and 218,822 persons have two or more spellings, 25,415 of them in two scripts or more (a spelling whose letters are in two scripts, or none, not counted as one). The population for spelling and transliteration variants of one name, read by the `groups` view, and for Arabic-script names. Licensed under the JRC-Names EULA, personal and non-transferable, with no redistribution or derivative works, so its views' output and its snapshot stay in the cache like every local corpus's, and here the licence, not only the names being real people's, is why | titles and credentials, since the names are harvested from running text (`parse()` read a title in 3,011 of its 1,315,051 names and a suffix in 1,940, 2026-10-24); the ASCII comma, which none holds (1,510 Arabic spellings hold an Arabic comma, U+060C, which the parser reads as one); the Indic scripts (Devanagari 827 spellings, Bengali 419, the others fewer); people the news did not name | | `nepal-candidates` | every candidate in Nepal's local elections of 2079 BS (2022), from the Election Commission's results site (result.election.gov.np), one file per local body: a record per distinct pair of the Commission's romanization (`name`, the one `parse()` reads) and the name in Devanagari as filed (`devanagari`). The corpus and the word index hold both, so a snapshot and a `name_usage.py` lookup read both scripts; `tally`, which counts what follows a comma, reads the romanized name. Of 126,981 pairs fetched 2026-10-24 (152,960 candidates in 753 files, 26 of whose romanizations were a placeholder such as `- - -` and are dropped), the romanized names carry surnames written as initials (`B.K.`, `K.C.`, `G.C.`), honorific words that are titles in front (`Guru`; `Shekh` until #752's phase 2j made it a title only in a title phrase) and `Md.`, Muhammad's abbreviation, which its period makes a title by shape; the Devanagari ones carry vowel signs and the virama, combining marks the letter-shape tests do not count as letters (decisions/nfd-shapes), and the joiners ZWJ and ZWNJ. The population for romanized South Asian names and for Devanagari. The site states no licence, which the credit line says | case and the comma, since the Commission's romanization is one consistent hand; names a person typed themselves; credentials and generational words; the parliament and provincial candidates, whose files carry Devanagari only; names outside Nepal | @@ -290,9 +291,11 @@ A live source has no fixed snapshot to pin, so the cached copy is the pin: `meta `psc-individuals` caches an extract of each of the snapshot's 33 parts rather than the zips: 168 MB against 2.44 GB on 2026-10-22, each extract holding only the records and fields the corpus keeps. A download that fails costs one part, and running the fetch again resumes, keeping the parts already read for the same day; the set replaces the cached one only once every part is read, and a part that disappears mid-fetch (the files are overwritten each morning) is refused naming the refresh. The fetch refuses a snapshot where more than one individual in 10,000 has no `name_elements`, or a display name that is not its four boxes joined by spaces, or where fewer than half of the lines are individuals (87% were on 2026-10-22; a kind Companies House renamed would read as another kind), and counts what it skips into `meta.json`'s detail. It reads the cached extracts only as one whole snapshot of one date, and an extract written by another version of the extraction (its file name carries the version) is neither resumed from nor read. Companies House's data-products guidance says it imposes no rules on how the register is used and leaves data protection to the user; the dataset's data.gov.uk entry gives its licence as "TBD", and gov.uk's footer licenses its own page text under the Open Government Licence. The credit line says the names are personal data. +`orcid-researchers` streams the summaries file in HTTP Range requests through `gzip` and `tarfile`'s stream mode, and keeps nothing of it but the extract: a dropped connection reopens at the byte it reached, with `ORCID_RETRIES` attempts at each read (243 s of waits, about four minutes of outage before each attempt's own 60 s timeout, where the other sources' few attempts suit a small request), every part must be a 206 with the `Content-Range` asked for and the `Last-Modified` the first answer gave (a first answer stating none is refused), and once the tar ends the rest of the gzip is read so its CRC is checked, then the whole body's length and MD5 against the figshare listing's (`ORCID_SIZE`, `ORCID_MD5`). The extract replaces the cached one only after that check and after the fetch has read it back through every check a load makes, so a fetch refused at the end keeps the last good one; reading it back took 2 min 23 s on 2026-10-27. The first draft checked its shares only on load, after the rename, where a file past them would have replaced a good extract with one every load refused (found in review, 2026-10-27): a check a fetch's output must pass belongs before the rename, not only on the read. A fetch took 78 minutes on 2026-10-27, about 10 MB/s, with the XML parsed by a worker per CPU. Its summaries are parsed with `xml.etree` and a summary declaring a DTD or an entity is refused and counted, rather than adding a dependency for a safer parser; the fetch refuses past `ORCID_MAX_BAD` of summaries that do not parse or are refused that way, of tar members that are not summaries, or of countries that are not two capital letters (none of the four on 2026-10-27), and below the `ORCID_MIN_` shares of summaries giving a record, of records holding a family name and of records holding a country (95.8%, 94.8% and 42.8% on 2026-10-27), which catch a release whose layout moved under the reader once the pin moves. The count of summaries is pinned too (`ORCID_SUMMARIES`): `tarfile` ends a stream at a damaged header without raising, so a tar read only in part would pass its MD5. The extract is cut at `"\n"` only (`newline="\n"`), as `corpus_build.jsonl_lines` cuts, and streamed rather than read whole. `tarfile` keeps every member it has read in `TarFile.members`, even reading a stream, so the reader clears it after each one: before it did, the fetch held 2.3 GB a GiB into the file. An extract written by another version of the extraction is not read (`ORCID_EXTRACT_VERSION`, in its file name). Of the extract's records, the country came from the address in 2,842,439, an employment in 7,025,517 and an education in 829,738 (2026-10-27): a country is where a holder works more often than where they come from, so read a country's agreement as its institutions' population, not its nationals'. + `nepal-candidates` reads each local body's file through the site's own JSON handler, which asks for the site's double-submit check: the `CsrfToken` cookie a GET of the site sets, echoed in an `X-CSRF-Token` header, with the `Referer` its pages send. The site refuses a User-Agent holding `python` or `curl` with a 403 (2026-10-24), so this source sends a browser's form naming the tool (`NEPAL_USER_AGENT`) where the others send the repository's URL. A 403 mid-fetch is what a lapsed token gets, and renews the session once before it is raised; a 404 counts the file as missing, its id kept in `meta.json`'s detail, and with the empty files (a list of no candidates, as a site that blanked its files would answer) past `NEPAL_MAX_MISSING` of the `NEPAL_BODIES` bodies refuses the fetch, the empty ones' ids kept too. The election is over and its files final, so the candidates are pinned as the bodies are: a fetch reading every file must read exactly `NEPAL_CANDIDATES`, so a site that trimmed its files (to the winners, or to one post) is refused however well formed what is left; with files missing or empty the pin cannot be exact, and a shortfall of up to `NEPAL_MAX_SHORT` of it is accepted, so a few candidates trimmed elsewhere are not seen then. Every listed file answered on 2026-10-24: the 18 a survey on 2026-10-23 saw refused, among them ids 5551 to 5554, answered when asked again. A file is JSON, so a body cut short does not parse, and is retried as one cut in transit rather than read as a body with fewer candidates. The fetch refuses a lookup listing other than `NEPAL_BODIES` bodies (the election is over, so its files are final), a file that is not a list of candidates, too many candidates with no romanized or no Devanagari name (`NEPAL_MAX_NAMELESS`) or with a name in the wrong script (`NEPAL_MAX_WRONG_SCRIPT`: two columns swapped, or a romanization dropped), and too few distinct names in either column (`NEPAL_MIN_DISTINCT`). A record is a distinct pair, as `tse-donors`' are, not a candidate row: a body lists a candidate once per post, and a common name recurs across bodies. The `pairs` view refuses it, its records carrying no reference name: that view compares the letters of two spellings, and a romanization shares none with its Devanagari, so the Devanagari is kept under a key of its own. `meta.json` records the keys the corpus was built from (`corpus_keys`), and a corpus written under other keys is refused on load. A fetch took 28 minutes on 2026-10-24 at `NEPAL_DELAY` (0.3 s) a request. -`local_agreement.py` does the SCORE job for a local corpus. It parses each filed display name and compares title, given, middle and family with the filer's boxes, folding case, periods and spacing. A suffix the parser peels is compared inside the surname box: the display name joins the surname last, so a trailing suffix was typed into it whenever it is filled. A title box the parser reads otherwise, holding a word the vocabulary lists as neither a title nor a connective (a job typed as a title), is set aside as filer error rather than scored, and the report lists the commonest such titles as leads for the vocabulary. The report gives agreement overall and by nationality, misses per field, and the commonest disagreement shapes, as counts, word counts and the commonest set-aside titles; `--list N` prints moved records, names and all, for reading locally. Its record, `agreement.jsonl` beside the corpus, holds every record that did not wholly agree, so a re-run reports what moved. As with the local snapshot, check it from the branch against a base you name (`--base `, whose record is kept under `agreements/` beside the corpus, keyed by the fetch, the base's `nameparser/` files, the interpreter's minor version and the scorer, the ten most recent kept), or, alone, record the shared one from the base checkout (`--record --tree `) and run it from the branch; it refuses a record from another fetch, or one written by another version of the scorer, whose own change would otherwise read as the parser's moves. No CI run sees it. +`local_agreement.py` does the SCORE job for a local corpus. It parses each filed display name and compares the parser's roles with the boxes the source states (title, given, middle and family for `psc-individuals`; given and middle joined against the given-names box, then family, for `orcid-researchers`), folding case, periods and spacing. A suffix the parser peels is compared inside the surname box: the display name joins the surname last, so a trailing suffix was typed into it whenever it is filled. Where a source has a title box, a title the parser reads otherwise, holding a word the vocabulary lists as neither a title nor a connective (a job typed as a title), is set aside as filer error rather than scored, and the report lists the commonest such titles as leads for the vocabulary. The report gives agreement overall and by the group the source states (nationality, country), misses per field, and the commonest disagreement shapes, as counts, word counts and the commonest set-aside titles; `--list N` prints moved records, names and all, for reading locally. Its record, `agreement.jsonl` beside the corpus, holds every record that did not wholly agree, so a re-run reports what moved. As with the local snapshot, check it from the branch against a base you name (`--base `, whose record is kept under `agreements/` beside the corpus, keyed by the fetch, the base's `nameparser/` files, the interpreter's minor version and the scorer, the ten most recent kept), or, alone, record the shared one from the base checkout (`--record --tree `) and run it from the branch; it refuses a record from another fetch, or one written by another version of the scorer, whose own change would otherwise read as the parser's moves. No CI run sees it. `pairs` reads a paired source (`tse-donors`, whose records carry the registry's name beside the filed one) as two spellings of one person, and sorts each pair by how the filed name departs from the reference: the first of its shapes that fits, from case, accents (case unchanged) and case and accents together, through words left out, shortened or misspelled, to few words in common. A word left out or added counts once per pair, and is listed only when ten pairs share it. `--agreement` also parses both spellings of every pair with a reference name that differs, and counts the pairs whose two readings end the family name in the same word (two readings with no family name share none). It lists the accents-only pairs read with two surnames by the two spellings' last words, the same ten-pair floor applied. @@ -328,7 +331,7 @@ Measured 2026-10-27 on a 12-CPU machine (load average 3 to 15 from other session uv run python tools/differential/measure.py --base ../master-checkout ``` -The steps: the committed snapshot (`snapshot.py`, the check `tests/v2/test_snapshot.py` makes), `cldr_agreement.py`, `snapshot.py --local --base` for every fetched local corpus, `local_agreement.py psc-individuals --base` where it is fetched, and `local_corpora.py alike --base` for every fetched source that pairs spellings. Only the local steps read `--base`: the first two compare the tree with the records committed on its branch, so on a branch that has re-recorded `snapshot.jsonl` the committed snapshot says no moves and the moves are in the PR's diff of that file, and the summary labels each step with what it compared against. A local corpus that is not fetched gets no step, the summary names it, and a run in which no step read `--base` exits 1. Each step's whole output goes to a file of its own under `--out`, by default a folder named for the time under the cache's `measure/`, outside the repository, since a local snapshot lists its moves by name. The summary, `summary.txt` beside them and printed, gives each step's status (no moves, moves, or failed with its exit code), its time, and its unindented lines (the first few and the last, where the verdicts are, up to `SUMMARY_LINES`, and any line saying part of a check was not measured), never the indented ones where the tools list names; a crash's traceback is indented below a line naming only its type, since its message may hold a name. A step that fails makes the run exit 1; one that finds moves does not. Each tool refuses with `SystemExit("message")`, which exits 1 as a move does, so measure runs every step through its own `--step` wrapper, which turns a message exit or an uncaught exception into exit 2: a refused base or a damaged corpus is reported as a failure, never as moves. +The steps: the committed snapshot (`snapshot.py`, the check `tests/v2/test_snapshot.py` makes), `cldr_agreement.py`, `snapshot.py --local --base` for every fetched local corpus, `local_agreement.py SOURCE --base` for every fetched source whose records carry their fields (`psc-individuals`, `orcid-researchers`), and `local_corpora.py alike --base` for every fetched source that pairs spellings. Only the local steps read `--base`: the first two compare the tree with the records committed on its branch, so on a branch that has re-recorded `snapshot.jsonl` the committed snapshot says no moves and the moves are in the PR's diff of that file, and the summary labels each step with what it compared against. A local corpus that is not fetched gets no step, the summary names it, and a run in which no step read `--base` exits 1. Each step's whole output goes to a file of its own under `--out`, by default a folder named for the time under the cache's `measure/`, outside the repository, since a local snapshot lists its moves by name. The summary, `summary.txt` beside them and printed, gives each step's status (no moves, moves, or failed with its exit code), its time, and its unindented lines (the first few and the last, where the verdicts are, up to `SUMMARY_LINES`, and any line saying part of a check was not measured), never the indented ones where the tools list names; a crash's traceback is indented below a line naming only its type, since its message may hold a name. A step that fails makes the run exit 1; one that finds moves does not. Each tool refuses with `SystemExit("message")`, which exits 1 as a move does, so measure runs every step through its own `--step` wrapper, which turns a message exit or an uncaught exception into exit 2: a refused base or a damaged corpus is reported as a failure, never as moves. For a vocabulary word whose reading acts on a name position, `tools/vocab/word_report.py` gives the per-word view before and alongside this: the word in a grid of shapes, its incidence in every fetched local corpus, and its `name_usage.py` rows (`nameparser/vocab/README.md`, "Adding a word"). diff --git a/tools/differential/local_agreement.py b/tools/differential/local_agreement.py index fd00f3660..358918005 100644 --- a/tools/differential/local_agreement.py +++ b/tools/differential/local_agreement.py @@ -3,7 +3,10 @@ The SCORE job (mechanisms.md#NAME-DATA-JOBS ("the job decides where its data lives")) for a local corpus whose records carry the name's fields beside it: each record's display name is parsed with the tree, -and each field is compared with the box the filer typed it into. What +and each field is compared with the box the filer typed it into +(`psc-individuals`' title, forename, middle name and surname boxes; +`orcid-researchers`' given-names box, holding the middle names too, and +family-name box, typed by the holder). What cldr_agreement.py does for the committed fields files, over names that are never committed -- so no CI run and no reviewer sees this score, and its record lives in the cache beside the corpus, where a re-run @@ -24,7 +27,7 @@ uv run python tools/differential/local_agreement.py psc-individuals The report prints counts and shapes, never a record: agreement overall -and by the group a source states (a nationality), the misses per field, +and by the group a source states (a nationality, a country), the misses per field, the commonest disagreement shapes, a shape being each disagreeing field's word count as filed and as parsed (`middle 0->1, family 2->1` is a two-word surname whose first word was read as a middle name), and the @@ -43,7 +46,7 @@ last in the display name, so it was typed into the last filled box, normally the surname: the parser's family is compared with its suffix appended, and the report counts the agreements that needed it. -- A filed title the parser does not read as the title, holding a word +- Where a source has a title box, a filed title the parser does not read as the title, holding a word the vocabulary lists in neither its titles nor its connectives, is set aside rather than scored: a job (`Managing Partner`) or a name typed into the title box is the filer's error, and it can move the @@ -84,20 +87,40 @@ from nameparser import Lexicon, Parser RECORD = "agreement.jsonl" +#: The parser's roles a Fielded box may name, held to nameparser's Role +#: by test_local_agreement (nameparser is imported only once --tree runs). +ROLES = frozenset({"title", "given", "middle", "family", "suffix", "nickname", "maiden"}) @dataclass(frozen=True) class Fielded: """How a local corpus states its fields: record key -> the parser's role it is compared with, in the order the display name joins them, - and the record key the report groups by.""" + and the record key the report groups by. A box holding more than one + role names them joined by "+", and is compared with those roles' + values joined by a space in that order: ORCID's given-names box holds + the middle names too. A source with no title box sets nothing aside. + The group is title-cased for the report unless `group_cased` says + it is written as it should print (a country code).""" fields: dict[str, str] group: str + group_cased: bool = False + + def __post_init__(self) -> None: + # A misspelled role would find no box (_box), and a misspelled + # title would quietly turn the set-aside off. + unknown = {r for role in self.fields.values() for r in role.split("+")} - ROLES + if unknown or "family" not in "+".join(self.fields.values()).split("+"): + raise ValueError(f"Fielded({self.fields}) names {sorted(unknown) or 'no family'}:" + f" each box names roles of {sorted(ROLES)}, a family among them") FIELDED = {local_corpora.PSC: Fielded( {"title": "title", "forename": "given", "middle_name": "middle", - "surname": "family"}, group="nationality")} + "surname": "family"}, group="nationality"), + local_corpora.ORCID: Fielded( + {"given": "given+middle", "family": "family"}, group="country", + group_cased=True)} #: The fewest records sharing a set-aside title for the report to print #: it: a word typed into the title box by this many filers is no one #: person's name. @@ -137,22 +160,25 @@ def read(record: dict[str, str], fielded: Fielded, parser: Parser, listed: frozenset[str]) -> Reading: """`record` parsed and compared box by box (see the module docstring).""" name = parser.parse(record["name"]) - got = {role: getattr(name, role) for role in fielded.fields.values()} + got = {role: " ".join(v for r in role.split("+") if (v := getattr(name, r))) + for role in fielded.fields.values()} by_suffix = False if name.suffix: - with_suffix = f"{got['family']} {name.suffix}".strip() + family = fielded.fields[_box(fielded, "family")] + with_suffix = f"{got[family]} {name.suffix}".strip() by_suffix = fold(with_suffix) == fold(record.get(_box(fielded, "family"), "")) - got["family"] = with_suffix + got[family] = with_suffix wrong = {role: got[role] for box, role in fielded.fields.items() if fold(record.get(box, "")) != fold(got[role])} title = record.get(_box(fielded, "title"), "") - if "title" in wrong and title and unlisted(title, listed): + if title and "title" in wrong and unlisted(title, listed): return Reading(wrong, set_aside=True) return Reading(wrong, by_suffix=by_suffix and not wrong) def _box(fielded: Fielded, role: str) -> str: - return next(box for box, r in fielded.fields.items() if r == role) + """The box `role` is compared with, or "" where none holds it.""" + return next((box for box, r in fielded.fields.items() if role in r.split("+")), "") def shape(record: dict[str, str], fielded: Fielded, got: dict[str, str]) -> str: @@ -166,7 +192,8 @@ def key(record: dict[str, str]) -> str: def group_of(record: dict[str, str], fielded: Fielded) -> str: - return " ".join(record.get(fielded.group, "").split()).title() or "(none)" + group = " ".join(record.get(fielded.group, "").split()) + return (group if fielded.group_cased else group.title()) or "(none)" def records(local: local_corpora.Local) -> Iterator[dict[str, str]]: diff --git a/tools/differential/local_core.py b/tools/differential/local_core.py index 08a890c45..29710f057 100644 --- a/tools/differential/local_core.py +++ b/tools/differential/local_core.py @@ -118,17 +118,18 @@ def _refuse_past(rows: Iterable[tuple[int, int, float, str]], why: str) -> None: raise ValueError(f"{count:,} of {of:,} {what}: {why}") -def _backed_off(exc: Exception, attempt: int, url: str) -> bool: +def _backed_off(exc: Exception, attempt: int, url: str, retries: int | None = None) -> bool: """The retry policy every source shares, after `exc` on try `attempt` of `url`: False where the caller should raise it (a 4xx other than - 429 is the request's fault, and the last attempt's error is the - fetch's), else True once a backoff of 2**attempt seconds, or the - Retry-After a server asks for up to a minute, has been slept, with a - line on stderr.""" + 429 is the request's fault, and the last of `retries` attempts' + error is the fetch's; RETRIES unless a caller whose attempt is cheap + beside what giving up loses asks for more), else True once a backoff + of 2**attempt seconds up to a minute, or the Retry-After a server + asks for up to a minute, has been slept, with a line on stderr.""" if (isinstance(exc, urllib.error.HTTPError) and exc.code < 500 - and exc.code != 429 or attempt == RETRIES - 1): + and exc.code != 429 or attempt == (RETRIES if retries is None else retries) - 1): return False - wait = 2 ** attempt + wait = min(2 ** attempt, 60) if isinstance(exc, urllib.error.HTTPError): after = exc.headers.get("Retry-After", "") if exc.headers else "" if after.isdigit(): diff --git a/tools/differential/local_corpora.py b/tools/differential/local_corpora.py index b3a79a9ff..4069e0ab0 100644 --- a/tools/differential/local_corpora.py +++ b/tools/differential/local_corpora.py @@ -37,11 +37,13 @@ name for the same donor; one distinct PSC filed name, with its four boxes and nationality; one distinct spelling of a JRC-Names person, with its entity and script; one distinct pair of a Nepali - candidate's romanized name and the Devanagari one. The counts are - counted from here. + candidate's romanized name and the Devanagari one; one distinct ORCID + record's two boxes joined, with the boxes, the credit name and the + country with where it came from. The counts are counted from here. - corpus.jsonl -- the distinct names, in the corpus line format snapshot.py and compare.py --corpus read: each record's "name", and - for nepal-candidates its Devanagari name too (Source.corpus_keys). + for nepal-candidates its Devanagari name too, and for + orcid-researchers the credit name (Source.corpus_keys). - meta.json -- the source's own date for the data, what was fetched, and the sha256 of both files, checked on every load: a truncated file would otherwise read as a run of false zeros. @@ -67,8 +69,9 @@ and what its terms allow is its module's docstring: local_ctgov.py (ctgov-officials), local_fec.py (fec-individuals), local_tse.py (tse-donors), local_tw.py (tw-donors), local_ukec.py (ukec-individuals), -local_psc.py (psc-individuals), local_jrc.py (jrc-names) and -local_nepal.py (nepal-candidates). +local_psc.py (psc-individuals), local_jrc.py (jrc-names), +local_nepal.py (nepal-candidates) and local_orcid.py +(orcid-researchers). """ from __future__ import annotations @@ -108,6 +111,7 @@ from local_fec import check_fec, fetch_fec, renew_fec # noqa: E402 from local_jrc import JRC, check_jrc, fetch_jrc, renew_jrc # noqa: E402 from local_nepal import check_nepal, fetch_nepal # noqa: E402 +from local_orcid import ORCID, check_orcid, fetch_orcid, renew_orcid # noqa: E402 from local_psc import PSC, check_psc, fetch_psc, renew_psc # noqa: E402 from local_tse import check_tse, fetch_tse # noqa: E402 from local_tw import fetch_tw # noqa: E402 @@ -218,6 +222,12 @@ def index(self) -> WordIndex: "entities (licensed under the JRC-Names EULA: personal use, no " "redistribution or derivative works)", unit="spellings", fetch=fetch_jrc, renew=renew_jrc, check=check_jrc), + ORCID: Source( + credit="ORCID Public Data File 2025, public names and countries " + "(CC0 1.0; ORCID asks for credit and respect for the privacy " + "of the people in it)", + unit="records", fetch=fetch_orcid, renew=renew_orcid, check=check_orcid, + corpus_keys=("name", "credit")), } diff --git a/tools/differential/local_orcid.py b/tools/differential/local_orcid.py new file mode 100644 index 000000000..1095b197b --- /dev/null +++ b/tools/differential/local_orcid.py @@ -0,0 +1,508 @@ +"""The orcid-researchers local corpus: the names ORCID's holders gave their own records. + +orcid-researchers -- the people holding an ORCID iD, from ORCID's +yearly Public Data File: each record's public name as its holder typed +it into the registry's two boxes (given names, family name), the +published name ("credit name") the holder typed beside them, and a +country. A local corpus whose names carry their fields +(local_agreement.FIELDED), so local_agreement.py scores the parser +against them; the person split the name, not a clerk or a filer's +software. The name the parser reads is the two boxes joined, given +first, as the registry displays it; the credit name is a corpus key of +its own, the same person's name as typed for publication. Most holders +are researchers, and they come from every country: from 217 countries +in the probe of the file's first GiB (2026-10-27). + +The country is the holder's own public address where there is one, else +the first employment that names one, else the first education that +does: of +605,872 records in that probe, 10.9%, 26.7% and 3.2%, and none for the +other 59.1%; where a record gave both an address and an employment the +two agreed in 92%. A country is where a holder works or lives, not +where the name comes from. The corpus keeps at most ORCID_CAP records +per country (records with none count as one more country), chosen by a +hash of the record, so a country with few holders is kept whole and one +with many is sampled; the corpus therefore shows what each country +holds, never how often anything occurs across them. + +The fetch streams the one file it needs, the summaries (46.3 GB for the +2025 release), through ranged requests that resume where a dropped +connection left off, and parses its records in a worker process per CPU +(renew_orcid), reading the extract back through every check a load +makes before it replaces the cached one. Only the extract is cached, never the tar: one line per +record with a public name, every country kept, so a change to the cap +reads the extract again rather than the file. The file is pinned by its +figshare id, size, MD5 and count of summaries (ORCID_URL, ORCID_SIZE, +ORCID_MD5, ORCID_SUMMARIES); a new year's release is measured and the +pin moved by hand, and the floors below catch a release whose layout +moved under the reader. ORCID releases the +file under CC0 1.0; its use policy asks that ORCID be credited and that +the privacy of the people in it be respected, which keeping it out of +the repository does. +""" +from __future__ import annotations + +import collections +import contextlib +import email.utils +import gzip +import hashlib +import heapq +import http.client +import importlib +import io +import json +import multiprocessing +import os +import re +import sys +import tarfile +import urllib.request +import xml.etree.ElementTree as ET +import zlib +from collections import Counter +from collections.abc import Callable, Iterator +from concurrent.futures import Future, ProcessPoolExecutor +from pathlib import Path +from typing import Any + +import corpus_build +import local_core +from local_core import Fetched, _again, _backed_off, _pinned, _refuse_past, directory + + +ORCID = "orcid-researchers" +#: The pinned release: the 2025 Public Data File's summaries +#: (figshare article 30375589, DOI 10.23640/07243.30375589.v1), +#: as figshare's API described it on 2026-10-27. +ORCID_FILE = "ORCID_2025_10_summaries.tar.gz" +ORCID_URL = "https://ndownloader.figshare.com/files/58834837" +ORCID_SIZE = 46_328_319_190 +ORCID_MD5 = "210edf71f4a2bb44dd33aaa3037b3f17" +#: The summaries the pinned file holds, read whole on 2026-10-27: tarfile +#: ends a stream at a damaged header without raising, so a count short of +#: this is a tar read only in part, which its MD5 cannot show. +ORCID_SUMMARIES = 26_078_951 +#: A record in the tar: one XML file per iD, under its iD's last three +#: characters, the check character (a digit or X) among them. +_SUMMARY = re.compile(r"ORCID_\d{4}_\d{2}_summaries/\d{2}[\dX]/\d{4}-\d{4}-\d{4}-\d{3}[\dX]\.xml") +#: The extract format, in the extract's file name: a change to what +#: _summary keeps or refuses bumps it, so an extract written by another +#: version is not read. +ORCID_EXTRACT_VERSION = 1 +#: The most records a country keeps (orcid_kept). +ORCID_CAP = 20_000 +#: The share of summaries a read of the extract (and so a fetch) skips +#: as not XML, or as XML declaring a DTD or an entity, the share of the +#: tar's files that are not summaries at all, and the share of the +#: countries kept that are not two capital letters, before it refuses +#: the file: the whole file met none of the four (2026-10-27). +ORCID_MAX_BAD = 1 / 10_000 +#: The least share of summaries that give a record, of records with a +#: family name, and of records with a country, before a read refuses the +#: file as one whose layout moved under the reader (an element renamed, +#: a namespace moved): 95.8%, 94.8% and 42.8% on 2026-10-27. +ORCID_MIN_RECORDS = 1 / 2 +ORCID_MIN_FAMILY = 1 / 2 +ORCID_MIN_COUNTRY = 1 / 5 +#: Attempts at each read of the stream: a reopen costs a request, and +#: giving up costs the 46 GB, so it rides out minutes of outage +#: (local_core._backed_off's nine waits, capped at a minute: 243 s, each +#: attempt's own 60 s timeout on top). +ORCID_RETRIES = 10 +#: Summaries handed to a worker at a time. +_BATCH = 2_000 + +_ADDRESS = "{http://www.orcid.org/ns/address}address" +_EMPLOYMENT = "{http://www.orcid.org/ns/employment}employment-summary" +_EDUCATION = "{http://www.orcid.org/ns/education}education-summary" +_LOCALE = "{http://www.orcid.org/ns/preferences}locale" +#: What a worker hands back: a batch's extract lines and counts. +_Batch = tuple[list[str], dict[str, int]] +#: Where a record's country is read from, in order of preference. +COUNTRY_FROM = ("address", "employment", "education") +_COUNTRY = re.compile(r"[A-Z]{2}") + + +def extract_path() -> Path: + """The extract cached for the pinned file.""" + stem = ORCID_FILE.removesuffix(".tar.gz") + return directory(ORCID) / f"{stem}.v{ORCID_EXTRACT_VERSION}.jsonl.gz" + + +def _local(tag: str) -> str: + return tag.rsplit("}", 1)[-1] + + +def _text(element: ET.Element) -> str: + return " ".join((element.text or "").split()) + + +def _summary(body: bytes) -> tuple[str, dict[str, Any] | None]: + """One summary's outcome and the record it gives: ("record", the + record) for a public name, else a count's key and None. The record + holds the boxes, the credit name, the other names, the country with + where it came from and the locale the holder set for ORCID's own + pages (a language, kept for a later view), the empty ones left out. A summary that + declares a DTD is refused unread ("dtd_refused"): ORCID's never + does, and one that did could define entities. The standard library's + parser expands no external entity, and the Expat it ships bounds + entity amplification; refusing the declaration keeps a parse from + depending on either.""" + if b" tuple[list[str], dict[str, int]]: + """A batch of summaries read (_summary): each record as an extract + line, and the counts of the rest. A worker's unit of work.""" + lines, counts = [], Counter[str]() + for body in bodies: + outcome, record = _summary(body) + counts[outcome] += 1 + if record is not None: + lines.append(corpus_build.jsonl_line(record, sort_keys=True)) + return lines, dict(counts) + + +class _RangeStream(io.RawIOBase): + """`url`'s body, `size` bytes, read through ranged requests: a + connection dropped or timed out mid-body is opened again from the + byte reached, under local_core's retry policy with ORCID_RETRIES + attempts, so an outage of a few minutes costs no byte already read. + Each response must be a 206 saying it is the range asked of a body of + `size` bytes, last modified when the first one was (which must say + when), or the file was replaced and the stream refuses it. + finish() checks the bytes read against `size` and `md5`.""" + + def __init__(self, url: str, size: int, md5: str) -> None: + self.url, self.size, self.md5 = url, size, md5 + self.pos = 0 + self.modified = "" + self._digest = hashlib.md5() + self._response: http.client.HTTPResponse | None = None + + def readable(self) -> bool: + return True + + def _open(self) -> http.client.HTTPResponse: + request = urllib.request.Request(self.url, headers={ + "User-Agent": local_core.USER_AGENT, "Range": f"bytes={self.pos}-"}) + response = urllib.request.urlopen(request, timeout=60) + stated = response.headers.get("Content-Range", "") + modified = response.headers.get("Last-Modified", "") + wanted = f"bytes {self.pos}-{self.size - 1}/{self.size}" + if response.status != 206 or stated != wanted or not modified or ( + self.modified and modified != self.modified): + response.close() + raise ValueError( + f"{self.url} answered {response.status} {stated or 'no range'}" + f" (last modified {modified or 'not stated'}) where {wanted}" + f"{' of the file last modified ' + self.modified if self.modified else ''}" + f" was asked: the file is not the pinned one") + self.modified = modified + return response + + def readinto(self, buffer: memoryview | bytearray) -> int: # type: ignore[override] + if self.pos >= self.size: + return 0 + for attempt in range(ORCID_RETRIES): + try: + if self._response is None: + self._response = self._open() + got = self._response.readinto(buffer) + if not got: + raise http.client.IncompleteRead(b"", self.size - self.pos) + except local_core._TRANSIENT as exc: + self._drop() + if not _backed_off(exc, attempt, self.url, ORCID_RETRIES): + raise + continue + self._digest.update(memoryview(buffer)[:got]) + self.pos += got + return got + raise AssertionError("unreachable: the last attempt raises") + + def _drop(self) -> None: + if self._response is not None: + self._response.close() + self._response = None + + def close(self) -> None: + self._drop() + super().close() + + def finish(self) -> None: + """ValueError naming the refresh unless every byte was read and + their MD5 is the pinned one.""" + if self.pos != self.size or self._digest.hexdigest() != self.md5: + raise ValueError(f"{self.url}: read {self.pos:,} of {self.size:,} bytes " + f"with MD5 {self._digest.hexdigest()}, where {self.md5} " + f"is pinned{_again(ORCID)}") + + +class _Inline: + """An executor that runs each call as it is submitted: one job.""" + + def submit(self, fn: Callable[[list[bytes]], _Batch], + batch: list[bytes]) -> Future[_Batch]: + done: Future[_Batch] = Future() + done.set_result(fn(batch)) + return done + + +def _drain(pending: collections.deque[Future[_Batch]], + out: io.TextIOBase, counts: Counter[str], keep: int) -> None: + """Write the oldest batches' lines until at most `keep` are pending.""" + while len(pending) > keep: + lines, batch_counts = pending.popleft().result() + counts.update(batch_counts) + out.writelines(f"{line}\n" for line in lines) + + +def _extract(raw: io.RawIOBase, out: io.TextIOBase, jobs: int) -> Counter[str]: + """Every summary of the tar.gz read from `raw`, parsed `jobs` at a + time (_summaries), its records written to `out` as they come; the + counts. The gzip stream is read to its end, past the tar's own end, + so that its CRC is checked.""" + counts: Counter[str] = Counter() + unzipped = gzip.GzipFile(fileobj=io.BufferedReader(raw, local_core.CHUNK)) + batch: list[bytes] = [] + pending: collections.deque[Future[_Batch]] = ( + collections.deque()) + # A worker unpickles the function by its module's bare name, which + # resolves there because this directory is on the inherited sys.path + # (snapshot.parallel says why). + read = getattr(importlib.import_module("local_orcid"), "_summaries") + with (ProcessPoolExecutor(jobs, mp_context=multiprocessing.get_context("spawn")) + if jobs > 1 else contextlib.nullcontext(_Inline())) as pool: + with tarfile.open(fileobj=unzipped, mode="r|") as tar: + for member in tar: + # A TarFile keeps every member it has read, even read as + # a stream: 26M of them would hold the whole file's + # headers in memory (2.3 GB a GiB in, measured 2026-10-27). + tar.members = [] + if not member.isfile(): + continue + if not _SUMMARY.fullmatch(member.name): + counts["members_not_summaries"] += 1 + continue + file = tar.extractfile(member) + assert file is not None, "a regular member has a body" + batch.append(file.read()) + if len(batch) == _BATCH: + pending.append(pool.submit(read, batch)) + batch = [] + _drain(pending, out, counts, 2 * jobs) + if batch: + pending.append(pool.submit(read, batch)) + _drain(pending, out, counts, 0) + while unzipped.read(local_core.CHUNK): + pass + return counts + + +def renew_orcid(jobs: int | None = None) -> None: + """Stream the pinned summaries file (_RangeStream) and keep its + extract: a header naming the file, its data date (the file's + Last-Modified) and the counts, then one line per record with a + public name, in the tar's order. Written beside its final name and + renamed into place only once the whole file was read, its MD5 + checked and the extract read back through every check a load makes + (_orcid_read), so a fetch that dies or is refused leaves the old + extract, or none. The + header is a gzip member of its own ahead of the records' (gzip reads + the members as one stream), since the counts are known only at the + end and the records are not compressed twice.""" + target = extract_path() + body, staged = (target.with_name(target.name + end) for end in (".part", ".new")) + stream = _RangeStream(ORCID_URL, ORCID_SIZE, ORCID_MD5) + print(f"streaming {ORCID_FILE} ({ORCID_SIZE / 1e9:.1f} GB) from {ORCID_URL} ...", + file=sys.stderr) + try: + with gzip.open(body, "wt", encoding="utf-8") as out: + counts = _extract(stream, out, jobs or os.cpu_count() or 1) + stream.finish() + header = corpus_build.jsonl_line({"file": ORCID_FILE, + "data_date": _iso(stream.modified), + "counts": dict(counts)}) + with staged.open("wb") as joined, body.open("rb") as records: + joined.write(gzip.compress(f"{header}\n".encode())) + while chunk := records.read(local_core.CHUNK): + joined.write(chunk) + _orcid_read(staged) + os.replace(staged, target) + finally: + stream.close() + body.unlink(missing_ok=True) + staged.unlink(missing_ok=True) + + +def _iso(http_date: str) -> str: + """An HTTP date as the corpora's data dates are written.""" + if not http_date: + raise ValueError(f"{ORCID_URL} stated no Last-Modified to date the data by") + return email.utils.parsedate_to_datetime(http_date).strftime("%Y-%m-%dT%H:%M:%S") + + +def orcid_record(line: dict[str, Any]) -> dict[str, str]: + """An extract line as a corpus record: the boxes joined given first + as "name", the boxes, the credit name and the country with where it + came from. The other names stay in the extract: a list, where a + record holds text.""" + record = {k: line[k] for k in ("given", "family", "credit", "country", + "country_from") if line.get(k)} + record["name"] = " ".join(v for v in (line.get("given"), line.get("family")) if v) + return record + + +def orcid_kept(records: Iterator[dict[str, str]], cap: int) -> list[dict[str, str]]: + """The distinct records, at most `cap` per country (no country is a + country of its own): those whose line hashes lowest, so a country + under the cap is kept whole and the same record is kept whatever + else the file holds.""" + heaps: dict[str, list[tuple[int, str]]] = collections.defaultdict(list) + seen: dict[str, set[int]] = collections.defaultdict(set) + for record in records: + line = corpus_build.jsonl_line(record, sort_keys=True) + rank = -int(hashlib.sha256(line.encode()).hexdigest()[:16], 16) + country = record.get("country", "") + heap, held = heaps[country], seen[country] + if rank in held: + continue + if len(heap) < cap: + heapq.heappush(heap, (rank, line)) + held.add(rank) + elif rank > heap[0][0]: + held.discard(heapq.heapreplace(heap, (rank, line))[0]) + held.add(rank) + return [json.loads(line) for heap in heaps.values() for _, line in sorted(heap)] + + +def _orcid_read(path: Path) -> Fetched: + """The corpus from the cached extract (orcid_record, orcid_kept), or + ValueError naming the refresh where it is not an extract of the + pinned file, where it holds another number of summaries than + ORCID_SUMMARIES or another number of records than its header counts, + where past ORCID_MAX_BAD of its summaries were not XML or declared a + DTD, of the tar's files were not summaries (a layout that moved + under _SUMMARY) or of its countries are not two capital letters, or + where fewer than the ORCID_MIN_ shares of its summaries gave a record + or of its records hold a family name or a country (an element the + reader looks for, renamed). The extract is cut at "\n" only, as + corpus_build.jsonl_lines cuts, and streamed rather than read whole.""" + seen: Counter[str] = Counter() + + def read(lines: Iterator[str]) -> Iterator[dict[str, str]]: + for line in lines: + record = orcid_record(json.loads(line)) + seen["records"] += 1 + seen["family"] += "family" in record + if country := record.get("country"): + seen["country"] += 1 + seen["country_malformed"] += not _COUNTRY.fullmatch(country) + yield record + + try: + with gzip.open(path, "rt", encoding="utf-8", newline="\n") as lines: + head = json.loads(next(lines)) + counts, data_date = head["counts"], head["data_date"] + if (head.get("file") != ORCID_FILE or not isinstance(counts, dict) + or not all(type(n) is int for n in counts.values()) + or not isinstance(data_date, str)): + raise ValueError("its header does not describe the pinned file") + records = orcid_kept(read(lines), ORCID_CAP) + except (OSError, EOFError, StopIteration, ValueError, KeyError, TypeError, + zlib.error) as exc: + raise ValueError(f"{path} is not an extract this tool wrote ({exc})" + f"{_again(ORCID)}") from None + summaries = sum(n for key, n in counts.items() if key != "members_not_summaries") + if summaries != ORCID_SUMMARIES or seen["records"] != counts.get("record", 0): + raise ValueError( + f"{path} holds {seen['records']:,} records of {summaries:,} summaries, where its " + f"header counts {counts.get('record', 0):,} records and {ORCID_SUMMARIES:,} " + f"summaries are pinned: the tar was read in part{_again(ORCID)}") + why = "the file is not the Public Data File this tool reads" + _refuse_past([(counts.get("not_xml", 0), summaries, ORCID_MAX_BAD, + "summaries are not XML"), + (counts.get("dtd_refused", 0), summaries, ORCID_MAX_BAD, + "summaries declare a DTD or an entity"), + (counts.get("members_not_summaries", 0), + summaries + counts.get("members_not_summaries", 0), ORCID_MAX_BAD, + "files in the tar are not summaries"), + (seen["country_malformed"], seen["country"], ORCID_MAX_BAD, + "countries are not two capital letters")], why) + for count, of, least, what in ( + (seen["records"], summaries, ORCID_MIN_RECORDS, "summaries give a record"), + (seen["family"], seen["records"], ORCID_MIN_FAMILY, "records hold a family name"), + (seen["country"], seen["records"], ORCID_MIN_COUNTRY, "records hold a country")): + if count < of * least: + raise ValueError(f"only {count:,} of {of:,} {what}: {why}") + by_country = Counter(r["country"] for r in records if "country" in r) + return Fetched(records, data_date, { + "kept": _kept(), "summaries": summaries, + **{key: counts[key] for key in sorted(counts)}, + "countries": len(by_country), + "countries_at_the_cap": sum(n == ORCID_CAP for n in by_country.values())}) + + +def _kept() -> str: + return f"{ORCID_FILE}, at most {ORCID_CAP:,} records a country" + + +def fetch_orcid() -> Fetched: + """The corpus (_orcid_read), read from the extract cached beside it, + streamed from the pinned file when missing and again by + `fetch --refresh`.""" + if not extract_path().exists(): + renew_orcid() + return _orcid_read(extract_path()) + + +check_orcid = _pinned("kept", _kept, "keeps {held}, and this tool keeps {wanted}") diff --git a/tools/differential/measure.py b/tools/differential/measure.py index 4b973568f..7e47f3eff 100644 --- a/tools/differential/measure.py +++ b/tools/differential/measure.py @@ -15,7 +15,8 @@ moved, grouped by field; 2. cldr_agreement.py, the CLDR and GND agreement records; 3. snapshot.py --local SOURCE --base, for every fetched local corpus; -4. local_agreement.py psc-individuals --base, where it is fetched; +4. local_agreement.py SOURCE --base, for every fetched corpus whose + records carry their fields (local_agreement.FIELDED); 5. local_corpora.py alike SOURCE --base, for every fetched corpus whose records pair two spellings of one person (ALIKE_SOURCES). @@ -60,6 +61,7 @@ if str(HERE) not in sys.path: sys.path.append(str(HERE)) import corpus_build # noqa: E402 +import local_agreement # noqa: E402 import local_corpora # noqa: E402 #: How many of a step's unindented lines the summary keeps: the first @@ -114,7 +116,7 @@ def plan(base: Path, jobs: int) -> list[Step]: [*py, tool["snapshot"], "--local", s, "--base", str(base), *j]) for s in sources] steps += [Step(f"agreement {s}", [*py, tool["local_agreement"], s, "--base", str(base), *j]) - for s in sources if s == local_corpora.PSC] + for s in sources if s in local_agreement.FIELDED] steps += [Step(f"alike {s}", [*py, tool["local_corpora"], "alike", s, "--base", str(base), *j]) for s in sources if s in local_corpora.ALIKE_SOURCES]