Skip to content

feat(datamart): Phase A public discovery (gov, lib, edu, search) over HTTP, MCP and CLI #28

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ralyodio merged 5 commits into from
Oct 8, 2026
Merged

feat(datamart): Phase A public discovery (gov, lib, edu, search) over HTTP, MCP and CLI #28

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
ec2cc27
feat(datamart): Phase A public discovery — gov, lib, edu, search over…
ralyodio Oct 7, 2026
f9099fe
fix(datamart): review findings — MCP GET 405, cooldown floor, item id…
ralyodio Oct 7, 2026
26ca997
chore(deps): override qs to ^6.16.0 (GHSA-x5fp-wj9c-mxmx, GHSA-4mjr-x…
ralyodio Oct 7, 2026
663518a
fix(datamart): return LoC rights notes as plain text; record live LoC…
ralyodio Oct 7, 2026
901b912
fix(datamart): CodeQL — anchor loc.gov host rewrite, decode & las…
ralyodio Oct 7, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .eslintrc.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,10 @@
"env": {
"mocha": true
},
"globals": {
"expect": "readonly",
"sinon": "readonly"
},
"rules": {
"no-unused-expressions": "off"
}
Expand Down
215 changes: 215 additions & 0 deletions bin/datamart.js
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,215 @@
#!/usr/bin/env node
/**
* datamart CLI — the same tools as the HTTP API and MCP endpoints.
*
* datamart search --zip 95032 --radius-miles 20 [--namespace lib|edu|gov] [--json]
* datamart lib search --zip 95032 [--type branch,central] [--json]
* datamart lib loc search "California history" [--format maps] [--json]
* datamart lib loc item 2021668470 [--json]
* datamart edu search --zip 95032 --type high,college [--json]
* datamart gov providers [--namespace gov] [--json]
* datamart gov search "business filing" [--json]
* datamart data search "public libraries" [--json] (needs DATAGOV_API_KEY)
* datamart resource us.ca.sos.bizfile [--json]
* datamart tools [--json]
* datamart call <tool> --key value ... [--json]
* datamart mcp serve [--namespace gov|lib|edu|contracts|search] (stdio MCP)
*
* Add --api https://mcp.profullstack.com to call a running server instead of
* running tools in-process. --json writes only the result to stdout;
* diagnostics go to stderr.
*/
import os from 'node:os';
import path from 'node:path';
import { z } from 'zod';
import { runTool, TOOLS, toolsFor, getTool } from '../src/datamart/tools.js';
import { usePersistentCooldowns } from '../src/datamart/http.js';
import { toErrorResponse } from '../src/datamart/errors.js';

const NOT_YET = {
contracts: 'Contract opportunity search, profiles and applications arrive in Phase B. Run `datamart call contracts_sources` for official sources.',
profiles: 'Applicant profiles arrive in Phase B.',
applications: 'Applications arrive in Phase B.',
finance: 'Private finance through CoinPay arrives in Phase C and will require your own authorized connection.',
tui: 'The TUI is not built yet.',
};

const BOOLEAN_FLAGS = new Set(['json', 'help', 'strict', 'include-unresolved']);

/** `--key value`, `--key=value` and boolean flags; everything else is positional. */
export function parseCli(argv) {
const values = {};
const positionals = [];
for (let i = 0; i < argv.length; i++) {
const a = argv[i];
if (!a.startsWith('--')) {
positionals.push(a);
continue;
}
const eq = a.indexOf('=');
const key = a.slice(2, eq === -1 ? undefined : eq);
let value;
if (eq !== -1) value = a.slice(eq + 1);
else if (BOOLEAN_FLAGS.has(key)) value = true;
else if (i + 1 < argv.length && !argv[i + 1].startsWith('--')) value = argv[++i];
else throw new Error(`--${key} needs a value`);
// Repeated flags (--keyword a --keyword b) collect into a list.
values[key] = key in values ? [].concat(values[key], value) : value;
}
return { values, positionals };
}

let values = {};
let positionals = [];
let json = process.argv.includes('--json');
let api = null;

/** --radius-miles 20 -> { radius_miles: '20' }; drops CLI-only flags. */
function toolArgs(extra = {}) {
const out = {};
for (const [k, v] of Object.entries(values)) {
if (['json', 'api', 'help'].includes(k)) continue;
out[k.replace(/-/g, '_')] = v;
}
return { ...out, ...extra };
}

/** Wrap single values for array-typed arguments (e.g. datagov_search keyword). */
function fitArgs(name, args) {
const tool = getTool(name);
if (!tool) return args;
const out = { ...args };
for (const [k, v] of Object.entries(out)) {
const def = tool.shape[k];
const inner = def?.unwrap ? def.unwrap() : def;
if (inner instanceof z.ZodArray && !Array.isArray(v)) out[k] = [v];
}
return out;
}

async function call(name, rawArgs) {
const args = fitArgs(name, rawArgs);
if (!api) return runTool(name, args);
const res = await fetch(`${api}/api/v1/tools/${encodeURIComponent(name)}`, {
method: 'POST',
headers: { 'content-type': 'application/json' },
body: JSON.stringify(args),
});
const body = await res.json();
if (!res.ok) throw Object.assign(new Error(body?.error?.message || `HTTP ${res.status}`), { body });
return body;
}

function print(result) {
if (json) {
process.stdout.write(JSON.stringify(result, null, 2) + '\n');
return;
}
const rows = result.data || [];
const m = result.meta || {};
if (m.origin) {
const o = m.origin;
console.log(`Within ${m.radius_miles} mi of ${o.label}${o.defaulted ? ' (default)' : ''} — ${m.total_matches} match(es), straight-line distance`);
}
for (const r of rows) {
const dist = r.distance_miles !== undefined ? `${r.distance_miles.toFixed(1).padStart(5)} mi ` : '';
const name = r.name || r.title || r.id;
const where = r.address ? ` ${r.address.street}, ${r.address.city}` : r.publisher ? ` — ${r.publisher}` : '';
const state = r.headline_state ? ` [${r.headline_state}]` : '';
console.log(`${dist}${name}${where}${state}`);
}
if (result.services?.length) {
console.log('\nGovernment services:');
for (const s of result.services) console.log(` ${s.name} [${s.headline_state}] ${s.official_url}`);
}
for (const w of result.warnings || []) console.error(`note: ${w}`);
if (m.next_cursor) console.error(`more: --cursor ${m.next_cursor}`);
}

async function serveMcp(namespace) {
const { McpServer } = await import('@modelcontextprotocol/sdk/server/mcp.js');
const { StdioServerTransport } = await import('@modelcontextprotocol/sdk/server/stdio.js');
const defs = namespace ? toolsFor({ namespace }) : TOOLS;
const server = new McpServer({ name: `datamart${namespace ? `-${namespace}` : ''}`, version: '0.1.0' });
for (const t of defs) {
server.registerTool(
t.name,
{ description: t.description, inputSchema: z.object(t.shape).strict(), annotations: { readOnlyHint: true, openWorldHint: true } },
async args => {
try {
const out = await runTool(t.name, args);
return { content: [{ type: 'text', text: JSON.stringify(out) }], structuredContent: out };
} catch (err) {
return { isError: true, content: [{ type: 'text', text: JSON.stringify(toErrorResponse(err).body) }] };
}
}
);
}
await server.connect(new StdioServerTransport());
console.error(`datamart MCP (stdio): ${defs.length} tools`);
}

async function main() {
({ values, positionals } = parseCli(process.argv.slice(2)));
json = values.json === true;
api = typeof values.api === 'string' ? values.api.replace(/\/+$/, '') : null;
const cacheHome = process.env.XDG_CACHE_HOME || path.join(os.homedir(), '.cache');
usePersistentCooldowns(path.join(cacheHome, 'datamart', 'cooldowns.json'));
const [cmd, sub, ...rest] = positionals;
if (!cmd || values.help) {
console.log((await import('node:fs')).readFileSync(new URL(import.meta.url), 'utf8').split('*/')[0].replace(/^#!.*\n\/\*\*\n/, '').replace(/^ \* ?/gm, ''));
return;
}
if (NOT_YET[cmd]) {
console.error(NOT_YET[cmd]);
process.exitCode = 2;
return;
}
const q = rest.length ? rest.join(' ') : undefined;
const withQ = (o = {}) => toolArgs(q ? { q, ...o } : o);
switch (`${cmd} ${sub ?? ''}`.trim()) {
case 'search':
return print(await call('datamart_search', toolArgs(sub ? { q: [sub, ...rest].join(' ') } : {})));
case 'lib search':
return print(await call('libraries_search', withQ()));
case 'lib loc': {
const [action, ...words] = rest;
if (action === 'search') return print(await call('loc_search', toolArgs(words.length ? { q: words.join(' ') } : {})));
if (action === 'collections') return print(await call('loc_list_collections', toolArgs()));
if (action === 'item') return print(await call('loc_get_item', toolArgs({ id: words[0] })));
if (action === 'files') return print(await call('loc_get_resource_links', toolArgs({ id: words[0] })));
throw new Error('usage: datamart lib loc search|collections|item|files');
}
case 'edu search':
return print(await call('education_search', withQ()));
case 'gov providers':
case 'providers': {
const { listProviders } = await import('../src/datamart/registry.js');
const data = listProviders({ namespace: values.namespace || (cmd === 'gov' ? 'gov' : undefined) });
return print({ data, meta: {} });
}
case 'gov search':
return print(await call('gov_services_search', withQ()));
case 'data search':
return print(await call('datagov_search', withQ()));
case 'data orgs':
return print(await call('datagov_list_organizations', withQ()));
case 'mcp serve':
if (api) throw new Error('mcp serve runs tools in-process; drop --api');
return serveMcp(values.namespace);
case 'tools':
return print({ data: TOOLS.map(t => ({ name: t.name, description: t.description })), meta: {} });
default:
if (cmd === 'resource' && sub) return print(await call('resource_get', { id: sub }));
if (cmd === 'call' && sub) return print(await call(sub, toolArgs()));
throw new Error(`unknown command: ${positionals.join(' ')} (try --help)`);
}
}

main().catch(err => {
const body = err.body || toErrorResponse(err).body;
const message = body?.error?.code === 'internal_error' ? err.message : `${body.error.code}: ${body.error.message}`;
if (json) process.stdout.write(JSON.stringify(body, null, 2) + '\n');
console.error(`datamart: ${message}`);
process.exitCode = 1;
});
43 changes: 43 additions & 0 deletions docs/datamart/capabilities.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
# Datamart capabilities (Phase A)

What works, what needs configuration, what is handoff-only, and what is not
built. Generated from `src/datamart/registry.js`; that file is the source of
truth and is validated at load time (a planned capability cannot claim a working
state).

## Providers

| Provider | Path | Works now | Not yet / never |
| ----------------------------------------------------------------- | -------------------- | ------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------- |
| Data.gov catalog (`us.gsa.datagov`) | `/gov/us/data` | `public_read`: search, harvest record, organizations — **needs `DATAGOV_API_KEY`** | dataset writes (publisher-only) |
| Library of Congress (`us.loc`) | `/lib/us/loc` | `public_read`: search, collections, item, resource links | record writes |
| Library outlets (`us.ca.libraries`) | `/lib` | `public_read` directory search over the IMLS FY2024 snapshot; records are `directory_only` | catalog reads, holds, renewals, card applications |
| Public schools & colleges (`us.ca.education`) | `/edu` | `public_read` directory search over NCES snapshots | private schools, preschools (coverage gap), enrollment prep, student records (excluded) |
| California BizFile (`us.ca.sos.bizfile`) | `/gov/us/ca/bizfile` | `directory_only`: official links, access facts | entity search, filing prep, submission |
| IRS (`us.irs`) | `/gov/us/irs` | `directory_only` | forms search, workflow prep; direct e-file not planned |
| California FTB (`us.ca.ftb`) | `/gov/us/ca/ftb` | `directory_only` | workflow prep; e-file not planned |
| California DMV (`us.ca.dmv`) | `/gov/us/ca/dmv` | `directory_only` | workflow prep, appointment booking |
| California EDD (`us.ca.edd`) | `/gov/us/ca/edd` | `directory_only` (employer and claimant kept apart) | employer prep; claimant certification never automated |
| SAM.gov, Cal eProcure, County of Santa Clara, City of Santa Clara | `/contracts` | `directory_only` official sources | opportunity ingestion, profiles, applications (Phase B) |
| CoinPay finance (`us.coinpay.finance`) | `/finance` | nothing | everything (Phase C); no finance tool exists on any endpoint |

## Requirement → test matrix

| PRD acceptance test | Covered by | Status |
| ------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------ |
| AT01 routes load without breaking existing MCP routes | `test/datamart/module.test.js` "existing root routes"; full suite 151 passing | local ✅, production ❌ not deployed |
| AT02 MCP initialize, tools/list, tools/call return real results | `module.test.js` SDK + root `/mcp`; `cli.test.js` stdio | local ✅ (Node and Bun) |
| AT03 Data.gov real records, cursor continuity | `providers.test.js` (mocked); live page 1 verified 2026-10-07 with DEMO_KEY | ⚠️ live page 2 not verified — DEMO_KEY quota exhausted; needs a real key |
| AT04 missing key → visible configuration state | `providers.test.js`, `registry.test.js` | ✅ |
| AT05 LoC ids, links, rights, upstream errors | `providers.test.js` (mocked); live 2026-10-07: `maps` search "los gatos" → 9 Sanborn records, item `sanborn00659_006` rights + JPEG/JP2 files | ✅ live |
| AT06 95032/20 mi returns only validated in-radius entries | `directory.test.js` | ✅ |
| AT07 inside / on / outside radius; no-coordinate records excluded | `geo.test.js` | ✅ |
| AT08 ZIP, coordinates, address, city consistent; conflicts explicit | `geo.test.js` | ✅ (address via mocked Census Geocoder) |
| AT09 public filter excludes private; sites vs offices; preschool evidence | `directory.test.js` | ✅ (preschool reported as a coverage gap) |
| AT10 branches distinct from systems; no implied eligibility | `directory.test.js` | ✅ |
| AT13 deep links normalize identically across surfaces | `module.test.js` (filters JSON + overrides, strict unknown keys on HTTP, root MCP, SDK) | ✅ for Phase A tools |
| AT19 finance unreachable anonymously | `registry.test.js`, `module.test.js` | ✅ (no finance tool exists) |
| AT23 no fetches to private/unknown hosts, no redirects | `providers.test.js` (allowlist, redirect refusal) | ✅ |
| AT25 quotas fail safely, no paid bypass | `providers.test.js` (cooldown, per-minute budget) | ✅ |
| AT11–AT12, AT14–AT18, AT20–AT22 | contracts, profiles, approvals, finance | Phase B/C, not started |
| AT24, AT26 | caches/logs audit, deploy evidence | at release |
Loading
Loading