diff --git a/AGENTS.md b/AGENTS.md index b5d15c8..e894b2a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -29,7 +29,8 @@ Modules: - `cli/` — root command registration, ordering, global options, verbose logging - `auth/` — token resolution (flag > env > config file), setup/reset/whoami commands -- `scrape/` — scrape/search/screenshot commands, schema loading, request building, the SDK client +- `scrape/` — scrape/screenshot commands, schema loading, request building, the SDK client +- `search/`, `map/`, `crawl/` — Data API commands (API key only); crawl reuses map's site options - `output/` — output format handling (text, JSON, NDJSON, PNG), request defaults - `platform/` — OS-level concerns: config paths, file/binary writes, hidden prompts, central error handler diff --git a/README.md b/README.md index ed52d65..fff0a98 100644 --- a/README.md +++ b/README.md @@ -45,7 +45,7 @@ The CLI is the right tool when you need direct shell access, scripting, or autom - **Structured output for automation**. Markdown (human-readable), JSON and NDJSON (pipe-friendly), and PNG screenshots built for scripts, data pipelines, and AI agent subprocesses. -- **Built-in support for popular targets**. Tier-1 commands for scrape, search, and screenshot, plus schema-driven subcommands for Google Search and Bing, Amazon, Walmart, and Target, Reddit, TikTok, YouTube, and more. Run `decodo targets` to list everything available. +- **Built-in support for popular targets**. Tier-1 commands for scrape, search, map, crawl, and screenshot, plus schema-driven subcommands for Google Search and Bing, Amazon, Walmart, and Target, Reddit, TikTok, YouTube, and more. Run `decodo targets` to list everything available. - **Global proxy infrastructure**. Access 115M+ residential IPs across 195+ locations with IP rotation and geo-targeting. All requests are powered by the Decodo Web Scraping API with a 99.99% success rate. @@ -113,7 +113,7 @@ npx @decodo/cli scrape https://ip.decodo.com --token "$DECODO_AUTH_TOKEN" Use the Web Data API key from your subscription on the Decodo [dashboard](https://dashboard.decodo.com/web-data/playground). Older plans only have a basic authentication token, which works the same way. The CLI detects which -one you provide, so the commands below take either. +one you provide, so the commands below take either. `search`, `map`, and `crawl` need an API key. ```bash # Interactive — saves the credential to config @@ -151,13 +151,38 @@ You should see Markdown or parsed JSON within seconds. If you see an auth error, | Command | Description | | --- | --- | | `decodo scrape ` | Scrape a URL (Markdown by default) | -| `decodo search ` | Web search (`--engine google\|bing`, `--geo`, `--limit`) | +| `decodo search ` | Web search, parsed JSON results (`--geo`, `--locale`, `--page`) | +| `decodo map ` | List a site's URLs, one per line (`--limit`, `--sitemap`, `--full`) | +| `decodo crawl ` | Start a crawl and print its id; `--wait` prints pages as NDJSON | | `decodo screenshot ` | Capture a PNG screenshot (`-o` file or directory) | | `decodo targets` | List all scrape targets by group | | `decodo setup` | Save your API key or auth token interactively | | `decodo whoami` | Show configured auth source | | `decodo reset` | Remove saved auth config | +### Search, map, and crawl + +These commands call the Decodo Web Data API and need an API key; a basic auth token is rejected with exit code `3`. + +```bash +decodo search "rust web scraping" --geo de | jq '.organic[].title' +decodo map https://example.com --limit 100 --select-paths "/blog/*" +decodo crawl https://example.com --limit 50 --wait --progress -o pages.ndjson +``` + +`decodo crawl ` returns right away with the crawl id. Manage a running crawl with: + +```bash +decodo crawl status +decodo crawl results -o pages.ndjson +decodo crawl results --wait -o pages.ndjson +decodo crawl cancel +``` + +`crawl results --wait` lets an agent start a crawl, keep working, and collect the pages later. With `--wait`, the CLI polls every `--poll-interval` seconds (default `5`) and gives up after `--timeout` seconds (default `3600`, exit code `6`); the crawl keeps running until you cancel it. The crawl id goes to stderr as soon as the crawl starts. A `truncated` or `cancelled` crawl prints a warning; a `faulted` crawl still prints its pages, then exits `7`. + +> **Upgrading from 1.x:** `decodo search` used the `google_search` scrape target. Use `decodo google-search` or `decodo bing-search` for the previous behaviour. + ### Schema-driven target commands Every API target is also available as its own subcommand using the kebab-case name shown by `decodo targets`: diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index f9c683f..583b31b 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -28,12 +28,16 @@ HTTP client, and typed errors. Node >= 18 (CI on Node 24), pnpm, Biome via `ultr Command assembly is two-layered: -- `cli/register.ts` composes the static commands (`setup`, `reset`, `whoami`) with the +- `cli/register.ts` composes the static commands (`setup`, `reset`, `whoami`, `search`, `map`, `crawl`) with the dynamically generated scrape commands, then sorts them with `sortCommandsByOrder` so help output has a stable, curated order (`ROOT_COMMAND_ORDER`). - `scrape/register.ts` loads the schema once and builds the scrape-family commands: - `scrape`, `search`, `screenshot`, `targets` (list), plus one generated command per API + `scrape`, `screenshot`, `targets` (list), plus one generated command per API target from `codegen-target-commands.ts`. +- The Data API commands `search`, `map`, and `crawl` (with `crawl status|results|cancel`) + live in their own modules and are registered directly in `cli/register.ts`. They don't + use the schema; each runs through `cli/services/run-data-action.ts`, which requires an + API key and hands the handler the SDK client. ## Schema loading @@ -48,7 +52,7 @@ types, enums, and which field is the "primary" input (URL/query/etc.). Every command is produced by a `createXCommand(schema)` factory that returns a Commander `Command`. There are two flavors: -- **Hand-written commands** (`scrape`, `search`, `screenshot`) define their own arguments, +- **Hand-written commands** (`scrape`, `screenshot`) define their own arguments, options, and a custom request-body builder. Example: `scrape.ts` adds `--country`, `--headers`, `--target`, then passes a `buildBody` closure to `createTargetAction`. - **Generated commands** (`codegen-target-commands.ts`) iterate `schema.listTargets()`, diff --git a/package.json b/package.json index fa6c5b2..ea6fb2e 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "@decodo/cli", - "version": "1.0.4", + "version": "2.0.0", "description": "Official CLI for the Decodo APIs", "license": "MIT", "type": "module", @@ -37,7 +37,7 @@ }, "packageManager": "pnpm@10.33.3", "dependencies": { - "@decodo/sdk-ts": "^2.3.0", + "@decodo/sdk-ts": "^2.6.1", "commander": "^14.0.0" }, "devDependencies": { diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index e2b28b0..7f4243c 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -13,8 +13,8 @@ importers: .: dependencies: '@decodo/sdk-ts': - specifier: ^2.3.0 - version: 2.3.0 + specifier: ^2.6.1 + version: 2.6.1 commander: specifier: ^14.0.0 version: 14.0.3 @@ -105,8 +105,8 @@ packages: resolution: {integrity: sha512-S0My7XPGIgpRWMDG8uRqalbgT+a6FmCUdOW+HaIOVVpUPHOb7RrpvjTjiODadKp06fsrVDJZlIzc6yCTp4AnxA==} engines: {node: '>= 20.12.0'} - '@decodo/sdk-ts@2.3.0': - resolution: {integrity: sha512-iTTmTvBhezY4SD9sQk+0jfKxMhNSTtUDGscw0J6cHDIhLsjpAtKE+duh5+HjTqjgMJ3o+8Be36ZLqbGTFBjWXA==} + '@decodo/sdk-ts@2.6.1': + resolution: {integrity: sha512-+QmLAz+81ge44VRjAwXbsM6rrGBp/EfBO9/degbDl5YuBs4EkiSYlo3LHgadGoEJT0ENrfKSUkAW+ghOpvfccQ==} engines: {node: '>=18.0.0'} '@esbuild/aix-ppc64@0.28.1': @@ -963,7 +963,7 @@ snapshots: fast-wrap-ansi: 0.2.2 sisteransi: 1.0.5 - '@decodo/sdk-ts@2.3.0': + '@decodo/sdk-ts@2.6.1': dependencies: zod: 4.4.3 diff --git a/src/auth/commands/reset.ts b/src/auth/commands/reset.ts index 76243a8..89745bf 100644 --- a/src/auth/commands/reset.ts +++ b/src/auth/commands/reset.ts @@ -1,9 +1,11 @@ import { Command } from "commander"; import { clearConfig } from "../services/config.js"; -export const resetCommand = new Command("reset") - .description("Remove saved CLI configuration") - .action(async () => { - await clearConfig(); - console.log("Configuration cleared."); - }); +export function createResetCommand(): Command { + return new Command("reset") + .description("Remove saved CLI configuration") + .action(async () => { + await clearConfig(); + console.log("Configuration cleared."); + }); +} diff --git a/src/auth/commands/setup.ts b/src/auth/commands/setup.ts index db6721e..dbe5931 100644 --- a/src/auth/commands/setup.ts +++ b/src/auth/commands/setup.ts @@ -58,26 +58,33 @@ async function verifyCredential(value: string): Promise { } } -export const setupCommand = new Command("setup") - .description("Configure the Decodo CLI with your API key or auth token") - .option("--token ", "Web Data API key or auth token (non-interactive)") - .action(async (options: SetupOptions, command) => { - const rootOpts = getRootOpts(command); - const token = ( - options.token?.trim() || - rootOpts.token?.trim() || - (await promptHidden(TOKEN_PROMPT)) - ).trim(); +export function createSetupCommand(): Command { + return new Command("setup") + .description("Configure the Decodo CLI with your API key or auth token") + .option( + "--token ", + "Web Data API key or auth token (non-interactive)" + ) + .action(async (options: SetupOptions, command) => { + const rootOpts = getRootOpts(command); + const token = ( + options.token?.trim() || + rootOpts.token?.trim() || + (await promptHidden(TOKEN_PROMPT)) + ).trim(); - if (!token) { - handleCliError(new CliUsageError("API key or auth token is required.")); - } + if (!token) { + handleCliError(new CliUsageError("API key or auth token is required.")); + } - try { - const credential = await verifyCredential(token); - await writeConfig(toConfig(credential)); - console.log(`Setup complete. Configuration saved to ${getConfigPath()}`); - } catch (err) { - handleCliError(err, { fallbackMessage: "Setup failed." }); - } - }); + try { + const credential = await verifyCredential(token); + await writeConfig(toConfig(credential)); + console.log( + `Setup complete. Configuration saved to ${getConfigPath()}` + ); + } catch (err) { + handleCliError(err, { fallbackMessage: "Setup failed." }); + } + }); +} diff --git a/src/auth/commands/whoami.ts b/src/auth/commands/whoami.ts index a982418..876e946 100644 --- a/src/auth/commands/whoami.ts +++ b/src/auth/commands/whoami.ts @@ -12,20 +12,22 @@ const CREDENTIAL_LABEL: Record = { [AUTH_TYPE.TOKEN]: "token", }; -export const whoamiCommand = new Command("whoami") - .description("Show the active auth source and masked credential") - .action(async (_options, command) => { - const rootOpts = getRootOpts(command); - const { credential, source } = await resolveAuthToken({ - token: rootOpts.token, - }); +export function createWhoamiCommand(): Command { + return new Command("whoami") + .description("Show the active auth source and masked credential") + .action(async (_options, command) => { + const rootOpts = getRootOpts(command); + const { credential, source } = await resolveAuthToken({ + token: rootOpts.token, + }); - if (!credential) { - handleCliError(new AuthRequiredError()); - } + if (!credential) { + handleCliError(new AuthRequiredError()); + } - console.log(`source: ${source}`); - console.log( - `${CREDENTIAL_LABEL[credential.type]}: ${mask(credential.value, 4, -4)}` - ); - }); + console.log(`source: ${source}`); + console.log( + `${CREDENTIAL_LABEL[credential.type]}: ${mask(credential.value, 4, -4)}` + ); + }); +} diff --git a/src/auth/constants.ts b/src/auth/constants.ts index b451d80..b0a7661 100644 --- a/src/auth/constants.ts +++ b/src/auth/constants.ts @@ -7,3 +7,6 @@ export const AUTH_TYPE = { TOKEN: "token", API_KEY: "apiKey", } as const; + +export const API_KEY_REQUIRED_MESSAGE = + "This command needs a Web Data API key; a basic auth token cannot call it."; diff --git a/src/auth/errors/api-key-required-error.ts b/src/auth/errors/api-key-required-error.ts new file mode 100644 index 0000000..c08c68b --- /dev/null +++ b/src/auth/errors/api-key-required-error.ts @@ -0,0 +1,8 @@ +import { API_KEY_REQUIRED_MESSAGE } from "../constants.js"; + +export class ApiKeyRequiredError extends Error { + constructor(message = API_KEY_REQUIRED_MESSAGE) { + super(message); + this.name = "ApiKeyRequiredError"; + } +} diff --git a/src/cli/register.ts b/src/cli/register.ts index e64ff02..3e16d0d 100644 --- a/src/cli/register.ts +++ b/src/cli/register.ts @@ -1,15 +1,21 @@ import type { Command } from "commander"; -import { resetCommand } from "../auth/commands/reset.js"; -import { setupCommand } from "../auth/commands/setup.js"; -import { whoamiCommand } from "../auth/commands/whoami.js"; +import { createResetCommand } from "../auth/commands/reset.js"; +import { createSetupCommand } from "../auth/commands/setup.js"; +import { createWhoamiCommand } from "../auth/commands/whoami.js"; +import { createCrawlCommand } from "../crawl/commands/crawl.js"; +import { createMapCommand } from "../map/commands/map.js"; import { createScrapeCommands } from "../scrape/register.js"; +import { createSearchCommand } from "../search/commands/search.js"; import { sortCommandsByOrder } from "./services/sort-commands-by-order.js"; export async function createCommands(): Promise { return sortCommandsByOrder([ - setupCommand, - resetCommand, - whoamiCommand, + createSetupCommand(), + createResetCommand(), + createWhoamiCommand(), + createSearchCommand(), + createMapCommand(), + createCrawlCommand(), ...(await createScrapeCommands()), ]); } diff --git a/src/cli/services/collect-values.ts b/src/cli/services/collect-values.ts new file mode 100644 index 0000000..66eb03c --- /dev/null +++ b/src/cli/services/collect-values.ts @@ -0,0 +1,6 @@ +export function collectValues( + value: string, + previous: string[] = [] +): string[] { + return [...previous, value]; +} diff --git a/src/cli/services/parse-positive-int.ts b/src/cli/services/parse-positive-int.ts new file mode 100644 index 0000000..27df618 --- /dev/null +++ b/src/cli/services/parse-positive-int.ts @@ -0,0 +1,9 @@ +import { InvalidArgumentError } from "commander"; + +export function parsePositiveInt(value: string): number { + const parsed = Number(value); + if (!Number.isInteger(parsed) || parsed < 1) { + throw new InvalidArgumentError("Must be a positive integer."); + } + return parsed; +} diff --git a/src/cli/services/run-data-action.ts b/src/cli/services/run-data-action.ts new file mode 100644 index 0000000..0707845 --- /dev/null +++ b/src/cli/services/run-data-action.ts @@ -0,0 +1,41 @@ +import type { Command } from "commander"; +import { AUTH_TYPE } from "../../auth/constants.js"; +import { ApiKeyRequiredError } from "../../auth/errors/api-key-required-error.js"; +import { AuthRequiredError } from "../../auth/errors/auth-required-error.js"; +import { resolveAuthToken } from "../../auth/services/resolve-token.js"; +import { handleCliError } from "../../platform/services/handle-cli-error.js"; +import { createDecodoClient } from "../../scrape/services/client.js"; +import type { DataActionHandler } from "../types/data-action.js"; +import { getRootOpts } from "./global-opts.js"; +import { verboseLog } from "./verbose-log.js"; + +export function createDataAction(handler: DataActionHandler) { + return async ( + input: string, + _localOptions: unknown, + command: Command + ): Promise => { + const options = command.optsWithGlobals() as Options; + const rootOpts = getRootOpts(command); + const verbose = rootOpts.verbose === true; + + try { + const auth = await resolveAuthToken({ token: rootOpts.token }); + verboseLog( + verbose, + `auth source=${auth.source} type=${auth.credential?.type ?? "none"}` + ); + if (!auth.credential) { + throw new AuthRequiredError(); + } + if (auth.credential.type !== AUTH_TYPE.API_KEY) { + throw new ApiKeyRequiredError(); + } + + const { webScrapingApi } = createDecodoClient(auth.credential); + await handler({ api: webScrapingApi, input, options, verbose }); + } catch (err) { + handleCliError(err, { fallbackMessage: "Request failed." }); + } + }; +} diff --git a/src/cli/services/sort-commands-by-order.ts b/src/cli/services/sort-commands-by-order.ts index 7333e75..d0d98b2 100644 --- a/src/cli/services/sort-commands-by-order.ts +++ b/src/cli/services/sort-commands-by-order.ts @@ -3,6 +3,8 @@ import type { Command } from "commander"; export const ROOT_COMMAND_ORDER = [ "scrape", "search", + "map", + "crawl", "screenshot", "setup", "reset", diff --git a/src/cli/types/data-action.ts b/src/cli/types/data-action.ts new file mode 100644 index 0000000..6b9aa06 --- /dev/null +++ b/src/cli/types/data-action.ts @@ -0,0 +1,12 @@ +import type { WebScrapingApi } from "@decodo/sdk-ts"; + +export interface DataActionContext { + api: WebScrapingApi; + input: string; + options: Options; + verbose: boolean; +} + +export type DataActionHandler = ( + context: DataActionContext +) => Promise; diff --git a/src/crawl/commands/attach-wait-options.ts b/src/crawl/commands/attach-wait-options.ts new file mode 100644 index 0000000..9d4e3e5 --- /dev/null +++ b/src/crawl/commands/attach-wait-options.ts @@ -0,0 +1,23 @@ +import type { Command } from "commander"; +import { parsePositiveInt } from "../../cli/services/parse-positive-int.js"; +import { + DEFAULT_CRAWL_TIMEOUT_SECONDS, + DEFAULT_POLL_INTERVAL_SECONDS, +} from "../constants.js"; +import { parsePollInterval } from "../services/parse-poll-interval.js"; + +export function attachWaitOptions(command: Command, waitHelp: string): Command { + return command + .option("--wait", waitHelp) + .option("--progress", "Print crawl progress to stderr (with --wait)") + .option( + "--poll-interval ", + `Seconds between status checks (with --wait, default: ${DEFAULT_POLL_INTERVAL_SECONDS})`, + parsePollInterval + ) + .option( + "--timeout ", + `Give up waiting after this many seconds (with --wait, default: ${DEFAULT_CRAWL_TIMEOUT_SECONDS})`, + parsePositiveInt + ); +} diff --git a/src/crawl/commands/crawl.ts b/src/crawl/commands/crawl.ts new file mode 100644 index 0000000..85a3702 --- /dev/null +++ b/src/crawl/commands/crawl.ts @@ -0,0 +1,124 @@ +import { Command, Option } from "commander"; +import { parsePositiveInt } from "../../cli/services/parse-positive-int.js"; +import { createDataAction } from "../../cli/services/run-data-action.js"; +import { attachSiteOptions } from "../../map/commands/attach-site-options.js"; +import { writeJsonOutput } from "../../output/services/write-json-output.js"; +import { + CRAWL_CONTENT_FORMATS, + CRAWL_START_WAIT_ONLY_FLAGS, +} from "../constants.js"; +import { assertCrawlSucceeded } from "../services/assert-crawl-succeeded.js"; +import { assertWaitFlags } from "../services/assert-wait-flags.js"; +import { buildCrawlRequest } from "../services/build-crawl-request.js"; +import { settleCrawl } from "../services/settle-crawl.js"; +import { writeCrawlPages } from "../services/write-crawl-pages.js"; +import type { + CrawlOptions, + CrawlResultsOptions, + CrawlStatusOptions, +} from "../types/crawl-command.js"; +import { attachWaitOptions } from "./attach-wait-options.js"; + +function createCrawlStatusCommand(): Command { + return new Command("status") + .description("Show the progress of a crawl as JSON") + .argument("", "Crawl id") + .option("--pretty", "Pretty-print JSON on stdout") + .action( + createDataAction(async ({ api, input, options }) => { + writeJsonOutput(await api.getCrawlStatus(input), options); + }) + ); +} + +function createCrawlResultsCommand(): Command { + const command = new Command("results") + .description("Print the pages of a crawl as NDJSON") + .argument("", "Crawl id") + .option("--no-content", "Skip page content, list page metadata only") + .option("-o, --output ", "Write NDJSON to file instead of stdout"); + + return attachWaitOptions( + command, + "Wait for the crawl to finish before printing its pages" + ).action( + createDataAction(async ({ api, input, options }) => { + assertWaitFlags(options); + const status = options.wait + ? await settleCrawl(api, input, options) + : undefined; + + const pages = await api.getCrawlResults(input, { + content: options.content, + }); + writeCrawlPages(pages, options.output); + if (status) { + assertCrawlSucceeded(status); + } + }) + ); +} + +function createCrawlCancelCommand(): Command { + return new Command("cancel") + .description("Stop a running crawl") + .argument("", "Crawl id") + .option("--pretty", "Pretty-print JSON on stdout") + .action( + createDataAction(async ({ api, input, options }) => { + writeJsonOutput(await api.cancelCrawl(input), options); + }) + ); +} + +export function createCrawlCommand(): Command { + const command = new Command("crawl") + .description( + "Start a crawl and print its id; --wait prints the pages as NDJSON" + ) + .enablePositionalOptions() + .argument("", "Site URL to crawl") + .option("--max-depth ", "Maximum link depth", parsePositiveInt) + .option( + "--max-duration ", + "Stop the crawl after this many seconds", + parsePositiveInt + ) + .addOption( + new Option("--content-format ", "Page content format").choices( + CRAWL_CONTENT_FORMATS + ) + ); + + attachSiteOptions(command); + + return attachWaitOptions( + command, + "Wait for the crawl to finish and print its pages" + ) + .option( + "-o, --output ", + "Write NDJSON to file instead of stdout (with --wait)" + ) + .action( + createDataAction(async ({ api, input, options }) => { + assertWaitFlags(options, CRAWL_START_WAIT_ONLY_FLAGS); + + const { crawl_id: crawlId } = await api.startCrawl( + buildCrawlRequest(input, options) + ); + if (!options.wait) { + process.stdout.write(`${crawlId}\n`); + return; + } + + process.stderr.write(`crawl ${crawlId} started\n`); + const status = await settleCrawl(api, crawlId, options); + writeCrawlPages(await api.getCrawlResults(crawlId), options.output); + assertCrawlSucceeded(status); + }) + ) + .addCommand(createCrawlStatusCommand()) + .addCommand(createCrawlResultsCommand()) + .addCommand(createCrawlCancelCommand()); +} diff --git a/src/crawl/constants.ts b/src/crawl/constants.ts new file mode 100644 index 0000000..e8bb4c9 --- /dev/null +++ b/src/crawl/constants.ts @@ -0,0 +1,25 @@ +export const DEFAULT_POLL_INTERVAL_SECONDS = 5; + +export const DEFAULT_CRAWL_TIMEOUT_SECONDS = 3600; + +export const CRAWL_CONTENT_FORMATS = ["markdown", "html"] as const; + +export const WAIT_ONLY_FLAGS = { + progress: "--progress", + pollInterval: "--poll-interval", + timeout: "--timeout", +} as const; + +export const CRAWL_START_WAIT_ONLY_FLAGS = { + ...WAIT_ONLY_FLAGS, + output: "--output", +} as const; + +export const MAX_POLL_INTERVAL_SECONDS = Math.floor((2 ** 31 - 1) / 1000); + +export const FAILED_CRAWL_STATUS = "faulted"; + +export const PARTIAL_CRAWL_STATUSES: ReadonlySet = new Set([ + "truncated", + "cancelled", +]); diff --git a/src/crawl/errors/crawl-failed-error.ts b/src/crawl/errors/crawl-failed-error.ts new file mode 100644 index 0000000..ada85d6 --- /dev/null +++ b/src/crawl/errors/crawl-failed-error.ts @@ -0,0 +1,13 @@ +export class CrawlFailedError extends Error { + readonly crawlId: string; + readonly status: string; + + constructor(crawlId: string, status: string, reason?: string) { + super( + `Crawl ${crawlId} ended with status ${status}${reason ? ` (${reason})` : ""}.` + ); + this.name = "CrawlFailedError"; + this.crawlId = crawlId; + this.status = status; + } +} diff --git a/src/crawl/services/assert-crawl-succeeded.ts b/src/crawl/services/assert-crawl-succeeded.ts new file mode 100644 index 0000000..507b086 --- /dev/null +++ b/src/crawl/services/assert-crawl-succeeded.ts @@ -0,0 +1,9 @@ +import type { GetCrawlStatusResponse } from "@decodo/sdk-ts"; +import { FAILED_CRAWL_STATUS } from "../constants.js"; +import { CrawlFailedError } from "../errors/crawl-failed-error.js"; + +export function assertCrawlSucceeded(status: GetCrawlStatusResponse): void { + if (status.status === FAILED_CRAWL_STATUS) { + throw new CrawlFailedError(status.crawl_id, status.status, status.reason); + } +} diff --git a/src/crawl/services/assert-wait-flags.ts b/src/crawl/services/assert-wait-flags.ts new file mode 100644 index 0000000..7396cf2 --- /dev/null +++ b/src/crawl/services/assert-wait-flags.ts @@ -0,0 +1,18 @@ +import { CliUsageError } from "../../platform/errors/cli-usage-error.js"; +import { WAIT_ONLY_FLAGS } from "../constants.js"; +import type { CrawlWaitOptions } from "../types/crawl-command.js"; + +export function assertWaitFlags( + options: CrawlWaitOptions & { output?: string }, + waitOnlyFlags: Readonly> = WAIT_ONLY_FLAGS +): void { + if (options.wait) { + return; + } + + for (const [key, flag] of Object.entries(waitOnlyFlags)) { + if (options[key as keyof typeof options] !== undefined) { + throw new CliUsageError(`${flag} requires --wait.`); + } + } +} diff --git a/src/crawl/services/build-crawl-request.ts b/src/crawl/services/build-crawl-request.ts new file mode 100644 index 0000000..e9716e7 --- /dev/null +++ b/src/crawl/services/build-crawl-request.ts @@ -0,0 +1,20 @@ +import type { StartCrawlRequest } from "@decodo/sdk-ts"; +import { buildSiteRequest } from "../../map/services/build-site-request.js"; +import type { CrawlOptions } from "../types/crawl-command.js"; + +export function buildCrawlRequest( + url: string, + options: CrawlOptions +): StartCrawlRequest { + const request: StartCrawlRequest = buildSiteRequest(url, options); + if (options.maxDepth !== undefined) { + request.max_depth = options.maxDepth; + } + if (options.maxDuration !== undefined) { + request.max_duration = options.maxDuration; + } + if (options.contentFormat !== undefined) { + request.output = options.contentFormat; + } + return request; +} diff --git a/src/crawl/services/format-crawl-progress.ts b/src/crawl/services/format-crawl-progress.ts new file mode 100644 index 0000000..230bf98 --- /dev/null +++ b/src/crawl/services/format-crawl-progress.ts @@ -0,0 +1,5 @@ +import type { GetCrawlStatusResponse } from "@decodo/sdk-ts"; + +export function formatCrawlProgress(status: GetCrawlStatusResponse): string { + return `crawl ${status.crawl_id} ${status.status}: ${status.completed}/${status.discovered} pages, ${status.failed} failed, ${status.in_flight} in flight`; +} diff --git a/src/crawl/services/parse-poll-interval.ts b/src/crawl/services/parse-poll-interval.ts new file mode 100644 index 0000000..18f46f5 --- /dev/null +++ b/src/crawl/services/parse-poll-interval.ts @@ -0,0 +1,13 @@ +import { InvalidArgumentError } from "commander"; +import { parsePositiveInt } from "../../cli/services/parse-positive-int.js"; +import { MAX_POLL_INTERVAL_SECONDS } from "../constants.js"; + +export function parsePollInterval(value: string): number { + const parsed = parsePositiveInt(value); + if (parsed > MAX_POLL_INTERVAL_SECONDS) { + throw new InvalidArgumentError( + `Must be at most ${MAX_POLL_INTERVAL_SECONDS}.` + ); + } + return parsed; +} diff --git a/src/crawl/services/settle-crawl.ts b/src/crawl/services/settle-crawl.ts new file mode 100644 index 0000000..18dbf4d --- /dev/null +++ b/src/crawl/services/settle-crawl.ts @@ -0,0 +1,32 @@ +import type { GetCrawlStatusResponse, WebScrapingApi } from "@decodo/sdk-ts"; +import { + DEFAULT_CRAWL_TIMEOUT_SECONDS, + DEFAULT_POLL_INTERVAL_SECONDS, + PARTIAL_CRAWL_STATUSES, +} from "../constants.js"; +import type { CrawlWaitOptions } from "../types/crawl-command.js"; +import { formatCrawlProgress } from "./format-crawl-progress.js"; +import { waitForCrawl } from "./wait-for-crawl.js"; + +export async function settleCrawl( + api: WebScrapingApi, + crawlId: string, + options: CrawlWaitOptions +): Promise { + const status = await waitForCrawl(api, crawlId, { + pollIntervalMs: + (options.pollInterval ?? DEFAULT_POLL_INTERVAL_SECONDS) * 1000, + timeoutMs: (options.timeout ?? DEFAULT_CRAWL_TIMEOUT_SECONDS) * 1000, + onStatus: options.progress + ? (current) => process.stderr.write(`${formatCrawlProgress(current)}\n`) + : undefined, + }); + + if (PARTIAL_CRAWL_STATUSES.has(status.status)) { + console.error( + `Warning: crawl ${crawlId} ended with status ${status.status}${status.reason ? ` (${status.reason})` : ""}` + ); + } + + return status; +} diff --git a/src/crawl/services/wait-for-crawl.ts b/src/crawl/services/wait-for-crawl.ts new file mode 100644 index 0000000..f68423f --- /dev/null +++ b/src/crawl/services/wait-for-crawl.ts @@ -0,0 +1,36 @@ +import { setTimeout as sleep } from "node:timers/promises"; +import { + type GetCrawlStatusResponse, + TimeoutError, + type WebScrapingApi, +} from "@decodo/sdk-ts"; + +export interface WaitForCrawlOptions { + onStatus?: (status: GetCrawlStatusResponse) => void; + pollIntervalMs: number; + timeoutMs: number; +} + +export async function waitForCrawl( + api: WebScrapingApi, + crawlId: string, + { onStatus, pollIntervalMs, timeoutMs }: WaitForCrawlOptions +): Promise { + const deadline = Date.now() + timeoutMs; + let status = await api.getCrawlStatus(crawlId); + onStatus?.(status); + + while (status.status === "pending") { + const remainingMs = deadline - Date.now(); + if (remainingMs <= 0) { + throw new TimeoutError( + `Crawl ${crawlId} did not finish within ${timeoutMs / 1000}s. It is still running; run \`decodo crawl cancel ${crawlId}\` to stop it.` + ); + } + await sleep(Math.min(pollIntervalMs, remainingMs)); + status = await api.getCrawlStatus(crawlId); + onStatus?.(status); + } + + return status; +} diff --git a/src/crawl/services/write-crawl-pages.ts b/src/crawl/services/write-crawl-pages.ts new file mode 100644 index 0000000..9684da5 --- /dev/null +++ b/src/crawl/services/write-crawl-pages.ts @@ -0,0 +1,13 @@ +import { writeFileSync } from "node:fs"; +import type { CrawlPage } from "@decodo/sdk-ts"; + +export function writeCrawlPages(pages: CrawlPage[], output?: string): void { + const text = pages.map((page) => `${JSON.stringify(page)}\n`).join(""); + + if (output !== undefined) { + writeFileSync(output, text, "utf8"); + return; + } + + process.stdout.write(text); +} diff --git a/src/crawl/types/crawl-command.ts b/src/crawl/types/crawl-command.ts new file mode 100644 index 0000000..7db277a --- /dev/null +++ b/src/crawl/types/crawl-command.ts @@ -0,0 +1,24 @@ +import type { StartCrawlRequest } from "@decodo/sdk-ts"; +import type { SiteOptions } from "../../map/types/site-options.js"; +import type { JsonOutputOptions } from "../../output/types/json-output-options.js"; + +export interface CrawlWaitOptions { + pollInterval?: number; + progress?: boolean; + timeout?: number; + wait?: boolean; +} + +export interface CrawlOptions extends SiteOptions, CrawlWaitOptions { + contentFormat?: StartCrawlRequest["output"]; + maxDepth?: number; + maxDuration?: number; + output?: string; +} + +export interface CrawlResultsOptions extends CrawlWaitOptions { + content: boolean; + output?: string; +} + +export type CrawlStatusOptions = JsonOutputOptions; diff --git a/src/map/commands/attach-site-options.ts b/src/map/commands/attach-site-options.ts new file mode 100644 index 0000000..451b8b2 --- /dev/null +++ b/src/map/commands/attach-site-options.ts @@ -0,0 +1,29 @@ +import { type Command, Option } from "commander"; +import { collectValues } from "../../cli/services/collect-values.js"; +import { parsePositiveInt } from "../../cli/services/parse-positive-int.js"; +import { DOMAIN_FILTERS, SITEMAP_MODES } from "../constants.js"; + +export function attachSiteOptions(command: Command): Command { + return command + .option("--limit ", "Maximum number of pages", parsePositiveInt) + .addOption( + new Option("--sitemap ", "Sitemap usage").choices(SITEMAP_MODES) + ) + .addOption( + new Option("--domain-filter ", "Which hosts to follow").choices( + DOMAIN_FILTERS + ) + ) + .option("--geo ", "Geo / country code") + .option("--locale ", "Locale, e.g. en-US") + .option( + "--select-paths ", + "Only include matching paths (repeatable)", + collectValues + ) + .option( + "--exclude-paths ", + "Skip matching paths (repeatable)", + collectValues + ); +} diff --git a/src/map/commands/map.ts b/src/map/commands/map.ts new file mode 100644 index 0000000..5c39c8c --- /dev/null +++ b/src/map/commands/map.ts @@ -0,0 +1,38 @@ +import { Command } from "commander"; +import { createDataAction } from "../../cli/services/run-data-action.js"; +import { writeJsonOutput } from "../../output/services/write-json-output.js"; +import { writeTextOutput } from "../../output/services/write-text-output.js"; +import { buildSiteRequest } from "../services/build-site-request.js"; +import type { MapOptions } from "../types/map-command.js"; +import { attachSiteOptions } from "./attach-site-options.js"; + +export function createMapCommand(): Command { + const command = new Command("map") + .description("List the URLs on a site, one per line") + .argument("", "Site URL to map") + .option("--js-rendering", "Render JavaScript while discovering links"); + + return attachSiteOptions(command) + .option("--full", "Print the full API response as JSON") + .option("--pretty", "Pretty-print JSON on stdout (with --full)") + .option("-o, --output ", "Write output to file instead of stdout") + .action( + createDataAction(async ({ api, input, options }) => { + const request = buildSiteRequest(input, options); + if (options.jsRendering) { + request.JS_rendering = true; + } + + const response = await api.map(request); + if (options.full) { + writeJsonOutput(response, options); + return; + } + + const urls = response.results.flatMap((result) => + result.content.links.map((link) => link.url) + ); + writeTextOutput(urls.join("\n"), options); + }) + ); +} diff --git a/src/map/constants.ts b/src/map/constants.ts new file mode 100644 index 0000000..69e41b2 --- /dev/null +++ b/src/map/constants.ts @@ -0,0 +1,3 @@ +export const SITEMAP_MODES = ["include", "only", "skip"] as const; + +export const DOMAIN_FILTERS = ["domain", "subdomain", "all"] as const; diff --git a/src/map/services/build-site-request.ts b/src/map/services/build-site-request.ts new file mode 100644 index 0000000..20cfce7 --- /dev/null +++ b/src/map/services/build-site-request.ts @@ -0,0 +1,23 @@ +import type { SiteOptions } from "../types/site-options.js"; + +export function buildSiteRequest( + url: string, + options: SiteOptions +): Record & { url: string } { + const fields: Record = { + limit: options.limit, + sitemap: options.sitemap, + domain_filter: options.domainFilter, + geo: options.geo, + locale: options.locale, + select_paths: options.selectPaths, + exclude_paths: options.excludePaths, + }; + + return { + url, + ...Object.fromEntries( + Object.entries(fields).filter(([, value]) => value !== undefined) + ), + }; +} diff --git a/src/map/types/map-command.ts b/src/map/types/map-command.ts new file mode 100644 index 0000000..412bb67 --- /dev/null +++ b/src/map/types/map-command.ts @@ -0,0 +1,7 @@ +import type { JsonOutputOptions } from "../../output/types/json-output-options.js"; +import type { SiteOptions } from "./site-options.js"; + +export interface MapOptions extends JsonOutputOptions, SiteOptions { + full?: boolean; + jsRendering?: boolean; +} diff --git a/src/map/types/site-options.ts b/src/map/types/site-options.ts new file mode 100644 index 0000000..5dc995c --- /dev/null +++ b/src/map/types/site-options.ts @@ -0,0 +1,9 @@ +export interface SiteOptions { + domainFilter?: string; + excludePaths?: string[]; + geo?: string; + limit?: number; + locale?: string; + selectPaths?: string[]; + sitemap?: string; +} diff --git a/src/output/services/write-json-output.ts b/src/output/services/write-json-output.ts new file mode 100644 index 0000000..0f1767b --- /dev/null +++ b/src/output/services/write-json-output.ts @@ -0,0 +1,13 @@ +import type { JsonOutputOptions } from "../types/json-output-options.js"; +import { resolvePrettyIndent } from "./resolve-pretty.js"; +import { writeTextOutput } from "./write-text-output.js"; + +export function writeJsonOutput( + payload: unknown, + options: JsonOutputOptions +): void { + writeTextOutput( + JSON.stringify(payload, null, resolvePrettyIndent(options)), + options + ); +} diff --git a/src/output/types/json-output-options.ts b/src/output/types/json-output-options.ts new file mode 100644 index 0000000..e6eb167 --- /dev/null +++ b/src/output/types/json-output-options.ts @@ -0,0 +1,4 @@ +export interface JsonOutputOptions { + output?: string; + pretty?: boolean; +} diff --git a/src/platform/services/handle-cli-error.ts b/src/platform/services/handle-cli-error.ts index 3c44737..471271a 100644 --- a/src/platform/services/handle-cli-error.ts +++ b/src/platform/services/handle-cli-error.ts @@ -6,7 +6,9 @@ import { ValidationError, } from "@decodo/sdk-ts"; import { PLAYGROUND_URL } from "../../auth/constants.js"; +import { ApiKeyRequiredError } from "../../auth/errors/api-key-required-error.js"; import { AuthRequiredError } from "../../auth/errors/auth-required-error.js"; +import { CrawlFailedError } from "../../crawl/errors/crawl-failed-error.js"; import { ScrapeFailedError } from "../../output/errors/scrape-failed-error.js"; import { EXIT } from "../constants.js"; import { CliUsageError } from "../errors/cli-usage-error.js"; @@ -48,7 +50,11 @@ export function resolveCliExitCode(err: unknown): number { return EXIT.USAGE; } - if (err instanceof AuthRequiredError || err instanceof AuthenticationError) { + if ( + err instanceof AuthRequiredError || + err instanceof ApiKeyRequiredError || + err instanceof AuthenticationError + ) { return EXIT.AUTH; } @@ -64,7 +70,7 @@ export function resolveCliExitCode(err: unknown): number { return EXIT.TIMEOUT; } - if (err instanceof ScrapeFailedError) { + if (err instanceof ScrapeFailedError || err instanceof CrawlFailedError) { return EXIT.NETWORK; } @@ -162,6 +168,10 @@ export function handleCliError( " 2. Run `decodo setup` to save it (or set DECODO_AUTH_TOKEN)\n" + " 3. Re-run your command" ); + } else if (err instanceof ApiKeyRequiredError) { + console.error( + `Hint: Get your Web Data API key at ${PLAYGROUND_URL}, then run \`decodo setup\` or pass --token.` + ); } else if (err instanceof AuthenticationError) { console.error( "Hint: Run `decodo setup` to configure your API key or auth token." diff --git a/src/scrape/commands/search.ts b/src/scrape/commands/search.ts deleted file mode 100644 index 8ac9da5..0000000 --- a/src/scrape/commands/search.ts +++ /dev/null @@ -1,93 +0,0 @@ -import { type DecodoSchema, Target, ValidationError } from "@decodo/sdk-ts"; -import { Command, Option } from "commander"; -import { attachScrapeOutputOptions } from "../../output/commands/attach-output-options.js"; -import { applyRequestDefaults } from "../../output/services/apply-request-defaults.js"; -import { resolveTarget } from "../services/resolve-target.js"; -import { createTargetAction } from "../services/run-target-scrape.js"; -import type { SearchOptions } from "../types/search-command.js"; - -const ENGINE_TARGETS = { - google: Target.GoogleSearch, - bing: Target.BingSearch, -} as const; - -const PAGE_COUNT_MIN = 1; -const PAGE_COUNT_MAX = 10; - -function parseLimit(value: string): number { - const limit = Number.parseInt(value, 10); - if (Number.isNaN(limit)) { - throw new ValidationError("Invalid --limit: must be an integer."); - } - return limit; -} - -function validatePageCount(limit: number): void { - if (limit < PAGE_COUNT_MIN || limit > PAGE_COUNT_MAX) { - throw new ValidationError( - `--limit must be between ${PAGE_COUNT_MIN} and ${PAGE_COUNT_MAX}.` - ); - } -} - -function resolveSearchTarget( - opts: SearchOptions, - schema: DecodoSchema -): string { - if (opts.target !== undefined) { - return resolveTarget(opts.target, schema, Target.GoogleSearch); - } - - const engine = opts.engine ?? "google"; - const target = ENGINE_TARGETS[engine as keyof typeof ENGINE_TARGETS]; - if (target === undefined) { - throw new ValidationError(`Unknown search engine: ${engine}`); - } - - return target; -} - -export function createSearchCommand(schema: DecodoSchema): Command { - const command = new Command("search") - .description( - "Search the web (default: Google). Use decodo google-search or decodo bing-search for full options." - ) - .argument("", "Search query") - .addOption( - new Option("--engine ", "Search engine") - .choices(["google", "bing"]) - .default("google") - ) - .option("--geo ", "Geo / country code") - .option("--limit ", "Number of result pages", parseLimit) - .option("--target ", "Scrape target override"); - - attachScrapeOutputOptions(command); - - return command.action( - createTargetAction(Target.GoogleSearch, schema, (query, options) => { - if (query === undefined) { - throw new Error("Missing required query."); - } - - const opts = options as SearchOptions; - const body: Record = { - target: resolveSearchTarget(opts, schema), - query, - }; - - if (opts.geo !== undefined) { - body.geo = opts.geo; - } - - if (opts.limit !== undefined) { - validatePageCount(opts.limit); - body.page_count = opts.limit; - } - - applyRequestDefaults(body, body.target as string, schema); - - return body; - }) - ); -} diff --git a/src/scrape/register.ts b/src/scrape/register.ts index 51d5e67..7219eec 100644 --- a/src/scrape/register.ts +++ b/src/scrape/register.ts @@ -3,7 +3,6 @@ import { createCodegenTargetCommands } from "./commands/codegen-target-commands. import { createListTargetsCommand } from "./commands/list-targets.js"; import { createScrapeCommand } from "./commands/scrape.js"; import { createScreenshotCommand } from "./commands/screenshot.js"; -import { createSearchCommand } from "./commands/search.js"; import { loadSchema } from "./services/schema-loader.js"; export async function createScrapeCommands(): Promise { @@ -11,7 +10,6 @@ export async function createScrapeCommands(): Promise { return [ createScrapeCommand(schema), - createSearchCommand(schema), createScreenshotCommand(schema), createListTargetsCommand(schema), ...createCodegenTargetCommands(schema), diff --git a/src/scrape/types/search-command.ts b/src/scrape/types/search-command.ts deleted file mode 100644 index 3827e0e..0000000 --- a/src/scrape/types/search-command.ts +++ /dev/null @@ -1,6 +0,0 @@ -export interface SearchOptions { - engine?: string; - geo?: string; - limit?: number; - target?: string; -} diff --git a/src/search/commands/search.ts b/src/search/commands/search.ts new file mode 100644 index 0000000..0d63b0a --- /dev/null +++ b/src/search/commands/search.ts @@ -0,0 +1,44 @@ +import type { RunSearchRequest } from "@decodo/sdk-ts"; +import { Command } from "commander"; +import { parsePositiveInt } from "../../cli/services/parse-positive-int.js"; +import { createDataAction } from "../../cli/services/run-data-action.js"; +import { ScrapeFailedError } from "../../output/errors/scrape-failed-error.js"; +import { writeJsonOutput } from "../../output/services/write-json-output.js"; +import type { SearchOptions } from "../types/search-command.js"; + +function buildSearchRequest( + query: string, + options: SearchOptions +): RunSearchRequest { + const request: RunSearchRequest = { query }; + if (options.geo !== undefined) { + request.geo = options.geo; + } + if (options.locale !== undefined) { + request.locale = options.locale; + } + if (options.page !== undefined) { + request.page = options.page; + } + return request; +} + +export function createSearchCommand(): Command { + return new Command("search") + .description("Search the web and print parsed results as JSON") + .argument("", "Search query") + .option("--geo ", "Geo / country code") + .option("--locale ", "Locale, e.g. en-US") + .option("--page ", "Result page number", parsePositiveInt) + .option("--pretty", "Pretty-print JSON on stdout") + .option("-o, --output ", "Write output to file instead of stdout") + .action( + createDataAction(async ({ api, input, options }) => { + const response = await api.search(buildSearchRequest(input, options)); + if (response.status === "failed") { + throw new ScrapeFailedError(response.message, response.status_code); + } + writeJsonOutput(response, options); + }) + ); +} diff --git a/src/search/types/search-command.ts b/src/search/types/search-command.ts new file mode 100644 index 0000000..b8081ba --- /dev/null +++ b/src/search/types/search-command.ts @@ -0,0 +1,7 @@ +import type { JsonOutputOptions } from "../../output/types/json-output-options.js"; + +export interface SearchOptions extends JsonOutputOptions { + geo?: string; + locale?: string; + page?: number; +} diff --git a/tests/auth/commands/reset.test.ts b/tests/auth/commands/reset.test.ts index eb5d843..2c627e1 100644 --- a/tests/auth/commands/reset.test.ts +++ b/tests/auth/commands/reset.test.ts @@ -2,7 +2,7 @@ import { Command } from "commander"; import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; import { isolateConfigHome } from "../../platform/helpers/config-home.js"; -describe("resetCommand", () => { +describe("createResetCommand", () => { let restoreConfigHome: () => void; let stdout: string[]; @@ -27,10 +27,10 @@ describe("resetCommand", () => { ); await writeConfig({ authToken: "saved-token" }); - const { resetCommand } = await import( + const { createResetCommand } = await import( "../../../src/auth/commands/reset.js" ); - const program = new Command().addCommand(resetCommand); + const program = new Command().addCommand(createResetCommand()); await program.parseAsync(["reset"], { from: "user" }); expect(await readConfig()).toBeUndefined(); diff --git a/tests/auth/commands/setup.test.ts b/tests/auth/commands/setup.test.ts index 0e9cd46..a24d020 100644 --- a/tests/auth/commands/setup.test.ts +++ b/tests/auth/commands/setup.test.ts @@ -15,16 +15,18 @@ async function runSetup( setupArgs: string[] = [], globalArgs: string[] = [] ): Promise { - const { setupCommand } = await import("../../../src/auth/commands/setup.js"); + const { createSetupCommand } = await import( + "../../../src/auth/commands/setup.js" + ); const program = new Command() .option("--token ", "global token") - .addCommand(setupCommand); + .addCommand(createSetupCommand()); await program.parseAsync([...globalArgs, "setup", ...setupArgs], { from: "user", }); } -describe("setupCommand", () => { +describe("createSetupCommand", () => { let restoreConfigHome: () => void; let exitCode: number | undefined; let stdout: string[]; diff --git a/tests/auth/commands/whoami.test.ts b/tests/auth/commands/whoami.test.ts index 54fcc34..8c41a3f 100644 --- a/tests/auth/commands/whoami.test.ts +++ b/tests/auth/commands/whoami.test.ts @@ -3,16 +3,16 @@ import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; import { isolateConfigHome } from "../../platform/helpers/config-home.js"; async function runWhoami(args: string[]): Promise { - const { whoamiCommand } = await import( + const { createWhoamiCommand } = await import( "../../../src/auth/commands/whoami.js" ); const program = new Command() .option("--token ", "global token") - .addCommand(whoamiCommand); + .addCommand(createWhoamiCommand()); await program.parseAsync(args, { from: "user" }); } -describe("whoamiCommand", () => { +describe("createWhoamiCommand", () => { let restoreConfigHome: () => void; let previousEnvToken: string | undefined; let exitCode: number | undefined; diff --git a/tests/auth/errors/api-key-required-error.test.ts b/tests/auth/errors/api-key-required-error.test.ts new file mode 100644 index 0000000..fbe3ad2 --- /dev/null +++ b/tests/auth/errors/api-key-required-error.test.ts @@ -0,0 +1,11 @@ +import { describe, expect, it } from "vitest"; +import { API_KEY_REQUIRED_MESSAGE } from "../../../src/auth/constants.js"; +import { ApiKeyRequiredError } from "../../../src/auth/errors/api-key-required-error.js"; + +describe("ApiKeyRequiredError", () => { + it("defaults to the API key message", () => { + const err = new ApiKeyRequiredError(); + expect(err.message).toBe(API_KEY_REQUIRED_MESSAGE); + expect(err.name).toBe("ApiKeyRequiredError"); + }); +}); diff --git a/tests/cli/helpers/run-data-command.ts b/tests/cli/helpers/run-data-command.ts new file mode 100644 index 0000000..0270f73 --- /dev/null +++ b/tests/cli/helpers/run-data-command.ts @@ -0,0 +1,10 @@ +import { Command } from "commander"; + +export function runDataCommand(command: Command, args: string[]) { + const program = new Command() + .option("--token ") + .exitOverride() + .addCommand(command.exitOverride()); + + return program.parseAsync(args, { from: "user" }); +} diff --git a/tests/cli/services/collect-values.test.ts b/tests/cli/services/collect-values.test.ts new file mode 100644 index 0000000..1f61bb7 --- /dev/null +++ b/tests/cli/services/collect-values.test.ts @@ -0,0 +1,15 @@ +import { describe, expect, it } from "vitest"; +import { collectValues } from "../../../src/cli/services/collect-values.js"; + +describe("collectValues", () => { + it("starts a list on first use", () => { + expect(collectValues("/blog/*")).toEqual(["/blog/*"]); + }); + + it("appends repeated values", () => { + expect(collectValues("/docs/*", ["/blog/*"])).toEqual([ + "/blog/*", + "/docs/*", + ]); + }); +}); diff --git a/tests/cli/services/parse-positive-int.test.ts b/tests/cli/services/parse-positive-int.test.ts new file mode 100644 index 0000000..1d27b2b --- /dev/null +++ b/tests/cli/services/parse-positive-int.test.ts @@ -0,0 +1,13 @@ +import { InvalidArgumentError } from "commander"; +import { describe, expect, it } from "vitest"; +import { parsePositiveInt } from "../../../src/cli/services/parse-positive-int.js"; + +describe("parsePositiveInt", () => { + it("parses positive integers", () => { + expect(parsePositiveInt("12")).toBe(12); + }); + + it.each(["0", "-1", "1.5", "abc", ""])("rejects %j", (value) => { + expect(() => parsePositiveInt(value)).toThrow(InvalidArgumentError); + }); +}); diff --git a/tests/cli/services/run-data-action.test.ts b/tests/cli/services/run-data-action.test.ts new file mode 100644 index 0000000..1683bb0 --- /dev/null +++ b/tests/cli/services/run-data-action.test.ts @@ -0,0 +1,107 @@ +import { Command } from "commander"; +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { AUTH_TYPE } from "../../../src/auth/constants.js"; +import { resolveAuthToken } from "../../../src/auth/services/resolve-token.js"; +import { createDataAction } from "../../../src/cli/services/run-data-action.js"; +import { createDecodoClient } from "../../../src/scrape/services/client.js"; + +vi.mock("../../../src/auth/services/resolve-token.js", () => ({ + resolveAuthToken: vi.fn(), +})); + +vi.mock("../../../src/scrape/services/client.js", () => ({ + createDecodoClient: vi.fn(), +})); + +function run(handler: Parameters[0]) { + const program = new Command() + .option("--token ") + .option("-v, --verbose") + .addCommand( + new Command("probe") + .argument("") + .option("--flag") + .action(createDataAction(handler)) + ); + + return program.parseAsync(["probe", "value", "--flag", "--token", "key"], { + from: "user", + }); +} + +describe("createDataAction", () => { + let exitCode: number | undefined; + let stderr: string[]; + const api = { search: vi.fn() }; + + beforeEach(() => { + exitCode = undefined; + stderr = []; + vi.mocked(createDecodoClient).mockReturnValue({ + webScrapingApi: api, + } as never); + vi.spyOn(process, "exit").mockImplementation((code) => { + exitCode = code as number; + throw new Error(`process.exit:${code}`); + }); + vi.spyOn(console, "error").mockImplementation((message) => { + stderr.push(String(message)); + }); + }); + + afterEach(() => { + vi.restoreAllMocks(); + }); + + it("passes the client, input and options to the handler", async () => { + vi.mocked(resolveAuthToken).mockResolvedValue({ + credential: { type: AUTH_TYPE.API_KEY, value: "key" }, + source: "flag", + }); + const handler = vi.fn().mockResolvedValue(undefined); + + await run(handler); + + expect(resolveAuthToken).toHaveBeenCalledWith({ token: "key" }); + expect(handler).toHaveBeenCalledWith({ + api, + input: "value", + options: { flag: true, token: "key" }, + verbose: false, + }); + }); + + it("exits 3 when no credential is configured", async () => { + vi.mocked(resolveAuthToken).mockResolvedValue({ + credential: undefined, + source: "none", + }); + + await expect(run(vi.fn())).rejects.toThrow("process.exit:3"); + expect(exitCode).toBe(3); + }); + + it("exits 3 with an API key hint for a basic token", async () => { + vi.mocked(resolveAuthToken).mockResolvedValue({ + credential: { type: AUTH_TYPE.TOKEN, value: "dTpw" }, + source: "config", + }); + const handler = vi.fn(); + + await expect(run(handler)).rejects.toThrow("process.exit:3"); + expect(handler).not.toHaveBeenCalled(); + expect(stderr[0]).toContain("needs a Web Data API key"); + }); + + it("routes handler errors through the CLI error handler", async () => { + vi.mocked(resolveAuthToken).mockResolvedValue({ + credential: { type: AUTH_TYPE.API_KEY, value: "key" }, + source: "flag", + }); + + await expect(run(() => Promise.reject(new Error("boom")))).rejects.toThrow( + "process.exit:1" + ); + expect(stderr[0]).toBe("Error: boom"); + }); +}); diff --git a/tests/crawl/commands/attach-wait-options.test.ts b/tests/crawl/commands/attach-wait-options.test.ts new file mode 100644 index 0000000..e0d5fa0 --- /dev/null +++ b/tests/crawl/commands/attach-wait-options.test.ts @@ -0,0 +1,37 @@ +import { Command } from "commander"; +import { describe, expect, it } from "vitest"; +import { attachWaitOptions } from "../../../src/crawl/commands/attach-wait-options.js"; + +describe("attachWaitOptions", () => { + it("parses wait flags with the given --wait help", () => { + const command = attachWaitOptions( + new Command("probe"), + "Wait for it" + ).exitOverride(); + + command.parse( + ["--wait", "--progress", "--poll-interval", "2", "--timeout", "30"], + { from: "user" } + ); + + expect(command.opts()).toEqual({ + wait: true, + progress: true, + pollInterval: 2, + timeout: 30, + }); + expect( + command.options.find((option) => option.long === "--wait")?.description + ).toBe("Wait for it"); + }); + + it("rejects a poll interval too large for a timer", () => { + const command = attachWaitOptions(new Command("probe"), "Wait for it") + .exitOverride() + .configureOutput({ writeErr: () => undefined }); + + expect(() => + command.parse(["--poll-interval", "3000000"], { from: "user" }) + ).toThrow("Must be at most 2147483."); + }); +}); diff --git a/tests/crawl/commands/crawl.test.ts b/tests/crawl/commands/crawl.test.ts new file mode 100644 index 0000000..09c568f --- /dev/null +++ b/tests/crawl/commands/crawl.test.ts @@ -0,0 +1,294 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { AUTH_TYPE } from "../../../src/auth/constants.js"; +import { resolveAuthToken } from "../../../src/auth/services/resolve-token.js"; +import { createCrawlCommand } from "../../../src/crawl/commands/crawl.js"; +import { createDecodoClient } from "../../../src/scrape/services/client.js"; +import { runDataCommand } from "../../cli/helpers/run-data-command.js"; + +vi.mock("../../../src/auth/services/resolve-token.js", () => ({ + resolveAuthToken: vi.fn(), +})); + +vi.mock("../../../src/scrape/services/client.js", () => ({ + createDecodoClient: vi.fn(), +})); + +const PAGES = [{ url: "https://example.com/", content: "home" }]; + +describe("createCrawlCommand", () => { + let exitCode: number | undefined; + let stdout: string[]; + let stderr: string[]; + const api = { + startCrawl: vi.fn(), + getCrawlStatus: vi.fn(), + getCrawlResults: vi.fn(), + cancelCrawl: vi.fn(), + }; + + beforeEach(() => { + exitCode = undefined; + stdout = []; + stderr = []; + for (const fn of Object.values(api)) { + fn.mockReset(); + } + api.startCrawl.mockResolvedValue({ crawl_id: "abc" }); + api.getCrawlResults.mockResolvedValue(PAGES); + Object.defineProperty(process.stdout, "isTTY", { + value: false, + configurable: true, + }); + vi.mocked(resolveAuthToken).mockResolvedValue({ + credential: { type: AUTH_TYPE.API_KEY, value: "key" }, + source: "flag", + }); + vi.mocked(createDecodoClient).mockReturnValue({ + webScrapingApi: api, + } as never); + vi.spyOn(process, "exit").mockImplementation((code) => { + exitCode = code as number; + throw new Error(`process.exit:${code}`); + }); + vi.spyOn(process.stdout, "write").mockImplementation((chunk) => { + stdout.push(String(chunk)); + return true; + }); + vi.spyOn(process.stderr, "write").mockImplementation((chunk) => { + stderr.push(String(chunk)); + return true; + }); + vi.spyOn(console, "error").mockImplementation((message) => { + stderr.push(String(message)); + }); + }); + + afterEach(() => { + vi.restoreAllMocks(); + }); + + it("starts a crawl and prints its id", async () => { + await runDataCommand(createCrawlCommand(), [ + "crawl", + "https://example.com", + "--limit", + "5", + "--max-depth", + "2", + "--content-format", + "html", + ]); + + expect(api.startCrawl).toHaveBeenCalledWith({ + url: "https://example.com", + limit: 5, + max_depth: 2, + output: "html", + }); + expect(api.getCrawlStatus).not.toHaveBeenCalled(); + expect(stdout).toEqual(["abc\n"]); + }); + + it("waits, reports progress and prints pages as NDJSON", async () => { + api.getCrawlStatus + .mockResolvedValueOnce({ + crawl_id: "abc", + status: "pending", + completed: 0, + discovered: 1, + failed: 0, + in_flight: 1, + }) + .mockResolvedValueOnce({ + crawl_id: "abc", + status: "done", + completed: 1, + discovered: 1, + failed: 0, + in_flight: 0, + }); + + await runDataCommand(createCrawlCommand(), [ + "crawl", + "https://example.com", + "--wait", + "--progress", + "--poll-interval", + "1", + ]); + + expect(stderr).toEqual([ + "crawl abc started\n", + "crawl abc pending: 0/1 pages, 0 failed, 1 in flight\n", + "crawl abc done: 1/1 pages, 0 failed, 0 in flight\n", + ]); + expect(api.getCrawlResults).toHaveBeenCalledWith("abc"); + expect(stdout).toEqual([`${JSON.stringify(PAGES[0])}\n`]); + }); + + it("warns when the crawl does not finish cleanly", async () => { + api.getCrawlStatus.mockResolvedValue({ + crawl_id: "abc", + status: "truncated", + reason: "max_duration", + }); + + await runDataCommand(createCrawlCommand(), [ + "crawl", + "https://example.com", + "--wait", + ]); + + expect(stderr).toEqual([ + "crawl abc started\n", + "Warning: crawl abc ended with status truncated (max_duration)", + ]); + expect(stdout).toHaveLength(1); + }); + + it("prints the pages, then exits non-zero when the crawl faults", async () => { + api.getCrawlStatus.mockResolvedValue({ + crawl_id: "abc", + status: "faulted", + }); + + await expect( + runDataCommand(createCrawlCommand(), [ + "crawl", + "https://example.com", + "--wait", + ]) + ).rejects.toThrow("process.exit:7"); + expect(stdout).toEqual([`${JSON.stringify(PAGES[0])}\n`]); + expect(stderr).toEqual([ + "crawl abc started\n", + "Error: Crawl abc ended with status faulted.", + ]); + }); + + it("names the crawl id before a failure while waiting", async () => { + api.getCrawlStatus.mockRejectedValue(new Error("Service unavailable")); + + await expect( + runDataCommand(createCrawlCommand(), [ + "crawl", + "https://example.com", + "--wait", + ]) + ).rejects.toThrow("process.exit:1"); + expect(stderr).toEqual([ + "crawl abc started\n", + "Error: Service unavailable", + ]); + }); + + it("exits 2 when a wait-only flag is used without --wait", async () => { + await expect( + runDataCommand(createCrawlCommand(), [ + "crawl", + "https://example.com", + "--timeout", + "10", + ]) + ).rejects.toThrow("process.exit:2"); + expect(exitCode).toBe(2); + expect(api.startCrawl).not.toHaveBeenCalled(); + }); + + it("prints crawl status", async () => { + api.getCrawlStatus.mockResolvedValue({ crawl_id: "abc", status: "done" }); + + await runDataCommand(createCrawlCommand(), ["crawl", "status", "abc"]); + + expect(api.getCrawlStatus).toHaveBeenCalledWith("abc"); + expect(api.startCrawl).not.toHaveBeenCalled(); + expect(stdout).toEqual(['{"crawl_id":"abc","status":"done"}\n']); + }); + + it("prints crawl results without content", async () => { + await runDataCommand(createCrawlCommand(), [ + "crawl", + "results", + "abc", + "--no-content", + ]); + + expect(api.getCrawlResults).toHaveBeenCalledWith("abc", { content: false }); + expect(stdout).toEqual([`${JSON.stringify(PAGES[0])}\n`]); + }); + + it("waits for a running crawl before printing results", async () => { + api.getCrawlStatus + .mockResolvedValueOnce({ crawl_id: "abc", status: "pending" }) + .mockResolvedValueOnce({ crawl_id: "abc", status: "done" }); + + await runDataCommand(createCrawlCommand(), [ + "crawl", + "results", + "abc", + "--wait", + "--poll-interval", + "1", + ]); + + expect(api.getCrawlStatus).toHaveBeenCalledTimes(2); + expect(api.getCrawlResults).toHaveBeenCalledWith("abc", { content: true }); + expect(stdout).toEqual([`${JSON.stringify(PAGES[0])}\n`]); + }); + + it("prints results, then exits non-zero when the awaited crawl faulted", async () => { + api.getCrawlStatus.mockResolvedValue({ + crawl_id: "abc", + status: "faulted", + }); + + await expect( + runDataCommand(createCrawlCommand(), [ + "crawl", + "results", + "abc", + "--wait", + ]) + ).rejects.toThrow("process.exit:7"); + expect(stdout).toEqual([`${JSON.stringify(PAGES[0])}\n`]); + }); + + it("warns but succeeds when the awaited crawl was cancelled", async () => { + api.getCrawlStatus.mockResolvedValue({ + crawl_id: "abc", + status: "cancelled", + }); + + await runDataCommand(createCrawlCommand(), [ + "crawl", + "results", + "abc", + "--wait", + ]); + expect(exitCode).toBeUndefined(); + expect(stderr).toEqual(["Warning: crawl abc ended with status cancelled"]); + expect(stdout).toEqual([`${JSON.stringify(PAGES[0])}\n`]); + }); + + it("exits 2 when results gets --timeout without --wait", async () => { + await expect( + runDataCommand(createCrawlCommand(), [ + "crawl", + "results", + "abc", + "--timeout", + "10", + ]) + ).rejects.toThrow("process.exit:2"); + expect(api.getCrawlResults).not.toHaveBeenCalled(); + }); + + it("cancels a crawl", async () => { + api.cancelCrawl.mockResolvedValue({ crawl_id: "abc", status: "cancelled" }); + + await runDataCommand(createCrawlCommand(), ["crawl", "cancel", "abc"]); + + expect(api.cancelCrawl).toHaveBeenCalledWith("abc"); + expect(stdout).toEqual(['{"crawl_id":"abc","status":"cancelled"}\n']); + }); +}); diff --git a/tests/crawl/errors/crawl-failed-error.test.ts b/tests/crawl/errors/crawl-failed-error.test.ts new file mode 100644 index 0000000..4f405ae --- /dev/null +++ b/tests/crawl/errors/crawl-failed-error.test.ts @@ -0,0 +1,20 @@ +import { describe, expect, it } from "vitest"; +import { CrawlFailedError } from "../../../src/crawl/errors/crawl-failed-error.js"; + +describe("CrawlFailedError", () => { + it("names the crawl, its status and the reason", () => { + const err = new CrawlFailedError("abc", "faulted", "upstream_error"); + expect(err.message).toBe( + "Crawl abc ended with status faulted (upstream_error)." + ); + expect(err.name).toBe("CrawlFailedError"); + expect(err.crawlId).toBe("abc"); + expect(err.status).toBe("faulted"); + }); + + it("omits the reason when there is none", () => { + expect(new CrawlFailedError("abc", "faulted").message).toBe( + "Crawl abc ended with status faulted." + ); + }); +}); diff --git a/tests/crawl/services/assert-crawl-succeeded.test.ts b/tests/crawl/services/assert-crawl-succeeded.test.ts new file mode 100644 index 0000000..d679af0 --- /dev/null +++ b/tests/crawl/services/assert-crawl-succeeded.test.ts @@ -0,0 +1,24 @@ +import type { GetCrawlStatusResponse } from "@decodo/sdk-ts"; +import { describe, expect, it } from "vitest"; +import { CrawlFailedError } from "../../../src/crawl/errors/crawl-failed-error.js"; +import { assertCrawlSucceeded } from "../../../src/crawl/services/assert-crawl-succeeded.js"; + +function statusOf(status: string): GetCrawlStatusResponse { + return { crawl_id: "abc", status } as GetCrawlStatusResponse; +} + +describe("assertCrawlSucceeded", () => { + it.each([ + "done", + "truncated", + "cancelled", + ])("accepts a %s crawl", (status) => { + expect(() => assertCrawlSucceeded(statusOf(status))).not.toThrow(); + }); + + it("throws for a faulted crawl", () => { + expect(() => assertCrawlSucceeded(statusOf("faulted"))).toThrow( + CrawlFailedError + ); + }); +}); diff --git a/tests/crawl/services/assert-wait-flags.test.ts b/tests/crawl/services/assert-wait-flags.test.ts new file mode 100644 index 0000000..d0b8c13 --- /dev/null +++ b/tests/crawl/services/assert-wait-flags.test.ts @@ -0,0 +1,40 @@ +import { describe, expect, it } from "vitest"; +import { CRAWL_START_WAIT_ONLY_FLAGS } from "../../../src/crawl/constants.js"; +import { assertWaitFlags } from "../../../src/crawl/services/assert-wait-flags.js"; +import { CliUsageError } from "../../../src/platform/errors/cli-usage-error.js"; + +describe("assertWaitFlags", () => { + it("allows wait-only flags with --wait", () => { + expect(() => + assertWaitFlags( + { + wait: true, + progress: true, + pollInterval: 2, + timeout: 30, + output: "pages.ndjson", + }, + CRAWL_START_WAIT_ONLY_FLAGS + ) + ).not.toThrow(); + }); + + it("allows plain crawls", () => { + expect(() => assertWaitFlags({})).not.toThrow(); + }); + + it("allows --output without --wait by default", () => { + expect(() => assertWaitFlags({ output: "pages.ndjson" })).not.toThrow(); + }); + + it.each([ + [{ progress: true }, "--progress requires --wait."], + [{ pollInterval: 2 }, "--poll-interval requires --wait."], + [{ timeout: 30 }, "--timeout requires --wait."], + [{ output: "pages.ndjson" }, "--output requires --wait."], + ])("rejects %j without --wait", (options, message) => { + expect(() => assertWaitFlags(options, CRAWL_START_WAIT_ONLY_FLAGS)).toThrow( + new CliUsageError(message) + ); + }); +}); diff --git a/tests/crawl/services/build-crawl-request.test.ts b/tests/crawl/services/build-crawl-request.test.ts new file mode 100644 index 0000000..c960892 --- /dev/null +++ b/tests/crawl/services/build-crawl-request.test.ts @@ -0,0 +1,29 @@ +import { describe, expect, it } from "vitest"; +import { buildCrawlRequest } from "../../../src/crawl/services/build-crawl-request.js"; + +describe("buildCrawlRequest", () => { + it("adds crawl-only fields to the site request", () => { + expect( + buildCrawlRequest("https://example.com", { + limit: 5, + maxDepth: 2, + maxDuration: 60, + contentFormat: "html", + wait: true, + output: "pages.ndjson", + }) + ).toEqual({ + url: "https://example.com", + limit: 5, + max_depth: 2, + max_duration: 60, + output: "html", + }); + }); + + it("sends only the url when no options are set", () => { + expect(buildCrawlRequest("https://example.com", {})).toEqual({ + url: "https://example.com", + }); + }); +}); diff --git a/tests/crawl/services/format-crawl-progress.test.ts b/tests/crawl/services/format-crawl-progress.test.ts new file mode 100644 index 0000000..9d8826f --- /dev/null +++ b/tests/crawl/services/format-crawl-progress.test.ts @@ -0,0 +1,20 @@ +import type { GetCrawlStatusResponse } from "@decodo/sdk-ts"; +import { describe, expect, it } from "vitest"; +import { formatCrawlProgress } from "../../../src/crawl/services/format-crawl-progress.js"; + +describe("formatCrawlProgress", () => { + it("summarises page counts", () => { + const status = { + crawl_id: "abc", + status: "pending", + completed: 3, + discovered: 10, + failed: 1, + in_flight: 2, + } as GetCrawlStatusResponse; + + expect(formatCrawlProgress(status)).toBe( + "crawl abc pending: 3/10 pages, 1 failed, 2 in flight" + ); + }); +}); diff --git a/tests/crawl/services/parse-poll-interval.test.ts b/tests/crawl/services/parse-poll-interval.test.ts new file mode 100644 index 0000000..c7945c4 --- /dev/null +++ b/tests/crawl/services/parse-poll-interval.test.ts @@ -0,0 +1,19 @@ +import { InvalidArgumentError } from "commander"; +import { describe, expect, it } from "vitest"; +import { parsePollInterval } from "../../../src/crawl/services/parse-poll-interval.js"; + +describe("parsePollInterval", () => { + it("accepts the largest interval a timer can hold", () => { + expect(parsePollInterval("2147483")).toBe(2_147_483); + }); + + it("rejects an interval a timer cannot hold", () => { + expect(() => parsePollInterval("2147484")).toThrow( + new InvalidArgumentError("Must be at most 2147483.") + ); + }); + + it("rejects a non-positive interval", () => { + expect(() => parsePollInterval("0")).toThrow(InvalidArgumentError); + }); +}); diff --git a/tests/crawl/services/settle-crawl.test.ts b/tests/crawl/services/settle-crawl.test.ts new file mode 100644 index 0000000..1e408dc --- /dev/null +++ b/tests/crawl/services/settle-crawl.test.ts @@ -0,0 +1,68 @@ +import type { WebScrapingApi } from "@decodo/sdk-ts"; +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { settleCrawl } from "../../../src/crawl/services/settle-crawl.js"; + +const DONE = { + crawl_id: "abc", + status: "done", + completed: 1, + discovered: 1, + failed: 0, + in_flight: 0, +}; + +describe("settleCrawl", () => { + let stderr: string[]; + + beforeEach(() => { + stderr = []; + vi.spyOn(process.stderr, "write").mockImplementation((chunk) => { + stderr.push(String(chunk)); + return true; + }); + vi.spyOn(console, "error").mockImplementation((message) => { + stderr.push(String(message)); + }); + }); + + afterEach(() => { + vi.restoreAllMocks(); + }); + + function apiWith(status: object): WebScrapingApi { + return { + getCrawlStatus: vi.fn().mockResolvedValue(status), + } as unknown as WebScrapingApi; + } + + it("stays quiet without --progress", async () => { + await settleCrawl(apiWith(DONE), "abc", {}); + expect(stderr).toEqual([]); + }); + + it("prints progress lines with --progress", async () => { + await settleCrawl(apiWith(DONE), "abc", { progress: true }); + expect(stderr).toEqual([ + "crawl abc done: 1/1 pages, 0 failed, 0 in flight\n", + ]); + }); + + it("returns the final status", async () => { + const faulted = { ...DONE, status: "faulted" }; + await expect(settleCrawl(apiWith(faulted), "abc", {})).resolves.toEqual( + faulted + ); + expect(stderr).toEqual([]); + }); + + it("warns when the crawl is truncated", async () => { + await settleCrawl( + apiWith({ ...DONE, status: "truncated", reason: "max_duration" }), + "abc", + {} + ); + expect(stderr).toEqual([ + "Warning: crawl abc ended with status truncated (max_duration)", + ]); + }); +}); diff --git a/tests/crawl/services/wait-for-crawl.test.ts b/tests/crawl/services/wait-for-crawl.test.ts new file mode 100644 index 0000000..3588e5d --- /dev/null +++ b/tests/crawl/services/wait-for-crawl.test.ts @@ -0,0 +1,63 @@ +import { TimeoutError, type WebScrapingApi } from "@decodo/sdk-ts"; +import { describe, expect, it, vi } from "vitest"; +import { waitForCrawl } from "../../../src/crawl/services/wait-for-crawl.js"; + +function apiReturning(...statuses: string[]): WebScrapingApi { + const getCrawlStatus = vi.fn(); + for (const status of statuses) { + getCrawlStatus.mockResolvedValueOnce({ crawl_id: "abc", status }); + } + return { getCrawlStatus } as unknown as WebScrapingApi; +} + +describe("waitForCrawl", () => { + it("polls until the crawl settles and reports each status", async () => { + const api = apiReturning("pending", "pending", "done"); + const onStatus = vi.fn(); + + const status = await waitForCrawl(api, "abc", { + pollIntervalMs: 1, + timeoutMs: 1000, + onStatus, + }); + + expect(status.status).toBe("done"); + expect(api.getCrawlStatus).toHaveBeenCalledTimes(3); + expect(onStatus).toHaveBeenCalledTimes(3); + }); + + it("returns a settled status without waiting", async () => { + const api = apiReturning("cancelled"); + + const status = await waitForCrawl(api, "abc", { + pollIntervalMs: 1000, + timeoutMs: 1, + }); + + expect(status.status).toBe("cancelled"); + }); + + it("uses the time left before the deadline instead of giving up early", async () => { + const api = apiReturning("pending", "done"); + + const status = await waitForCrawl(api, "abc", { + pollIntervalMs: 60_000, + timeoutMs: 20, + }); + + expect(status.status).toBe("done"); + expect(api.getCrawlStatus).toHaveBeenCalledTimes(2); + }); + + it("throws a TimeoutError naming the cancel command", async () => { + const api = apiReturning("pending", "pending"); + + await expect( + waitForCrawl(api, "abc", { pollIntervalMs: 2000, timeoutMs: 1000 }) + ).rejects.toThrow( + new TimeoutError( + "Crawl abc did not finish within 1s. It is still running; run `decodo crawl cancel abc` to stop it." + ) + ); + }); +}); diff --git a/tests/crawl/services/write-crawl-pages.test.ts b/tests/crawl/services/write-crawl-pages.test.ts new file mode 100644 index 0000000..d6d683e --- /dev/null +++ b/tests/crawl/services/write-crawl-pages.test.ts @@ -0,0 +1,49 @@ +import { mkdtempSync, readFileSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import type { CrawlPage } from "@decodo/sdk-ts"; +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { writeCrawlPages } from "../../../src/crawl/services/write-crawl-pages.js"; + +const PAGES = [ + { url: "https://example.com/", content: "home" }, + { url: "https://example.com/about", content: "about" }, +] as CrawlPage[]; + +const NDJSON = + '{"url":"https://example.com/","content":"home"}\n' + + '{"url":"https://example.com/about","content":"about"}\n'; + +describe("writeCrawlPages", () => { + let written: string | undefined; + + beforeEach(() => { + written = undefined; + vi.spyOn(process.stdout, "write").mockImplementation((chunk) => { + written = String(chunk); + return true; + }); + }); + + afterEach(() => { + vi.restoreAllMocks(); + }); + + it("writes one page per line to stdout", () => { + writeCrawlPages(PAGES); + expect(written).toBe(NDJSON); + }); + + it("writes NDJSON to a file with -o", () => { + const dir = mkdtempSync(join(tmpdir(), "decodo-crawl-")); + const path = join(dir, "pages.ndjson"); + + try { + writeCrawlPages(PAGES, path); + expect(readFileSync(path, "utf8")).toBe(NDJSON); + expect(written).toBeUndefined(); + } finally { + rmSync(dir, { recursive: true, force: true }); + } + }); +}); diff --git a/tests/docs/readme-commands.test.ts b/tests/docs/readme-commands.test.ts index dd8d361..379191a 100644 --- a/tests/docs/readme-commands.test.ts +++ b/tests/docs/readme-commands.test.ts @@ -3,14 +3,16 @@ import { fileURLToPath } from "node:url"; import { BundledSchema } from "@decodo/sdk-ts"; import { Command } from "commander"; import { describe, expect, it } from "vitest"; -import { resetCommand } from "../../src/auth/commands/reset.js"; -import { setupCommand } from "../../src/auth/commands/setup.js"; -import { whoamiCommand } from "../../src/auth/commands/whoami.js"; +import { createResetCommand } from "../../src/auth/commands/reset.js"; +import { createSetupCommand } from "../../src/auth/commands/setup.js"; +import { createWhoamiCommand } from "../../src/auth/commands/whoami.js"; +import { createCrawlCommand } from "../../src/crawl/commands/crawl.js"; +import { createMapCommand } from "../../src/map/commands/map.js"; import { createCodegenTargetCommands } from "../../src/scrape/commands/codegen-target-commands.js"; import { createListTargetsCommand } from "../../src/scrape/commands/list-targets.js"; import { createScrapeCommand } from "../../src/scrape/commands/scrape.js"; import { createScreenshotCommand } from "../../src/scrape/commands/screenshot.js"; -import { createSearchCommand } from "../../src/scrape/commands/search.js"; +import { createSearchCommand } from "../../src/search/commands/search.js"; const README_PATH = fileURLToPath(new URL("../../README.md", import.meta.url)); @@ -34,11 +36,13 @@ function buildProgram(): Command { .option("--token "); for (const command of [ - setupCommand, - resetCommand, - whoamiCommand, + createSetupCommand(), + createResetCommand(), + createWhoamiCommand(), createScrapeCommand(schema), - createSearchCommand(schema), + createSearchCommand(), + createMapCommand(), + createCrawlCommand(), createScreenshotCommand(schema), createListTargetsCommand(schema), ...createCodegenTargetCommands(schema), @@ -127,6 +131,12 @@ function knownFlags(command: Command, program: Command): Set { return flags; } +function findCommand(parent: Command, name: string): Command | undefined { + return parent.commands.find( + (cmd) => cmd.name() === name || cmd.aliases().includes(name) + ); +} + function resolveSubcommand( args: string[], program: Command @@ -136,10 +146,18 @@ function resolveSubcommand( return { command: program, name: "decodo" }; } - const command = program.commands.find( - (cmd) => cmd.name() === name || cmd.aliases().includes(name) - ); - return command ? { command, name } : undefined; + const command = findCommand(program, name); + if (!command) { + return; + } + + const subName = args + .slice(args.indexOf(name) + 1) + .find((arg) => !arg.startsWith("-")); + const subcommand = subName ? findCommand(command, subName) : undefined; + return subcommand + ? { command: subcommand, name: `${name} ${subName}` } + : { command, name }; } describe("README command examples", () => { diff --git a/tests/map/commands/attach-site-options.test.ts b/tests/map/commands/attach-site-options.test.ts new file mode 100644 index 0000000..18cd70b --- /dev/null +++ b/tests/map/commands/attach-site-options.test.ts @@ -0,0 +1,48 @@ +import { Command } from "commander"; +import { describe, expect, it } from "vitest"; +import { attachSiteOptions } from "../../../src/map/commands/attach-site-options.js"; + +describe("attachSiteOptions", () => { + it("parses shared site flags", () => { + const command = attachSiteOptions(new Command("probe")).exitOverride(); + + command.parse( + [ + "--limit", + "3", + "--sitemap", + "skip", + "--domain-filter", + "domain", + "--geo", + "us", + "--locale", + "en-US", + "--exclude-paths", + "/a/*", + "--exclude-paths", + "/b/*", + ], + { from: "user" } + ); + + expect(command.opts()).toEqual({ + limit: 3, + sitemap: "skip", + domainFilter: "domain", + geo: "us", + locale: "en-US", + excludePaths: ["/a/*", "/b/*"], + }); + }); + + it("rejects unknown sitemap modes", () => { + const command = attachSiteOptions(new Command("probe")) + .exitOverride() + .configureOutput({ writeErr: () => undefined }); + + expect(() => + command.parse(["--sitemap", "always"], { from: "user" }) + ).toThrow(); + }); +}); diff --git a/tests/map/commands/map.test.ts b/tests/map/commands/map.test.ts new file mode 100644 index 0000000..085e2fb --- /dev/null +++ b/tests/map/commands/map.test.ts @@ -0,0 +1,100 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { AUTH_TYPE } from "../../../src/auth/constants.js"; +import { resolveAuthToken } from "../../../src/auth/services/resolve-token.js"; +import { createMapCommand } from "../../../src/map/commands/map.js"; +import { createDecodoClient } from "../../../src/scrape/services/client.js"; +import { runDataCommand } from "../../cli/helpers/run-data-command.js"; + +vi.mock("../../../src/auth/services/resolve-token.js", () => ({ + resolveAuthToken: vi.fn(), +})); + +vi.mock("../../../src/scrape/services/client.js", () => ({ + createDecodoClient: vi.fn(), +})); + +const RESPONSE = { + results: [ + { + content: { + links: [ + { url: "https://example.com/" }, + { url: "https://example.com/a" }, + ], + }, + }, + ], +}; + +describe("createMapCommand", () => { + let stdout: string | undefined; + const map = vi.fn(); + + beforeEach(() => { + stdout = undefined; + map.mockReset().mockResolvedValue(RESPONSE); + Object.defineProperty(process.stdout, "isTTY", { + value: false, + configurable: true, + }); + vi.mocked(resolveAuthToken).mockResolvedValue({ + credential: { type: AUTH_TYPE.API_KEY, value: "key" }, + source: "flag", + }); + vi.mocked(createDecodoClient).mockReturnValue({ + webScrapingApi: { map }, + } as never); + vi.spyOn(process.stdout, "write").mockImplementation((chunk) => { + stdout = String(chunk); + return true; + }); + }); + + afterEach(() => { + vi.restoreAllMocks(); + }); + + it("prints one URL per line", async () => { + await runDataCommand(createMapCommand(), ["map", "https://example.com"]); + + expect(map).toHaveBeenCalledWith({ url: "https://example.com" }); + expect(stdout).toBe("https://example.com/\nhttps://example.com/a\n"); + }); + + it("maps flags to the request", async () => { + await runDataCommand(createMapCommand(), [ + "map", + "https://example.com", + "--limit", + "50", + "--sitemap", + "only", + "--domain-filter", + "all", + "--select-paths", + "/blog/*", + "--select-paths", + "/docs/*", + "--js-rendering", + ]); + + expect(map).toHaveBeenCalledWith({ + url: "https://example.com", + limit: 50, + sitemap: "only", + domain_filter: "all", + select_paths: ["/blog/*", "/docs/*"], + JS_rendering: true, + }); + }); + + it("prints the full response with --full", async () => { + await runDataCommand(createMapCommand(), [ + "map", + "https://example.com", + "--full", + ]); + + expect(stdout).toBe(`${JSON.stringify(RESPONSE)}\n`); + }); +}); diff --git a/tests/map/services/build-site-request.test.ts b/tests/map/services/build-site-request.test.ts new file mode 100644 index 0000000..ad6441d --- /dev/null +++ b/tests/map/services/build-site-request.test.ts @@ -0,0 +1,33 @@ +import { describe, expect, it } from "vitest"; +import { buildSiteRequest } from "../../../src/map/services/build-site-request.js"; + +describe("buildSiteRequest", () => { + it("maps options to snake_case request fields", () => { + expect( + buildSiteRequest("https://example.com", { + limit: 10, + sitemap: "skip", + domainFilter: "subdomain", + geo: "us", + locale: "en-US", + selectPaths: ["/blog/*"], + excludePaths: ["/tag/*"], + }) + ).toEqual({ + url: "https://example.com", + limit: 10, + sitemap: "skip", + domain_filter: "subdomain", + geo: "us", + locale: "en-US", + select_paths: ["/blog/*"], + exclude_paths: ["/tag/*"], + }); + }); + + it("omits unset options", () => { + expect(buildSiteRequest("https://example.com", {})).toEqual({ + url: "https://example.com", + }); + }); +}); diff --git a/tests/output/services/write-json-output.test.ts b/tests/output/services/write-json-output.test.ts new file mode 100644 index 0000000..c01cf37 --- /dev/null +++ b/tests/output/services/write-json-output.test.ts @@ -0,0 +1,48 @@ +import { mkdtempSync, readFileSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { writeJsonOutput } from "../../../src/output/services/write-json-output.js"; + +describe("writeJsonOutput", () => { + let written: string | undefined; + + beforeEach(() => { + written = undefined; + Object.defineProperty(process.stdout, "isTTY", { + value: false, + configurable: true, + }); + vi.spyOn(process.stdout, "write").mockImplementation((chunk) => { + written = String(chunk); + return true; + }); + }); + + afterEach(() => { + vi.restoreAllMocks(); + }); + + it("writes compact JSON when piped", () => { + writeJsonOutput({ ok: true }, {}); + expect(written).toBe('{"ok":true}\n'); + }); + + it("indents JSON with --pretty", () => { + writeJsonOutput({ ok: true }, { pretty: true }); + expect(written).toBe('{\n "ok": true\n}\n'); + }); + + it("writes to a file with -o", () => { + const dir = mkdtempSync(join(tmpdir(), "decodo-json-")); + const path = join(dir, "out.json"); + + try { + writeJsonOutput({ ok: true }, { output: path }); + expect(readFileSync(path, "utf8")).toBe('{"ok":true}'); + expect(written).toBeUndefined(); + } finally { + rmSync(dir, { recursive: true, force: true }); + } + }); +}); diff --git a/tests/platform/services/handle-cli-error.test.ts b/tests/platform/services/handle-cli-error.test.ts index b0ca8b1..04c5c22 100644 --- a/tests/platform/services/handle-cli-error.test.ts +++ b/tests/platform/services/handle-cli-error.test.ts @@ -6,7 +6,9 @@ import { ValidationError, } from "@decodo/sdk-ts"; import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { ApiKeyRequiredError } from "../../../src/auth/errors/api-key-required-error.js"; import { AuthRequiredError } from "../../../src/auth/errors/auth-required-error.js"; +import { CrawlFailedError } from "../../../src/crawl/errors/crawl-failed-error.js"; import { ScrapeFailedError } from "../../../src/output/errors/scrape-failed-error.js"; import { CliUsageError } from "../../../src/platform/errors/cli-usage-error.js"; import { handleCliError } from "../../../src/platform/services/handle-cli-error.js"; @@ -44,6 +46,16 @@ describe("handleCliError", () => { expect(stderr.join("\n")).toContain("DECODO_AUTH_TOKEN"); }); + it("maps API-key-required errors to exit code 3 with an API key hint", () => { + expect(() => handleCliError(new ApiKeyRequiredError())).toThrow( + "process.exit:3" + ); + + expect(exitCode).toBe(3); + expect(stderr.join("\n")).toContain("Web Data API key"); + expect(stderr.join("\n")).toContain("decodo setup"); + }); + it("maps SDK authentication errors to exit code 3 with setup hint", () => { const err = new AuthenticationError("Unauthorized"); @@ -111,6 +123,13 @@ describe("handleCliError", () => { expect(stderr.join("\n")).toContain("Trends request was rejected"); }); + it("maps crawl failures to exit code 7", () => { + const err = new CrawlFailedError("abc", "faulted"); + + expect(() => handleCliError(err)).toThrow("process.exit:7"); + expect(stderr.join("\n")).toContain("Crawl abc ended with status faulted."); + }); + it("maps syscall-coded network failures in the cause chain to exit code 7", () => { const cause = Object.assign( new Error("getaddrinfo ENOTFOUND api.decodo.com"), diff --git a/tests/scrape/commands/search.test.ts b/tests/scrape/commands/search.test.ts deleted file mode 100644 index 397b208..0000000 --- a/tests/scrape/commands/search.test.ts +++ /dev/null @@ -1,226 +0,0 @@ -import { BundledSchema, ValidationError } from "@decodo/sdk-ts"; -import { Command } from "commander"; -import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; -import { AUTH_TYPE } from "../../../src/auth/constants.js"; -import { resolveAuthToken } from "../../../src/auth/services/resolve-token.js"; -import { createSearchCommand } from "../../../src/scrape/commands/search.js"; -import { createDecodoClient } from "../../../src/scrape/services/client.js"; - -vi.mock("../../../src/auth/services/resolve-token.js", () => ({ - resolveAuthToken: vi.fn(), -})); - -vi.mock("../../../src/scrape/services/client.js", () => ({ - createDecodoClient: vi.fn(), -})); - -describe("createSearchCommand", () => { - let exitCode: number | undefined; - let stdout: string | undefined; - - beforeEach(() => { - exitCode = undefined; - stdout = undefined; - - Object.defineProperty(process.stdout, "isTTY", { - value: false, - configurable: true, - }); - - vi.mocked(resolveAuthToken).mockResolvedValue({ - credential: { type: AUTH_TYPE.TOKEN, value: "test-token" }, - source: "flag", - }); - vi.spyOn(process, "exit").mockImplementation((code) => { - exitCode = code as number; - throw new Error(`process.exit:${code}`); - }); - vi.spyOn(process.stdout, "write").mockImplementation((chunk) => { - stdout = String(chunk); - return true; - }); - vi.spyOn(console, "error").mockImplementation(vi.fn()); - }); - - afterEach(() => { - vi.restoreAllMocks(); - }); - - it("posts tier-1 search body and prints JSON response", async () => { - const scrape = vi.fn().mockResolvedValue({ - results: [{ content: { ok: true } }], - }); - vi.mocked(createDecodoClient).mockReturnValue({ - webScrapingApi: { scrape }, - } as never); - - const program = new Command() - .option("--token ") - .addCommand(createSearchCommand(BundledSchema.shared)); - - await program.parseAsync(["search", "coffee", "--token", "test-token"], { - from: "user", - }); - - expect(scrape).toHaveBeenCalledWith({ - target: "google_search", - query: "coffee", - parse: true, - markdown: false, - }); - expect(stdout).toBe('{"ok":true}\n'); - }); - - it("maps --engine bing to bing_search target", async () => { - const scrape = vi.fn().mockResolvedValue({ - results: [{ content: "ok" }], - }); - vi.mocked(createDecodoClient).mockReturnValue({ - webScrapingApi: { scrape }, - } as never); - - const program = new Command() - .option("--token ") - .addCommand(createSearchCommand(BundledSchema.shared)); - - await program.parseAsync( - ["search", "coffee", "--engine", "bing", "--token", "test-token"], - { from: "user" } - ); - - expect(scrape).toHaveBeenCalledWith({ - target: "bing_search", - query: "coffee", - parse: true, - markdown: false, - }); - }); - - it("maps --geo and --limit to geo and page_count", async () => { - const scrape = vi.fn().mockResolvedValue({ - results: [{ content: "ok" }], - }); - vi.mocked(createDecodoClient).mockReturnValue({ - webScrapingApi: { scrape }, - } as never); - - const program = new Command() - .option("--token ") - .addCommand(createSearchCommand(BundledSchema.shared)); - - await program.parseAsync( - [ - "search", - "coffee", - "--geo", - "us", - "--limit", - "3", - "--token", - "test-token", - ], - { from: "user" } - ); - - expect(scrape).toHaveBeenCalledWith({ - target: "google_search", - query: "coffee", - geo: "us", - page_count: 3, - parse: true, - markdown: false, - }); - }); - - it("resolves --target kebab-case override", async () => { - const scrape = vi.fn().mockResolvedValue({ - results: [{ content: "ok" }], - }); - vi.mocked(createDecodoClient).mockReturnValue({ - webScrapingApi: { scrape }, - } as never); - - const program = new Command() - .option("--token ") - .addCommand(createSearchCommand(BundledSchema.shared)); - - await program.parseAsync( - [ - "search", - "coffee", - "--engine", - "bing", - "--target", - "google-search", - "--token", - "test-token", - ], - { from: "user" } - ); - - expect(scrape).toHaveBeenCalledWith({ - target: "google_search", - query: "coffee", - parse: true, - markdown: false, - }); - }); - - it("maps validation errors to validation exit code", async () => { - const scrape = vi - .fn() - .mockRejectedValue(new ValidationError("invalid params")); - vi.mocked(createDecodoClient).mockReturnValue({ - webScrapingApi: { scrape }, - } as never); - - const program = new Command() - .option("--token ") - .addCommand(createSearchCommand(BundledSchema.shared)); - - await expect( - program.parseAsync(["search", "coffee", "--token", "test-token"], { - from: "user", - }) - ).rejects.toThrow("process.exit:4"); - - expect(exitCode).toBe(4); - }); - - it("maps unknown target to validation exit code", async () => { - const program = new Command() - .option("--token ") - .addCommand(createSearchCommand(BundledSchema.shared)); - - await expect( - program.parseAsync( - [ - "search", - "coffee", - "--target", - "not-a-target", - "--token", - "test-token", - ], - { from: "user" } - ) - ).rejects.toThrow("process.exit:4"); - - expect(exitCode).toBe(4); - }); - - it("maps out-of-range --limit to validation exit code", async () => { - const program = new Command() - .option("--token ") - .addCommand(createSearchCommand(BundledSchema.shared)); - - await expect( - program.parseAsync( - ["search", "coffee", "--limit", "11", "--token", "test-token"], - { from: "user" } - ) - ).rejects.toThrow("process.exit:4"); - - expect(exitCode).toBe(4); - }); -}); diff --git a/tests/search/commands/search.test.ts b/tests/search/commands/search.test.ts new file mode 100644 index 0000000..084bcb0 --- /dev/null +++ b/tests/search/commands/search.test.ts @@ -0,0 +1,99 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { AUTH_TYPE } from "../../../src/auth/constants.js"; +import { resolveAuthToken } from "../../../src/auth/services/resolve-token.js"; +import { createDecodoClient } from "../../../src/scrape/services/client.js"; +import { createSearchCommand } from "../../../src/search/commands/search.js"; +import { runDataCommand } from "../../cli/helpers/run-data-command.js"; + +vi.mock("../../../src/auth/services/resolve-token.js", () => ({ + resolveAuthToken: vi.fn(), +})); + +vi.mock("../../../src/scrape/services/client.js", () => ({ + createDecodoClient: vi.fn(), +})); + +describe("createSearchCommand", () => { + let exitCode: number | undefined; + let stdout: string | undefined; + let stderr: string[]; + const search = vi.fn(); + + beforeEach(() => { + exitCode = undefined; + stdout = undefined; + stderr = []; + search.mockReset(); + Object.defineProperty(process.stdout, "isTTY", { + value: false, + configurable: true, + }); + vi.mocked(resolveAuthToken).mockResolvedValue({ + credential: { type: AUTH_TYPE.API_KEY, value: "key" }, + source: "flag", + }); + vi.mocked(createDecodoClient).mockReturnValue({ + webScrapingApi: { search }, + } as never); + vi.spyOn(process, "exit").mockImplementation((code) => { + exitCode = code as number; + throw new Error(`process.exit:${code}`); + }); + vi.spyOn(process.stdout, "write").mockImplementation((chunk) => { + stdout = String(chunk); + return true; + }); + vi.spyOn(console, "error").mockImplementation((message) => { + stderr.push(String(message)); + }); + }); + + afterEach(() => { + vi.restoreAllMocks(); + }); + + it("calls the data API search and prints the response", async () => { + search.mockResolvedValue({ status: "done", organic: [] }); + + await runDataCommand(createSearchCommand(), [ + "search", + "coffee", + "--geo", + "de", + "--locale", + "de-DE", + "--page", + "2", + ]); + + expect(search).toHaveBeenCalledWith({ + query: "coffee", + geo: "de", + locale: "de-DE", + page: 2, + }); + expect(stdout).toBe('{"status":"done","organic":[]}\n'); + }); + + it("sends only the query by default", async () => { + search.mockResolvedValue({ status: "done" }); + + await runDataCommand(createSearchCommand(), ["search", "coffee"]); + + expect(search).toHaveBeenCalledWith({ query: "coffee" }); + }); + + it("exits 7 when the search fails", async () => { + search.mockResolvedValue({ + status: "failed", + status_code: 502, + message: "upstream failed", + }); + + await expect( + runDataCommand(createSearchCommand(), ["search", "coffee"]) + ).rejects.toThrow("process.exit:7"); + expect(exitCode).toBe(7); + expect(stderr[0]).toBe("Error: upstream failed"); + }); +});