[CyMule]

What

Jobs the new preflight check rejects no longer break the SDK. Workflows can also opt out of preflight from the SDK.

Before, after a job is rejected:

client.jobs.list_jobs(...)   → ValidationError: 'REJECTED' is not a valid JobStatus   (whole page lost)
client.jobs.get_job(...)     → ValidationError

After:

job.status   → JobStatus.REJECTED
job.status   → "SOME_FUTURE_STATUS"   # a status this SDK version doesn't know comes back as a plain str instead of raising

details.node_stats[0].failure_category → PreflightFailureCategory.AUTH_INVALID_CREDENTIAL
details.node_stats[0].failure_reason   → "The AWS Access Key Id you provided does not exist."

• JobStatus and JobProcessingStatus gain REJECTED. JobInformation.status and JobDetails.processing_status also accept unknown values. Known values are still enum members, so job.status == JobStatus.COMPLETED and .value behave as before.
• JobNodeDetails gains failure_category (Union[PreflightFailureCategory, str]) and failure_reason, so a caller can see why a job was rejected. The server sets them only on the node that failed preflight; before this, pydantic dropped them.
• skip_preflight is now on CreateWorkflow, UpdateWorkflow (unset = unchanged) and WorkflowInformation. For a one-off create_job, the API defines the flag inside the request_data JSON. The docs now say to set it there; the SDK does not rewrite that string.
• Releases as 0.47.0.

Why

The SDK models used to be regenerated from the API spec every night. That generator has been dead since January (removed in #349). Since then, new server values reach the SDK only when someone adds them by hand. REJECTED didn't get added, and a closed enum turns a new value into a hard failure.

This PR also adds a way to catch that kind of drift before a customer does (DTPL-696 acceptance criterion 4):

weekly CI → fetch live OpenAPI specs → compare with SDK models → new missing field/enum value → open/update a GitHub issue
                                                                    (a failed download fails the run but opens no issue)

• make check-spec-drift runs the check locally. Gaps we already know about are recorded in scripts/spec_drift_baseline.json (20 schemas today, e.g. JobInformation.processing_status, the teradata connector type). The check fails only on new drift, or on a baseline entry that has since been covered.
• CONTRIBUTING.md documents how to resolve a drift issue.

Replaces #353, which got here through a custom _missing_ enum hook and code that spliced bytes into request_data. This version uses a plain pydantic union and leaves request_data alone.

Notes for consumers

• Type checkers now see these two status fields as Union[JobStatus, str] / Union[JobProcessingStatus, str]. Code that reads .value or .name on them needs to handle the str case. Runtime behavior for known statuses is unchanged.
• Follow-up, not in this PR: the SDK's other "open" enums (connector types, partition Strategy, etc.) use OpenEnumMeta / validate_open_enum. On Python 3.12+ those return a raw str even for known values, and they emit pydantic serializer warnings. The long-term fix is to move them to the same union pattern used here. That changes their runtime types, so it belongs in its own PR.

Validation

• make test-unit: 419 passed. make test-contract: 72 passed. The new job/workflow contract tests pass on Python 3.11, 3.12 and 3.13, and they fail without the model changes.
• make lint is clean. make check-spec-drift passes against the live specs.
• Two rounds of Codex review. Round 1 found that enums declared inline on a property escaped the drift check; that is fixed and has a test. Round 2 was clean.
• cubic: three findings fixed (no drift issue on a fetch failure, no duplicate issues from overlapping runs, changelog wording). Re-review clean, and full CI is green, including integration on 3.11–3.13.

Maintenance validation (2026-10-19, keepalive base refresh)

• Synchronized main through its 0.46.5 keepalive release while retaining the authored 0.47.0 version and user-agent. Manual conflict resolution touched only the changelog and version file; main runtime and test changes merged automatically.
• Local validation: 419 unit tests and 71 contract tests passed (one existing expected unit failure). Four localhost socket tests needed bind permission and passed without source changes.
• Exact maintenance-head CI: https://github.com/Unstructured-IO/unstructured-python-client/actions/runs/39329146842.

Review follow-up (2026-10-20)

• Added failure_category / failure_reason to JobNodeDetails with a new PreflightFailureCategory enum, and dropped both from the drift baseline. A new contract test covers a known category, an unknown category (plain str) and a passing node (both null); it passes on 3.11, 3.12 and 3.13.
• Removed the JobStatus / JobProcessingStatus entries from overlay_client.yaml. Nothing reads the overlay since the generator was removed, and x-speakeasy-unknown-values would describe the OpenEnumMeta style, not the union shipped here.
• make test-unit 419 passed, make test-contract 72 passed, make lint clean, make check-spec-drift clean against the live specs.

[cubic-dev-ai]

All reported issues were addressed across 26 files

<sub>Shadow auto-approve: would not auto-approve because issues were found.

View guided diff | Re-trigger cubic</sub>

[cubic-dev-ai]

0 issues found across 2 files (changes from recent commits).

<sub>Shadow auto-approve: would not auto-approve. This PR does not meet the repository auto-approval settings.

View guided diff | Re-trigger cubic</sub>

[paulkarayan]

Approving with one real ask and two nits.

The contract checks out against platform-api main and the live spec: REJECTED is spelled the same on both enums, create treats null/false as one state, update treats null as unchanged, and the SDK only puts skip_preflight on the wire when it's set. Reverting the model files makes the new contract tests fail, so they're testing the change. 419 unit / 71 contract pass locally on the head.

The ask: a caller who gets JobStatus.REJECTED still can't find out why. The server only puts the reason on each node of /details (failure_category, failure_reason), JobNodeDetails doesn't have either field, and pydantic drops them. JobDetails.message is only set when that detail is missing. So we fixed the crash, but the reason still isn't reachable. I'd add both fields to JobNodeDetails here (failure_category as Union[PreflightFailureCategory, UnrecognizedStr], same pattern as status) and drop them from the baseline. If you'd rather do it as a follow-up, say so in the body, because right now that gap only shows up in spec_drift_baseline.json.

Nits:

1. The body still says the 0.46.3 split-PDF fix is folded into 0.47.0, and that's no longer true now that main ships 0.46.3 on its own. The first Validation block also still says 324/74.
2. The overlay_client.yaml entries for JobStatus/JobProcessingStatus are dead now that the generator is gone, and x-speakeasy-unknown-values would give the OpenEnumMeta style rather than the Union you actually shipped. I'd drop them.

Withdrawing my approval.

[cubic-dev-ai]

All reported issues were addressed across 9 files (changes from recent commits).

<sub>Shadow auto-approve: would not auto-approve because issues were found.
Tip: Review your code locally with the cubic CLI to iterate faster.

View guided diff | Re-trigger cubic</sub>

[cragwolfe]

Reviewed exact head 4aeb58adb1d4d6c9da34684b50cafeb55dfb62ae with GPT Pro. No actionable defects found.

Known job statuses retain their enum identity; future values remain strings. Rejection diagnostics and workflow skip_preflight serialization have meaningful contract coverage. The prior inline-enum drift, fetch-failure reporting, and overlapping-run findings are addressed in the current source.

All 20 reported checks on this head passed. The scheduled spec-drift workflow has no execution result in that rollup. No local tests were run.

(authored by codex)