fix: stop failing on rejected jobs and add skip_preflight (0.47.0)

.github/workflows/spec_drift.yaml

@@ -0,0 +1,55 @@
+name: API spec drift
+
+on:
+  schedule:
+    - cron: "0 13 * * 1"
+  workflow_dispatch:
+
+permissions:
+  contents: read
+  issues: write
+
+concurrency:
+  group: ${{ github.workflow }}
+  cancel-in-progress: false
+
+jobs:
+  check:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v5
+    - uses: astral-sh/setup-uv@v7
+    - name: Set up Python 3.13
+      uses: actions/setup-python@v6
+      with:
+        python-version: "3.13"
+    - name: Install dependencies
+      env:
+        UV_LOCKED: "1"
+        UV_PYTHON: "3.13"
+      run: make install
+    - name: Check spec drift
+      id: drift
+      env:
+        UV_PYTHON: "3.13"
+      run: |
+        set -o pipefail
+        make check-spec-drift | tee drift.txt
+    - name: Report drift in an issue
+      if: failure() && steps.drift.outcome == 'failure'
+      env:
+        GH_TOKEN: ${{ github.token }}
+        RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
+      run: |
+        if ! grep -qE '^(New drift|Stale baseline):' drift.txt; then
+          echo "The check failed without reporting drift; see the previous step."
+          exit 0
+        fi
+        title="SDK models have drifted from the API spec"
+        body="$(printf '%s\n\n```\n%s\n```\n\nSee CONTRIBUTING.md, \"Keeping models in sync with the API\".' "$RUN_URL" "$(cat drift.txt)")"
+        number="$(gh issue list --state open --search "in:title \"$title\"" --json number --jq '.[0].number')"
+        if [ -n "$number" ]; then
+          gh issue comment "$number" --body "$body"
+        else
+          gh issue create --title "$title" --body "$body"
+        fi

CHANGELOG.md

@@ -1,3 +1,12 @@
+## 0.47.0
+
+### Features
+* Add `skip_preflight` to `CreateWorkflow` and `UpdateWorkflow`. Set it to `True` to skip the job preflight check for a workflow's jobs; on update, leaving it unset keeps the current value. `WorkflowInformation.skip_preflight` reports the workflow's current setting. For a one-off job, set `"skip_preflight": true` in the `request_data` JSON passed to `jobs.create_job`.
+
+### Fixes
+* Deserialize jobs the preflight check rejected. `JobStatus` and `JobProcessingStatus` gain `REJECTED`; previously `jobs.get_job` raised on a rejected job, and one rejected job failed the whole `jobs.list_jobs` page. `JobInformation.status` and `JobDetails.processing_status` now also accept a status this SDK version does not know, returning it as a plain `str` instead of raising. Known statuses are still returned as enum members, but the fields are typed `Union[JobStatus, str]` and `Union[JobProcessingStatus, str]`, so type-checked code that reads `.value` or `.name` from them needs to handle the `str` case.
+* Expose why a job was rejected. `JobNodeDetails` gains `failure_category` (a `PreflightFailureCategory`, or a plain `str` for a category this SDK version does not know) and `failure_reason`, set on the node that failed preflight in `jobs.get_job_details`. Previously both were dropped during deserialization.
+
 ## 0.46.6
 
 ### Fixes

CONTRIBUTING.md

@@ -17,6 +17,19 @@ If you encounter any bugs or have suggestions for improvements, please open an i
 
 We will review and triage issues as quickly as possible. Our goal is to address bugs and incorporate improvements in the next release.
 
+## Keeping models in sync with the API
+
+The models under `src/unstructured_client/models/` are maintained by hand. They are no longer regenerated from the API's OpenAPI spec, so a field or enum value added on the server does not reach this SDK until someone adds it.
+
+`make check-spec-drift` compares the public specs (`https://platform.unstructuredapp.io/openapi.json` and `https://api.unstructuredapp.io/general/openapi.json`) with the models in `unstructured_client.models.shared`. It reports every property and enum value the spec has that the matching model lacks. Gaps the SDK does not cover yet are recorded in `scripts/spec_drift_baseline.json`, so the check fails only on new drift, or on a baseline entry that no longer applies.
+
+The `API spec drift` workflow runs the check every Monday and opens or comments on an issue when it fails. To resolve it:
+
+1. Add new fields and enum values to the models, and update the matching page under `docs/models/`.
+   * A new enum value on a response model is the case that breaks callers. A closed enum raises on a value it does not know, and one bad item fails a whole list response. Response fields whose enum is expected to grow should accept unknown values. See `JobInformation.status` for the pattern: `Union[<Enum>, UnrecognizedStr]` with `union_mode="left_to_right"`.
+2. Acknowledge any gap you are deliberately not covering, and clear entries the SDK now covers, with `uv run python scripts/check_spec_drift.py --update-baseline`.
+3. Add a `CHANGELOG.md` entry and bump `src/unstructured_client/_version.py`.
+
 ## Contact
 
 If you have any questions or need further assistance, please feel free to reach out by opening an issue.

Makefile

@@ -68,6 +68,10 @@ lint:
 	uv run pylint --rcfile=pylintrc src
 	uv run mypy src
 
+.PHONY: check-spec-drift
+check-spec-drift:
+	uv run python scripts/check_spec_drift.py
+
 .PHONY: publish
 publish:
 	./scripts/publish.sh

_test_contract/platform_api/test_jobs.py

@@ -222,3 +222,118 @@ def test_create_job(httpx_mock, platform_client: UnstructuredClient, platform_ap
     assert job.status == "SCHEDULED"
     assert job.job_type == "template"
     assert job.created_at == datetime.fromisoformat("2025-06-22T11:37:21.648+00:00")
+
+
+def test_list_jobs_tolerates_rejected_and_unrecognized_statuses(
+    httpx_mock, platform_client: UnstructuredClient, platform_api_url: str
+):
+    url = f"{platform_api_url}/api/v1/jobs/"
+
+    def job(job_id: str, status: str) -> dict:
+        return {
+            "created_at": "2025-06-22T11:37:21.648Z",
+            "id": job_id,
+            "status": status,
+            "workflow_id": "16b80fee-64dc-472d-8f26-1d7729b6423d",
+            "workflow_name": "test_workflow",
+        }
+
+    httpx_mock.add_response(
+        method="GET",
+        headers={"Content-Type": "application/json"},
+        json=[
+            job("fcdc4994-eea5-425c-91fa-e03f2bd8030d", "REJECTED"),
+            job("a1b2c3d4-eea5-425c-91fa-e03f2bd8030d", "SOME_FUTURE_STATUS"),
+            job("b2c3d4e5-eea5-425c-91fa-e03f2bd8030d", "COMPLETED"),
+        ],
+        url=url,
+    )
+
+    jobs = platform_client.jobs.list_jobs(request=operations.ListJobsRequest()).response_list_jobs
+
+    assert jobs[0].status is shared.JobStatus.REJECTED
+    assert type(jobs[1].status) is str
+    assert jobs[1].status == "SOME_FUTURE_STATUS"
+    assert jobs[2].status is shared.JobStatus.COMPLETED
+
+
+@pytest.mark.parametrize(
+    ("processing_status", "expected"),
+    [
+        ("REJECTED", shared.JobProcessingStatus.REJECTED),
+        ("SOME_FUTURE_STATUS", "SOME_FUTURE_STATUS"),
+    ],
+)
+def test_get_job_details_tolerates_rejected_and_unrecognized_statuses(
+    httpx_mock,
+    platform_client: UnstructuredClient,
+    platform_api_url: str,
+    processing_status: str,
+    expected,
+):
+    url = f"{platform_api_url}/api/v1/jobs/fcdc4994-eea5-425c-91fa-e03f2bd8030d/details"
+
+    httpx_mock.add_response(
+        method="GET",
+        headers={"Content-Type": "application/json"},
+        json={
+            "id": "fcdc4994-eea5-425c-91fa-e03f2bd8030d",
+            "node_stats": [],
+            "processing_status": processing_status,
+        },
+        url=url,
+    )
+
+    details = platform_client.jobs.get_job_details(
+        request=operations.GetJobDetailsRequest(job_id="fcdc4994-eea5-425c-91fa-e03f2bd8030d")
+    ).job_details
+
+    assert details.processing_status == expected
+    assert type(details.processing_status) is type(expected)
+
+
+def test_get_job_details_exposes_preflight_failure(
+    httpx_mock, platform_client: UnstructuredClient, platform_api_url: str
+):
+    url = f"{platform_api_url}/api/v1/jobs/fcdc4994-eea5-425c-91fa-e03f2bd8030d/details"
+
+    def node(name: str, **failure) -> dict:
+        return {
+            "node_name": name,
+            "ready": 0,
+            "in_progress": 0,
+            "success": 0,
+            "failure": 0,
+            **failure,
+        }
+
+    httpx_mock.add_response(
+        method="GET",
+        headers={"Content-Type": "application/json"},
+        json={
+            "id": "fcdc4994-eea5-425c-91fa-e03f2bd8030d",
+            "processing_status": "REJECTED",
+            "node_stats": [
+                node(
+                    "s3-source",
+                    failure_category="AUTH_INVALID_CREDENTIAL",
+                    failure_reason="The AWS Access Key Id you provided does not exist.",
+                ),
+                node("future-destination", failure_category="SOME_FUTURE_CATEGORY"),
+                node("partitioner", failure_category=None, failure_reason=None),
+            ],
+        },
+        url=url,
+    )
+
+    details = platform_client.jobs.get_job_details(
+        request=operations.GetJobDetailsRequest(job_id="fcdc4994-eea5-425c-91fa-e03f2bd8030d")
+    ).job_details
+
+    source, destination, partitioner = details.node_stats
+    assert source.failure_category is shared.PreflightFailureCategory.AUTH_INVALID_CREDENTIAL
+    assert source.failure_reason == "The AWS Access Key Id you provided does not exist."
+    assert type(destination.failure_category) is str
+    assert destination.failure_category == "SOME_FUTURE_CATEGORY"
+    assert partitioner.failure_category is None
+    assert partitioner.failure_reason is None

_test_contract/platform_api/test_workflows.py

@@ -1,3 +1,4 @@
+import json
 from datetime import datetime
 
 import pytest
@@ -256,4 +257,66 @@ def test_run_workflow(httpx_mock, platform_client: UnstructuredClient, platform_
     new_job = run_workflow_response.job_information
     assert new_job.id == "fcdc4994-eea5-425c-91fa-e03f2bd8030d"
     assert new_job.workflow_name == "test_workflow"
-    assert new_job.status == "IN_PROGRESS"
+    assert new_job.status == "IN_PROGRESS"
+
+WORKFLOW_RESPONSE = {
+    "created_at": "2025-06-22T11:37:21.648Z",
+    "destinations": [],
+    "id": "16b80fee-64dc-472d-8f26-1d7729b6423d",
+    "name": "test_workflow",
+    "sources": [],
+    "workflow_nodes": [],
+    "status": "active",
+}
+
+
+def test_create_workflow_sends_skip_preflight(
+    httpx_mock, platform_client: UnstructuredClient, platform_api_url: str
+):
+    httpx_mock.add_response(
+        method="POST",
+        url=f"{platform_api_url}/api/v1/workflows/",
+        json={**WORKFLOW_RESPONSE, "skip_preflight": True},
+    )
+
+    response = platform_client.workflows.create_workflow(
+        request=operations.CreateWorkflowRequest(
+            create_workflow=shared.CreateWorkflow(
+                name="test_workflow",
+                workflow_type="custom",
+                skip_preflight=True,
+            )
+        )
+    )
+
+    assert json.loads(httpx_mock.get_requests()[0].content)["skip_preflight"] is True
+    assert response.workflow_information.skip_preflight is True
+
+
+@pytest.mark.parametrize(
+    ("update_workflow", "expected_body"),
+    [
+        (shared.UpdateWorkflow(name="renamed"), {"name": "renamed"}),
+        (shared.UpdateWorkflow(skip_preflight=False), {"skip_preflight": False}),
+        (shared.UpdateWorkflow(skip_preflight=True), {"skip_preflight": True}),
+    ],
+)
+def test_update_workflow_sends_skip_preflight_only_when_set(
+    httpx_mock,
+    platform_client: UnstructuredClient,
+    platform_api_url: str,
+    update_workflow: shared.UpdateWorkflow,
+    expected_body: dict,
+):
+    url = f"{platform_api_url}/api/v1/workflows/16b80fee-64dc-472d-8f26-1d7729b6423d"
+    httpx_mock.add_response(method="PUT", url=url, json=WORKFLOW_RESPONSE)
+
+    response = platform_client.workflows.update_workflow(
+        request=operations.UpdateWorkflowRequest(
+            workflow_id="16b80fee-64dc-472d-8f26-1d7729b6423d",
+            update_workflow=update_workflow,
+        )
+    )
+
+    assert json.loads(httpx_mock.get_requests()[0].content) == expected_body
+    assert response.workflow_information.skip_preflight is False

_test_unstructured_client/unit/test_check_spec_drift.py

@@ -0,0 +1,52 @@
+from scripts.check_spec_drift import find_drift
+
+
+def _spec(schemas: dict) -> dict:
+    return {"components": {"schemas": schemas}}
+
+
+def test_matching_spec_reports_no_drift():
+    spec = _spec(
+        {
+            "JobStatus": {"type": "string", "enum": ["SCHEDULED", "REJECTED"]},
+            "CreateWorkflow": {
+                "properties": {
+                    "name": {"type": "string"},
+                    "schedule": {"anyOf": [{"type": "string", "enum": ["daily"]}, {"type": "null"}]},
+                }
+            },
+        }
+    )
+    assert find_drift([spec]) == {}
+
+
+def test_reports_missing_enum_values_and_properties():
+    spec = _spec(
+        {
+            "JobStatus": {"type": "string", "enum": ["SCHEDULED", "PAUSED"]},
+            "CreateWorkflow": {"properties": {"name": {}, "brand_new_field": {}}},
+            "SomeSchemaTheSdkDoesNotModel": {"properties": {"x": {}}},
+        }
+    )
+    assert find_drift([spec]) == {
+        "CreateWorkflow": ["brand_new_field"],
+        "JobStatus": ["PAUSED"],
+    }
+
+
+def test_reports_missing_values_of_an_inline_property_enum():
+    spec = _spec(
+        {
+            "CreateWorkflow": {
+                "properties": {
+                    "schedule": {
+                        "anyOf": [
+                            {"type": "string", "enum": ["daily", "every 30 minutes"]},
+                            {"type": "null"},
+                        ]
+                    }
+                }
+            }
+        }
+    )
+    assert find_drift([spec]) == {"CreateWorkflow.schedule": ["every 30 minutes"]}

docs/models/shared/bodycreatejob.md

@@ -6,4 +6,4 @@
 | Field                                                        | Type                                                         | Required                                                     | Description                                                  |
 | ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ |
 | `input_files`                                                | List[[shared.InputFiles](../../models/shared/inputfiles.md)] | :heavy_minus_sign:                                           | N/A                                                          |
-| `request_data`                                               | *str*                                                        | :heavy_check_mark:                                           | N/A                                                          |
+| `request_data`                                               | *str*                                                        | :heavy_check_mark:                                           | JSON-encoded object. Provide either `template_id` or `job_nodes`, not both. Set `"skip_preflight": true` to skip the job preflight check for this job. |

docs/models/shared/createworkflow.md

@@ -9,6 +9,7 @@
 | `name`                                                               | *str*                                                                | :heavy_check_mark:                                                   | N/A                                                                  |
 | `reprocess_all`                                                      | *OptionalNullable[bool]*                                             | :heavy_minus_sign:                                                   | N/A                                                                  |
 | `schedule`                                                           | [OptionalNullable[shared.Schedule]](../../models/shared/schedule.md) | :heavy_minus_sign:                                                   | N/A                                                                  |
+| `skip_preflight`                                                     | *OptionalNullable[bool]*                                             | :heavy_minus_sign:                                                   | Skip the job preflight check for this workflow.                      |
 | `source_id`                                                          | *OptionalNullable[str]*                                              | :heavy_minus_sign:                                                   | N/A                                                                  |
 | `template_id`                                                        | *OptionalNullable[str]*                                              | :heavy_minus_sign:                                                   | N/A                                                                  |
 | `workflow_nodes`                                                     | List[[shared.WorkflowNode](../../models/shared/workflownode.md)]     | :heavy_minus_sign:                                                   | N/A                                                                  |

docs/models/shared/jobdetails.md

@@ -8,4 +8,4 @@
 | `id`                                                                     | *str*                                                                    | :heavy_check_mark:                                                       | N/A                                                                      |
 | `message`                                                                | *OptionalNullable[str]*                                                  | :heavy_minus_sign:                                                       | N/A                                                                      |
 | `node_stats`                                                             | List[[shared.JobNodeDetails](../../models/shared/jobnodedetails.md)]     | :heavy_check_mark:                                                       | N/A                                                                      |
-| `processing_status`                                                      | [shared.JobProcessingStatus](../../models/shared/jobprocessingstatus.md) | :heavy_check_mark:                                                       | N/A                                                                      |
+| `processing_status`                                                      | Union[[shared.JobProcessingStatus](../../models/shared/jobprocessingstatus.md), *str*] | :heavy_check_mark:                                                       | N/A                                                                      |