diff --git a/.github/AGENTS.md b/.github/AGENTS.md index 015fb44..61aee70 100644 --- a/.github/AGENTS.md +++ b/.github/AGENTS.md @@ -3,7 +3,7 @@ CI/CD workflows, automation, security scanning, and package distribution. ## Workflows -All build/test workflows cover Python 3.10-3.15 plus free-threaded 3.14t and 3.15t, except `conda-package-cf.yml`, which takes the versions below 3.14 from the feedstock (its `python-versions` input). +The conda workflows build and test the Python versions in the `python:` list of their recipe's `conda_build_config.yaml`; the other build/test workflows list theirs in the workflow file. - **conda-package.yml** — main build/test pipeline (Linux/Windows) - **conda-package-cf.yml** — build/test using only conda-forge channel (Linux/Windows/macOS), set up like `conda-forge/mkl-service-feedstock`: Windows uses `windows-2022` (Visual Studio 2022) @@ -14,16 +14,20 @@ All build/test workflows cover Python 3.10-3.15 plus free-threaded 3.14t and 3.1 - **openssf-scorecard.yml** — security posture scanning - **coverity.yml** — weekly Coverity Scan static analysis of the C/Cython extensions (scan.coverity.com) +## Actions +- **actions/conda-python-matrix/** — composite action that reads the `python:` list of a recipe's `conda_build_config.yaml` and outputs the build/test matrix. `action.yml` and `conda_python_matrix.py` (standard library only) refer to nothing outside the directory, so it can move unchanged to a shared repository together with `conda-build-test.yml`. The `variants` job references it as `./.github/actions/conda-python-matrix`, which resolves against the caller's checkout; after the move that line must become `$/.github/actions/conda-python-matrix`, which resolves against the repository of the workflow file itself. It is not `$/` already because actionlint 1.7.12, used by pre-commit, rejects that syntax (rhysd/actionlint#711). Run it locally with `python .github/actions/conda-python-matrix/conda_python_matrix.py conda-recipe-cf/conda_build_config.yaml` + ## CI/CD policy -- Keep build matrix (Python versions, platforms) in workflow files only +- Platforms are set in the caller workflows. The conda workflows read their Python versions from the `python:` list in the recipe's `conda_build_config.yaml`: the `variants` job of `conda-build-test.yml` turns it into the build/test matrix with the `.github/actions/conda-python-matrix` composite action. The other workflows keep their Python matrix in the workflow file +- Every entry of a recipe's `python:` list is a full spec on its own line, without selectors (`3.12.* *_cpython`, `3.13.* *_cp313`, `3.14.* *_cp314`, `3.14.* *_cp314t`); the `variants` job rejects bare versions, selectors, an ABI that doesn't match the version (`3.13.* *_cp314t`) and a second entry for the same build, whose artifact names would collide +- The `variants` job keeps the check names of the former static matrix: `Build (3.12)` for a version with one build, `Build (3.14, 3.14.* *_cp314t, 3.14t)` for a version with GIL and free-threaded builds. The master ruleset requires the 3.14 `Test` checks by these exact names - Required checks: conda build + test on supported Python versions/platforms in CI -- Python 3.14 is expressed as two `include` rows carrying `python_spec` (`3.14.* *_cp314` and `3.14.* *_cp314t`); a bare `--python 3.14` has been observed to select the free-threaded ABI and must not be relied on +- In workflow-file matrices (`build_pip.yml`), Python 3.14 is expressed as two `include` rows carrying `python_spec` (`3.14.* *_cp314` and `3.14.* *_cp314t`); a bare `--python 3.14` has been observed to select the free-threaded ABI and must not be relied on - Python 3.15 follows the same pattern as 3.14: two `include` rows with `python_spec` `3.15.* *_cp315` and `3.15.* *_cp315t` -- `include` rows create new matrix combinations and inherit nothing, so `experimental` must be repeated wherever the job reads them -- Artifact naming: `$PACKAGE_NAME $OS Python $VERSION`, where `$VERSION` is `python_tag` (`3.14`, `3.14t`, `3.15`, or `3.15t`) when set, otherwise `python` +- Artifact naming: `$PACKAGE_NAME $OS Python $VERSION`, where `$VERSION` is `python_tag`: the Python version, with a `t` suffix for free-threaded builds (`3.12`, `3.14`, `3.14t`) - Channels: `conda-forge`, Intel channel - Build jobs run `conda build --no-test`; test jobs run `conda build --test` on the built package on a fresh runner, so CI runs the test section of the recipe, as the feedstock does -- `conda-package-cf.yml` and `conda-recipe-cf/` follow `conda-forge/mkl-service-feedstock`. The feedstock's pins are kept only in `conda-recipe-cf/conda_build_config.yaml` (compiler versions, `mkl_devel`) and in the `python-versions` input of `conda-package-cf.yml` (Python versions below 3.14): when the feedstock is re-rendered with new pins, update them there, and don't restate the numbers in the `AGENTS.md` files +- `conda-package-cf.yml` and `conda-recipe-cf/` follow `conda-forge/mkl-service-feedstock`. The feedstock's pins are kept only in `conda-recipe-cf/conda_build_config.yaml` (compiler versions, `mkl_devel`, and Python versions in its `python:` list): when the feedstock is re-rendered with new pins, update them there, and don't restate the numbers in the `AGENTS.md` files ## Security - OpenSSF Scorecard runs automatically diff --git a/.github/actions/conda-python-matrix/action.yml b/.github/actions/conda-python-matrix/action.yml new file mode 100644 index 0000000..e9ba0a2 --- /dev/null +++ b/.github/actions/conda-python-matrix/action.yml @@ -0,0 +1,24 @@ +name: Conda Python matrix +description: >- + Build the conda build/test matrix from the python list of a recipe's + conda_build_config.yaml + +inputs: + recipe-dir: + description: Path to the conda recipe directory + required: true + +outputs: + matrix: + description: Matrix for strategy.matrix, as JSON + value: ${{ steps.read.outputs.matrix }} + +runs: + using: composite + steps: + - name: Read Python versions from conda_build_config.yaml + id: read + shell: bash + env: + CBC_PATH: ${{ inputs.recipe-dir }}/conda_build_config.yaml + run: python3 "${GITHUB_ACTION_PATH}/conda_python_matrix.py" "${CBC_PATH}" diff --git a/.github/actions/conda-python-matrix/conda_python_matrix.py b/.github/actions/conda-python-matrix/conda_python_matrix.py new file mode 100644 index 0000000..5fdcf1e --- /dev/null +++ b/.github/actions/conda-python-matrix/conda_python_matrix.py @@ -0,0 +1,127 @@ +"""Build the conda CI matrix from the python list of a conda_build_config.yaml. + +Usage: + python conda_python_matrix.py RECIPE_DIR/conda_build_config.yaml + +Prints the matrix entries and, when $GITHUB_OUTPUT is set, writes +matrix= to it for strategy.matrix of later jobs. Standard library only, +so it runs on any runner without conda. + +The python: key must hold full specs, one per line and without selectors, +such as "3.12.* *_cpython", "3.13.* *_cp313" or "3.14.* *_cp314t", quoted or +not, so that every job gets an explicit ABI. The ABI must be cpython or name +the entry's version (cp314 or cp314t for 3.14), and each build may be listed +only once, since its python_tag names its artifacts. Each entry becomes one +matrix entry with: +- python_spec: the entry itself, for conda build --python +- python_tag: the version, with a "t" suffix for free-threaded builds, for + artifact names +- label: the check name suffix. A version with a single build shows only the + version ("3.12"); a version with GIL and free-threaded builds shows the + version, spec and tag ("3.14, 3.14.* *_cp314t, 3.14t"). These are the + names of the former static matrix, which branch rulesets may require. +""" + +import argparse +import collections +import json +import os +import re +import sys + +SPEC_RE = re.compile(r"(\d+\.\d+)\.\* \*_(cpython|cp\d+t?)") + + +class DeclarationError(Exception): + """The python: list cannot be turned into a matrix.""" + + +def read_specs(path): + """Return the entries of the python: key of a conda_build_config.yaml.""" + specs, in_list = [], False + with open(path, encoding="utf-8") as f: + for line in f: + # In YAML, a comment line doesn't end the list, even at column 0 + if line.lstrip().startswith("#"): + continue + if line.startswith("python:"): + if specs or line.strip() != "python:": + raise DeclarationError( + "expected a single 'python:' key without selectors" + ) + in_list = True + elif in_list and re.match(r"\s+-", line): + if "#" in line: + raise DeclarationError( + "selectors on python entries are not supported: " + + line.strip() + ) + spec = line.split("-", 1)[1].strip() + if len(spec) > 1 and spec[0] == spec[-1] and spec[0] in "'\"": + spec = spec[1:-1] + specs.append(spec) + elif in_list and line.strip() and not line[0].isspace(): + in_list = False + if not specs: + raise DeclarationError("no python entries found") + return specs + + +def build_matrix(specs): + """Return the strategy.matrix for the given python entries.""" + parsed, seen = [], {} + for spec in specs: + m = SPEC_RE.fullmatch(spec) + if not m: + raise DeclarationError( + f"unsupported python entry '{spec}', " + "expected e.g. '3.14.* *_cp314t'" + ) + version, abi = m.groups() + cp = "cp" + version.replace(".", "") + if abi not in ("cpython", cp, cp + "t"): + raise DeclarationError( + f"python entry '{spec}': ABI {abi} doesn't match Python " + f"{version}, expected cpython, {cp} or {cp}t" + ) + tag = version + ("t" if abi.endswith("t") else "") + if tag in seen: + raise DeclarationError( + f"python entries '{seen[tag]}' and '{spec}' both build " + f"Python {tag}" + ) + seen[tag] = spec + parsed.append((spec, version, tag)) + + builds = collections.Counter(version for _, version, _ in parsed) + include = [] + for spec, version, tag in parsed: + if builds[version] == 1: + label = version + else: + label = f"{version}, {spec}, {tag}" + include.append({"python_spec": spec, "python_tag": tag, "label": label}) + return {"include": include} + + +def main(argv=None): + parser = argparse.ArgumentParser(description=__doc__.splitlines()[0]) + parser.add_argument("cbc", help="path to conda_build_config.yaml") + args = parser.parse_args(argv) + + try: + matrix = build_matrix(read_specs(args.cbc)) + except DeclarationError as e: + print(f"{args.cbc}: {e}", file=sys.stderr) + return 1 + + print(json.dumps(matrix["include"], indent=2)) + output = os.environ.get("GITHUB_OUTPUT") + if output: + with open(output, "a", encoding="utf-8") as f: + f.write("matrix=" + json.dumps(matrix) + "\n") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/.github/workflows/conda-build-test.yml b/.github/workflows/conda-build-test.yml index 55e5977..41a4be2 100644 --- a/.github/workflows/conda-build-test.yml +++ b/.github/workflows/conda-build-test.yml @@ -24,11 +24,6 @@ on: required: false default: false type: boolean - python-versions: - description: JSON list of Python versions to build and test in addition to 3.14 and 3.15, which always get both the GIL and free-threaded builds - required: false - default: '["3.10", "3.11", "3.12", "3.13"]' - type: string permissions: contents: read @@ -37,26 +32,34 @@ env: package-name: mkl-service jobs: + variants: + name: Variants + + runs-on: ubuntu-latest + timeout-minutes: 5 + + outputs: + matrix: ${{ steps.matrix.outputs.matrix }} + + steps: + - name: Checkout repo + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + + - name: Read Python versions from conda_build_config.yaml + id: matrix + uses: ./.github/actions/conda-python-matrix + with: + recipe-dir: ${{ inputs.recipe-dir }} + build: - name: Build + name: Build (${{ matrix.label }}) + needs: variants strategy: fail-fast: false - matrix: - python: ${{ fromJSON(inputs.python-versions) }} - include: - - python: '3.14' - python_spec: '3.14.* *_cp314' - python_tag: '3.14' - - python: '3.14' - python_spec: '3.14.* *_cp314t' - python_tag: '3.14t' - - python: '3.15' - python_spec: '3.15.* *_cp315' - python_tag: '3.15' - - python: '3.15' - python_spec: '3.15.* *_cp315t' - python_tag: '3.15t' + matrix: ${{ fromJSON(needs.variants.outputs.matrix) }} runs-on: ${{ inputs.runs-on }} timeout-minutes: 15 @@ -95,7 +98,7 @@ jobs: env: CHANNELS_LIST: ${{ inputs.channels-list }} RECIPE_DIR: ${{ inputs.recipe-dir }} - PYTHON_SPEC: ${{ matrix.python_spec || matrix.python }} + PYTHON_SPEC: ${{ matrix.python_spec }} run: | # CHANNELS_LIST intentionally word-splits into multiple -c args # shellcheck disable=SC2086 @@ -104,37 +107,23 @@ jobs: - name: Upload artifact uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 with: - name: ${{ env.package-name }} ${{ runner.os }} Python ${{ matrix.python_tag || matrix.python }} + name: ${{ env.package-name }} ${{ runner.os }} Python ${{ matrix.python_tag }} path: ${{ env.CONDA_BLD }}/${{ env.package-name }}-*.conda - name: Upload wheels artifact if: ${{ inputs.build-wheels }} uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 with: - name: ${{ env.package-name }} ${{ runner.os }} Wheels Python ${{ matrix.python_tag || matrix.python }} + name: ${{ env.package-name }} ${{ runner.os }} Wheels Python ${{ matrix.python_tag }} path: ${{ env.WHEELS_OUTPUT_FOLDER }}/mkl_service-*.whl test: - name: Test - needs: build + name: Test (${{ matrix.label }}) + needs: [variants, build] strategy: fail-fast: false - matrix: - python: ${{ fromJSON(inputs.python-versions) }} - include: - - python: '3.14' - python_spec: '3.14.* *_cp314' - python_tag: '3.14' - - python: '3.14' - python_spec: '3.14.* *_cp314t' - python_tag: '3.14t' - - python: '3.15' - python_spec: '3.15.* *_cp315' - python_tag: '3.15' - - python: '3.15' - python_spec: '3.15.* *_cp315t' - python_tag: '3.15t' + matrix: ${{ fromJSON(needs.variants.outputs.matrix) }} runs-on: ${{ inputs.runs-on }} timeout-minutes: 15 @@ -150,7 +139,7 @@ jobs: - name: Download artifact uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2 with: - name: ${{ env.package-name }} ${{ runner.os }} Python ${{ matrix.python_tag || matrix.python }} + name: ${{ env.package-name }} ${{ runner.os }} Python ${{ matrix.python_tag }} path: ${{ env.pkg-dir }} - name: Setup miniconda diff --git a/.github/workflows/conda-package-cf.yml b/.github/workflows/conda-package-cf.yml index 0e71ed8..38ec80e 100644 --- a/.github/workflows/conda-package-cf.yml +++ b/.github/workflows/conda-package-cf.yml @@ -13,8 +13,8 @@ concurrency: group: ${{ github.workflow }}-${{ github.ref }} cancel-in-progress: true -# Build like conda-forge/mkl-service-feedstock: the feedstock's Python versions -# plus 3.15t, which it doesn't build yet, and Visual Studio 2022 on Windows +# Build like conda-forge/mkl-service-feedstock, whose Python versions are listed +# in conda-recipe-cf/conda_build_config.yaml, with Visual Studio 2022 on Windows jobs: linux: name: linux @@ -24,7 +24,6 @@ jobs: conda-subdir: linux-64 channels-list: -c conda-forge --override-channels recipe-dir: conda-recipe-cf - python-versions: '["3.11", "3.12", "3.13"]' windows: name: windows @@ -34,7 +33,6 @@ jobs: conda-subdir: win-64 channels-list: -c conda-forge --override-channels recipe-dir: conda-recipe-cf - python-versions: '["3.11", "3.12", "3.13"]' osx: name: osx @@ -44,4 +42,3 @@ jobs: conda-subdir: osx-64 channels-list: -c conda-forge --override-channels recipe-dir: conda-recipe-cf - python-versions: '["3.11", "3.12", "3.13"]' diff --git a/AGENTS.md b/AGENTS.md index fe8f640..17bfea6 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -48,7 +48,7 @@ python -m pip install --no-deps --no-build-isolation . ## CI/CD - **Platforms in CI workflows:** Linux, Windows, macOS (conda-forge workflow only) -- **Python versions:** 3.10, 3.11, 3.12, 3.13, 3.14, 3.15, and free-threaded 3.14t and 3.15t (`conda-package-cf.yml` takes the versions below 3.14 from the feedstock instead) +- **Python versions:** the conda workflows build and test the entries of the `python:` list in the recipe's `conda_build_config.yaml` (`conda-recipe/` or `conda-recipe-cf/`), one build and one test job per entry; the other workflows list their Python versions in the workflow file - **Python 3.14 ABI selection:** conda jobs must request an explicit ABI, `3.14.* *_cp314` (GIL) or `3.14.* *_cp314t` (free-threaded). Once the recipes stopped constraining `python-gil`, a bare `--python 3.14` was observed to resolve to the free-threaded build, so it must not be relied on. - **Python 3.15 ABI selection:** same as 3.14; conda jobs request `3.15.* *_cp315` (GIL) or `3.15.* *_cp315t` (free-threaded) explicitly. - **Workflows:** `.github/workflows/` diff --git a/conda-recipe-cf/conda_build_config.yaml b/conda-recipe-cf/conda_build_config.yaml index b0c7156..a40a793 100644 --- a/conda-recipe-cf/conda_build_config.yaml +++ b/conda-recipe-cf/conda_build_config.yaml @@ -22,3 +22,13 @@ c_compiler: # [win] - vs2022 # [win] mkl_devel: # [not osx] - '2026' # [not osx] +# Python versions of conda-forge/mkl-service-feedstock, plus 3.15t; CI builds +# and tests one job per entry, see .github/workflows/conda-build-test.yml +python: + - 3.11.* *_cpython + - 3.12.* *_cpython + - 3.13.* *_cp313 + - 3.14.* *_cp314 + - 3.14.* *_cp314t + - 3.15.* *_cp315 + - 3.15.* *_cp315t diff --git a/conda-recipe/AGENTS.md b/conda-recipe/AGENTS.md index 3f9c821..64a675b 100644 --- a/conda-recipe/AGENTS.md +++ b/conda-recipe/AGENTS.md @@ -6,11 +6,11 @@ Conda package build recipe for conda-forge and Intel channel distribution. - **meta.yaml** — package metadata, dependencies, build requirements - **build.sh** — Linux/macOS build script - **bld.bat** — Windows build script -- **conda_build_config.yaml** — compiler / stdlib pinning used by conda-build variants +- **conda_build_config.yaml** — compiler / stdlib pinning used by conda-build variants, and the `python:` list of versions that CI builds and tests (one job per entry) ## Build configuration - **Channels:** `conda-forge`, Intel channel -- **Python versions in CI build matrix:** 3.10, 3.11, 3.12, 3.13, 3.14, 3.15, and free-threaded 3.14t and 3.15t +- **Python versions in CI build matrix:** from the `python:` list in each recipe's `conda_build_config.yaml`. A plain `conda build conda-recipe` therefore builds every listed version; pass `--python "3.12.* *_cpython"` to build one - **Compilers:** pinned via `conda_build_config.yaml` (GCC/GXX on Linux, VS2022 on Windows) - **Dependencies:** `mkl` (runtime), `mkl-devel` (build), `cython >=3.1.0` diff --git a/conda-recipe/conda_build_config.yaml b/conda-recipe/conda_build_config.yaml index 49c17b0..bae0c6f 100644 --- a/conda-recipe/conda_build_config.yaml +++ b/conda-recipe/conda_build_config.yaml @@ -8,3 +8,13 @@ c_stdlib: # [win] - vs # [win] c_compiler: # [win] - vs2022 # [win] +# CI builds and tests one job per entry, see .github/workflows/conda-build-test.yml +python: + - 3.10.* *_cpython + - 3.11.* *_cpython + - 3.12.* *_cpython + - 3.13.* *_cp313 + - 3.14.* *_cp314 + - 3.14.* *_cp314t + - 3.15.* *_cp315 + - 3.15.* *_cp315t