Skip to content

simplify build workflow and move cibuildwheel configuration to pyproject.toml - #1371

Open
zacharyburnett wants to merge 9 commits into
GalSim-developers:releases/2.8from
zacharyburnett:simplify_build_workflow
Open

zacharyburnett wants to merge 9 commits into
GalSim-developers:releases/2.8from
zacharyburnett:simplify_build_workflow

Conversation

@zacharyburnett

@zacharyburnett zacharyburnett commented Oct 20, 2026 •

Copy link
Copy Markdown

we can move these specific cibuildwheel configurations into pyproject.toml, and it will run them for each platform / arch

also, cibuildwheel can use emulation to build macOS Intel x86 wheels on the Apple Silicon platform

also also, we can run the wheels.yml workflow on every pull request to build wheels (with on: pull_request) and use an if conditional to only attempt publishing to PyPI on an actual release

@zacharyburnett
zacharyburnett marked this pull request as ready for review October 20, 2026 18:46
@zacharyburnett
zacharyburnett force-pushed the simplify_build_workflow branch from 87babc2 to b1f643b Compare October 20, 2026 18:47
@zacharyburnett
zacharyburnett force-pushed the simplify_build_workflow branch from b1f643b to 1843404 Compare October 20, 2026 18:52
@rmjarvis

Copy link
Copy Markdown
Member

Why? What was wrong with the previous version? This is a pretty big overhaul with no (apparent to me) benefit. I'll let you make your case though.

also also, we can run the wheels.yml workflow on every pull request to build wheels (with on: pull_request) and use an if conditional to only attempt publishing to PyPI on an actual release

This in particular seems purely wasteful. Why would we want to build wheels on every random PR?

@zacharyburnett

zacharyburnett commented Oct 21, 2026 •

Copy link
Copy Markdown
Author

Why? What was wrong with the previous version? This is a pretty big overhaul with no (apparent to me) benefit. I'll let you make your case though.

This reduces the amount of jobs actually running, and GitHub runners spun up, from 4 (glibc Linux, muslc Linux, macOS x86, macOS ARM, plus the Linux containers that cibuildwheel starts) to 2 (GitHub-supplied ubuntu-latest and macos-latest, plus manylinux and musllinux spun up by cibuildwheel) by letting cibuildwheel handle the build matrix and also build the macOS x86 wheel with emulation alongside the macOS ARM64 wheel. We also don't need to install fftw on the final publish job in order to build the source dist, because we can build it alongside the wheel in the build jobs

Additionally, moving the meaningful configuration to pyproject.toml makes it a lot more explicit on what is actually different from defaults, and makes it much easier to maintain. In my experience configuration should be in one place or it will diverge and people will forget to update it. For instance, setting requires_python once in the pyproject.toml is much better IMO than copying CIBW_SKIP: cp36* cp37* cp38* pp* four times buried in the workflow YAML; if GalSim ever needs new syntax features that aren't backported and needs to update the minimum Python, there's much less configuration change required

also also, we can run the wheels.yml workflow on every pull request to build wheels (with on: pull_request) and use an if conditional to only attempt publishing to PyPI on an actual release

This in particular seems purely wasteful. Why would we want to build wheels on every random PR?

I personally like to run a build job on every pull request so I know about build failures early, otherwise I'm blindsided by build failures when it comes to release time and have to scramble. I can take that out though if you prefer; there's always the workflow_dispatch that can be run manually

@rmjarvis

Copy link
Copy Markdown
Member

This reduces the amount of jobs actually running, and GitHub runners spun up, from 4 (glibc Linux, muslc Linux, macOS x86, macOS ARM, plus the Linux containers that cibuildwheel starts) to 2

This IMO is not particularly desirable. The reason for having 4 there was to reduce the human wait time for them all to finish building.

@zacharyburnett
zacharyburnett force-pushed the simplify_build_workflow branch from 24309a2 to feeb083 Compare October 23, 2026 13:05
@zacharyburnett
zacharyburnett force-pushed the simplify_build_workflow branch from feeb083 to c790f48 Compare October 23, 2026 13:06
@zacharyburnett
zacharyburnett force-pushed the simplify_build_workflow branch 12 times, most recently from 84ef419 to 0a0a325 Compare October 23, 2026 18:23
try ccache
@zacharyburnett
zacharyburnett force-pushed the simplify_build_workflow branch from 0a0a325 to 4d2c51b Compare October 23, 2026 18:31

@rmjarvis rmjarvis left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I seem some benefits to the simplification here. But I have some significant concerns with the current implementation.

Comment thread .github/workflows/wheels.yml Outdated
Comment thread .github/workflows/wheels.yml Outdated
cancel-in-progress: true

- uses: actions/setup-python@v5
permissions: {}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the point of this? AFAIK, we only need permissions later when publishing. And specifying none here seems vaguely redundant.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the default permissions given to GITHUB_TOKEN are very broad and have been misused in security breaches where someone added malicious code to an action. We can mitigate the amount of access given by setting permissions: {} globally and then adding permissions to individual jobs (see https://docs.zizmor.sh/audits/#excessive-permissions)

with:
fetch-tags: true
fetch-depth: 0
persist-credentials: false

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pretty sure these are unnecessary. And fetch-depth=0 is surely a waste of bandwidth.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right, I usually work with projects that use setuptools_scm to retrieve the version from the Git tags in which case fetch-tags is necessary, and sometimes with projects that use release_branch_semver with release branches diverging from main, which needs fetch-depth: 0. But those are not needed here, I'll take them out

persist-credentials: false is recommended by zizmor to mitigate artifact packing that may exfiltrate credentials, either accidental or maliciously from upstream (see https://docs.zizmor.sh/audits/#remediation_2)


permissions:
id-token: write
attestations: write

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the point of attestations?

@zacharyburnett zacharyburnett Oct 26, 2026 •

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://docs.pypi.org/attestations/ attestations gives strong cryptographic proof that a binary was created by and signed by a specific source, in this case this github actions workflow. I don't think most users are actually checking attestations currently, but it's part of PyPI now so it doesn't hurt to have, especially if that ever changes. I can remove the attestation step if desired though

Comment thread pyproject.toml Outdated
Comment thread pyproject.toml Outdated
Comment thread .github/workflows/wheels.yml Outdated
- name: cache C++ build objects
uses: hendrikmuhs/ccache-action@f09c25b45002a07be2955cbe52e8cee55643f89d # v1.2.24
with:
key: ccache-${{ runner.os }}-${{ github.sha }}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this is doing anything useful. Including the sha here means the cache won't be reused I think. We only run this workflow once per commit, so I don't see the benefit of this.

Comment thread .github/workflows/wheels.yml Outdated

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants