Skip to content

Update to v2.1.0-alpha.1 and mache v5.2.0 - #1008

Draft
xylar wants to merge 12 commits into
MPAS-Dev:mainfrom
xylar:update-to-2.1.0
Draft

xylar wants to merge 12 commits into
MPAS-Dev:mainfrom
xylar:update-to-2.1.0

Conversation

@xylar

@xylar xylar commented Oct 23, 2026 •

Copy link
Copy Markdown
Collaborator

This pull request updates Compass to v2.1.0-alpha.1 and to mache 5.2.0, which builds Spack environments with Spack 1.x (see the mache 5.0.0 release notes). The shared Spack environments need to be redeployed under dev_compass_2.1.0 on each machine.

Updates:

  • mache 5.2.0 -- Spack 1.x, with deploy.py and deploy/cli_spec.json synced (adds --spack-pins)
  • e3sm-spack-packages pinned to 65739a6 for the trilinos-for-albany and albany fixes in Declare language dependencies for trilinos-for-albany and albany E3SM-Project/e3sm-spack-packages#4, until a mache release includes them
  • mpas_tools 2.1.0
  • netcdf-c 4.10.1, netcdf-fortran 4.6.3 and pnetcdf 1.15.0
  • Katara Spack template migrated to the Spack 1.x compiler model
  • Perlmutter: Spack builds on login nodes are limited to 8 jobs (4 on pm-gpu), and pm-gpu's Albany environment is deployed from a compute node
  • CI uses the pinned mache, and the GIS mesh test stays off RDMA transports and prints its log on failure

For reviewers:

Testing

MALI with full_integration, using a test Spack deployment in scratch:

Deployed

MALI with full_integration, using the shared Spack deployment:

  • Chrysalis (@xylar)
    • gnu and openmpi
  • Perlmutter (@xylar)
    • gnu and mpich
    • gnugpu and mpich

Checklist

  • Document (in a comment titled Testing in this PR) any testing that was used to verify the changes

Closes E3SM-Project/mache#493


Posted by Claude Code on @xylar's behalf. The testing, analysis and wording above are AI-authored; please check them accordingly.

@xylar xylar added the dependencies and deployment Changes relate to creating conda and Spack environments, and creating a load script label Oct 23, 2026
This reverts commit 0a8c5da, which pointed CI at an old mache branch
whose jigsaw build pins libnetcdf 4.10.0, conflicting with the 4.10.1
pin in deploy/pins.cfg.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@xylar xylar mentioned this pull request Oct 23, 2026
1 task
xylar and others added 2 commits October 23, 2026 10:11
Some GitHub runners have a Microsoft Azure Network Adapter whose RDMA
device UCX cannot open, so MPI_Init fails in ESMF_RegridWeightGen on
those runners only.  Limit UCX to self, shared-memory and TCP
transports.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Perlmutter login nodes cap each user's memory at 30 GiB, and Albany's
build was OOM-killed with Spack's default of 16 jobs.  A new
spack_build_jobs option in a machine config's [deploy] section sets
spack.build_jobs through the pre_spack hook; Perlmutter uses 8.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@xylar

xylar commented Oct 23, 2026

Copy link
Copy Markdown
Collaborator Author

Testing: Chrysalis

The full_integration suite passed (all 30 tests) on Chrysalis with gnu and openmpi, using MALI built with Albany against a test Spack deployment in scratch. There was no baseline to compare against.


Posted by Claude Code on @xylar's behalf. The testing, analysis and wording above are AI-authored; please check them accordingly.

xylar and others added 4 commits October 23, 2026 15:09
Albany's CUDA build was still OOM-killed on a Perlmutter login node
with 8 jobs.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Inside a Slurm job the deployment has a whole node's memory, so
spack_build_jobs no longer applies there.  The Spack deployment guide
now says to deploy pm-gpu's Albany environment from a compute node,
since its CUDA build only fits on a login node with 4 jobs.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The trilinos-for-albany and albany recipes need the Spack 1.x fixes from
E3SM-Project/e3sm-spack-packages#4, which no tag or mache release
includes yet.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
pixi cannot lock files in its default cache in the home directory from a
Perlmutter compute node, so the compute-node example sets PIXI_CACHE_DIR
under $SCRATCH first.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@xylar

xylar commented Oct 23, 2026

Copy link
Copy Markdown
Collaborator Author

Testing: Perlmutter

I ran the full_integration suite with MALI (MALI-Dev 013c38724) built with Albany against test Spack deployments in scratch. There was no baseline to compare against.

pm-cpu, gnu and mpich pm-gpu, gnugpu and mpich
Result all 30 passed 21 of 30 passed
compass 99bb9da 83851f7
Spack /pscratch/sd/x/xylar/compass_2.1/spack_test/pm-cpu/dev_compass_2.1.0 /pscratch/sd/x/xylar/compass_2.1/spack_test/pm-gpu/dev_compass_2.1.0
Work directory /pscratch/sd/x/xylar/compass_2.1/pm-cpu/test_20261023/full_integration_scratch_spack /pscratch/sd/x/xylar/compass_2.1/pm-gpu/test_20261023/full_integration_scratch_spack

On pm-gpu, the seven FO restart tests fail validation as in #909, and both MISMIP+ smoke tests abort in Albany (#1011).

Both deployments used E3SM-Project/e3sm-spack-packages#4 through --spack-pins (d9bfd2d, the same recipes as the merged 65739a6). pm-gpu also used E3SM-Project/mache#541, built from a compute node, and ran the suite in an interactive allocation.


Posted by Claude Code on @xylar's behalf. The testing, analysis and wording above are AI-authored; please check them accordingly.

@xylar

xylar commented Oct 23, 2026

Copy link
Copy Markdown
Collaborator Author

We're holding off on deployment waiting for new albany and trilinos-for-albany tags.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dependencies and deployment Changes relate to creating conda and Spack environments, and creating a load script

Projects

None yet

Development

Successfully merging this pull request may close these issues.

compass main cannot solve its pixi env (jigsawpy vs libnetcdf 4.10.1); blocks compass deploy tests

1 participant