Skip to content

Fix/mitigate SISEGV11 errors on python+rust builds on haswell nodes with 2025.06 - #329

Merged
bedroge merged 5 commits into
EESSI:mainfrom
Crivella:feature-rustc_2025_hook
Oct 30, 2026
Merged

bedroge merged 5 commits into
EESSI:mainfrom
Crivella:feature-rustc_2025_hook

Conversation

@Crivella

@Crivella Crivella commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Ugly but working fix for issue observed in:

Should also work for:

if the failures are truly random and just rerunning the build multiple times will cause it to succeed.
In this case we leverage the fact that re-running the pip install command will skip already installed crates and just process the not yet compiled ones

Manually tested by running it 3 times successfully on top of EESSI/software-layer#1871
The last iteration with the print of the failures failed 3 times before succeeding.

TODO

  • Cleanup comments (i've left them there for now to show some of the things that have been tried)
  • Potentially properly find the source of the failure but i think it is something very specific to our nodes + Rust/LLVM version combination. This failure does not seem to occur in our 2026.1 builds and i assume was never observed in the 2023.06 ones, or on other haswell machines with the same software stack
  • Potentially have a more complete map like in Add hook to include --cpu-target for julia wrappers #327 so that we are sure that the produced binaries are always compatible with the expected target

Neves-P added a commit to Neves-P/software-layer that referenced this pull request Oct 29, 2026
@Crivella Crivella changed the title PoC: Fix python+rust builds on haswell nodes with 2025.06 Fix/mitigate SISEGV11 errors on python+rust builds on haswell nodes with 2025.06 Oct 29, 2026
@Crivella
Crivella force-pushed the feature-rustc_2025_hook branch from f79f53c to e099299 Compare October 29, 2026 15:33

@ocaisa ocaisa left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed the problem so going in

@ocaisa

ocaisa commented Oct 29, 2026

Copy link
Copy Markdown
Member

bot: build repo:eessi.io-2023.06-software instance:eessi-bot-deucalion for:arch=aarch64/a64fx
bot: build repo:eessi.io-2025.06-software instance:eessi-bot-deucalion for:arch=aarch64/a64fx
bot: build repo:eessi.io-2026.06-software instance:eessi-bot-deucalion for:arch=aarch64/a64fx

@eessi-bot-deucalion

eessi-bot-deucalion Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

New job on instance eessi-bot-deucalion for repository eessi.io-2023.06-software
Building on: a64fx
Building for: aarch64/a64fx
Job dir: /home/eessibot/new-bot/jobs/2026.10/pr_329/2165446

date job status comment
Oct 29 20:59:13 UTC 2026 submitted job id 2165446 awaits release by job manager
Oct 29 21:00:21 UTC 2026 released job awaits launch by Slurm scheduler
Oct 29 21:01:27 UTC 2026 running job 2165446 is running
Oct 29 21:06:02 UTC 2026 finished
😁 SUCCESS (click triangle for details)
Details
✅ job output file slurm-2165446.out
✅ no message matching FATAL:
✅ no message matching ERROR:
✅ no message matching FAILED:
✅ no message matching required modules missing:
✅ found message(s) matching No missing installations
✅ found message matching .tar.* created!
Artefacts
eessi-2023.06-software-linux-aarch64-a64fx-17933079050.tar.zstsize: 0 MiB (37053 bytes)
entries: 1
modules under 2023.06/software/linux/aarch64/a64fx/modules/all
no module files in tarball
software under 2023.06/software/linux/aarch64/a64fx/software
no software packages in tarball
reprod directories under 2023.06/software/linux/aarch64/a64fx/reprod
no reprod directories in tarball
other under 2023.06/software/linux/aarch64/a64fx
no other files in tarball
Oct 29 21:06:02 UTC 2026 test result
😢 FAILURE (click triangle for details)
Reason
EESSI test suite was not run, test step itself failed to execute.
Details
✅ job output file slurm-2165446.out
❌ found message matching ERROR:
✅ no message matching [\s*FAILED\s*].*Ran .* test case
Oct 29 22:00:18 UTC 2026 uploaded transfer of eessi-2023.06-software-linux-aarch64-a64fx-17933079050.tar.zst to S3 bucket succeeded

@eessi-bot-deucalion

eessi-bot-deucalion Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

New job on instance eessi-bot-deucalion for repository eessi.io-2025.06-software
Building on: a64fx
Building for: aarch64/a64fx
Job dir: /home/eessibot/new-bot/jobs/2026.10/pr_329/2165448

date job status comment
Oct 29 20:59:19 UTC 2026 submitted job id 2165448 awaits release by job manager
Oct 29 21:00:18 UTC 2026 released job awaits launch by Slurm scheduler
Oct 29 21:01:30 UTC 2026 running job 2165448 is running
Oct 29 21:03:51 UTC 2026 finished
😁 SUCCESS (click triangle for details)
Details
✅ job output file slurm-2165448.out
✅ no message matching FATAL:
✅ no message matching ERROR:
✅ no message matching FAILED:
✅ no message matching required modules missing:
✅ found message(s) matching No missing installations
✅ found message matching .tar.* created!
Artefacts
eessi-2025.06-software-linux-aarch64-a64fx-17933077740.tar.zstsize: 0 MiB (37056 bytes)
entries: 1
modules under 2025.06/software/linux/aarch64/a64fx/modules/all
no module files in tarball
software under 2025.06/software/linux/aarch64/a64fx/software
no software packages in tarball
reprod directories under 2025.06/software/linux/aarch64/a64fx/reprod
no reprod directories in tarball
other under 2025.06/software/linux/aarch64/a64fx
no other files in tarball
Oct 29 21:03:51 UTC 2026 test result
😢 FAILURE (click triangle for details)
Reason
EESSI test suite was not run, test step itself failed to execute.
Details
✅ job output file slurm-2165448.out
❌ found message matching ERROR:
✅ no message matching [\s*FAILED\s*].*Ran .* test case
Oct 29 22:00:28 UTC 2026 uploaded transfer of eessi-2025.06-software-linux-aarch64-a64fx-17933077740.tar.zst to S3 bucket succeeded

@eessi-bot-deucalion

eessi-bot-deucalion Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

New job on instance eessi-bot-deucalion for repository eessi.io-2026.06-software
Building on: a64fx
Building for: aarch64/a64fx
Job dir: /home/eessibot/new-bot/jobs/2026.10/pr_329/2165449

date job status comment
Oct 29 20:59:25 UTC 2026 submitted job id 2165449 awaits release by job manager
Oct 29 21:00:15 UTC 2026 released job awaits launch by Slurm scheduler
Oct 29 21:01:33 UTC 2026 running job 2165449 is running
Oct 29 21:06:05 UTC 2026 finished
😁 SUCCESS (click triangle for details)
Details
✅ job output file slurm-2165449.out
✅ no message matching FATAL:
✅ no message matching ERROR:
✅ no message matching FAILED:
✅ no message matching required modules missing:
✅ found message(s) matching No missing installations
✅ found message matching .tar.* created!
Artefacts
eessi-2026.06-software-linux-aarch64-a64fx-17933077880.tar.zstsize: 0 MiB (37014 bytes)
entries: 1
modules under 2026.06/software/linux/aarch64/a64fx/modules/all
no module files in tarball
software under 2026.06/software/linux/aarch64/a64fx/software
no software packages in tarball
reprod directories under 2026.06/software/linux/aarch64/a64fx/reprod
no reprod directories in tarball
other under 2026.06/software/linux/aarch64/a64fx
no other files in tarball
Oct 29 21:06:05 UTC 2026 test result
😁 SUCCESS (click triangle for details)
ReFrame Summary
[ PASSED ] Ran 0/0 test case(s) from 0 check(s) (0 failure(s), 0 expected failure(s), 0 skipped, 0 aborted)
Details
✅ job output file slurm-2165449.out
✅ no message matching ERROR:
✅ no message matching [\s*FAILED\s*].*Ran .* test case
Oct 29 22:00:38 UTC 2026 uploaded transfer of eessi-2026.06-software-linux-aarch64-a64fx-17933077880.tar.zst to S3 bucket succeeded

@bedroge

bedroge commented Oct 30, 2026

Copy link
Copy Markdown
Contributor

Staging PR merged

@bedroge
bedroge merged commit 46e1e9c into EESSI:main Oct 30, 2026
122 of 125 checks passed
@Crivella
Crivella deleted the feature-rustc_2025_hook branch October 30, 2026 10:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants