[ggorman]

Refs devitocodespro/devitopro#1051

What

_merge_halospots and _hoist_invariant (devito/passes/iet/mpi.py) built a Scope for every HaloSpot pair.
Building one means a FindWithin visit of the enclosing Iteration, and it happened before the cheap checks
that reject most pairs. Since #3013 (0038c8c), an Operator with many Eqs over distinct SubDomains carries one
HaloSpot per Eq, so this became quadratic in the number of Eqs.

This PR derives the Scope lazily (_lazy_scope, a functools.cache over partial(_derive_scope, ...)):

• _merge_halospots derives it only once a pair passes _is_compatible. That check is the void,
  dimension and shared-function test, split out of _is_mergeable, which keeps its contract for
  _hoist_redundant_from_conditionals.
• _hoist_invariant derives it only once hsf1.issubset(hsf0) holds.

The Scope is built from the same (it, hs0, hs1) as before, so every merge and hoist decision is unchanged.
#3013's HaloSpot placement is untouched.

Effect

case	_derive_scope calls	build
per-row SubDomain leapfrog (devitopro#1051), before	127,221	506.25 s
same, after	7,077	58.70 s (old pin, before #3013: 27.04 s)
new test (n=8 SubDomains, 2n Eqs), before / after	156 / 7

Tests

• New: tests/test_mpi.py::TestCodeGeneration::test_merge_many_subdomains_linear. It builds 2n Eqs over
  n row SubDomains: n of them share one Function, so they merge, and n exchange distinct Functions, so they
  can only hoist. It spies on _derive_scope with mock.patch(wraps=...). A spy is needed because the
  pair count is a compile-time cost that the Operator does not expose. The test asserts fewer than 2*n calls.
  • Before this change: 156 calls, fails.
  • After: 7 calls, passes.
  • Reverting only the merge pass: 120 calls, fails.
  • Reverting only the hoist pass: 43 calls, fails.
• Generated code identical before and after, compared by sha256 of op.ccode under mpirun -n 2:
  • modes basic, diag2 and overlap2;
  • nine Operators: acoustic, TTI, elastic, the subset-comms case, an n-SubDomain case with advanced and
    noop, and the per-row leapfrog.
• DevitoPRO tests/test_cpml.py: codegen-only, with the CUDA target and identifiers normalized. All 185
  generated kernels are identical, including the MPI ones.
• Suites with the change:
  • -m "not parallel" on test_mpi, test_subdomains, test_dle, test_operator and test_ir:
    887 passed, 1 xfailed.
  • -m parallel on test_mpi, test_subdomains, test_operator and test_dle: 580 passed.

https://claude.ai/code/session_01TT4Kt2oZNzhQrSw8SfVoaw

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 84.46%. Comparing base (33633a0) to head (2ccfa70).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3098      +/-   ##
==========================================
+ Coverage   84.43%   84.46%   +0.02%     
==========================================
  Files         260      260              
  Lines       58262    58293      +31     
  Branches     4956     4955       -1     
==========================================
+ Hits        49194    49236      +42     
+ Misses       8237     8228       -9     
+ Partials      831      829       -2     

Flag	Coverage Δ
pytest-gpu-aomp-amdgpuX	68.44% <90.90%> (+0.01%)	⬆️
pytest-gpu-gcc-	79.17% <100.00%> (+0.01%)	⬆️
pytest-gpu-icx-	79.09% <100.00%> (+<0.01%)	⬆️
pytest-gpu-nvc-nvidiaX	69.15% <90.90%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[mloubout]

This didn't follow devito style. Caching should go through memoized_func for example. Peter patch also doesn't make sense

[ggorman]

Thanks @mloubout, this was the right question. We went back and measured whether the change matters for real programs.

• The slowdown needs hundreds of Eqs, each over its own SubDomain, in one Operator. Across 900+ Operators we measured, there are at most 12 HaloSpots in one Operator: the devito SubDomain/PML example notebooks, the CPML-style tests, and production acoustic/elastic/VTI/TTI propagators. Scope derivation stays in the tens of calls there, so the cost is negligible.
• The only program that reached ~480 HaloSpots was one of our own: it emitted an assembled sparse operator one row at a time, each row over its own SubDomain. A separable stencil form of the same scheme needs about 18 Eqs and no SubDomains. We're fixing that on our side instead.
• The generated code was identical with and without this change everywhere, so no test written from real use can tell the two apart. That is another reason not to merge it.

Closing. We'll reopen the question only if a realistic program hits it.

https://claude.ai/code/session_01TT4Kt2oZNzhQrSw8SfVoaw