feat(csr): sparse one-term expressions and sparse broadcast in merges by FabianHofmann · Pull Request #1034 · PyPSA/linopy

FabianHofmann · 2026-10-18T20:32:28Z

Making the CSR path faster and bridging to a full data backend.
Related: #1019, #1033, #1011, #969.

Note

The following content was generated by AI.

Changes proposed in this Pull Request

In a sparse model (Model(sparse=True)), linear expressions now stay sparse from the start, and sums over operands with different dimensions no longer densify.

Variable arithmetic is sparse. Variable.to_linexpr builds the CSR store directly from the variable labels (CSRLinearExpression.from_labels), so 2 * x, x + y and 1*x + 2*y + 3*z are CSR-backed. This runs at dense parity: the earlier cost of one-term CSR expressions came from the generic from_dense scatter (54 ms vs 13 ms for 2 * x on a 2000 × 500 grid).
Sparse broadcast in merges. Merging operands over different dimensions (e.g. x.groupby(g).sum() + w with w over a subset of the dims) broadcasts rows on the sparse grid (Grid.broadcast_rows, CSRLinearExpression.broadcast) instead of densifying. This works for all joins and in xarray's dimension order. MultiIndex dims and non-unique labels still densify with an explicit reason.
Lazy canonical form. Sums and merges no longer sort rows or sum duplicate variables after every operation. Like a dense expression, a sparse one may hold the same variable more than once in a cell, and nterm counts stored terms. Rows are brought into canonical form once, when a constraint is added.
Safety. Stored CSR arrays are shared between expressions and read-only, so an accidental in-place write (e.g. through scipy's sum_duplicates) raises instead of corrupting another expression. to_dense() keeps the order of repeated terms. fillna(expr) warns once.
Release notes and the sparse-models guide are updated.

Behavior change

In a sparse model, operations on variable arithmetic that have no sparse path now densify with a PerformanceWarning, where the expression used to be dense and silent: simplify, assign_coords, drop_sel, expand_dims, shift(fill_value=...), where(callable), multiplication or comparison with a new dimension, merge along a new dimension, fillna(expr). x * y, x ** 2, rolling and cumsum stay dense without a warning.

Known open points

(x + y).sum("a") is about 12 % slower than on master (row gather of 2M terms).
1*x + 2*y + 3*z uses 111 MB instead of 95 MB.

Benchmarks (2000 × 500 grid; master sparse / branch sparse / branch dense)

On master, variable arithmetic is dense even in a sparse model.

Operation	master sparse	branch sparse	branch dense
`2*x`	4.2 ms	4.5 ms	3.9 ms
`x + y`	23.0 ms	23.5 ms	22.3 ms
`1x + 2y + 3*z`	48.6 ms, 95 MB	47.6 ms, 111 MB	43.8 ms, 95 MB
`(x + y).sum("b")`	29.8 ms	25.4 ms	27.4 ms
`(x + y).sum("a")`	35.5 ms	39.8 ms	32.5 ms
`e.sum()`	3.2 ms	1.5 ms	3.1 ms
`1x + 2y >= 1`	34.7 ms	31.3 ms	33.3 ms
add constraints + objective	315 ms	192 ms	–

Tests: 7598 passed, 1522 skipped (Xpress excluded locally because of a license limit).

Checklist

AI-generated content is marked (see AGENTS.md).
Code changes are sufficiently documented; i.e. new functions contain docstrings and further explanations may be given in doc.
Unit tests for new features were added (if applicable).
A note for the release notes doc/release_notes.rst of the upcoming release is included.
I consent to the release of this PR's code under the MIT license.

github-actions · 2026-10-18T20:35:56Z

Build cost — v1 vs legacy

v1 build peak & time relative to legacy, on this commit — not a comparison against master (that is CodSpeed).

peak — v1 / legacy	time — v1 / legacy

Full table (time + peak, mean)

benchmarks/drivers/test_build.py::test_build[basic-n=10]
                  time (s)         peak (KiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.06357 (1.16)   │   15.03 (1.00) 
 (v1)        0.05501 (1.0)   │    15.00 (1.0) 

benchmarks/drivers/test_build.py::test_build[basic-n=250]
                  time (s)         peak (MiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.07132 (1.20)   │   12.04 (1.00) 
 (v1)        0.05939 (1.0)   │    12.04 (1.0) 

benchmarks/drivers/test_build.py::test_build[cumsum-severity=0]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.02704 (1.12)   │   15.20 (1.0) 
 (v1)        0.02407 (1.0)   │   15.20 (1.0) 

benchmarks/drivers/test_build.py::test_build[cumsum-severity=100]
                  time (s)        peak (MiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.04215 (1.15)   │   44.93 (1.0) 
 (v1)        0.03675 (1.0)   │   44.93 (1.0) 

benchmarks/drivers/test_build.py::test_build[cumsum-severity=50]
                  time (s)        peak (MiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.02989 (1.13)   │   11.51 (1.0) 
 (v1)        0.02643 (1.0)   │   11.51 (1.0) 

benchmarks/drivers/test_build.py::test_build[expression_arithmetic-n=10]
                  time (s)         peak (KiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.07095 (1.12)   │   24.45 (1.06) 
 (v1)        0.06321 (1.0)   │    23.04 (1.0) 

benchmarks/drivers/test_build.py::test_build[expression_arithmetic-n=250]
                  time (s)         peak (MiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.08019 (1.13)   │   16.12 (1.00) 
 (v1)         0.0712 (1.0)   │    16.12 (1.0) 

benchmarks/drivers/test_build.py::test_build[knapsack-n=10000]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.01673 (1.09)   │   752.18 (1.10) 
 (v1)        0.01537 (1.0)   │    685.15 (1.0) 

benchmarks/drivers/test_build.py::test_build[knapsack-n=100]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.01645 (1.12)   │   3.12 (1.33) 
 (v1)        0.01464 (1.0)   │    2.34 (1.0) 

benchmarks/drivers/test_build.py::test_build[kvl_cycles-severity=0]
                  time (s)          peak (MiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.04805 (1.28)   │   126.16 (1.44) 
 (v1)        0.03747 (1.0)   │     87.71 (1.0) 

benchmarks/drivers/test_build.py::test_build[kvl_cycles-severity=100]
                  time (s)          peak (MiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.04713 (1.29)   │   126.16 (1.44) 
 (v1)        0.03653 (1.0)   │     87.71 (1.0) 

benchmarks/drivers/test_build.py::test_build[kvl_cycles-severity=50]
                  time (s)          peak (MiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.04748 (1.29)   │   126.16 (1.44) 
 (v1)        0.03686 (1.0)   │     87.71 (1.0) 

benchmarks/drivers/test_build.py::test_build[masked-n=100]
                 time (s)          peak (KiB) 
 name                mean   │            mean 
──────────────────────────────────────────────
 (legacy)   0.0388 (1.07)   │    712.45 (1.0) 
 (v1)       0.03611 (1.0)   │   787.73 (1.11) 

benchmarks/drivers/test_build.py::test_build[masked-n=10]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.03747 (1.16)   │   4.62 (1.29) 
 (v1)        0.03243 (1.0)   │    3.57 (1.0) 

benchmarks/drivers/test_build.py::test_build[merge_balance-severity=0]
                 time (s)          peak (KiB) 
 name                mean   │            mean 
──────────────────────────────────────────────
 (legacy)   0.2572 (1.08)   │   707.91 (1.10) 
 (v1)        0.2387 (1.0)   │    643.85 (1.0) 

benchmarks/drivers/test_build.py::test_build[merge_balance-severity=100]
                time (s)        peak (MiB) 
 name               mean   │          mean 
───────────────────────────────────────────
 (legacy)   0.269 (1.07)   │   18.34 (1.0) 
 (v1)       0.2522 (1.0)   │   18.34 (1.0) 

benchmarks/drivers/test_build.py::test_build[merge_balance-severity=50]
                 time (s)       peak (MiB) 
 name                mean   │         mean 
───────────────────────────────────────────
 (legacy)   0.2645 (1.06)   │   9.54 (1.0) 
 (v1)         0.249 (1.0)   │   9.54 (1.0) 

benchmarks/drivers/test_build.py::test_build[milp-n=10]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.05375 (1.15)   │   3.85 (1.14) 
 (v1)        0.04654 (1.0)   │    3.37 (1.0) 

benchmarks/drivers/test_build.py::test_build[milp-n=50]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.05367 (1.16)   │   216.98 (1.11) 
 (v1)        0.04646 (1.0)   │    196.23 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance-severity=0]
                  time (s)         peak (KiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.02704 (1.14)   │   938.49 (1.0) 
 (v1)        0.02376 (1.0)   │   938.49 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance-severity=100]
                 time (s)       peak (MiB) 
 name                mean   │         mean 
───────────────────────────────────────────
 (legacy)   0.0282 (1.13)   │   9.66 (1.0) 
 (v1)       0.02492 (1.0)   │   9.66 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance-severity=50]
                  time (s)       peak (MiB) 
 name                 mean   │         mean 
────────────────────────────────────────────
 (legacy)   0.02768 (1.14)   │   5.32 (1.0) 
 (v1)        0.02432 (1.0)   │   5.32 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance_sparse-severity=0]
                 time (s)       peak (MiB) 
 name                mean   │         mean 
───────────────────────────────────────────
 (legacy)   0.0117 (1.06)   │   1.29 (1.0) 
 (v1)       0.01106 (1.0)   │   1.29 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance_sparse-severity=100]
                  time (s)       peak (MiB) 
 name                 mean   │         mean 
────────────────────────────────────────────
 (legacy)   0.01135 (1.03)   │   1.29 (1.0) 
 (v1)        0.01105 (1.0)   │   1.29 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance_sparse-severity=50]
                 time (s)       peak (MiB) 
 name                mean   │         mean 
───────────────────────────────────────────
 (legacy)   0.0113 (1.02)   │   1.29 (1.0) 
 (v1)       0.01106 (1.0)   │   1.29 (1.0) 

benchmarks/drivers/test_build.py::test_build[piecewise-n=1000]
                 time (s)          peak (KiB) 
 name                mean   │            mean 
──────────────────────────────────────────────
 (legacy)   0.1295 (1.08)   │   946.85 (1.06) 
 (v1)        0.1204 (1.0)   │    891.54 (1.0) 

benchmarks/drivers/test_build.py::test_build[piecewise-n=10]
                 time (s)         peak (KiB) 
 name                mean   │           mean 
─────────────────────────────────────────────
 (legacy)   0.1296 (1.10)   │   12.01 (1.00) 
 (v1)        0.1175 (1.0)   │    11.99 (1.0) 

benchmarks/drivers/test_build.py::test_build[qp-n=1000]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.03358 (1.12)   │   147.70 (1.06) 
 (v1)        0.02991 (1.0)   │    139.87 (1.0) 

benchmarks/drivers/test_build.py::test_build[qp-n=10]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.03343 (1.10)   │   2.60 (1.09) 
 (v1)        0.03026 (1.0)   │    2.38 (1.0) 

benchmarks/drivers/test_build.py::test_build[rolling-severity=0]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.02799 (1.13)   │   704.59 (1.05) 
 (v1)        0.02477 (1.0)   │    673.70 (1.0) 

benchmarks/drivers/test_build.py::test_build[rolling-severity=100]
                  time (s)         peak (MiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.06465 (1.08)   │   137.97 (1.0) 
 (v1)        0.06011 (1.0)   │   137.97 (1.0) 

benchmarks/drivers/test_build.py::test_build[rolling-severity=50]
                  time (s)        peak (MiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.04456 (1.07)   │   69.22 (1.0) 
 (v1)         0.0415 (1.0)   │   69.22 (1.0) 

benchmarks/drivers/test_build.py::test_build[sos-n=1000]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.03221 (1.16)   │   402.37 (1.00) 
 (v1)        0.02768 (1.0)   │    402.30 (1.0) 

benchmarks/drivers/test_build.py::test_build[sos-n=10]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.03172 (1.15)   │   3.23 (1.20) 
 (v1)        0.02764 (1.0)   │    2.69 (1.0) 

benchmarks/drivers/test_build.py::test_build[sparse_network-n=10]
                  time (s)         peak (KiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.03474 (1.12)   │   29.00 (1.54) 
 (v1)        0.03099 (1.0)   │    18.84 (1.0) 

benchmarks/drivers/test_build.py::test_build[sparse_network-n=250]
                  time (s)         peak (MiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.04455 (1.18)   │   37.95 (1.43) 
 (v1)         0.0377 (1.0)   │    26.51 (1.0) 

benchmarks/drivers/test_build.py::test_build[storage-n=10]
                 time (s)          peak (KiB) 
 name                mean   │            mean 
──────────────────────────────────────────────
 (legacy)   0.0663 (1.02)   │    410.14 (1.0) 
 (v1)        0.0651 (1.0)   │   427.84 (1.04) 

benchmarks/drivers/test_build.py::test_build[storage-n=250]
                  time (s)         peak (MiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.07101 (1.01)   │     9.94 (1.0) 
 (v1)        0.07015 (1.0)   │   10.22 (1.03)

📊 Interactive plots + CSV: download the semantics-report-v1-vs-legacy artifact from this run.

_{Report-only · not a gate · refreshed on every push · obsolete once legacy is dropped.}

codspeed · 2026-10-18T20:35:57Z

Merging this PR will not alter performance

✅ 181 untouched benchmarks
⏩ 181 skipped benchmarks¹

_{Comparing feat/sparse-broadcast-from-labels (12c6b29) with master (09d4fde)²}

181 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
No successful run was found on master (44ff525) during the generation of this report, so 09d4fde was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

In a sparse model, Variable.to_linexpr builds CSR directly from labels (from_labels), and merges over different dims broadcast sparsely instead of densifying. CSR rows are canonicalized lazily when a constraint is added; stored arrays are read-only.

…from_labels

FabianHofmann added performance This improves performance while not (meaningfully) altering behaviour for users sparse Sparse / CSR-backed expressions and constraints labels Oct 18, 2026

FabianHofmann mentioned this pull request Oct 20, 2026

Meta: separate dense and sparse stores from the expression and constraint frontend #1019

Open

8 tasks

FabianHofmann added 3 commits October 21, 2026 09:43

refactor(csr): fold broadcast_rows into Grid.indexer, single path in …

b9e6389

…from_labels

test(csr): narrow join results to LinearExpression for mypy

12c6b29

FabianHofmann force-pushed the feat/sparse-broadcast-from-labels branch from 2235980 to 12c6b29 Compare October 21, 2026 07:45

FabianHofmann marked this pull request as ready for review October 21, 2026 08:37

FabianHofmann merged commit 22bfc8e into master Oct 21, 2026
24 checks passed

FabianHofmann deleted the feat/sparse-broadcast-from-labels branch October 21, 2026 08:41

This was referenced Oct 21, 2026

release: v0.10.1 #1037

Merged

docs: release notes section for v0.10.1 #1038

Merged

Navigation Menu

Repository navigation

feat(csr): sparse one-term expressions and sparse broadcast in merges - #1034

feat(csr): sparse one-term expressions and sparse broadcast in merges#1034
FabianHofmann merged 3 commits into
masterfrom
feat/sparse-broadcast-from-labels

FabianHofmann commented Oct 18, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 18, 2026 •

edited

Loading

Uh oh!

codspeed Bot commented Oct 18, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

FabianHofmann commented Oct 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes proposed in this Pull Request

Behavior change

Known open points

Checklist

Uh oh!

github-actions Bot commented Oct 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Build cost — v1 vs legacy

Uh oh!

codspeed Bot commented Oct 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will not alter performance

Footnotes

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

FabianHofmann commented Oct 18, 2026 •

edited

Loading

github-actions Bot commented Oct 18, 2026 •

edited

Loading

codspeed Bot commented Oct 18, 2026 •

edited

Loading