[avehtari]

This PR was assisted by Claude

Summary

#409 makes loo_model_weights() reject objects from loo_subsample(), because their pointwise values cover only the subsampled observations. Pseudo-BMA without the Bayesian bootstrap does not need pointwise values: the weights are proportional to exp(elpd_loo) of each model, and loo_subsample() estimates elpd_loo for all observations. With this PR, loo_model_weights(x, method = "pseudobma", BB = FALSE) accepts objects from loo_subsample() and uses their estimates of elpd_loo. The subsamples and their sizes can differ between the models, and a list can mix subsampled and full objects.

Stacking and pseudo-BMA+ (BB = TRUE, the default) still reject subsampled objects. Both need the pointwise values of the same observations for all models. The error message now names the supported combination: "loo_model_weights() supports objects from loo_subsample() only with method = "pseudobma" and BB = FALSE, because they have pointwise values only for the subsampled observations. Use loo() for all observations."

Change

• validate_psis_loo_list() has a new argument allow_subsampling, which loo_model_weights() sets for method = "pseudobma" and BB = FALSE. The check that all models have the same number of observations uses length(x$loo_subsampling$elpd_loo_approx) for a subsampled object, because nobs() returns the size of the subsample.
• A new internal function pseudobma_weights_from_elpd() computes the weights from a vector of elpd_loo values. pseudobma_weights(BB = FALSE) uses it too, so its results do not change.
• The documentation of x describes which methods accept objects from loo_subsample().

Tests

The test for subsampled objects from #409 now checks:

• stacking, and method = "pseudobma" with BB = TRUE, reject two subsampled objects and a mix of a subsampled and a full object;
• method = "pseudobma", BB = FALSE gives exp(elpd_loo)-proportional weights from the subsampling estimates, for two subsampled objects with different subsamples of 50 and 80 observations, and for a mix of a full and a subsampled object;
• subsampled objects of data sets with different numbers of observations give the error "same number of observations".

With the R code of #409, this test fails. devtools::test() with NOT_CRAN=true passes all 1279 tests and gives no warnings (R 4.6.1, testthat 3.3.2, Linux x86_64).

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.54%. Comparing base (f6df6b2) to head (8a7462a).
⚠️ Report is 4 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #414      +/-   ##
==========================================
+ Coverage   92.49%   92.54%   +0.04%     
==========================================
  Files          31       31              
  Lines        3143     3164      +21     
==========================================
+ Hits         2907     2928      +21     
  Misses        236      236              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

This is how benchmark results would change (along with a 95% confidence interval in relative change) if 8a7462a is merged into master:

• ❗🐌loo_function: 1.64s -> 1.89s [+15.17%, +15.74%]
• ❗🐌loo_matrix: 1.62s -> 1.85s [+14.01%, +14.66%]
  Further explanation regarding interpretation and methodology can be found in the documentation.

[jgabry]

Looks good, just a few small suggestions

[avehtari]

Applied / added all your suggestions