Skip to content

[WIP] Feature/gpu sharing factor - #1798

Open
jlnav wants to merge 8 commits into
developfrom
feature/gpu-sharing-factor
Open

jlnav wants to merge 8 commits into
developfrom
feature/gpu-sharing-factor

Conversation

@jlnav

@jlnav jlnav commented Oct 27, 2026 •

Copy link
Copy Markdown
Member

Addresses #1394

  • Implements logical GPU sharing via libE_specs["gpu_sharing_factor"] (shares per physical GPU) with physical-exclusive, packing-aware scheduling.
  • num_gpus now supports fractional values (e.g. 0.5 → 1 share when sharing_factor is 2, 2 → 2*sharing_factor shares)
  • scheduler_opts["gpu_packing"]="pack"|"spread" controls co-location.
  • Worker/executor GPU visibility dedups to physical IDs and correctly handles fractional requests
  • tests and docstrings

@jlnav
jlnav marked this pull request as ready for review October 27, 2026 19:28
@jlnav
jlnav requested a review from shuds13 October 28, 2026 14:53
@codecov

codecov Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 81.06996% with 46 lines in your changes missing coverage. Please review.
✅ Project coverage is 88.70%. Comparing base (303e333) to head (f5d73e0).
⚠️ Report is 1 commits behind head on develop.

Files with missing lines Patch % Lines
libensemble/resources/scheduler.py 81.53% 16 Missing and 8 partials ⚠️
libensemble/tools/alloc_support.py 72.97% 5 Missing and 5 partials ⚠️
libensemble/executors/mpi_runner.py 66.66% 5 Missing and 1 partial ⚠️
libensemble/resources/worker_resources.py 76.92% 4 Missing and 2 partials ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #1798      +/-   ##
===========================================
- Coverage    88.95%   88.70%   -0.26%     
===========================================
  Files           74       74              
  Lines         8795     9015     +220     
  Branches      1421     1470      +49     
===========================================
+ Hits          7824     7997     +173     
- Misses         723      754      +31     
- Partials       248      264      +16     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

jlnav added 8 commits October 29, 2026 15:59
…gned logical "shares" of physical gpus). libE_specs["gpu_sharing_factor"] requests number of logical shares to make out of each physical gpu. gpu_sharing_factor of 1 (default) produces traditional behavior. shares assigned round-robin. workers convert assigned shares into physical GPU ids. associated tests.
…PU was assigned. new "pack" policy for scheduling fractional gpu shares (multiple jobs get packed onto a GPU by default). "pack" or "spread" options for gpu_packing. scheduler builds "gpu_bins" for allocating into. jobs can still request a whole GPU. tests.
…two GPUs. unit tests for gpu_sharing_factor, gpus_per_group, building gpu bins, capacity handling for gpus, "pack" and "spread" policies, and fractional-gpu support in alloc_support
Use local Resources instance instead of Resources.resources (None when
run via run_tests.py -u isolation). Return Resources not just manager,
pass directly to AllocSupport, and override scheduler for pack policy.
Add test_select_rsets_fragmented_remainder for sf=4 where only partial
GPUs are free. No single GPU satisfies rem, so _select_rsets_for_gpu_group
must fragment across multiple GPUs (pack vs spread ordering) and raise
InsufficientFreeResources when fragments insufficient. Covers previously
uncovered else block scheduler.py:256-269.
@jlnav
jlnav force-pushed the feature/gpu-sharing-factor branch from 0f8b4f5 to f5d73e0 Compare October 29, 2026 20:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant