Repository navigation
[ML] Split the filtered lazy dataset with multiple clusters like the unfiltered one - #23843
Merged
Merged
Conversation
siliataider
changed the title
siliataider
force-pushed
the
rdataloader
branch
from
October 26, 2026 11:13
57dd033 to
f73a18c
Compare
vepadulano
approved these changes
Oct 26, 2026
vepadulano
left a comment
vepadulano
left a comment
Member
There was a problem hiding this comment.
Thank you! Changes LGTM overall, but consider a few minor comments before merging.
siliataider
force-pushed
the
rdataloader
branch
2 times, most recently
from
October 26, 2026 13:04
ffdb990 to
bb3d3a6
Compare
Test Results 24 files 24 suites 3d 22h 58m 20s ⏱️ Results for commit 43960a9. ♻️ This comment has been updated with latest results. |
The shuffled branch of `RClusterLoader::SplitDataset()` rounded the training entries down (`int(N * (1 - split))`), while the unshuffled branch and the eager loading round the validation entries down (`N - int(split * N)`). The same dataset could therefore get one more validation entry when shuffled, e.g. 3 instead of 2 for 6 entries and a split of 0.4. All modes now round the validation entries down.
…unfiltered one For filtered RDataFrames, the split is discovered while loading the first lazy epoch, and it always split every cluster proportionally, which is what's supposed to happen when shuffling is enabled only. This is now changed to take the first part of the whole dataset and attribute it to training, and the tail to validation, like the other loading paths do. Assisted-by: Claude Opus 5.5
siliataider
force-pushed
the
rdataloader
branch
from
October 27, 2026 09:50
bb3d3a6 to
43960a9
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This happens because for the filtered dataframes, we discover the cluster counts as we go, and attribute the train/test splits as we go, therefore the logic takes a separate part in the code, that slightly drifted from the unfiltered path :)
This PR fixes #23842