Skip to content

[DO NOT MERGE] Reproduce flaky cuvs-java randomized tests - #2792

Closed
bdice wants to merge 10 commits into
NVIDIA:mainfrom
bdice:repro-java-randomized-crash
Closed

bdice wants to merge 10 commits into
NVIDIA:mainfrom
bdice:repro-java-randomized-crash

Conversation

@bdice

@bdice bdice commented Oct 10, 2026

Copy link
Copy Markdown
Contributor

Draft PR to reproduce flaky cuvs-java failures in CI. Do not merge.

CI is reduced to the amd64 CUDA 13.3.0 cpp and java builds plus 4 copies of the java test job. Each copy runs a single-row CAGRA build reproducer, then loops HnswRandomizedIT and the re-enabled CagraRandomizedIT with diagnostics.

Coordination: #2783

Reduce CI to the amd64 CUDA 13 cpp/java build and 4 copies of the java
test job, which loops HnswRandomizedIT and the re-enabled
CagraRandomizedIT with mismatch diagnostics, and adds a deterministic
single-row CAGRA build reproducer.
@copy-pr-bot

copy-pr-bot Bot commented Oct 10, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@bdice

bdice commented Oct 10, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 0d29719

With one row, check_graph_degree reduced both the intermediate and the
final graph degree to 0, and the kNN graph build (NN-descent or IVF-PQ)
then hit an illegal memory access, which aborted the process from a
stream synchronization in a destructor. Throw a clear error up front
instead, and keep HnswRandomizedIT from generating single-row datasets.
@bdice

bdice commented Oct 10, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 1501cf1

@bdice

bdice commented Oct 10, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 1501cf1

@bdice

bdice commented Oct 10, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 95a6371

@bdice

bdice commented Oct 10, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 17b5523

@bdice

bdice commented Oct 10, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test d9ee449

@bdice

bdice commented Oct 11, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 663d1e0

@bdice

bdice commented Oct 11, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 8d5131b

@bdice

bdice commented Oct 11, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test b8afbce

1 similar comment
@bdice

bdice commented Oct 11, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test b8afbce

@bdice

bdice commented Oct 11, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test f324803

@bdice

bdice commented Oct 11, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 9a3fc1e

@bdice

bdice commented Oct 15, 2026

Copy link
Copy Markdown
Contributor Author

Closing this reproducer: it confirmed the crash and verified the fix (run 38077388744). The fixes are in #2805 and #2830; tracked in #2783.

@bdice bdice closed this Oct 15, 2026
rapids-bot Bot pushed a commit that referenced this pull request Oct 21, 2026
Building a CAGRA index from a single row reduced both the intermediate and the final graph degree to 0. The kNN graph build (NN-descent or IVF-PQ) then hit an illegal memory access, which aborted the process (`terminate called ... cuda_error`, exit code 134). This is the intermittent `HnswRandomizedIT` JVM crash: that test draws `datasetSize = nextInt(10_000) + 1`.

- `check_graph_degree` now rejects datasets with fewer than 2 rows by throwing an exception.
- Adds `CagraSingleRowIT`, which checks that a single-row build throws instead of crashing.
- `HnswRandomizedIT` now draws at least 2 rows, as `CagraRandomizedIT` and cuvs-lucene (`MIN_CAGRA_INDEX_SIZE = 2`) already do.

Reproduced and verified in #2792. Part of #2783.

Authors:
  - Bradley Dice (https://github.com/bdice)

Approvers:
  - Julian Miller (https://github.com/julianmi)
  - Igor Motov (https://github.com/imotov)
  - Anupam (https://github.com/aamijar)
  - Tarang Jain (https://github.com/tarang-jain)

URL: #2805
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

1 participant