Repository navigation
Conversation
…rrays OwnershipModel.HoldsCore memoized a "holds NDArrays" answer only when its walk had cut no cycle, and HolderMembersOf put the asking type itself on the walk. In a model whose types reference each other (entities with navigation properties) nearly every answer involves a cycle, so nothing was memoized and every type re-walked every path through the graph - exponential in the number of types. NDArrayHolderAnalyzer (NDW016/NDW017) runs this for every named type of the compilation. Seen in a consumer that references NumSharp only indirectly: NDArrayHolderAnalyzer took 321.6 s of analyzer time on one project (99 % of all analyzers, a 108 s compile) that declares no NDArray field at all; with this change 0.025 s and a 5 s compile, with the same diagnostics (none). HoldsCore now runs Tarjan's strongly connected components over the "stores a value of" graph: each type is visited once per walk, a true answer is memoized as soon as it is found, and the types of one cycle are memoized together with the answer of the type the walk entered the cycle through - "holds" is the least fixed point of an OR over the graph, so a cycle holds exactly when its entry does. HolderMembersOf answers a source class/struct that does not hold NDArrays as a whole from that memo instead of walking its members; for a holding type it keeps the type on the walk's stack as before, so a member that reaches arrays only through the type itself still adds nothing. Two tests pin it: a model of 14 types that all reference each other completes (no NDArray: clean; one NDArray: all 14 flagged NDW016). Both time out after 60 s without the change. All 330 tests of NumSharp.Tests.Build.Analyzer pass on net8.0 and net10.0.
Member
|
Hey @campersau, appreciate your issue openning and fix. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
NDArrayHolderAnalyzer(NDW016/NDW017) can take minutes on projects whose types reference each other in cycles, even when those projects don't use NDArray at all. The analyzer ships in the NumSharp package, so it runs in every project that references NumSharp, directly or transitively.Seen in a project that references NumSharp only transitively and doesn't use NDArray itself, but contains a larger domain model whose types reference each other in cycles. With
-p:ReportAnalyzer=true:NDArrayHolderAnalyzer, 99 % of all analyzers)Both versions report the same diagnostics on that project (none).
Workaround until a release with this fix
ExcludeAssets="analyzers"on the package reference doesn't help: the SDK still passes the analyzer to csc, asbuild/NumSharp.targetsnotes. What works is removing it from@(Analyzer)before compiling, for example in aDirectory.Build.targets:This turns off all NDW diagnostics, so it only suits projects that use neither the NDArray ownership rules nor
[NDScoped].Cause
OwnershipModel.HoldsCorememoized a "holds NDArrays" answer only when its walk had cut no cycle.HolderMembersOfalso put the asking type itself on the walk. In a model whose types reference each other, nearly every answer involves a cycle, so almost nothing was memoized. Every type then re-walked every path through the graph, which grows exponentially with the size and density of the cycle. The holder analyzer does this for every named type of the compilation.Fix
HoldsCoreruns Tarjan's strongly connected components over the "stores a value of" graph, so each type is visited once per walk.trueanswer is memoized as soon as it is found. Finding one never depends on a cut cycle, so it is always exact.HolderMembersOfanswers a source class/struct that holds no NDArrays straight from that memo instead of walking its members.HolderMembersOfkeeps the existing behaviour: the type stays on the walk's stack, so a member that reaches arrays only by coming back to the type itself adds nothing.Tests
Two new tests in
OwnershipRobustnessTestsuse a model of 14 types that all reference each other:DenselyCyclicModelWithoutArrays_CompletesClean: no diagnostics.DenselyCyclicModelWithAnArray_FlagsEveryType: one type stores anNDArray, and all 14 are flagged NDW016.Both time out after 60 s on
masterand finish at once with this change. All 330 tests ofNumSharp.Tests.Build.Analyzerpass on net8.0 and net10.0.