Repository navigation
Fix rounding in SSE byte stores - #357
Open
Nathan Chandrasekar (vera-rubin) wants to merge 1 commit into
Open
Nathan Chandrasekar (vera-rubin) wants to merge 1 commit into
Nathan Chandrasekar (vera-rubin) wants to merge 1 commit into
Conversation
Collaborator
|
I'll need to look closely at these proposed changes. There are a number of issues where I need to ensure that ARM matches, and that it doesn't negatively impact DirectXTex, etc. which is the primary consumer of these functions. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On SSE, storing
XMVectorReplicate(0.75f)withXMStoreByte4orXMStoreUByte4produces0x00000001instead of0x01010101.XMStoreByteN4similarly packs0.25fas0x1f1f1f1finstead of0x20202020. The documented Byte4, UByte4, and ByteN4 operations round each component before packing.We convert each clamped component to an integer before packing with SSE2 instructions, following the short-store implementation.
We checked the reproducer on SSE2, SSE4.1, and AVX2 and added a math3 regression that fails before and passes after, while the full suite retains its five existing math3 failures.
— ChipLab