Skip to content

Fix rounding in SSE byte stores - #357

Open
Nathan Chandrasekar (vera-rubin) wants to merge 1 commit into
microsoft:mainfrom
vera-rubin:fix/byte-store-rounding
Open

Nathan Chandrasekar (vera-rubin) wants to merge 1 commit into
microsoft:mainfrom
vera-rubin:fix/byte-store-rounding

Conversation

@vera-rubin

Copy link
Copy Markdown
Contributor

On SSE, storing XMVectorReplicate(0.75f) with XMStoreByte4 or XMStoreUByte4 produces 0x00000001 instead of 0x01010101. XMStoreByteN4 similarly packs 0.25f as 0x1f1f1f1f instead of 0x20202020. The documented Byte4, UByte4, and ByteN4 operations round each component before packing.

We convert each clamped component to an integer before packing with SSE2 instructions, following the short-store implementation.

We checked the reproducer on SSE2, SSE4.1, and AVX2 and added a math3 regression that fails before and passes after, while the full suite retains its five existing math3 failures.

— ChipLab

@walbourn

Copy link
Copy Markdown
Collaborator

I'll need to look closely at these proposed changes. There are a number of issues where I need to ensure that ARM matches, and that it doesn't negatively impact DirectXTex, etc. which is the primary consumer of these functions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants