Skip to content

Normalize producer memoryviews to byte-oriented buffers - #3176

Open
edsonsims wants to merge 2 commits into
dpkp:masterfrom
edsonsims:fix/producer-memoryview-byte-layout
Open

edsonsims wants to merge 2 commits into
dpkp:masterfrom
edsonsims:fix/producer-memoryview-byte-layout

Conversation

@edsonsims

Copy link
Copy Markdown

Typed and multidimensional memoryview payloads are accepted by KafkaProducer.send, but their len() counts elements or leading dimensions. The partitioner and record encoders then interpret that count as bytes, producing incorrect partition hashes, size estimates and record lengths. For example, a two-element H view contains four bytes; its current legacy batches fail CRC validation, while its v2 record fails decoding.

Normalize serialized views before metadata-result sizing, partitioning and accumulation. C-contiguous views become unsigned byte views over the same backing object; strided views use their logical bytes. Existing bytes, bytearrays, nulls and header validation are unchanged.

Validation on Python 3.13.1/macOS:

  • The new public-send/wire regression selection produced 31 failures and 44 passing controls before the fix; all 75 pass afterward.
  • Real magic 0/1/2 encoding, CRC and decoding match bytes(view) after the existing MockBroker/accumulator fixture, with partition, size-limit and no-copy controls. This uses no real Kafka broker.
  • Full non-integration suite: 2,826 passed, 1 skipped. make lint and git diff --check pass.

The two mock bootstraps check refresh-loop identity rather than handshake speed. Use the existing fixture request_timeout_ms for both calls, retaining the real bootstrap operations and identity assertions.

Production timeouts and the memoryview regressions are unchanged.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant