Skip to content

perf(protocol): speed up repr -- 4.3% less DEBUG exchange time - #3178

Open
dexhunter wants to merge 1 commit into
dpkp:masterfrom
dexhunter:perf/data-container-repr
Open

dexhunter wants to merge 1 commit into
dpkp:masterfrom
dexhunter:perf/data-container-repr

Conversation

@dexhunter

Copy link
Copy Markdown

TL;DR

  • Speed up DataContainer.__repr__ during offset-commit DEBUG logging.
  • Use direct field loops and reuse getattr within each call.
  • Median DEBUG exchange batch time: 211.54 ms -> 202.49 ms, 4.3% less time.
  • Native tests: pytest consumer coordinator/fetcher, producer sender/transaction manager, and protocol suites; 1,473 passed, one skipped.

What & Why

Before, __repr__ called a filter callback for every field, even when the object had no version.
Use separate loops for versioned and unversioned objects, and reuse getattr within each call.
The loops keep the same version checks, field order, value conversion, and output format.

The measured benefit applies to protocol DEBUG logging during offset commits.
The change removes callback work each time a request or response is formatted for a log message.

Validation

Measured with CPython 3.12.13 on Linux x86_64.
The DEBUG row reports median batch times across three runs.
Each batch repeats 56 exchanges and representation edge cases 64 times.
The exchange mix covers versions 0, 1, 2, 3, 4, 6, and 8.
It includes encoding, decoding, and in-memory DEBUG formatting; socket IO and fixture setup are excluded.

Situation Before After Change
Offset-commit DEBUG exchange batch, two partitions per request 211.54 ms 202.49 ms 4.3% less time
ApiVersions v0 request encoding, no fields 1.405 us 1.429 us 1.7% more time
ApiVersions v0 response decoding, 50 API keys 11.269 us 11.202 us No significant change
Fetch v12 response decoding, five topics and three partitions each 19.929 us 19.913 us No significant change

The encode/decode rows are pyperf means from the repository's protocol benchmark.
These paths do not call repr; both shown decode cases have no significant change.
ApiVersions encoding measured 1.7% more time, so no improvement is claimed for that case.

Reproduce the repository protocol benchmarks before and after applying the patch:

PYTHONPATH=. python kafka/benchmarks/protocol_old_vs_new.py --fast

The DEBUG workload follows the offset-commit exchanges exercised by test_send_offset_commit_request_versions.
The native validation command passed with 1,473 passed, one skipped:

python -m pytest test/consumer/test_coordinator.py test/consumer/test_fetcher.py test/producer/test_sender.py test/producer/test_transaction_manager_mock_broker.py test/protocol -q

pylint --errors-only kafka/protocol/data_container.py and git diff --check also pass.

Why this is safe and easy to maintain

The change is small and uses two simple loops.
It keeps field order, version checks, defaults, exception behavior, and the exact output.
The native consumer, producer, and protocol tests pass.
Exact-output checks also cover nested values, missing fields, Unicode, and exceptions.

AI assistance

Codex gpt-6.1-sol generated the candidate changes, tracked with Weco Observe.
Automated evaluation and local tests checked the selected change.

Weco share link: https://dashboard.weco.ai/share/OVatIaYwk1yutr81Fe4LWzSbAO5u4-Xe

Replace per-field filter callbacks with direct loops in
DataContainer.__repr__. Reuse getattr within each call and keep
version checks, field order, defaults, and output formatting.

Offset-commit DEBUG batches measured 211.54 ms before and 202.49 ms
after, using median batch times across three runs.

Validation: consumer coordinator/fetcher, producer sender/transaction
manager, and protocol tests: 1473 passed, 1 skipped. Changed-file
pylint errors-only and git diff --check pass.

Co-Authored-By: Aiden <aiden@weco.ai>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant