Repository navigation
Conversation
EXPLAIN (ANALYZE, BUFFERS) did not include buffer usage collected by query executors in the instrumentation deposited on the coordinator. Send per-node buffer usage to the coordinator and accumulate block counters across all workers within each slice. Node statistics are inclusive, so a parent also contains buffer usage accumulated by its descendants within the same slice.
leborchuk
left a comment
leborchuk
left a comment
There was a problem hiding this comment.
What it we add regression tests for the new behaviour?
We already have function get_explain_analyze_wal_output in gp_explain.sql. We could add function to analyze buffers too. Not exact number, but presence line with buffer usage in explain output.
|
I've added a regression test for this behavior: |
|
Maybe we should add per-segment buffer usage alongside the aggregated totals, perhaps behind a dedicated flag? This would help identify uneven buffer usage across segments while keeping the default output concise. |
There was a problem hiding this comment.
the comment is outdated. It is implementing now)
Co-authored-by: Alena Rybakina <alenka.rybakina@gmail.com> Co-authored-by: Alena Rybakina <58230554+Alena0704@users.noreply.github.com>
Yes, it would be nice to identify such situations : ) Do you mean reporting the maximum buffer usage across segments (as we currently do for the number of tuples processed, ntuples), or reporting buffer usage separately for each segment? I assume you mean the latter. And by a dedicated flag, you mean a GUC, right? |
Yes, I mean reporting buffer usage for each segments) but now I'm not sure about guc - it is fine to report it every time when report buffer usage. |
|
@Alena0704 Alena, is it how you want to see it in a plan? I want to synchronize expectations / formatting / naming before sending a commit. |
How about this format? Each segment’s counters should be the sum across its workers. Omit zero-valued counters and rows with no nonzero counters; don’t print |
|
Then I will open a new PR where I'll implement VERBOSE version for buffer accesses output. This one we can merge independently. |
|
I suggest adding regression coverage for the per-node Buffers: output, using regular expressions to normalize unstable values. See explain_filter() in PostgreSQL’s regression tests for an example. |
|
I'll update the current test suite |
Alena0704
left a comment
Alena0704
left a comment
There was a problem hiding this comment.
Everything looks correct. Thank you!
|
Thank you all for the review! : ) |
What does this PR do?
EXPLAIN (ANALYZE, BUFFERS) did not include buffer usage collected by query executors in the instrumentation deposited on the coordinator.
Send per-node buffer usage to the coordinator and accumulate block counters across all workers within each slice. Node statistics are inclusive, so a parent also contains buffer usage accumulated by its descendants within the same slice.
It adds text like:
Buffers: shared hit=2for each node plan.For example, the query from
gporca.sql:To verify where the buffer accesses reported by this feature occur, I created a local diagnostic patch that logs every increment of the shared and local buffer hit/read counters while a plan node is running. The output below was produced from those log records.
(i) A row can be matched to the execution plan using its Slice and Node name. The Segment identifies the worker that produced the record, while Node is the internal plan-node ID and helps distinguish nodes with the same name.
(ii) The first table groups direct buffer accesses by plan node and relation. For example, it shows which tables and indexes were accessed by a particular Sort or Seq Scan node. The second table aggregates these records per plan node across all relations.
(iii) Direct hits and Direct reads are accesses performed by the node itself. Direct calls is their sum:
Inclusive statistics also include accesses performed by descendant nodes in the same slice. Therefore, a parent node contains its own direct accesses plus the inclusive accesses of its children:
For example, if a Seq Scan performs one buffer hit, its parent Aggregate and Gather Motion nodes in that slice also report one inclusive hit.
(iv) Inclusive statistics do not cross a slice boundary. A node does not include accesses made by a descendant running in another slice. The coordinator separately accumulates the statistics for each plan node across all workers executing that slice.
(v) The displayed output was extracted from the log of one worker instance.
It therefore represents only that worker. To compare the result with the coordinator's EXPLAIN output, records from all workers participating in the slice must be combined (in this case we have two worker instances, so we need to multiply by 2).
(vi) The diagnostic logging calls are placed directly next to the increments of pgBufferUsage.shared_blks_hit, pgBufferUsage.shared_blks_read, pgBufferUsage.local_blks_hit, and pgBufferUsage.local_blks_read. Each record is attributed to the plan node that was executing when the counter was incremented.
(vii) The diagnostic output covers does not include the separate planning buffer statistics shown under the Planning section of EXPLAIN.
(vii) My cluster topology is:
Type of Change
Test Plan
make installcheckmake -C src/test installcheck-cbdb-parallelImpact
Make
BUFFERSoption available inEXPLAIN (ANALYZE, BUFFERS).Checklist