[OmarGamal10]

This PR implements the streaming window functions path as part of my GSoC project (MDEV-38970). Instead of materializing rows into a temporary table and sorting there, this executes window functions on the fly when the streaming criteria are met.

What's been added

• Window_funcs_sort_streaming object, attached to JOIN_TAB via window_funcs_streaming_step. Contains:

  • setup(): Sets up cursor managers and partition trackers (Group_bound_tracker), sets phase to computation, and configures the simple aggregator for aggregate functions.
  • process_row(): Iterates over window functions for the current row in TABLE::record[0]. Checks if a new partition boundary was crossed to clear and notify cursors; otherwise, notifies cursors of the next row.
  • cleanup(): Cleans up trackers and cursor managers.

• have_streaming_window_funcs(): Checks if the query's window functions can be streamed in a single pass:

  • Verifies all window functions share compatible ordering (find_longest_compatible_order()).
  • Checks that functions are streamable: ranking functions (ROW_NUMBER, RANK, DENSE_RANK) and running aggregates (SUM, COUNT, AVG, MIN, MAX).
  • Rejects DISTINCT aggregates and aggregates inside PARTITION BY or ORDER BY.
  • Validates frame specifications:
    • Ranking functions (prohibited frame).
    • Explicit ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW.
    • Default RANGE frames whose ordering covers a unique key.
  • Verifies order compatibility with the main query ORDER BY and GROUP BY.
  • If the window order is longer than the main query ORDER BY, ensures the extra columns belong to the first table, are deterministic, and are not expensive.

• Single sort order replacement: When the window order is longer than the query ORDER BY (streaming_wf_order_is_longer), replaces this->order with win_func_longest_order so filesort on the first table satisfies both in one go.

• set_window_order_uniqueness() / order_contains_unique_index(): For single-table queries, checks if the window ordering covers a UNIQUE NOT NULL index. When true, no peers can exist, allowing default RANGE to behave like ROWS and stream.

• test_if_need_tmp_table(): Skips creating a temporary table if window functions can stream, falling back to materialization only if a temp table is required for other reasons (e.g., const tables only, or GROUP BY without a loose index scan).

• Group by with a unique index can be optimized away when window functions exist, an existing optimization that was skipped for window functions

• Loose index scan support: When a loose index scan is used, rows come out already grouped, allowing window functions to stream over them. HAVING is evaluated before process_row() to filter rows out prior to computing window functions.

• end_compute_win_func(): Attached to the last real table before end_send. Runs process_row() on the live row and passes it directly to end_send() without materializing into a temporary table.

• is_streamable(): Virtual method added to Item_sum, overridden for supported functions.

• Tests (win_streaming.test / win_streaming.result):

  • Single-table queries, joins, and derived tables.
  • Loose index scan interactions and unique index detection.
  • Running aggregates and LIMIT early-exit optimization.
  • Comparative validation against SQL_BUFFER_RESULT to ensure streaming matches materialized output.
  • Negative test cases verifying proper fallback to temporary tables.

[gemini-code-assist]

Code Review

This pull request introduces support for streaming window functions, allowing certain functions like row number, rank, and dense rank to be computed on the fly without materializing into a temporary table. The code review feedback highlights several critical issues, including potential null pointer dereferences (crashes) due to missing checks on partition_list and order_list, a logic bug in compare_order_lists when handling trailing constants, and an ignored return value in cursor setup. Additionally, the reviewer suggests caching the THD pointer to avoid expensive thread-local lookups in the performance-critical per-row execution path, and resolving an inconsistency in how default versus explicit frames are handled.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[mariadb-OlegSmirnov]

win_streaming.test does not prove correctness of results. Would be nice to see the same query executed in both variants: materialized vs streaming so results are comparable (maybe we can dismiss the streaming by adding SQL_BUFFER_RESULT after SELECT?).
As I mentioned earlier, do we really need EXPLAIN FORMAT=JSON everywhere or EXPLAIN EXTENDED is enough?
I also suggest using the pattern

let $q = SELECT ...;
eval $q;
eval explain format=json $q;

in tests to avoid query text duplication.

[OmarGamal10]

win_streaming.test does not prove correctness of results. Would be nice to see the same query executed in both variants: materialized vs streaming so results are comparable (maybe we can dismiss the streaming by adding SQL_BUFFER_RESULT after SELECT?). As I mentioned earlier, do we really need EXPLAIN FORMAT=JSON everywhere or EXPLAIN EXTENDED is enough? I also suggest using the pattern

let $q = SELECT ...;
eval $q;
eval explain format=json $q;

in tests to avoid query text duplication.

I thought about this actually (running queries side by side) but thought I would need an optimizer switch just for that, I will try SQL_BUFFER_RESULT

[Olernov]

GROUP_BY relaxation breaks multi-table joins (wrong results). Repro:

CREATE TABLE tg (a INT, b INT, KEY(a,b));
INSERT INTO tg VALUES (1,1),(1,2),(2,1),(2,2),(2,3),(3,1);
CREATE TABLE t2 (a INT, x INT, KEY(a));
INSERT INTO t2 VALUES (1,10),(2,20),(2,21),(3,30);

SELECT tg.a, rank() OVER (ORDER BY tg.a) FROM tg JOIN t2 ON tg.a=t2.a GROUP BY tg.a;
--   streamed:      9 rows, a = NULL everywhere, rnk = 1,1,3,3,3,3,3,3,9
SELECT SQL_BUFFER_RESULT tg.a, rank() OVER (ORDER BY tg.a) FROM tg JOIN t2 ON tg.a=t2.a GROUP BY tg.a;
--   materialized:  3 rows, 1 -> 1, 2 -> 2, 3 -> 3   (correct)

Streaming such queries can provide correct results only when the rows arriving at the streaming callback are already grouped which is only true for a single-table loose index scan (using_index_for_group_by). For a multi-table join, grouping is performed by end_send_group, which the streaming attach overrides when it sets last_real_tab->next_select = end_compute_win_func. As a result, the grouping is silently bypassed.

This concern also applies not only to multi-table plans, but also to single-table plans that group via end_send_group. The last table emits ungrouped rows and the grouping executor collapses, overriding its next_select throws the grouping away.

If we still want to cover GROUP BY cases, the eligibility of streaming must be tightened, for example: confirm the plan is using_index_for_group_by and there's a single non-const table, rather than inferring it from simple_group/!need_tmp.

[OmarGamal10]

GROUP_BY relaxation breaks multi-table joins (wrong results). Repro:

CREATE TABLE tg (a INT, b INT, KEY(a,b));
INSERT INTO tg VALUES (1,1),(1,2),(2,1),(2,2),(2,3),(3,1);
CREATE TABLE t2 (a INT, x INT, KEY(a));
INSERT INTO t2 VALUES (1,10),(2,20),(2,21),(3,30);

SELECT tg.a, rank() OVER (ORDER BY tg.a) FROM tg JOIN t2 ON tg.a=t2.a GROUP BY tg.a;
--   streamed:      9 rows, a = NULL everywhere, rnk = 1,1,3,3,3,3,3,3,9
SELECT SQL_BUFFER_RESULT tg.a, rank() OVER (ORDER BY tg.a) FROM tg JOIN t2 ON tg.a=t2.a GROUP BY tg.a;
--   materialized:  3 rows, 1 -> 1, 2 -> 2, 3 -> 3   (correct)

Streaming such queries can provide correct results only when the rows arriving at the streaming callback are already grouped which is only true for a single-table loose index scan (using_index_for_group_by). For a multi-table join, grouping is performed by end_send_group, which the streaming attach overrides when it sets last_real_tab->next_select = end_compute_win_func. As a result, the grouping is silently bypassed.

This concern also applies not only to multi-table plans, but also to single-table plans that group via end_send_group. The last table emits ungrouped rows and the grouping executor collapses, overriding its next_select throws the grouping away.

If we still want to cover GROUP BY cases, the eligibility of streaming must be tightened, for example: confirm the plan is using_index_for_group_by and there's a single non-const table, rather than inferring it from simple_group/!need_tmp.

Yes, you're right I focused on cases where group by should stream and missed this. I handled this and added some tests. I looked into what makes a group by use end_send other than using a loose index scan. The other case that happens is when GROUP BY is rewritten into an ORDER BY when a unique non-null index exists on the group key. But for now this is skipped explicitly whenever grouping exists, I am experimenting with it to make sure I don't miss something.

  /*
     Check if we can optimize away GROUP BY/DISTINCT.
     We can do that if there are no aggregate functions, the
     fields in DISTINCT clause (if present) and/or columns in GROUP BY
     (if present) contain direct references to all key parts of
     an unique index (in whatever order) and if the key parts of the
     unique index cannot contain NULLs.
     Note that the unique keys for DISTINCT and GROUP BY should not
     be the same (as long as they are unique).

     The FROM clause must contain a single non-constant table.
  */
  if (table_count - const_tables == 1 && (group || select_distinct) &&
      !tmp_table_param.sum_func_count &&
      (!join_tab[const_tables].select ||
       !join_tab[const_tables].select->quick ||
       join_tab[const_tables].select->quick->get_type() != 
       QUICK_SELECT_I::QS_TYPE_GROUP_MIN_MAX) &&
      !select_lex->have_window_funcs())

[Olernov]

@OmarGamal10 , please also apply this cleanup.patch

[Olernov]

Next cleanup patch:
cleanup2.patch, please apply too

[Olernov]

@OmarGamal10 , everything looks good now. One more detail to add: a proper commit message, and you can remove the "Draft" flag from the PR.