Skip to content

ORDER BY inside aggregate calls in the expression API - #713

Open
evertlammerts wants to merge 3 commits into
duckdb:v2.0-cyanopterafrom
evertlammerts:ordered-aggregates
Open

evertlammerts wants to merge 3 commits into
duckdb:v2.0-cyanopterafrom
evertlammerts:ordered-aggregates

Conversation

@evertlammerts

Copy link
Copy Markdown
Member

The expression API had no way to write an aggregate with an ORDER BY inside its call, such as first(x ORDER BY t). Func supported FILTER and IGNORE NULLS but not ORDER BY. So a deterministic first, last or string_agg per group had to be raw SQL through SQLExpression, which is what Narwhals does today (narwhals-dev/narwhals#4099, duckdb/duckdb#19252).

The obvious spelling does not work. fn() passes every keyword through as a named SQL argument, so order_by= produces the wrong SQL:

fn("first", col("x"), order_by=col("t"))
# before: "first"("x", "order_by" := "t"), a named argument
col("x").first(order_by=col("t"))
# after:  "first"("x" ORDER BY "t")

What changes

Aggregate calls (Func and Distinct) get an .order_by(*keys) method. It puts the keys inside the parentheses and works alongside the other clauses, also under .over():

"first"("x" ORDER BY "t" DESC NULLS LAST) FILTER (WHERE ...)

Every generated aggregate method also accepts order_by= as a shortcut for the same thing. Keywords passed to fn() are still named SQL arguments. To order a call built with fn(), use the method.

Design choices

  • A method, not just a keyword. fn() has no keyword to spare, because all of its keywords become named SQL arguments. A method also combines with where(), ignore_nulls(), n_unique(), the .str() wrappers and over() the same way where() does.
  • Keys are checked like over() keys. For example, a bare string is a TypeError.
  • Python catches what it can and the binder catches the rest. Python rejects .order_by() on a node that is not a call, and an empty key list. What only the binder can judge fails at bind time, as it does for where(). Examples are ORDER BY on a scalar function, or a DISTINCT aggregate ordered by a key that is not one of its arguments.
  • Clause order follows the SQL grammar. ORDER BY comes before IGNORE NULLS.

Tested behavior

col("x").first(order_by=col("t").desc()) renders "first"("x" ORDER BY "t" DESC) and gives the same results as that SQL in these cases:

  • NULLs in the key or the value
  • combined with FILTER
  • under OVER, with and without IGNORE NULLS
  • on an empty group

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant