Skip to content

Expressions are lazily resolved via session - #10763

Draft
robert3005 wants to merge 4 commits into
developfrom
rk/session-call
Draft

robert3005 wants to merge 4 commits into
developfrom
rk/session-call

Conversation

@robert3005

Copy link
Copy Markdown
Contributor

ScalarFnPlugin gains bind_any, which builds the function from options passed as &dyn Any, downcasting them to the function's own ScalarFnVTable::Options type and failing with an error naming that type otherwise.

ScalarFnSessionExt uses it to look a function up in the session's scalar function registry by name and construct the call:

  • scalar_fn(name, options) builds the registered function
  • call(name, options, args) applies it to args, which are arrays (giving the lazy ScalarFnArray) or expressions (giving the expression), through the new ScalarFnArg trait

`ScalarFnPlugin` gains `bind_any`, which builds the function from options passed
as `&dyn Any`, downcasting them to the function's own `ScalarFnVTable::Options`
type and failing with an error naming that type otherwise.

`ScalarFnSessionExt` uses it to look a function up in the session's scalar
function registry by name and construct the call:

- `scalar_fn(name, options)` builds the registered function
- `call(name, options, args)` applies it to `args`, which are arrays (giving
  the lazy `ScalarFnArray`) or expressions (giving the expression), through the
  new `ScalarFnArg` trait

A function without options takes `&EmptyOptions`. This lets code call functions
that another crate registers with the session without depending on that crate.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ps4mp7KF5c4p66aoSoXx5j
Signed-off-by: Claude <noreply@anthropic.com>
@robert3005 robert3005 added the changelog/feature A new feature label Oct 23, 2026
@codspeed

codspeed Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

Merging this PR will degrade performance by 2.33%

4 warnings, 1 note

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚠️ Unable to measure the execution of 1 benchmark

This benchmark measured no execution time, so its result is not comparable and counts as unchanged. This usually happens when the compiler removes the code under test.

Prevent compiler optimizations

See affected benchmarks

⚠️ 21 benchmarks are dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

⚠️ Base and head ran on different runner types

Some benchmarks with significant performance changes were compared across different runner types, so the changes might not come from the code. Run base and head on the same runner type, or use CodSpeed Macro Runners for more stable results.

Learn more about Macro Runners

Open the report in CodSpeed to investigate

ℹ️ 40 benchmarks have no flamegraph

The flamegraph wasn't generated: the symbols of the profiling trace could not be resolved, because no information about the language frames was collected.

See affected benchmarks

⚡ 4 improved benchmarks
❌ 6 regressed benchmarks
✅ 2394 untouched benchmarks
🆕 4 new benchmarks
⏩ 16 skipped benchmarks1
🗄️ 19 archived benchmarks run2

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
❌ CPU Simulation bench_compare_primitive[(10000, 32)] 72.9 µs 97.2 µs -25.05%
❌ CPU Simulation filter_powerlaw_by_mostly_false[250000] 138.7 µs 158.4 µs -12.39%
❌ CPU Simulation test_scan_filter[int] 3.2 ms 3.6 ms -11.1%
❌ CPU Simulation filter_random_by_mostly_false[250000] 163.1 µs 183.3 µs -11.01%
❌ CPU Simulation filter_powerlaw_by_power_law[250000] 166.1 µs 185.1 µs -10.25%
❌ CPU Simulation filter_random_by_power_law[250000] 166 µs 184.9 µs -10.24%
⚡ CPU Simulation test_repeated_scan_scalar_at[int] 318.8 µs 239.7 µs +33.03%
⚡ Walltime bitpack_blocked_compress_avx2 7.6 µs 6.7 µs +13.69%
⚡ CPU Simulation bench_compare_sliced_dict_primitive[(2000, 10000)] 79.9 µs 70.9 µs +12.75%
⚡ CPU Simulation prepare_and_compile[1000] 2.4 ms 2.2 ms +10.74%
🆕 CPU Simulation pack_bind[100] N/A 398.1 µs N/A
🆕 CPU Simulation pack_bind[200] N/A 594.5 µs N/A
🆕 CPU Simulation pack_bind[350] N/A 1.1 ms N/A
🆕 CPU Simulation pack_bind[500] N/A 1.5 ms N/A

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing rk/session-call (ceaaf07) with develop (91b38f4)

Open in CodSpeed

Footnotes

  1. 16 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

  2. 19 benchmarks were run, but are now archived. If they were deleted in another branch, consider rebasing to remove them from the report. Instead if they were added back, click here to restore them. ↩

`lazy(name, options, children)` builds an expression node that holds only the
function id and its options, so it needs neither a session nor the crate that
defines the function. `Expression::resolve(session)` later replaces each such
node with the function registered under that id, built from the options through
`ScalarFnPlugin::bind_any`.

A lazy node is a scalar function vtable, `LazyScalarFn`, like the foreign
placeholder: it reports the target function's id so it displays and hashes like
that function, accepts any arity, and fails to bind, execute, or serialize with
an error asking for resolution. `LazyScalarFnOptions` type-erases the options
behind a trait dispatching equality and hashing by concrete type, so any
`ScalarFnVTable::Options` type fits.

Unknown names, options of the wrong type, and arity mismatches are all reported
by `resolve`. `ScalarFnSessionExt::scalar_fn_by_id` looks a function up by an
already interned id; `scalar_fn(name, ...)` delegates to it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EAanvjKHbMGyQyN2nQAwU
Signed-off-by: Claude <noreply@anthropic.com>
An `Expression::Scalar` node no longer holds a `ScalarFnRef`. It holds the id
of the scalar function it calls, the function's options with their type erased
(`ErasedOptions`), and its children. Building an expression therefore needs
neither a session nor the crate that defines the function. `Expression::new`
builds a node from an id and erased options; the vtable constructors
(`new_expr`, `try_new_expr`) still check the arity up front.

Binding resolves the function: `Expression::bind(dtype, session)` and
`Scope`, which now carries the session, look each node's id up in the
session's scalar function registry, build the function from the options
through `ScalarFnPlugin::bind_any`, and check the options type and the number
of children, producing a `BoundExpression` as before. Options read for an id
the session did not register bind to the foreign placeholder.

Consequences across the API:

- `ScalarFnVTable::bind_options` takes `Option<&VortexSession>`: `Some` when an
  expression binds, so options that hold an expression, such as a lambda, can
  bind it; `None` when a bound node is rebuilt or a `ScalarFnArray` is built.
  `Lambda::bind` takes the session, and `Lambda::return_dtype` and
  `Lambda::is_infallible` are gone, since neither can be answered unbound.
- `Expression::return_dtype` is gone; bind and read the dtype instead.
- `ExprSerializeProtoExt::serialize_proto` takes the session, which serializes
  each function's options through the new `ScalarFnPlugin::serialize_any`.
- `ArrayRef::apply` takes the session and binds first.
  `vx_array_apply` and the C++ `Array::apply` take a session too.
- `Expression::is`, `as_opt`, `as_` and `contains` need `V: Default` to know
  the vtable's id, so every unit vtable derives `Default`.
- Unbound expressions display generically, as `id(child, ..., opts=options)`,
  with numbered children in `display_tree`; binding gives each function its
  own formatting. Tests that assert on display bind first.
- `DynamicComparison` and `RowCount` are registered by default, and the new
  `vortex_layout::initialize` registers `RowIdx`, since binding now needs
  every function an expression calls to be registered.

`ScalarFnSessionExt` gains `scalar_fn_by_id`, `scalar_fn_plugin` and
`resolve_scalar_fn`. The lazy expression modules from the previous commit are
removed, since every expression is now built this way.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EAanvjKHbMGyQyN2nQAwU
Signed-off-by: Claude <noreply@anthropic.com>
@github-actions
github-actions Bot temporarily deployed to docs-preview/pr-10763 October 24, 2026 01:24 Inactive
@robert3005
robert3005 marked this pull request as draft October 24, 2026 12:22
@robert3005 robert3005 changed the title Call scalar functions by name through the session Expressions are lazily resolved via session Oct 24, 2026
Binding now resolves each node's scalar function in the session, so timing
`ArrayRef::apply` would time resolution and binding along with execution. The
execution benchmarks bind, and optimize as a scan does, once outside the timed
region and time `apply_bound` on the bound expression instead: the case_when
benches through a shared `bench_bound`, and the list_length, list_sum,
dict_compare and FSST URL benches directly.

The large struct pack bench, which timed `Expression::return_dtype`, now times
binding, the step that replaced it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EAanvjKHbMGyQyN2nQAwU
Signed-off-by: Claude <noreply@anthropic.com>

This branch was successfully deployed

1 active deployment
docs-preview/pr-10763 — ceaaf073 Deployed Oct 24, 2026 by github-actions[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/feature A new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants