[rapour]

Summary

• Tracking Issue: Tracking Issue: Elias-Fano Encoding #10605

This is the first step towards Elias-Fano: the Unary encoding, which will hold the bucket counts of EF's upper bits. It lives in a new encodings/elias-fano crate, which will also hold the EliasFano parent, as discussed on #10377.

It is deliberately naive. There is no index, no slice kernel and no compute kernel, so the PR is mostly boilerplate, and the benchmarks set the baselines that later work is measured against. It is in no file edition and vortex-file does not register it, so the default writers reject it.

Next up is slicing for Unary, then the naive EliasFano parent with upper and lower children.

Changes

Layout. One BitBuffer. Each value v is written as v set bits, then one unset bit that closes it. The array takes len + num_ones bits, where num_ones is the sum of the values, and the value at index k is the number of set bits between the k-th unset bit and the one before it. For [2, 0, 3], the bits are 110 0 1110.

The metadata is one u64, num_ones, the number of set bits, which gives the exact bit length together with the array's length. The dtype must be a non-nullable unsigned integer, and the input need not be sorted.

As EF's upper child. Unary will hold one count per bucket, so its length is the number of buckets, not the parent's length. Its bits are then the upper-bits array of Vigna's quasi-succinct indices (section 4), plus one unset bit that closes the last bucket. EF will read those bits through an extension trait, the way FoR reads BitPacked.

Benchmarks. Each shape is the bucket counts that EF's upper child would hold for 16,384 sorted values:

• Dense, about one value per bucket;
• Spread, about half a value per bucket, the sparsest EF gets;
• Runs, 32 values in every bucket;
• Gap, with a long run of empty buckets.

Wiring. The crate is a workspace member and dependency, and runs in CodSpeed shard 8.

[codspeed]

Merging this PR will degrade performance by 10.95%

3 warnings, 1 note

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚠️ 17 benchmarks are dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

⚠️ Base and head ran on different runner types

Some benchmarks with significant performance changes were compared across different runner types, so the changes might not come from the code. Run base and head on the same runner type, or use CodSpeed Macro Runners for more stable results.

Learn more about Macro Runners

Open the report in CodSpeed to investigate

ℹ️ 40 benchmarks have no flamegraph

The flamegraph wasn't generated: the symbols of the profiling trace could not be resolved, because no information about the language frames was collected.

See affected benchmarks

❌ 1 regressed benchmark
✅ 2407 untouched benchmarks
🆕 20 new benchmarks
⏩ 12 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌	CPU Simulation	test_scan_filter[int]	3.1 ms	3.4 ms	-10.95%
🆕	CPU Simulation	decode[Dense]	N/A	1.4 ms	N/A
🆕	CPU Simulation	decode[Gap]	N/A	2.1 ms	N/A
🆕	CPU Simulation	decode[Runs]	N/A	91.1 µs	N/A
🆕	CPU Simulation	decode[Spread]	N/A	2.8 ms	N/A
🆕	CPU Simulation	encode[Dense]	N/A	1.4 ms	N/A
🆕	CPU Simulation	encode[Gap]	N/A	2 ms	N/A
🆕	CPU Simulation	encode[Runs]	N/A	77 µs	N/A
🆕	CPU Simulation	encode[Spread]	N/A	2.7 ms	N/A
🆕	CPU Simulation	scalar_at_random[Dense]	N/A	477.9 µs	N/A
🆕	CPU Simulation	scalar_at_random[Gap]	N/A	562 µs	N/A
🆕	CPU Simulation	scalar_at_random[Runs]	N/A	344 µs	N/A
🆕	CPU Simulation	scalar_at_random[Spread]	N/A	643.6 µs	N/A
🆕	CPU Simulation	scalar_at_sequential[Dense]	N/A	470.2 µs	N/A
🆕	CPU Simulation	scalar_at_sequential[Gap]	N/A	545.7 µs	N/A
🆕	CPU Simulation	scalar_at_sequential[Runs]	N/A	359.3 µs	N/A
🆕	CPU Simulation	scalar_at_sequential[Spread]	N/A	629.8 µs	N/A
🆕	CPU Simulation	slice_decode[Dense]	N/A	1.5 ms	N/A
🆕	CPU Simulation	slice_decode[Gap]	N/A	2.1 ms	N/A
🆕	CPU Simulation	slice_decode[Runs]	N/A	108.6 µs	N/A
...	...	...	...	...	...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing rapour:feat/elias-fano-unary-delta (e96311a) with develop (d270200)}

Footnotes

1. 12 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[connortsui20]

Ok so I just realized that there are multiple ways that we can imagine the canonical / semantic values of the upper bits.

For example, these two arrays can be thought of as the same:

"delta" encode upper values:   [0, 0, 2, 2, 2]
                                0, 0,  2,  0, 0
->                              1, 1, 001, 1, 1, (0)


                    bucket      00   01   10
bucket counts:                 [2,   0,    3]
->                             110  0    1110 


                                11001110

Notice how these are the same thing!

So I think what we need to decide is what we actually want this logical array to represent, should it be the "delta" encoded upper values (which are not exactly delta but close), or the "bucket counts", by which I mean how many upper bit values fall into each bucket.

I was thinking about the buckets, but I can see how both of these are valid representations of the data. However, in my mind, a general Unary encoding should probably be represented as n 1s followed by a closing 0.

I don't think would change basically anything about the Elias-Fano encoding, but it does mean that this implementation of Unary would have to change. For example, we should not require the inputs to be sorted, as a start.

[rapour]

Thanks, that makes sense. Let's go with bucket counts: Unary becomes a general unsigned encoding where each value v is written as v ones and a closing zero, and the input doesn't need to be sorted. EF's upper child then holds one count per bucket instead of one value per element.