Skip to content

feat(flow-php/parquet): DELTA_LENGTH_BYTE_ARRAY / DELTA_BYTE_ARRAY and arrow-rs default encodings #2702

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
norberttech merged 2 commits into from
Oct 21, 2026
Merged

feat(flow-php/parquet): DELTA_LENGTH_BYTE_ARRAY / DELTA_BYTE_ARRAY and arrow-rs default encodings #2702

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
9e7bffc
feat(flow-php/parquet): DELTA_LENGTH_BYTE_ARRAY / DELTA_BYTE_ARRAY an…
norberttech Oct 21, 2026
86192ef
test(flow-php/flow-php-ext): expect the delta encoding fixtures in th…
norberttech Oct 21, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
58 changes: 49 additions & 9 deletions documentation/components/libs/parquet.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -391,8 +391,11 @@ $writer->close();
### Writer Options

- `COLUMN_INDEX_TRUNCATE_LENGTH` - default: `64` - bytes a string/binary page's min/max in the column index are cut to.
- `DICTIONARY_ENABLED` - default: `true` - columns without a `COLUMNS_ENCODINGS` entry are written `RLE_DICTIONARY`,
except BOOLEAN and, under `WRITER_VERSION` 1, FIXED_LEN_BYTE_ARRAY (as arrow-rs and parquet-java); `false` writes the
fallback encoding instead.
- `DICTIONARY_PAGE_SIZE` - default: `1Mb` - maximum size of dictionary page; the pages of a column chunk after it is
reached are written `PLAIN`.
reached use the fallback encoding.
- `GZIP_COMPRESSION_LEVEL` - default: `9` - compression level for GZIP compression (applied only when GZIP compression
is enabled).
- `COLUMNS_BLOOM_FILTERS` - default: `null` - `BloomFilterOptions` by column flat path: those columns get a bloom filter
Expand All @@ -409,7 +412,9 @@ $writer->close();
column index); the offset index is written at every level.
- `STATISTICS_TRUNCATE_LENGTH` - default: `64` - bytes a string/binary column chunk's min/max statistics are cut to.
- `VALIDATE_DATA` - default: `true` - if set to `true` then writer will validate data against schema.
- `WRITER_VERSION` - default `1` - tells writer which version of parquet format should be used.
- `WRITER_VERSION` - default `1` - `2` writes data page v2. It also picks the fallback encoding used without a
dictionary: `PLAIN` under `1`; under `2` `RLE` (BOOLEAN), `DELTA_BINARY_PACKED` (INT32/INT64), `DELTA_BYTE_ARRAY`
(BYTE_ARRAY/FIXED_LEN_BYTE_ARRAY), `PLAIN` otherwise, as arrow-rs and parquet-java fall back.

Two most important options that can heavily affect memory usage are:

Expand Down Expand Up @@ -715,9 +720,12 @@ You can specify custom encodings for individual columns using flat path notation

### Available Encodings

Without a `COLUMNS_ENCODINGS` entry both engines write `RLE_DICTIONARY` (see `DICTIONARY_ENABLED`), and the
`WRITER_VERSION` fallback encoding past `DICTIONARY_PAGE_SIZE`.

#### PLAIN

The default encoding that stores values as-is without any compression scheme.
Stores values as-is without any compression scheme.

**When to use:**

Expand Down Expand Up @@ -748,7 +756,7 @@ Run Length Encoding with Dictionary compression. Values are stored in a dictiona
- Enumeration-like data
- Significant file size reduction (often 50-90% smaller)

**Supported types:** All types except `FIXED_LEN_BYTE_ARRAY`
**Supported types:** All types except `BOOLEAN`; `FIXED_LEN_BYTE_ARRAY` only under `WRITER_VERSION` 2

```php
$options = Options::default()->set(Option::COLUMNS_ENCODINGS, [
Expand All @@ -758,6 +766,8 @@ $options = Options::default()->set(Option::COLUMNS_ENCODINGS, [
]);
```

Past `DICTIONARY_PAGE_SIZE`, the rest of the chunk uses the `WRITER_VERSION` fallback encoding (PLAIN under v1).

#### DELTA_BINARY_PACKED

Delta encoding with binary packing for integer columns. Stores differences between consecutive values.
Expand All @@ -778,8 +788,35 @@ $options = Options::default()->set(Option::COLUMNS_ENCODINGS, [
]);
```

#### DELTA_LENGTH_BYTE_ARRAY

**Supported types:** Only `BYTE_ARRAY`

```php
$options = Options::default()->set(Option::COLUMNS_ENCODINGS, [
'comment' => Encodings::DELTA_LENGTH_BYTE_ARRAY,
]);
```

**When to use:** many distinct strings.

#### DELTA_BYTE_ARRAY

**Supported types:** `BYTE_ARRAY` and `FIXED_LEN_BYTE_ARRAY`

```php
$options = Options::default()->set(Option::COLUMNS_ENCODINGS, [
'url' => Encodings::DELTA_BYTE_ARRAY,
]);
```

**When to use:** sorted strings sharing prefixes.

### Using Custom Encodings

Both engines write exactly the encoding given and refuse one the column type cannot take; only `RLE_DICTIONARY` adds a
dictionary.

#### Basic Column Encoding

```php
Expand Down Expand Up @@ -914,11 +951,14 @@ $options = Options::default()->set(Option::COLUMNS_ENCODINGS, [

### Encoding Compatibility

| Encoding | INT32/INT64 | BYTE_ARRAY | BOOLEAN | FLOAT/DOUBLE | FIXED_LEN_BYTE_ARRAY |
|---------------------|-------------|------------|---------|--------------|----------------------|
| PLAIN | yes | yes | yes | yes | yes |
| RLE_DICTIONARY | yes | yes | yes | yes | no |
| DELTA_BINARY_PACKED | yes | no | no | no | no |
| Encoding | INT32/INT64 | BYTE_ARRAY | BOOLEAN | FLOAT/DOUBLE | FIXED_LEN_BYTE_ARRAY |
|-------------------------|-------------|------------|---------|--------------|----------------------|
| PLAIN | yes | yes | yes | yes | yes |
| RLE | no | no | yes | no | no |
| RLE_DICTIONARY | yes | yes | no | yes | v2 only |
| DELTA_BINARY_PACKED | yes | no | no | no | no |
| DELTA_LENGTH_BYTE_ARRAY | no | yes | no | no | no |
| DELTA_BYTE_ARRAY | no | yes | no | no | yes |

### Performance Guidelines

Expand Down
25 changes: 25 additions & 0 deletions documentation/upgrading.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -147,6 +147,31 @@ public function scan(FlowContext $context, FileScan $scan): Generator
| `from_parquet(...)->withSchema()`, `from_floe(...)->withSchema()` threw `InvalidArgumentException` | read every file by name against the declared schema: undeclared columns are not read, a missing nullable column is null |
| `from_cache($id)` without a cache instance described an empty schema | `SchemaNotDerivableException`; the plan runs unbound |

### 10) `flow-php/parquet` - the PHP engine reads and writes DELTA_LENGTH_BYTE_ARRAY / DELTA_BYTE_ARRAY; both engines write the same default encodings (dictionary on, arrow-rs fallbacks); COLUMNS_ENCODINGS is literal on both engines

| Before | After |
|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `WRITER_VERSION 2` fallback encoding of BYTE_ARRAY / FIXED_LEN_BYTE_ARRAY: PLAIN | `DELTA_BYTE_ARRAY`, as arrow-rs and parquet-java |
| dictionary overflow past `DICTIONARY_PAGE_SIZE`: the rest of the chunk PLAIN | the `WRITER_VERSION` fallback: under v2 `DELTA_BINARY_PACKED` (INT32/INT64), `DELTA_BYTE_ARRAY` (BYTE_ARRAY/FLBA); PLAIN under v1 |
| arrow writer: a non-dictionary `COLUMNS_ENCODINGS` entry kept the column's dictionary, used only once it fell back | turns the column's dictionary off: the entry is the column's encoding, as the PHP writer writes it |
| `DeltaBinaryPackedDecoder::decode(string $data, int $valueCount)` | `decode(BinaryBufferReader $reader, int $valueCount)`, leaving the reader at the end of its stream |
| `new PlainFlatColumnChunkBuilder($column, $options, $compression, $statistics)`; `DeltaBinaryPackedColumnChunkBuilder` | `new FlatColumnChunkBuilder($column, $options, $compression, $statistics, ValueStorage $valueStorage)`; `DeltaBinaryPackedColumnChunkBuilder` removed |
| `ValueStorage` without an encoding | `ValueStorage::encoding(): Encodings`; `ValueStorage::size()` renamed to `plainSize()`, the PLAIN-encoded byte size |
| `new RLEDictionaryChunkBuilder($column, $options, $compression, $statistics)` | `new RLEDictionaryChunkBuilder($column, $options, $compression, $statistics, ValueStorage $fallback)` |
| a failed `Writer::write()` / `Writer::open()` left a 4-byte `PAR1` file behind | the file is removed; the exception is rethrown |
| PHP writer: columns without `COLUMNS_ENCODINGS` written without a dictionary | `RLE_DICTIONARY` (except BOOLEAN, and FIXED_LEN_BYTE_ARRAY under v1), as the arrow writer, arrow-rs and parquet-java; `Option::DICTIONARY_ENABLED` `false` restores the fallback encodings |
| - | `Option::DICTIONARY_ENABLED` (default `true`), on both engines |
| PHP writer `WRITER_VERSION 2`: BOOLEAN written PLAIN | `RLE`, as the arrow writer |
| `RLE_DICTIONARY` on a FIXED_LEN_BYTE_ARRAY column threw | allowed under `WRITER_VERSION 2`; still refused under 1 |
| PHP writer fell back once the dictionary exceeded `DICTIONARY_PAGE_SIZE` | once it reaches it (`>=`), as arrow-rs |
| PHP writer v2 chunks listed only the data page encoding | also `RLE` for the levels, as arrow-rs |
| PHP writer `DELTA_BINARY_PACKED`: a miniblock whose largest delta sits at or just above a power of two from about 2^47, or needs more than 55 bits, was packed wrong and read back wrong values | packed exactly; rewrite `WRITER_VERSION 2` files written by earlier versions whose INT32/INT64 columns hold such deltas |
| `COLUMNS_ENCODINGS` `RLE_DICTIONARY` on a BOOLEAN column: the PHP writer wrote a dictionary, the arrow writer ignored it | refused by both engines (`InvalidArgumentException`), as arrow-rs and parquet-java never dictionary-encode booleans |
| arrow writer: a `COLUMNS_ENCODINGS` entry the column type cannot take panicked (`\Error: Rust panic`) or was ignored (FIXED_LEN_BYTE_ARRAY dictionary under v1) | `InvalidArgumentException` with the PHP writer's message |
| INT32 `DELTA_BINARY_PACKED` with consecutive values more than 2^31 apart: written with 64-bit deltas arrow-rs refuses, read back as wrong values | 32-bit wrapping deltas, as arrow-rs and parquet-java; such INT32 files from earlier versions must be rewritten |

The `arrow` extension must be rebuilt.

---

## Upgrading from 0.45.x to 0.46.x
Expand Down
7 changes: 6 additions & 1 deletion src/extension/arrow-ext/src/parquet/library.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ use ext_php_rs::flags::{ClassFlags, DataType};
use ext_php_rs::prelude::*;
use ext_php_rs::types::{ZendClassObject, ZendHashTable, Zval};
use ext_php_rs::zend::ce;
use parquet::arrow::ArrowSchemaConverter;
use parquet::basic::Type as PhysicalType;
use parquet::file::metadata::ParquetMetaData;

Expand Down Expand Up @@ -911,10 +912,14 @@ impl RustParquetFileWriter {
.collect::<Result<Vec<_>, Error>>()
.map_err(refused)?;
let scalar = columns.iter().all(|column| column.cell.is_scalar());
let leaves = ArrowSchemaConverter::new()
.convert(&schema)
.map_err(Error::Parquet)
.map_err(refused)?;
let writer = ParquetWriter::open(
PhpSink::new(Arc::clone(&stream)),
schema,
options::properties(compression, options).map_err(refused)?,
options::properties(compression, options, &leaves).map_err(refused)?,
)
.map_err(refused)?;

Expand Down
Loading
Loading