[fjlinz]

Users of TaskManager can now choose the compression of the output file and the auto-flush setting of the output tree:

• SetCompressionSettings(int) — ROOT compression setting (100 × algorithm + level), e.g. 505 for ZSTD level 5 (ROOT::RCompressionSetting::EDefaults::kUseGeneralPurpose). Also applied to cloned branches in kCopyTree mode, which otherwise keep the compression of the input file.
• SetAutoFlush(Long64_t) — forwarded to TTree::SetAutoFlush().

Default behaviour is unchanged: if the setters are not called, ROOT's compiled default is used (101 = zlib level 1 since ROOT 6.20).

Some results from my study using ZSTD 505 instead of default zlib 101 using ROOT 6.32:

File	Compression	Size	Full read time
Simulation (1k events)	zlib 1 (current)	188 MB	2.5 s
	ZSTD 5	165 MB (−12 %)	1.3 s
Reconstructed data (20k events)	zlib 1 (current)	561 MB	6.3 s
	ZSTD 5	523 MB (−7 %)	3.9 s

Read time is the best of three passes over all branches. Other settings were also tested. LZ4 reads fastest but produces slightly larger files. LZMA gives the smallest files but reads 5–9× slower. ZSTD 9 compresses only 1–2 % better than ZSTD 5 at about 3× the write time.

Requirements

• ROOT ≥ 6.16 to build (ROOT::RCompressionSetting was introduced in 6.16).
• ROOT ≥ 6.20 to use ZSTD (505), which is the recommended setting

Tests

New unit test TaskManager.CompressionAndAutoFlush checks the compression of the output file and of all branches, as well as the auto-flush value.

[fjlinz]

@lubynets I think this is very useful option to have for large data productions to be able to control and optimize these parameters.

Fyi: Before the next tag, I have 2-3 more requests in the pipeline which should go in before.