ADR 0021: bzip2 and LZMA join the compressors

ADR 0020 made PPMd the default and left every other compressor deferred. This adds two, bzip2 and lzma, both from the Python standard library, because past 32 KiB DEFLATE loses shared content entirely and PPMd keeps less of it as objects grow, while LZMA keeps it at any size its dictionary holds. PPMd stays the default.

The measurement

Columns written as the normalizer writes them, one JSON value per line. Distribution is the distance across two vocabularies minus the distance within one; copy is the distance to an unrelated column minus the distance to the same column with 1 % of its bytes changed.

object

zlib

bzip2, 900 kB block

lzma, 8 MiB dictionary

ppmd

16 KiB, distribution / copy

0.127 / 0.828

0.454 / 0.661

0.169 / 0.884

0.413 / 0.531

128 KiB

0.028 / 0.019

0.313 / 0.590

0.084 / 0.906

0.238 / 0.298

1 MiB

0.005 / 0.003

0.079 / 0.070

0.085 / 0.907

0.160 / 0.178

4 MiB

0.001 / 0.001

0.025 / 0.025

0.030 / 0.906

0.090 / 0.090

throughput

~45 MB/s

~14 MB/s

1.7 to 3.4 MB/s

~15 MB/s

Each covers what the others miss. LZMA finds a copy at any size its dictionary holds the pair in, and little of a distribution. PPMd keeps the distribution of large objects and little of a copy. bzip2 sees both, but only within its block, and nothing past it. So the choice is the caller’s, and the default stays the one columns of a dataset need.

What each one is

  • compressor="bzip2" is bz2.BZ2Compressor at bzip2_block_size_100k (1 to 9, default 9, the largest block). The block is its window: two objects in different blocks share nothing.

  • compressor="lzma" is LZMA2 in the raw format, preset 6, with lzma_dictionary_bytes (4096 to 1610612736, the codec’s own bounds; default 8388608, preset 6’s own). The dictionary is its window. The raw format carries no container: the .xz one adds 59 bytes to every stream, a constant that inflates the distance between small objects, the same defect ADR 0020 found in the original binary’s file-name header.

A compressor exposes the settings that change what C(xy) can see across a pair – a window, a block, a model’s order or memory – and fixes the rest here. That is why the LZMA preset is not a setting: preset 9 measured the same as preset 6 at every size above. compression_level remains for zlib and gzip as the published contract it already is, and is not a precedent for this rule.

In damicore_distance each compressor is its own settings model, and DistanceConfig holds one of them, chosen by name: PpmdCompressor, DeflateCompressor for zlib and gzip, which share their one setting, Bzip2Compressor and LzmaCompressor. A setting for another compressor is then not refused by a rule but absent: the model has no field to hold it, and a result’s identity holds no default of a compressor it did not use. Each setting keeps one name on every surface – the model field, damicore.run’s argument, the CLI flag and the manifest field – so damicore, whose signature stays flat (ADR 0013), derives which arguments each compressor accepts from the models rather than restating them, and a test holds the four surfaces to the models. A compressor added later is a model added to that union.

What it costs

Every worker holds one stream, and preflight charges each compressor’s stream memory once per worker. Measured peak resident memory, on incompressible data, on Linux x86_64:

stream

measured

charged

bzip2, block 9

9.5 MiB

10 MiB at every block size

lzma, dictionary 4 KiB / 1 / 4 / 8 / 16 / 64 MiB

1.5 / 14.5 / 50.7 / 94 / 187 / 675 MiB

12 x dictionary + 4 MiB

zlib, gzip, level 6

–

256 KiB, the two tables zconf.h sizes; the few kilobytes it adds are not charged

ppmd

–

the configured model memory (ADR 0020)

At the defaults, four workers stay inside the default 512 MiB limit for every compressor. DEFLATE’s state was never charged before this record; it is now, so a zlib estimate rises by 256 KiB per worker.

How a result names its codec

bz2 and lzma call the system’s libbz2 and liblzma, and Python exposes the version of neither. A result is a function of the encoder, so the runtime fingerprint names each by behavior instead: the compressed size of a fixed payload, recorded beside the zlib versions and the pyppmd version. A system library that encodes that payload differently changes that size, and a checkpoint or a completed run written under the old one is refused rather than resumed. A difference that leaves the payload’s size alone, or shows only at larger blocks or dictionaries than the probe uses, goes unseen.

Status

ADR 0005 stands: like zlib and PPMd, both compressors open a fresh stream per pair and keep no state across pairs, so the process pool is unchanged; what their larger streams change is the memory per worker, which preflight now charges. The window each compressor declares for a gate is deferred to issue 59, which has the gate to use it. Revisit the LZMA charge if the preset changes, since the match finder’s memory is what the factor of 12 measures.