# ADR 0020: PPMd is the default compressor `AGENTS.md` defers alternative compressors. This reopens that decision for one compressor, PPMd variant I with model restart, and makes it the default. Two facts decide it: the 0.3 default measures the wrong thing above 32 KiB without saying so, and the implementation DAMICORE was first published with already defaulted to PPMd. ## The original implementation `brunokim/damicore-python` (2016, ) has no DOI, so this record is where it is named. Its `ncd.py` defaults to `ppmd` and runs Dmitry Shkarin's `ppmd` binary as `ppmd e -o6 -f `. It never passes `-m` or `-r`: its `--memory` option is parsed and dropped, so memory and restoration are the binary's defaults. The binary of that era, Debian `ppmd` 10.1-5 (2012), states them as 10 MB and `-r0`, restart, and its `-r` switch identifies it as variant I. Everything else matches this implementation: unordered pairs `x < y`, the concatenation `xy`, equation (3.1) of [cilibrasi2005] unclamped, and one value written to both triangles. ## The measurement DEFLATE sees 32 KiB: past it, the second object of a pair is compressed with the first out of view, and every distance nears 1. On columns written as the normalizer writes them, one JSON value per line drawn from a vocabulary, two from one vocabulary and a third from another, the gap between the distance across vocabularies and the distance within one is: | object size | `zlib` | PPMd variant I, order 6, 64 MiB | |---|---|---| | 16 KiB | 0.127 | 0.413 | | 128 KiB | 0.028 | 0.238 | | 1 MiB | 0.005 | 0.160 | What PPMd keeps is the distribution, not the copy. An object repeated verbatim is found again by neither compressor at 128 KiB when it is one independently drawn value per line -- distance to itself 0.94 to 0.99 under both -- while on lines of twelve words from a 5,000-word vocabulary PPMd finds it (0.242, against 0.970 under `zlib`). Which content is which was measured, not derived, so the gap above, not the distance of an object to itself, is the measure that decides this record. The original was run with that binary on the same objects, produced by this repository's normalizer, and every matrix went through this repository's Neighbor Joining and FastGreedy. Separation is the mean NCD between groups minus the mean within them. | corpus | measure | original | `zlib` | PPMd variant I, `pyppmd` | |---|---|---|---|---| | 30 columns of 21 KB | separation | 0.225 | 0.123 | 0.286 | | | Spearman against the original | 1 | 0.642 | 0.902 | | 30 columns of 636 KB | separation | 0.176 | 0.009 | 0.187 | | | Spearman against the original | 1 | 0.650 | 0.778 | | 18 texts of 200 KB sharing only a distribution | separation | 0.286 | 0.037 | 0.288 | | | Spearman against the original | 1 | 0.488 | 0.932 | The distance between groups that `zlib` loses above its window is what the original measured and what PPMd keeps. The original is not reproducible byte for byte: its archive format stores the input file name, so its sizes depend on name lengths, and its revision of variant I diverges from `pyppmd`'s after the first ten bytes of output, with sizes 0.4 to 0.8 % apart. No partition chosen by modularity matched the original's on any corpus under any compressor, the closest configuration included; that is the cut, not the compressor, and it is why the matrix and not the partition is the comparison above. ## What the dependency is `pyppmd>=1.3.1,<1.4` becomes a runtime dependency of `damicore-distance`. It is a wheel, not a system binary, published for CPython 3.10 to 3.14 on manylinux and musllinux (x86_64, aarch64), macOS (x86_64, arm64) and Windows (x86, x64, arm64). `ppmd-cffi`, its predecessor, publishes nothing past CPython 3.9. It is not an extra: a default that may be absent is either a failure on the default path or a silent fallback, and this project allows neither. ## What is part of the value `compressor="ppmd"` is variant I with restart, `pyppmd.Ppmd8Encoder`. Restoration is fixed rather than exposed, because the original never changed it; cut-off, or variant H, would be another compressor, never a parameter. The model order (2 to 64, default 6, the original's) and the model memory (2048 to 4294967259 bytes, default 64 MiB) change every NCD value, so they are configuration: covered by the configuration hash, recorded in the manifest, and validated here, because `pyppmd` clamps a value outside those ranges without reporting it and two configurations would give one result. A parameter set for a compressor it does not apply to is refused, not ignored (ADR 0021 says how). The default memory is not the original's 10 MB: the two agree until the model fills, which on text happens around a few MiB per object, and past that the larger model keeps measuring where the smaller one restarts and saturates. `ppmd_memory_bytes=10_485_760` reproduces the original's setting. Model memory is also the resource bound. A model can fill at 25 times its input on incompressible data, so preflight charges the configured memory once per worker rather than any multiple of the object size; four workers at the default fit the default working-memory limit. The `pyppmd` version joins the runtime fingerprint beside the zlib versions, so a checkpoint written by another build is refused rather than resumed. The distance rule stays `ncd-max/v1`: the formula is unchanged, and the compressor `Literal` is the rule's domain, which this record extends. ADR 0005 is not reopened: PPMd, like zlib, gets a fresh encoder per pair and keeps no state across pairs. ## Status Revisit when the window gate of issue 59 lands, which has to state a limit for a compressor that has a model instead of a window, or when LZMA is measured on real corpora: on objects that share content rather than a distribution, a 64 MiB LZMA2 dictionary separated groups by 0.940 where PPMd at order 6 separated them by 0.215.