Distance

damicore_distance computes the exact normalized compression distance between every pair of objects, in bounded memory, resumably.

Functions

damicore_distance.compute_distance_matrix(normalization_manifest: str | Path, output_dir: str | Path, *, config: DistanceConfig | None = None, progress: ProgressCallback | None = None) → DistanceResult

Compute or resume the exact NCD matrix from validated normalized objects.

Takes the normalizer’s manifest.json and re-verifies every object’s path containment, size, and SHA-256 before reading it. Writes distance.npy and labels.json into output_dir, plus checkpoints/compressed-sizes.json and checkpoints/distance-shards.json; config.save_diagnostics adds a diagnostics/ directory. distance.npy is a float64 square memory map, validated as finite, zero-diagonal, and bitwise symmetric before this returns, and it is what damicore_tree_builder.build_tree() consumes together with labels.json. That validation pass also measures the matrix range, reported as ncd_min, ncd_max and ncd_out_of_range_count on the result, and the largest object is compressed once more after itself for ncd_self_largest.

Pairs are computed shard by shard straight into the memory map, and each finished shard is checkpointed with a digest, so an interrupted run resumes from the shards already done. With config.resume disabled, existing outputs are a conflict instead of a starting point. With more than one worker the per-object compressed sizes and the shards run in one pool of spawned processes that re-import the caller’s __main__, so a script-level call must sit under if __name__ == "__main__":.

Parameters:

progress – Called once before work begins – so a resumed run reports the pairs it inherited – and again after each completed shard, as (completed, total, "distance").

Raises:

DistanceError – The normalization manifest or an object it names is invalid (artifact_validation_error); a checkpoint or an existing distance.npy is incompatible, corrupt, or disagrees with the matrix (checkpoint_mismatch_error); outputs exist but resume is disabled (output_directory_conflict_error); the compressor failed (compression_error); a worker died or returned an invalid shard (distance_computation_error); the finished matrix breaks an invariant (distance_matrix_validation_error).

damicore_distance.codec_fingerprint() → dict[str, str]

Name every codec this stage can measure with, as a result’s identity needs it.

zlib and pyppmd report their versions. libbz2 and liblzma come from the system and report none, so each is named by the size it gives a fixed payload: an encoder that compresses that payload differently changes that size, and a checkpoint or a run recorded under the other one is refused rather than resumed. A difference the payload does not reach goes unseen [adr-0021].

Models

class damicore_distance.DistanceConfig(*, compressor: PpmdCompressor | DeflateCompressor | Bzip2Compressor | LzmaCompressor = PpmdCompressor(name='ppmd', ppmd_order=6, ppmd_memory_bytes=67108864), compression_chunk_bytes: Annotated[int, Gt(gt=0)] = 4194304, workers: int | Literal['auto'] = 'auto', pairs_per_shard: int | Literal['auto'] = 'auto', resume: bool = True, save_diagnostics: bool = False)

What the distance stage measures with, and how it schedules the work.

compressor and its settings change every NCD value; everything else changes only cost and recovery. See ADR 0020 for the default, ADR 0021 for bzip2 and LZMA, and docs/method/ncd.md for what each compressor computes.

compressor: Compressor
compression_chunk_bytes: int
workers: int | Literal['auto']
pairs_per_shard: int | Literal['auto']
resume: bool
save_diagnostics: bool
property effective_workers: int
effective_pairs_per_shard(pair_count: int) → int

Return the shard size for pair_count pairs.

"auto" depends on the pair count alone, never on the machine, so the shard layout a checkpoint records stays valid when a run resumes elsewhere.

class damicore_distance.PpmdCompressor(*, name: Literal['ppmd'] = 'ppmd', ppmd_order: Annotated[int, Ge(ge=2), Le(le=64)] = 6, ppmd_memory_bytes: Annotated[int, Ge(ge=2048), Le(le=4294967259)] = 67108864)

PPMd variant I with model restart, at ppmd_order with ppmd_memory_bytes of model [adr-0020]. The bounds are the codec’s own: pyppmd would clamp a value outside them without saying so.

name: Literal['ppmd']
ppmd_order: int
ppmd_memory_bytes: int
property stream_memory_bytes: int

The memory one stream is charged, once per worker, by preflight.

Stated rather than measured per run: a stream fills to it on incompressible data [adr-0021].

property window_bytes: None

PPMd declares no window, because it has none to declare [adr-0023].

Its reach across a pair is a property of the content, not a byte count: the model’s size does not decide what it finds, which is measured in that record. What it loses with size is the copy, not the distribution [adr-0020], and DistanceResult.ncd_self_largest is where that shows.

class damicore_distance.DeflateCompressor(*, name: Literal['zlib', 'gzip'] = 'zlib', compression_level: Annotated[int, Ge(ge=0), Le(le=9)] = 6)

DEFLATE through zlib: "zlib" in the zlib container, "gzip" in the gzip one.

Both see a 32 KiB window, the most either format allows: past it, the first object of a pair is out of view while the second is compressed.

name: Literal['zlib', 'gzip']
compression_level: int
property stream_memory_bytes: int

The memory one stream is charged, once per worker, by preflight.

Stated rather than measured per run: a stream fills to it on incompressible data [adr-0021].

property window_bytes: int

The bytes of a pair this compressor can still match across [adr-0023].

32 KiB, which is 2 ** zlib.MAX_WBITS and the most either container allows.

class damicore_distance.Bzip2Compressor(*, name: Literal['bzip2'] = 'bzip2', bzip2_block_size_100k: Annotated[int, Ge(ge=1), Le(le=9)] = 9)

bzip2 at bzip2_block_size_100k; the block is its window [adr-0021].

name: Literal['bzip2']
bzip2_block_size_100k: int
property stream_memory_bytes: int

The memory one stream is charged, once per worker, by preflight.

Stated rather than measured per run: a stream fills to it on incompressible data [adr-0021].

property window_bytes: int

The bytes of a pair this compressor can still match across [adr-0023].

The block, which bzip2 sizes in units of 100,000 bytes rather than of 100 KiB. Two objects in different blocks share nothing [adr-0021].

class damicore_distance.LzmaCompressor(*, name: Literal['lzma'] = 'lzma', lzma_dictionary_bytes: Annotated[int, Ge(ge=4096), Le(le=1610612736)] = 8388608)

Raw LZMA2 at preset 6 with lzma_dictionary_bytes; the dictionary is its window [adr-0021].

name: Literal['lzma']
lzma_dictionary_bytes: int
property stream_memory_bytes: int

The memory one stream is charged, once per worker, by preflight.

Stated rather than measured per run: a stream fills to it on incompressible data [adr-0021].

property window_bytes: int

The bytes of a pair this compressor can still match across [adr-0023].

The dictionary. Measured, a copy still survives at twice it [adr-0023], so declaring the dictionary gates earlier than the measurement requires, never later.

class damicore_distance.DistanceMatrixView(path: str | ~pathlib.Path, labels: list[str], *, materialization_limit_bytes: int = 268435456, materialization_error: ~collections.abc.Callable[[str], Exception] = <class 'ValueError'>)

A read-only view of a persisted distance.npy, backed by a memory map.

The array is never loaded whole. shape, dtype, and indexing read through NumPy slicing, so a submatrix costs only the pages it touches; the file is opened with allow_pickle=False, so an artifact cannot execute code. The map stays open until close(), after which every accessor raises ValueError – except head and to_pandas, which test for pandas before the closed check and so report the missing extra first when it is absent.

head and to_pandas are the only members that need pandas, which is an extra of this distribution rather than a dependency. Without it they raise DistanceError with code missing_dependency_error. to_pandas additionally refuses to materialize a matrix larger than materialization_limit_bytes unless force=True, raising the injected materialization_error – ValueError by default, and damicore.MaterializationError for a view built by damicore.load_result.

property shape: tuple[int, ...]
property dtype: dtype[Any]
head(n: int = 5) → pd.DataFrame
to_pandas(force: bool = False) → pd.DataFrame
close() → None
class damicore_distance.DistanceResult(*, matrix_path: Path, labels_path: Path, object_count: Annotated[int, Ge(ge=0)], pair_count: Annotated[int, Ge(ge=0)], timing: Annotated[float, Ge(ge=0)], ncd_min: Annotated[float, AllowInfNan(allow_inf_nan=False)] = 0.0, ncd_max: Annotated[float, AllowInfNan(allow_inf_nan=False)] = 0.0, ncd_out_of_range_count: Annotated[int, Ge(ge=0)] = 0, ncd_self_largest: Annotated[float, AllowInfNan(allow_inf_nan=False)], pairs_over_window: Annotated[int | None, Ge(ge=0)] = None)

What one distance stage produced, including the range of the matrix it wrote.

The three ncd_* fields are measured during the validation pass this stage already performs over every cell, so reporting the range costs no additional traversal. They describe the whole matrix, diagonal included, which is why ncd_min is 0.0 for any matrix whose values are all non-negative.

ncd_self_largest is NCD(x, x) for the largest object – the first in manifest order among equals – which is (C(xx) - C(x)) / C(x): how much of an object the compressor recognizes when it reads it a second time, the idempotency [cilibrasi2005] requires of a normal compressor. Zero is full recognition. It is measured, never gated, and a high value is not by itself a fault: on one independently drawn value per line neither DEFLATE nor PPMd recognizes a repeat of 128 KiB, while PPMd still separates such columns by their distributions (ADR 0020). What it does show is a compressor that stops recognizing repetition at the size of this run’s objects, past a window or a filled model.

pairs_over_window is how many of this run’s pairs are longer than what the compressor can match across, counted from the object sizes. Preflight gates on the largest pair before a run starts; this says, afterwards and for every pair, how much of the matrix was measured with one object out of view. The window it is counted against is config.compressor.window_bytes, which the caller already holds. See [adr-0023].

matrix_path: Path
labels_path: Path
object_count: int
pair_count: int
timing: float
ncd_min: float
ncd_max: float
ncd_out_of_range_count: int
ncd_self_largest: float
pairs_over_window: int | None

Errors

class damicore_distance.DistanceError(message: str, *, code: str = 'distance_error', **context: object)

Base error raised by the standalone distance package.

code is the stable machine-readable identifier a caller branches on, and it is part of this package’s contract rather than a message detail: the orchestrator keys its own public error classes off these values, so renaming one changes what a damicore user catches.

Neither the message nor context may carry object bytes or a cell value. This stage reads the objects it measures, and an error reaches logs and issue trackers that the data itself is not allowed to reach; name the object, the shard, or the path instead.