Distance¶
damicore_distance computes the exact normalized compression distance between every pair of objects, in bounded memory, resumably.
Functions¶
- damicore_distance.compute_distance_matrix(normalization_manifest: str | Path, output_dir: str | Path, *, config: DistanceConfig | None = None, progress: ProgressCallback | None = None) DistanceResult¶
Compute or resume the exact NCD matrix from validated normalized objects.
Takes the normalizer’s
manifest.jsonand re-verifies every object’s path containment, size, and SHA-256 before reading it. Writesdistance.npyandlabels.jsonintooutput_dir, pluscheckpoints/compressed-sizes.jsonandcheckpoints/distance-shards.json;config.save_diagnosticsadds adiagnostics/directory.distance.npyis a float64 square memory map, validated as finite, zero-diagonal, and bitwise symmetric before this returns, and it is whatdamicore_tree_builder.build_tree()consumes together withlabels.json. That validation pass also measures the matrix range, reported asncd_min,ncd_maxandncd_out_of_range_counton the result, and the largest object is compressed once more after itself forncd_self_largest.Pairs are computed shard by shard straight into the memory map, and each finished shard is checkpointed with a digest, so an interrupted run resumes from the shards already done. With
config.resumedisabled, existing outputs are a conflict instead of a starting point. With more than one worker the per-object compressed sizes and the shards run in one pool of spawned processes that re-import the caller’s__main__, so a script-level call must sit underif __name__ == "__main__":.- Parameters:
progress – Called once before work begins – so a resumed run reports the pairs it inherited – and again after each completed shard, as
(completed, total, "distance").- Raises:
DistanceError – The normalization manifest or an object it names is invalid (
artifact_validation_error); a checkpoint or an existingdistance.npyis incompatible, corrupt, or disagrees with the matrix (checkpoint_mismatch_error); outputs exist but resume is disabled (output_directory_conflict_error); the compressor failed (compression_error); a worker died or returned an invalid shard (distance_computation_error); the finished matrix breaks an invariant (distance_matrix_validation_error).
- damicore_distance.codec_fingerprint() dict[str, str]¶
Name every codec this stage can measure with, as a result’s identity needs it.
zlib and pyppmd report their versions. libbz2 and liblzma come from the system and report none, so each is named by the size it gives a fixed payload: an encoder that compresses that payload differently changes that size, and a checkpoint or a run recorded under the other one is refused rather than resumed. A difference the payload does not reach goes unseen [adr-0021].
Models¶
- class damicore_distance.DistanceConfig(*, compressor: PpmdCompressor | DeflateCompressor | Bzip2Compressor | LzmaCompressor = PpmdCompressor(name='ppmd', ppmd_order=6, ppmd_memory_bytes=67108864), compression_chunk_bytes: Annotated[int, Gt(gt=0)] = 4194304, workers: int | Literal['auto'] = 'auto', pairs_per_shard: int | Literal['auto'] = 'auto', resume: bool = True, save_diagnostics: bool = False)¶
What the distance stage measures with, and how it schedules the work.
compressorand its settings change every NCD value; everything else changes only cost and recovery. See ADR 0020 for the default, ADR 0021 for bzip2 and LZMA, anddocs/method/ncd.mdfor what each compressor computes.- compressor: Compressor¶
- class damicore_distance.PpmdCompressor(*, name: Literal['ppmd'] = 'ppmd', ppmd_order: Annotated[int, Ge(ge=2), Le(le=64)] = 6, ppmd_memory_bytes: Annotated[int, Ge(ge=2048), Le(le=4294967259)] = 67108864)¶
PPMd variant I with model restart, at
ppmd_orderwithppmd_memory_bytesof model [adr-0020]. The bounds are the codec’s own: pyppmd would clamp a value outside them without saying so.- name: Literal['ppmd']¶
- property stream_memory_bytes: int¶
The memory one stream is charged, once per worker, by preflight.
Stated rather than measured per run: a stream fills to it on incompressible data [adr-0021].
- property window_bytes: None¶
PPMd declares no window, because it has none to declare [adr-0023].
Its reach across a pair is a property of the content, not a byte count: the model’s size does not decide what it finds, which is measured in that record. What it loses with size is the copy, not the distribution [adr-0020], and
DistanceResult.ncd_self_largestis where that shows.
- class damicore_distance.DeflateCompressor(*, name: Literal['zlib', 'gzip'] = 'zlib', compression_level: Annotated[int, Ge(ge=0), Le(le=9)] = 6)¶
DEFLATE through
zlib:"zlib"in the zlib container,"gzip"in the gzip one.Both see a 32 KiB window, the most either format allows: past it, the first object of a pair is out of view while the second is compressed.
- name: Literal['zlib', 'gzip']¶
- class damicore_distance.Bzip2Compressor(*, name: Literal['bzip2'] = 'bzip2', bzip2_block_size_100k: Annotated[int, Ge(ge=1), Le(le=9)] = 9)¶
bzip2 at
bzip2_block_size_100k; the block is its window [adr-0021].- name: Literal['bzip2']¶
- class damicore_distance.LzmaCompressor(*, name: Literal['lzma'] = 'lzma', lzma_dictionary_bytes: Annotated[int, Ge(ge=4096), Le(le=1610612736)] = 8388608)¶
Raw LZMA2 at preset 6 with
lzma_dictionary_bytes; the dictionary is its window [adr-0021].- name: Literal['lzma']¶
- class damicore_distance.DistanceMatrixView(path: str | ~pathlib.Path, labels: list[str], *, materialization_limit_bytes: int = 268435456, materialization_error: ~collections.abc.Callable[[str], Exception] = <class 'ValueError'>)¶
A read-only view of a persisted
distance.npy, backed by a memory map.The array is never loaded whole.
shape,dtype, and indexing read through NumPy slicing, so a submatrix costs only the pages it touches; the file is opened withallow_pickle=False, so an artifact cannot execute code. The map stays open untilclose(), after which every accessor raisesValueError– exceptheadandto_pandas, which test for pandas before the closed check and so report the missing extra first when it is absent.headandto_pandasare the only members that need pandas, which is an extra of this distribution rather than a dependency. Without it they raiseDistanceErrorwith codemissing_dependency_error.to_pandasadditionally refuses to materialize a matrix larger thanmaterialization_limit_bytesunlessforce=True, raising the injectedmaterialization_error–ValueErrorby default, anddamicore.MaterializationErrorfor a view built bydamicore.load_result.
- class damicore_distance.DistanceResult(*, matrix_path: Path, labels_path: Path, object_count: Annotated[int, Ge(ge=0)], pair_count: Annotated[int, Ge(ge=0)], timing: Annotated[float, Ge(ge=0)], ncd_min: Annotated[float, AllowInfNan(allow_inf_nan=False)] = 0.0, ncd_max: Annotated[float, AllowInfNan(allow_inf_nan=False)] = 0.0, ncd_out_of_range_count: Annotated[int, Ge(ge=0)] = 0, ncd_self_largest: Annotated[float, AllowInfNan(allow_inf_nan=False)], pairs_over_window: Annotated[int | None, Ge(ge=0)] = None)¶
What one distance stage produced, including the range of the matrix it wrote.
The three
ncd_*fields are measured during the validation pass this stage already performs over every cell, so reporting the range costs no additional traversal. They describe the whole matrix, diagonal included, which is whyncd_minis0.0for any matrix whose values are all non-negative.ncd_self_largestis NCD(x, x) for the largest object – the first in manifest order among equals – which is(C(xx) - C(x)) / C(x): how much of an object the compressor recognizes when it reads it a second time, the idempotency [cilibrasi2005] requires of a normal compressor. Zero is full recognition. It is measured, never gated, and a high value is not by itself a fault: on one independently drawn value per line neither DEFLATE nor PPMd recognizes a repeat of 128 KiB, while PPMd still separates such columns by their distributions (ADR 0020). What it does show is a compressor that stops recognizing repetition at the size of this run’s objects, past a window or a filled model.pairs_over_windowis how many of this run’s pairs are longer than what the compressor can match across, counted from the object sizes. Preflight gates on the largest pair before a run starts; this says, afterwards and for every pair, how much of the matrix was measured with one object out of view. The window it is counted against isconfig.compressor.window_bytes, which the caller already holds. See [adr-0023].- matrix_path: Path¶
- labels_path: Path¶
Errors¶
- class damicore_distance.DistanceError(message: str, *, code: str = 'distance_error', **context: object)¶
Base error raised by the standalone distance package.
codeis the stable machine-readable identifier a caller branches on, and it is part of this package’s contract rather than a message detail: the orchestrator keys its own public error classes off these values, so renaming one changes what a damicore user catches.Neither the message nor
contextmay carry object bytes or a cell value. This stage reads the objects it measures, and an error reaches logs and issue trackers that the data itself is not allowed to reach; name the object, the shard, or the path instead.