Orchestration¶
damicore runs the whole pipeline, estimates what a run will cost before it starts, and loads a finished run back into verified Python objects.
Functions¶
- damicore.estimate(source: str | Path | Sequence[str | Path], *, source_kind: str = 'delimited', split: str | None = None, delimiter: str | None = None, encoding: str | None = None, sheet: str | None = None, recursive: bool | None = None, include_hidden: bool | None = None, compressor: str = 'ppmd', compression_level: int | None = None, ppmd_order: int | None = None, ppmd_memory_bytes: int | None = None, bzip2_block_size_100k: int | None = None, lzma_dictionary_bytes: int | None = None, keep_normalized: bool = False, save_diagnostics: bool = False, execution: ExecutionConfig | None = None) ResourceEstimate¶
Inspect exact resource requirements without creating run artifacts.
The whole input is hashed and scanned, but nothing is written: no run directory, no materialized objects. Exceeding a limit is not a failure here, which is what makes this the cheap way to decide whether a run is worth starting.
- Parameters:
source – A dataset file, a directory of files, or a sequence of file paths. Which of those is accepted follows
source_kind.source_kind – Exactly
"delimited","xlsx", or"files". It decides where objects come from, and therefore which of the remaining arguments apply; supplying one that does not apply is aConfigurationErrorrather than a silently ignored setting.split – Exactly
"columns"or"rows", defaulting to"columns". It decides what one object is for a dataset, and therefore the object count that every resource gate is quadratic or cubic in. It does not apply to"files", where each file is an object.compressor – As for
run(), and validated the same way. The compressor decides the memory every worker’s stream holds, which the working-memory projection charges once per worker.compression_level – As for
run(), and validated the same way. The compressor decides the memory every worker’s stream holds, which the working-memory projection charges once per worker.ppmd_order – As for
run(), and validated the same way. The compressor decides the memory every worker’s stream holds, which the working-memory projection charges once per worker.ppmd_memory_bytes – As for
run(), and validated the same way. The compressor decides the memory every worker’s stream holds, which the working-memory projection charges once per worker.bzip2_block_size_100k – As for
run(), and validated the same way.lzma_dictionary_bytes – As for
run(), and validated the same way.keep_normalized – Accepted for parity with
run()and deliberately ignored: object bytes exist on disk while a run is in progress either way, so they are always counted.execution –
Noneuses the defaults. Free disk is measured against./damicore-results, since no output directory has been chosen yet; a run writing elsewhere may see a different amount of free space.
- Returns:
Returned whether or not the run fits.
within_limitsisFalseexactly whenviolationsis non-empty, andviolationsnames each failed gate.- Return type:
- Raises:
ConfigurationError –
source_kindorsplitis not one of its accepted values, an argument does not apply to this source, ordelimiter,encodingorexecutionis rejected by validation.InputValidationError – An input path is not a readable regular file, a corpus breaks the corpus rules, or an input changed while it was being hashed and scanned (code
input_drift).DatasetFormatError – The dataset violates the input contract.
- damicore.load_result(output_dir: str | Path) DamicoreResult¶
Load and verify a completed DAMICORE result without executing artifact code.
Every artifact the manifest declares is re-checked against its recorded size and SHA-256, and any entry that is absolute or resolves outside the run directory is rejected before it is read. Containment is the rule, not file kind: an entry is resolved before it is checked, so a link and the file it points at are one path by then – a link out of the directory is refused because its target is outside, and one within it is read like any other artifact, its bytes still held to the recorded digest. The matrix is memory-mapped with
allow_pickle=False, so no artifact can execute code on load. Only runs whose manifest and report both saycompleted, at the current schema version, can be loaded; a failed or interrupted run is diagnostic only.- Returns:
A fresh result that owns its own open memory map of
distance.npy. Every call produces an independent one, and each has to be closed by whoever received it.- Return type:
- Raises:
ArtifactValidationError – The only public failure of this function: a missing, unreadable, incomplete, or wrong-version manifest or report; a hash or size mismatch; a path that escapes the run directory; or a membership, cluster, or Newick artifact that does not parse.
- damicore.run(source: str | Path | Sequence[str | Path], *, source_kind: str = 'delimited', split: str | None = None, delimiter: str | None = None, encoding: str | None = None, sheet: str | None = None, recursive: bool | None = None, include_hidden: bool | None = None, compressor: str = 'ppmd', compression_level: int | None = None, ppmd_order: int | None = None, ppmd_memory_bytes: int | None = None, bzip2_block_size_100k: int | None = None, lzma_dictionary_bytes: int | None = None, num_clusters: int | None = None, output_dir: str | Path | None = None, keep_normalized: bool = False, save_diagnostics: bool = False, progress: bool = True, execution: ExecutionConfig | None = None) DamicoreResult¶
Execute, verify, and if possible resume the complete DAMICORE pipeline.
Preflight, normalization, the exact NCD matrix, Neighbor Joining, and FastGreedy run in that order, and the artifacts they leave behind are cross-checked against one another before the run is marked
completed. Once the run directory exists, a failure or an interruption still writesreport.jsonand updatesmanifest.jsonbefore the exception propagates, so the partial state stays diagnosable and, where the checkpoints allow it, resumable.The distance stage spawns worker processes, which re-import the calling module, whenever
execution.workersresolves above1;"auto"resolves from the CPU count, so only an explicitworkers=1rules the pool out. A call at module level in a.pyscript must therefore sit underif __name__ == "__main__":; a notebook or REPL already satisfies this.- Parameters:
source – A dataset file, a directory of files, or a sequence of file paths, according to
source_kind.source_kind – Exactly
"delimited","xlsx", or"files". It decides where objects come from and therefore which of the remaining arguments apply; one that does not apply is aConfigurationErrorrather than a silently ignored setting. It also enters the run identity, so the same corpus split differently is a different run.split – Exactly
"columns"or"rows", defaulting to"columns". It decides what one object is for a dataset, and therefore the object count that the resource gates are quadratic and cubic in. It does not apply to"files", where each file is one object already.sheet – The worksheet to read from an
"xlsx"source. Required when the workbook holds more than one, because defaulting to the first would silently decide which data was analyzed.recursive – How a
"files"source is enumerated. Both are recorded in the manifest, so a corpus can be reconstructed from the run rather than from the command that produced it.include_hidden – How a
"files"source is enumerated. Both are recorded in the manifest, so a corpus can be reconstructed from the run rather than from the command that produced it.compressor – Exactly
"ppmd", the default,"zlib","gzip","bzip2"or"lzma". NCD values are compressor-dependent, so changing it changes the run identity and the results, not merely their cost."ppmd"is PPMd variant I, the original DAMICORE compressor, and keeps a shared distribution across large objects;"lzma"finds shared content at any size its dictionary holds;"bzip2"sees both within its 900 kB block;"zlib"and"gzip"see a 32 KiB window, past which objects look unrelated whatever they are. See ADRs 0020 and 0021.compression_level – The settings of the chosen compressor:
compression_level(0 to 9, default 6) for"zlib"and"gzip";ppmd_order(2 to 64, default 6) andppmd_memory_bytes(2048 to 4294967259, default 67108864) for"ppmd";bzip2_block_size_100k(1 to 9, default 9) for"bzip2"; andlzma_dictionary_bytes(4096 to 1610612736, default 8388608) for"lzma".Noneuses the default. A setting for a compressor that does not read it is aConfigurationError, never ignored, and every setting in force enters the run identity. Each worker holds one compressor stream, and preflight charges its stated memory per worker againstmax_working_memory_bytes.ppmd_order – The settings of the chosen compressor:
compression_level(0 to 9, default 6) for"zlib"and"gzip";ppmd_order(2 to 64, default 6) andppmd_memory_bytes(2048 to 4294967259, default 67108864) for"ppmd";bzip2_block_size_100k(1 to 9, default 9) for"bzip2"; andlzma_dictionary_bytes(4096 to 1610612736, default 8388608) for"lzma".Noneuses the default. A setting for a compressor that does not read it is aConfigurationError, never ignored, and every setting in force enters the run identity. Each worker holds one compressor stream, and preflight charges its stated memory per worker againstmax_working_memory_bytes.ppmd_memory_bytes – The settings of the chosen compressor:
compression_level(0 to 9, default 6) for"zlib"and"gzip";ppmd_order(2 to 64, default 6) andppmd_memory_bytes(2048 to 4294967259, default 67108864) for"ppmd";bzip2_block_size_100k(1 to 9, default 9) for"bzip2"; andlzma_dictionary_bytes(4096 to 1610612736, default 8388608) for"lzma".Noneuses the default. A setting for a compressor that does not read it is aConfigurationError, never ignored, and every setting in force enters the run identity. Each worker holds one compressor stream, and preflight charges its stated memory per worker againstmax_working_memory_bytes.bzip2_block_size_100k – The settings of the chosen compressor:
compression_level(0 to 9, default 6) for"zlib"and"gzip";ppmd_order(2 to 64, default 6) andppmd_memory_bytes(2048 to 4294967259, default 67108864) for"ppmd";bzip2_block_size_100k(1 to 9, default 9) for"bzip2"; andlzma_dictionary_bytes(4096 to 1610612736, default 8388608) for"lzma".Noneuses the default. A setting for a compressor that does not read it is aConfigurationError, never ignored, and every setting in force enters the run identity. Each worker holds one compressor stream, and preflight charges its stated memory per worker againstmax_working_memory_bytes.lzma_dictionary_bytes – The settings of the chosen compressor:
compression_level(0 to 9, default 6) for"zlib"and"gzip";ppmd_order(2 to 64, default 6) andppmd_memory_bytes(2048 to 4294967259, default 67108864) for"ppmd";bzip2_block_size_100k(1 to 9, default 9) for"bzip2"; andlzma_dictionary_bytes(4096 to 1610612736, default 8388608) for"lzma".Noneuses the default. A setting for a compressor that does not read it is aConfigurationError, never ignored, and every setting in force enters the run identity. Each worker holds one compressor stream, and preflight charges its stated memory per worker againstmax_working_memory_bytes.num_clusters –
Nonelets FastGreedy choose the cut. Otherwise the upper bound is the object count this source produces, which is only known after preflight, so an oversized value is rejected there rather than at call time.output_dir –
Nonewrites to./damicore-results/<run_id>, whererun_idis derived from the input hash and the configuration, so repeating a call lands in the same directory and reuses or resumes it. An explicit directory is used as given; it may be absent, empty, or a compatible earlier run, and is never overwritten.keep_normalized – Keeps
normalization/in the run directory. Otherwise it is deleted once verification succeeds, since the objects are reproducible from the source.progress – Renders a progress bar for the distance stage through
tqdm. Purely presentational.execution –
Noneuses the defaults: automatic worker count, resume and completed-run reuse enabled, and the defaultResourceLimits.
- Returns:
The verified result, loaded through
load_result(). It owns an open memory map ofdistance.npy; callclose()when finished with it.- Return type:
- Raises:
ConfigurationError – An argument or configuration value is invalid, including
num_clustersabove the object count. Raised before any run directory is created.InputValidationError – An input path is not a readable regular file, a corpus breaks the corpus rules, or an input changed between preflight and materialization (code
input_drift).DatasetFormatError – The dataset violates the input contract.
ResourceLimitError – Preflight projected the run outside
execution.limits.OutputDirectoryConflictError –
output_diris not a directory, holds no readable DAMICORE manifest, belongs to a different input or configuration, holds a compatible run thatreuse_completedorresumeforbids continuing, or holds a completed run that a different runtime produced, which is never returned as this build’s result.CheckpointMismatchError – An incomplete run in
output_direxists but cannot be trusted to resume.CompressionError – The compressor rejected an object.
DistanceComputationError – The NCD stage failed, including a worker pool that died.
DistanceMatrixValidationError – The NCD stage’s own check rejected the matrix it computed.
NormalizationError, TreeBuildError, TreeFormatError, ClusterizationError – The corresponding stage failed with no more specific cause.
TreeBuildErroralso covers the tree stage rejecting the matrix it was given, which does not surface asDistanceMatrixValidationError.ArtifactValidationError – An artifact failed one of its checks, including the cross-artifact verification a run must pass before it is marked
completed.DamicoreError – Any other failure inside the pipeline, named by the underlying exception type.
KeyboardInterrupt – Re-raised unchanged, after the run is recorded as interrupted.
Models¶
- class damicore.ArtifactPaths(*, run_dir: Path, manifest: Path, report: Path, distance_matrix: Path, labels: Path, tree_json: Path, tree_newick: Path, membership: Path, clusters: Path, normalization_dir: Path | None, diagnostics_dir: Path | None)¶
Where a run’s artifacts live on disk, as absolute paths under
run_dir.The two optional directories are
Nonewhen the run did not keep them; the remaining paths are the run’s fixed layout and are reported by name, not by probing the filesystem.- run_dir: Path¶
- manifest: Path¶
- report: Path¶
- distance_matrix: Path¶
- labels: Path¶
- tree_json: Path¶
- tree_newick: Path¶
- membership: Path¶
- clusters: Path¶
- class damicore.DamicoreResult(membership: DataFrame, clusters: dict[int, list[str]], tree_newick: str, distance_matrix: DistanceMatrixView, report: RunReport, artifacts: ArtifactPaths)¶
A verified, completed run, together with the memory map it holds open.
distance_matrixkeepsdistance.npymapped for as long as this object lives.close()releases that map and leaves the view unusable; the other fields are ordinary in-memory values and stay valid afterwards. Everyload_resultcall – including the onedamicore.run()returns through – hands back a distinct result with its own map, so each one is owned, and has to be closed, by whoever received it.- distance_matrix: DistanceMatrixView¶
- artifacts: ArtifactPaths¶
- save(output_dir: str | Path) ArtifactPaths¶
Copy exactly the artifacts this run’s manifest declares to a new directory.
The destination must be absent or empty, so nothing is ever overwritten. Every file is re-hashed as it is copied and written through a same-directory temporary file, so a partially copied or drifted artifact never appears under its final name. The saved manifest records the new location.
- Raises:
OutputDirectoryConflictError – The destination exists and is not an empty directory.
ArtifactValidationError – The manifest is unreadable or not
completed, an artifact is missing, escapes the run directory, is duplicated in the inventory, or changed since it was recorded.
- class damicore.ExecutionConfig(*, workers: int | ~typing.Literal['auto'] = 'auto', csv_chunk_rows: ~typing.Annotated[int, ~annotated_types.Gt(gt=0)] = 50000, compression_chunk_bytes: ~typing.Annotated[int, ~annotated_types.Gt(gt=0)] = 4194304, pairs_per_shard: int | ~typing.Literal['auto'] = 'auto', resume: bool = True, reuse_completed: bool = True, pandas_materialization_limit_bytes: ~typing.Annotated[int, ~annotated_types.Gt(gt=0)] = 268435456, limits: ~damicore.config.ResourceLimits = <factory>)¶
How a run is executed, as opposed to what it computes.
These settings change cost and restart behavior, not the clustering: the pipeline is exact and deterministic under any of them, so none enters the configuration hash that names the run.
manifest.jsonrecords the values of the invocation that created the run directory; a later invocation that resumes or reuses it – on another machine, whereworkers="auto"resolves differently, for instance – adopts the same directory. A resume with anotherpairs_per_shardis refused while the distance stage is incomplete, because its checkpoint is indexed by that layout;pairs_per_shard="auto"resolves from the pair count alone, so it gives the same layout on every machine. See ADR 0022.- limits: ResourceLimits¶
- class damicore.ResourceEstimate(*, source_kind: Literal['delimited', 'xlsx', 'files'], source_paths: tuple[Path, ...], input_sha256: str, input_size_bytes: Annotated[int, Ge(ge=0)], split: Literal['columns', 'rows'] | None, object_count: Annotated[int, Ge(ge=0)], pair_count: Annotated[int, Ge(ge=0)], effective_workers: Annotated[int, Gt(gt=0)], matrix_bytes: Annotated[int, Ge(ge=0)], tree_workspace_bytes: Annotated[int, Ge(ge=0)], normalized_bytes: Annotated[int, Ge(ge=0)], max_serialized_chunk_bytes: Annotated[int, Ge(ge=0)], compression_input_bytes: Annotated[int | None, Ge(ge=0)] = None, max_pair_bytes: Annotated[int | None, Ge(ge=0)] = None, compressor_window_bytes: Annotated[int | None, Ge(ge=0)] = None, estimated_working_memory_bytes: Annotated[int, Ge(ge=0)], estimated_final_metadata_bytes: Annotated[int, Ge(ge=0)], estimated_diagnostic_bytes: Annotated[int, Ge(ge=0)], estimated_artifact_bytes: Annotated[int, Ge(ge=0)], required_free_disk_bytes: Annotated[int, Ge(ge=0)], available_free_disk_bytes: Annotated[int, Ge(ge=0)], within_limits: bool, violations: list[str])¶
What a run of this source under this configuration would cost, measured before it starts.
The counts and the input and object byte totals come from the same traversal the run itself performs, so they are exact rather than sampled; the memory and disk figures are conservative projections, and free disk is the one number read from the live filesystem.
within_limitsisFalseexactly whenviolationsis non-empty, andviolationsnames the gates in a fixed order.estimate()returns this either way;run()turns a non-emptyviolationsintoResourceLimitError.input_sha256identifies the whole input: one file’s digest for a dataset, and a digest over every adopted file for a corpus, since no single file identifies that run.compression_input_bytesis the exact number of bytes a distance stage that starts from no checkpoint feeds its compressor: every object once for its own size, both objects of every pair, and the largest object twice forncd_self_largest, which isobject_count * normalized_bytesplus twice the largest object. Divided by a measured throughput it is the stage’s time, and nothing gates on it.max_pair_bytesis the longest concatenation the compressor is given – the two largest objects – which is what a compressor’s window or model memory has to hold for an NCD value to see across the pair.compressor_window_bytesis what that compressor declares it can match across, and thecompressor_windowviolation is raised when the first exceeds the second timeslimits.max_pair_window_factor: the run would measure, and report nothing unusual about, a matrix the compressor could not see. See ADR 0023.- source_kind: Literal['delimited', 'xlsx', 'files']¶
- class damicore.ResourceLimits(*, max_objects: Annotated[int, Gt(gt=0)] = 1000, max_pairs: Annotated[int, Gt(gt=0)] = 500000, max_matrix_bytes: Annotated[int, Gt(gt=0)] = 536870912, max_working_memory_bytes: Annotated[int, Gt(gt=0)] = 536870912, required_free_disk_factor: Annotated[float, Ge(ge=1.0)] = 1.25, max_pair_window_factor: Annotated[float, Gt(gt=0)] = 1.0)¶
The gates preflight applies before a run starts, sized for the exact 0.2 algorithm.
Raise one only after reading an
estimate(): NCD is quadratic and Neighbor Joining cubic in the object count, so a limit lifted without that check buys a run that does not finish. Two gates are less visible than the object, pair, and matrix caps.max_working_memory_bytesbounds the peak RAM the distance stage is projected to need from the largest serialized chunk, the worker count, the shard size, and the compressor stream each worker holds, charged at its stated size.required_free_disk_factoris the headroom multiplier applied to the projected artifact bytes; the free-disk check it produces is enforced no matter how the limits are set.max_pair_window_factoris the only gate on what a run measures rather than on what it costs: the largest pair may be at most this multiple of the window the compressor declares, because past that the compressor is given a pair it cannot match across and every distance in the matrix moves toward 1 with nothing in the result saying so. The default of 1.0 is the window itself. Raise it to measure anyway – the value is recorded in the manifest, so a run says which gate it ran under – or choose a compressor whose window holds the pair. A compressor that declares no window, which is PPMd, is not gated here at all. See ADR 0023.
- class damicore.RunReport(*, status: ~typing.Literal['completed', 'failed', 'interrupted'], failed_stage: str | None = None, object_count: int = 0, pair_count: int = 0, community_count: int | None = None, cluster_count: int | None = None, effective_workers: int = 1, csv_chunk_rows: int = 50000, compression_chunk_bytes: int = 4194304, pairs_per_shard: int = 10000, matrix_bytes: int = 0, required_free_disk_bytes: int = 0, peak_rss_bytes: int | None = None, ncd_min: float | None = None, ncd_max: float | None = None, ncd_out_of_range_count: int = 0, ncd_self_largest: float | None = None, compressor_window_bytes: int | None = None, pairs_over_window: int | None = None, negative_branch_count: int = 0, modularity: float | None = None, timings_seconds: dict[str, float] = <factory>, verification: dict[str, bool] = <factory>, warnings: list[str] = <factory>, error: dict[str, object] | None = None)¶
The
report.jsonpayload: what one run did, or how far it got before it stopped.statusis the only success signal.completedfills in the measurements and the verification checks;failedandinterruptedare diagnostic, and describe the stop throughfailed_stage,error, and whichever timings had already been recorded.- status: Literal['completed', 'failed', 'interrupted']¶
Errors¶
- class damicore.ArtifactValidationError(message: str, *, code: str | None = None, **context: object)¶
A persisted artifact failed schema validation, its recorded size or SHA-256, path containment inside the run directory, or the cross-artifact verification a run must pass before it becomes
completed.
- class damicore.CheckpointMismatchError(message: str, *, code: str | None = None, **context: object)¶
Resumable state exists but cannot be trusted: a different runtime fingerprint, or a checkpoint that disagrees with the artifacts beside it. Start over in a fresh directory.
- class damicore.ClusterizationError(message: str, *, code: str | None = None, **context: object)¶
The FastGreedy stage failed, or its communities did not cover every leaf exactly once.
- class damicore.CompressionError(message: str, *, code: str | None = None, **context: object)¶
The compressor rejected an object while measuring its compressed size.
- class damicore.ConfigurationError(message: str, *, code: str | None = None, **context: object)¶
An argument or configuration value is invalid, so no run directory was created.
- class damicore.DamicoreError(message: str, *, code: str | None = None, **context: object)¶
Base class of every public DAMICORE failure, and the only one worth catching broadly.
codeis the stable machine-readable identifier for the failure: the class name in snake_case, withinput_driftas the single 0.2 exception. It is what the CLI’s JSON error envelope reports, and it never carries a stage’s internal vocabulary.contextholds bounded diagnostic values supplied at the raise site; it never contains dataset cell contents, whole input rows, or the contents of an adopted file.
- class damicore.DatasetFormatError(message: str, *, code: str | None = None, **context: object)¶
A dataset violates the input contract: bad, empty, or duplicated header names, a record whose field count disagrees with the header, too few columns or data rows to cluster, a workbook that cannot be opened or names no single worksheet, or a cell holding a value the cell-text rule does not span.
- class damicore.DistanceComputationError(message: str, *, code: str | None = None, **context: object)¶
The NCD stage failed, including a worker process that died.
A dead pool has two realistic causes and the message names both, because the traceback a caller reads cannot show which one it was.
- class damicore.DistanceMatrixValidationError(message: str, *, code: str | None = None, **context: object)¶
The computed NCD matrix broke an invariant: wrong shape or dtype, a non-finite value, a non-zero diagonal entry, or asymmetry.
- class damicore.InputValidationError(message: str, *, code: str | None = None, **context: object)¶
The input is unusable: a missing, wrong-kind, or unreadable path, a corpus that breaks the corpus rules, or an input that changed on disk mid-run (
input_drift).
- class damicore.MaterializationError(message: str, *, code: str | None = None, **context: object)¶
DistanceMatrixView.to_pandas()would exceed the configured materialization limit. Read throughhead()or NumPy slicing, or passforce=Trueto accept the cost.
- class damicore.NormalizationError(message: str, *, code: str | None = None, **context: object)¶
The normalization stage failed for a reason with no more specific class.
- class damicore.OutputDirectoryConflictError(message: str, *, code: str | None = None, **context: object)¶
The output path exists and is neither empty nor a resumable run of the same input and configuration, or reuse or resume of a compatible one is disabled. Nothing is overwritten.
- class damicore.ResourceLimitError(message: str, *, code: str | None = None, **context: object)¶
Preflight projected a run outside the configured
ResourceLimits.context["estimate"]holds theResourceEstimate, whoseviolationsnames every gate that failed. The message carries what to do about it, anddocs/scalability.mdexplains why the object count is what decides feasibility.
- class damicore.TreeBuildError(message: str, *, code: str | None = None, **context: object)¶
The Neighbor Joining stage failed, including rejecting the distance matrix it was given as input.
- class damicore.TreeFormatError(message: str, *, code: str | None = None, **context: object)¶
A tree artifact is not a valid unrooted binary tree: duplicate or dangling node ids, a non-internal or non-binary root, non-finite branch lengths, or a disconnected graph.
DistanceMatrixView is re-exported here from damicore_distance; it is described once, in distance.