Roadmap¶
DAMICORE 0.3 implemented the canonical 2011 pipeline – exact NCD over zlib/gzip,
deterministic Neighbor Joining, FastGreedy community detection – with engineering the
published work on the method never had: content-addressed runs, a manifest that records input,
configuration and artifact hashes, resumable shards, preflight resource gates, decision
records, strict typing. That release is the baseline this plan was measured against, and the
one the evidence page audits.
What it did not implement is what the literature added to the method after [sanches2011]. A
run returns membership, clusters, tree_newick, the distance matrix and a modularity
value: a partition with no meaning attached and no measure of how stable it is.
0.4.0 closed one of those gaps and the whole scientific record. The distance stage measures
with more than one compressor, each exposing the settings that change what it can see across
a pair – which is two of G1’s seven items, #60 and #61, and the compressors themselves are
named by ADRs 0020 and 0021 rather than restated here; and docs/ carries the method, the
references and the provenance, which is G7, closed. The rest of G1 and every other group
below is still ahead.
This page is the plan for closing that gap. The evidence it rests on – which capability is attested by how many published works, and the data behind each count – is the evidence page. Live status lives on the issue tree it links to; this page carries the structure and the rules, which do not change when an item closes.
Rules that apply to every item¶
Cite what you implement. Every method, index, criterion or encoding added here comes from a specific published work. The implementing pull request records the work in
docs/refs.json, cites the key on thedocs/method/page for its stage, and names it in the docstring of the unit that computes it, with what this implementation does not take from that work. An implementation whose source cannot be named does not belong here. See ADR 0015 and ADR 0019.A result carries its own provenance. When a change alters what is computed, the run manifest records the named, versioned rule that produced it – the pattern ADR 0008 set for
object_encoding, ADR 0014 forpartition_ruleand ADR 0018 for themethodblock. A run directory from a year ago must still say which variant of the method produced it.Deferred decisions need a record.
AGENTS.mdnames the decisions this project has deferred – approximate NCD, Dask or Ray, PyArrow or Polars, GPU, compressors beyond the ones ADRs 0020 and 0021 settle, DataFrame or stream input, Parquet output, a remote service, legacy spreadsheet formats, archive expansion. An item that reopens one carries a decision record in its acceptance criteria, and that record states the measurement, not the intention. ADRs 0020 and 0021 are what that looks like done: both reopened the compressor deferral, and both carry the measured table that decided it.The quality-driver order settles trade-offs. Correctness above bounded memory above reproducibility, down to portability (quality drivers). Correctness items come first because they are correctness, not features.
No silent sampling, ever. Scale is bought with incremental computation, declared parallelism and measured limits – never by quietly reducing the problem.
How the tree is organised¶
One group per layer of the pipeline, matching the package boundaries of ADR 0001. Milestones carry the sequence (0.4 to 1.0); groups carry the architecture. Every item states its own scientific basis and its own documentation obligations.
The whole tree hangs from the epic, Roadmap: bring DAMICORE from the 2011 core to the published state of the art.
G1 — #48 Distance layer: compression distance beyond a single fixed compressor¶
Item |
Milestone |
What it adds |
|---|---|---|
0.5 |
refuse or flag objects larger than the compressor window |
|
0.4 |
add bzip2 and LZMA compressors (standard library, no new dependency; ADR 0021) |
|
0.4 |
add PPMd (pyppmd) as the default compressor and a required dependency (ADR 0020) |
|
0.5 |
interleaved pairing for objects larger than the compressor window |
|
0.5 |
incremental NCD — extend an existing run with new objects |
|
0.9 |
pluggable metric interface (Levenshtein, DTW, Euclidean) alongside NCD |
|
0.9 |
combine several representations of the same object into one matrix |
G2 — #49 Tree layer: from one deterministic tree to a defensible one¶
G3 — #50 Partition layer: how many groups, and by which criterion¶
G4 — #51 Interpretation layer: what the groups mean and whether they hold¶
Item |
Milestone |
What it adds |
|---|---|---|
0.6 |
attribute-range labels with a non-overlap guarantee |
|
0.6 |
dendrogram and phylogram rendering with cluster and label coloring |
|
0.7 |
decision-tree backend |
|
0.7 |
internal and external cluster indices |
|
0.7 |
stability by resampling — congruence index and jackknife |
|
0.8 |
neural relevance backend (MRA/ALM) and labeling of internal tree nodes |
G5 — #52 Input layer: how raw data becomes comparable bytes¶
G6 — #53 Scale, prediction and the benchmark that closes the loop¶
Item |
Milestone |
What it adds |
|---|---|---|
0.9 |
re-derive the resource limits from measurement, per compressor and stage |
|
1.0 |
classify a new object against a finished run (quartet test / BCC) |
|
1.0 |
cold-start sample selection from the phylogram (PhyIL) |
|
1.0 |
canonical cases and an ablation harness (regression infrastructure) |
G7 — #54 Scientific record: make docs/ carry the method, the references and the memory¶
Item |
Milestone |
What it adds |
|---|---|---|
0.4 |
create docs/references.bib and make citation an enforced convention |
|
0.4 |
add docs/method/ — one page per pipeline stage, stating what it computes and citing it |
|
0.4 |
the manifest records which method variant produced a run |
|
0.4 |
publish the roadmap and the evidence behind it in the repository |
|
0.4 |
give a basis that is a project decision somewhere to point |
|
0.4 |
a ratchet, so an undeclared basis cannot regress |
|
0.4 |
twelve internal types appear in public signatures and a reader cannot follow them |
|
0.4 |
exercise Read the Docs or delete its configuration |
|
— |
the bibliography is fetched, never written (mechanism) |
|
— |
every critical module declares its basis and its delta |
Not in scope here¶
Writing the methodological paper. The benchmark and ablation harness is in scope as regression infrastructure; the study it enables is deferred until this tree is implemented.