Scalability and resource gates

For n objects, DAMICORE computes n(n-1)/2 NCD pairs, stores an n × n float64 matrix, uses another same-sized on-disk Neighbor Joining workspace, and runs Neighbor Joining in cubic time. Memory maps avoid loading those matrices as ordinary arrays; they do not change algorithmic complexity.

Preflight performs the run’s own traversal without creating object files and calculates exact object bytes plus conservative memory and disk bounds. For a files source those object bytes include the copy the run makes of the corpus. Default limits cap objects at 1,000, pairs at 500,000, and each matrix at 512 MiB. estimate always returns its ordered violations. run turns any violation into ResourceLimitError; free disk can never be bypassed.

The supported large-input shape is many input bytes but a moderate object count, typically columns. Splitting a large file into rows can create millions of objects and is deliberately rejected, and so is a directory holding more files than the object cap allows.

In that shape the cost is compression, not the object count. Every object is compressed once and every pair once more, so the distance stage feeds its compressor n times the object bytes, plus the largest object twice for ncd_self_largest: 50 columns of 20 MB are 50 GB of compressor input. estimate reports that figure exactly as compression_input_bytes, which a measured compressor throughput turns into time, and max_pair_bytes, the longest concatenation the compressor must hold for an NCD value to see across a pair. The first is not a gate; the second is, against the window the compressor declares: zlib and gzip match across 32 KiB, bzip2 across its block, lzma across its dictionary, and a pair longer than that is refused with the compressor_window violation, because past it the compressor is given a pair it cannot see across and every distance moves toward 1 with nothing saying so. ResourceLimits.max_pair_window_factor lifts the gate – the value is recorded in the manifest – and ppmd, the default, declares no window at all, which [adr-0023] measures rather than assumes. After the run, ncd_self_largest in report.json says how much of its largest object the compressor recognized when it read it twice, which is where a window or a filled model shows at the size the run actually used, and pairs_over_window says how many pairs were longer than the window to begin with. The work is split into at least 64 shards whatever the object count, so every worker has work even when a few large objects make few pairs.