ADR 0014: Dataset partitioning is upstream of the source axis and emits datasets¶
damicore_normalizer gains one public function that splits a dataset into
high and low subsets by a numeric column. This records where that capability
lives, what its artifact is, and the rules that fix its results, because the
product map in AGENTS.md is closed and adding a domain to it needs a
decision, not a pull request. The specification the implementation is checked
against is docs/dataset-partitioning.md; this ADR records only
what would be reopened by accident without it.
Placement¶
The capability lives in damicore_normalizer, upstream of the object source
axis of ADR 0006, and emits datasets, not objects. Three alternatives were
weighed. A fourth object source fails on semantics: the emitted subsets
overlap by construction, the top quarter being inside the top half, so they
cannot be objects of one distance matrix. A script under scripts/ has no
public contract, no error class, no manifest, and would re-implement the
readers. A separate distribution meets the packaging economics ADR 0012
measured and adds a sixth lockstep version for one function. The normalizer
already owns reading both dataset formats, the input contract, the error
class, and the atomic writer, so the capability reuses every one of them and
adds none. Each emitted file re-enters the pipeline through the existing
delimited source; the 0.2 manifests and schemas are unchanged.
The artifact is user-facing, not inter-stage¶
No stage consumes partition.json; the researcher does. It therefore carries
its own schema and input models rather than reusing the normalization
manifest’s, whose split field does not apply here and could not be carried
without being ignored, which the input contract forbids. The link to a later
run is the emitted file’s SHA-256, which the run manifest already records, so
no cross-manifest field exists to drift. The schema persists only what
reproduces or verifies the partition. Every derived or diagnostic field was
left out, because a field in a strict schema is cheap to add and expensive
to remove, as ADR 0011 measured.
One seam, two consumers¶
The readers’ validated text rows are the seam between what a file says and
what is done with it. split_table and the partition are its two consumers;
neither parses a file, and a third consumer would meet the same seam. Each
reader’s row iterator therefore becomes a package-internal function rather
than a private detail of one caller.
The rules are one versioned name¶
Everything an emitted file depends on is partition_rule v1: the numeric
grammar, separator resolution by falsification over the whole column against
the closed hypothesis space {".", ","}, the total order by value descending
then file position, the three method definitions with their tie-breaks, and
the one output form. Any change is v2 under a new name, never a silent
redefinition. Methods carry no version of their own.
Quantile and percentile are two spellings of one rank-based rule on exact
fractions. Jenks is value-based, computed over weighted distinct values with
exact rational arithmetic and a declarative tie-break, so equal values never
separate and the result is identical on every platform. The grammar bounds
form and the exponent width Decimal can construct, six digits, so that the
pattern is a total gate and no library exception can escape the package’s
failure contract; within that, every decimal is accepted as written, no
Decimal arithmetic is ever performed because it rounds to context
precision, and only Jenks carries a magnitude bound, stated as the span exact
arithmetic can afford.
Emitted files are always the library’s default dataset form, , and UTF-8
with LF, whatever the input was. The pipeline measures cell text through
UTF-8 JSON, so container encoding and delimiter never reach object bytes.
Defaults¶
Quantiles (2, 4, 8, 16, 32) are the maintainer’s request. Percentiles
(5, 10, 20) are the canonical top and bottom cuts of descriptive statistics
and portfolio sorting, chosen so that any n satisfying the quantile default
satisfies them. Jenks (2, 3, 4, 5) spans the single natural break, the
high-middle-low split, and the class count cartographic practice defaults
to. A default a column cannot satisfy is refused with the disabling argument
named, never skipped.
Accepted costs¶
Exact arithmetic over performance. Measured on the implementation with the default methods: one hundred thousand rows partition in 1.7 s when the column holds a thousand distinct values and in 11.8 s when every value is distinct, the difference being Jenks; one million rows with a thousand distinct values partition in 22.5 s with a 127 MB peak. Both are dominated by reading the input at the sizes the downstream pipeline can cluster, and are accepted for this release.
Revision condition¶
Reopen the exact float64 coarsening of the sort key if the key memory is
measured as a limiting cost, or a resource gate for Jenks if its time is
measured as infeasible at a realistic number of distinct values. Reopen the
reader seam’s contract if a third consumer needs different validation. A
request to expose partitioning through damicore.run reopens ADR 0013’s
signature question for that surface only.