Object materialization: what an object is¶
What this stage computes¶
An object is a sequence of bytes. Everything downstream — the compressed sizes, the distances, the tree, the clusters — is a function of those bytes and of nothing else about the input, so this stage is where the scientific identity of a run is decided.
Objects reach the matrix by one of two sources, which is an axis rather than a set of cases:
a dataset is split into objects, and files already are the objects. See
[adr-0006]. The source decides the encoding, recorded in the manifest as
object_encoding, because an NCD value is only meaningful relative to it.
json-lines/1 — a dataset split into objects¶
Every object’s bytes come from serializer.py, which is the only producer of them:
serialize_cell("a,b") -> b'"a,b"\n'
serialize_cell("ç") -> b'"\xc3\xa7"\n'
serialize_row(["x","y"]) -> b'["x","y"]\n'
One compact JSON string per cell, one compact JSON array per row, UTF-8 without escaping
non-ASCII, LF-terminated. Compact means no space after , or :. That form is what makes
the bytes independent of chunking and of the platform, and therefore the objects
reproducible and hashable; [adr-0001] is its basis.
A .csv and an .xlsx of the same logical table produce identical object bytes,
because a typed spreadsheet cell crosses to text through one named rule before it reaches
the serializer. That rule is versioned as cell_text_rule and recorded in the manifest; see
[adr-0010].
raw-bytes/1 — a corpus of files¶
Each file is one object and its bytes are unchanged. Objects may be binary. Files are ordered by relative POSIX path and labelled by it, so two files sharing a basename stay distinct, and the corpus is identified by a digest over the whole ordered set.
What makes a run reproducible¶
The bytes are canonical, not incidental. Two runs over the same input produce the same objects byte for byte, which is what lets a compressed size be compared at all.
The order is fixed. A dataset’s objects follow the split; a corpus is sorted by relative path. Nothing depends on filesystem enumeration order.
The input is digested. The manifest records a SHA-256 over the input, so a result can be attributed to the bytes that produced it.
The encoding is recorded, never assumed.
object_encodingis in the manifest, so a distance is attributable to the bytes it measured.
What this stage does not do¶
No published source. Nothing here implements a method from the literature; the encoding is this project’s own and [adr-0001] is its basis. A reader looking for the scientific content of DAMICORE should start at
ncd.md.No normalisation of content. Case, whitespace, number formatting and encoding of the source cells are passed through as text. The only transformation is the one
cell_text_rulenames.No feature extraction or discretisation. An object is the data, not a representation derived from it.
Status in this implementation¶
Available |
delimited text with any single-character delimiter ( |
Not available |
legacy |
Encodings |
|