Object materialization: what an object is

What this stage computes

An object is a sequence of bytes. Everything downstream — the compressed sizes, the distances, the tree, the clusters — is a function of those bytes and of nothing else about the input, so this stage is where the scientific identity of a run is decided.

Objects reach the matrix by one of two sources, which is an axis rather than a set of cases: a dataset is split into objects, and files already are the objects. See [adr-0006]. The source decides the encoding, recorded in the manifest as object_encoding, because an NCD value is only meaningful relative to it.

json-lines/1 — a dataset split into objects

Every object’s bytes come from serializer.py, which is the only producer of them:

serialize_cell("a,b")   -> b'"a,b"\n'
serialize_cell("ç")      -> b'"\xc3\xa7"\n'
serialize_row(["x","y"]) -> b'["x","y"]\n'

One compact JSON string per cell, one compact JSON array per row, UTF-8 without escaping non-ASCII, LF-terminated. Compact means no space after , or :. That form is what makes the bytes independent of chunking and of the platform, and therefore the objects reproducible and hashable; [adr-0001] is its basis.

A .csv and an .xlsx of the same logical table produce identical object bytes, because a typed spreadsheet cell crosses to text through one named rule before it reaches the serializer. That rule is versioned as cell_text_rule and recorded in the manifest; see [adr-0010].

raw-bytes/1 — a corpus of files

Each file is one object and its bytes are unchanged. Objects may be binary. Files are ordered by relative POSIX path and labelled by it, so two files sharing a basename stay distinct, and the corpus is identified by a digest over the whole ordered set.

What makes a run reproducible

  • The bytes are canonical, not incidental. Two runs over the same input produce the same objects byte for byte, which is what lets a compressed size be compared at all.

  • The order is fixed. A dataset’s objects follow the split; a corpus is sorted by relative path. Nothing depends on filesystem enumeration order.

  • The input is digested. The manifest records a SHA-256 over the input, so a result can be attributed to the bytes that produced it.

  • The encoding is recorded, never assumed. object_encoding is in the manifest, so a distance is attributable to the bytes it measured.

What this stage does not do

  • No published source. Nothing here implements a method from the literature; the encoding is this project’s own and [adr-0001] is its basis. A reader looking for the scientific content of DAMICORE should start at ncd.md.

  • No normalisation of content. Case, whitespace, number formatting and encoding of the source cells are passed through as text. The only transformation is the one cell_text_rule names.

  • No feature extraction or discretisation. An object is the data, not a representation derived from it.

Status in this implementation

Available

delimited text with any single-character delimiter (.csv, .tsv, .txt); .xlsx and .xlsm; a corpus of files; split by column or by row

Not available

legacy .xls, .xlsb, .ods; archive expansion; a DataFrame or stream as input; any symbolic or discretised encoding of numeric attributes

Encodings

json-lines/1 for a dataset, raw-bytes/1 for a file corpus