Normalizer¶
damicore_normalizer turns an input into objects, and is the only producer of the versioned normalization manifest. It also partitions a dataset, upstream of that.
Functions¶
- damicore_normalizer.materialize_objects(source: str | Path | Sequence[str | Path], output_dir: str | Path, *, config: NormalizationConfig | None = None) NormalizationResult¶
Turn an input source into deterministic versioned object artifacts.
Writes
manifest.jsonand one object file underobjects/inoutput_dir, which must be absent or empty. What one object is followsconfig.source: one column or one data row of a dataset, or one adopted file. Every written object is re-read and checked against its recorded size and SHA-256, and every input file is re-stat’d afterwards, so the manifest is only written once the artifacts and their sources have been shown to agree. That manifest is the inputdamicore_distance.compute_distance_matrix()expects.- Raises:
NormalizerError – An input path is missing, of the wrong kind, or unreadable (
input_validation_error);output_direxists and is not empty (output_conflict_error); a dataset violates the input contract (dataset_format_error); a corpus violates the corpus rules (corpus_validation_error); an input changed while it was being read (input_drift); a written object does not match its digest (artifact_validation_error).
- damicore_normalizer.normalize_csv(csv_path: str | Path, output_dir: str | Path, *, config: NormalizationConfig | None = None) NormalizationResult¶
Normalize one delimited-text file. A thin wrapper over
materialize_objects().Kept for callers written against 0.1. It accepts only a delimited source, which is what its name promises; every other source goes through
materialize_objects().
- damicore_normalizer.partition_dataset(source: str | Path, output_dir: str | Path, *, column: str, quantiles: Sequence[int] = (2, 4, 8, 16, 32), percentiles: Sequence[int] = (5, 10, 20), jenks_classes: Sequence[int] = (2, 3, 4, 5), decimal: str | None = None, source_kind: str = 'delimited', delimiter: str = ',', encoding: str = 'utf-8', sheet: str | None = None, chunk_rows: int = 50000) PartitionResult¶
Split a dataset into high and low subsets by a numeric column, with a manifest.
For every requested partition –
quantileq,percentilep, orjenksk classes – writes the rows with the highest values and the rows with the lowest values to{method}_{parameter:02d}_{high|low}.csvunderoutput_dir, which must be absent or empty, thenpartition.json. Each file is a pure subset of the input in input order, cells unchanged, in the library’s default dataset form (,, UTF-8, LF), so it feedsmaterialize_objects()ordamicore.runwith no further argument. Rows are ordered by (value descending, file position ascending) underpartition_rulev1, whose numeric grammar, separator resolution, method definitions and tie-breaks are specified indocs/dataset-partitioning.mdand decided in ADR 0014.decimalis the target column’s decimal separator,.or,;Noneresolves it by testing both against every cell and refusing when neither fits. Parameters are lists or tuples of ints, sorted here, an empty one disabling its method.delimiterandencodingapply to a delimited source andsheetto anxlsxone; a non-default setting that does not apply is rejected.Every data-dependent refusal happens before any file is created, and a failure after that leaves a directory without
partition.json, which is the incomplete state. No message or artifact carries a cell value.- Raises:
NormalizerError – An argument is invalid (
input_validation_error);output_direxists and is not empty (output_conflict_error); the header, the column, a cell of it, or a precondition of a requested method is violated (dataset_format_error); the input changed while it was being read (input_drift); an emitted file does not match its digest (artifact_validation_error).
Models¶
- class damicore_normalizer.DelimitedSource(*, kind: Literal['delimited'] = 'delimited', split: Literal['columns', 'rows'] = 'columns', delimiter: str = ',', encoding: str = 'utf-8')¶
Split one delimited-text file. Any single character is a delimiter, so .csv, .tsv, and .txt are the same source with different separators.
- kind: Literal['delimited']¶
- split: Literal['columns', 'rows']¶
- class damicore_normalizer.FileCorpusSource(*, kind: Literal['files'] = 'files', recursive: bool = True, include_hidden: bool = False)¶
Adopt files that are already the objects. No split, no delimiter, no encoding.
- kind: Literal['files']¶
- class damicore_normalizer.NormalizationConfig(*, source: DelimitedSource | SpreadsheetSource | FileCorpusSource = DelimitedSource(kind='delimited', split='columns', delimiter=',', encoding='utf-8'), chunk_rows: Annotated[int, Gt(gt=0)] = 50000, max_open_files: Annotated[int, Gt(gt=0)] = 64)¶
What the objects are (
source) and how the run is executed (everything else).- source: ObjectSource¶
- class damicore_normalizer.NormalizationManifest(*, schema_version: Literal[2], object_encoding: Literal['json-lines/1', 'raw-bytes/1'], input: DelimitedDatasetInput | SpreadsheetDatasetInput | FileCorpusInput, objects: tuple[ObjectDescriptor, ...])¶
- schema_version: Literal[2]¶
- object_encoding: ObjectEncoding¶
- input: NormalizationInput¶
- objects: tuple[ObjectDescriptor, ...]¶
- class damicore_normalizer.NormalizationResult(*, manifest_path: Path, object_count: Annotated[int, Ge(ge=0)], total_bytes: Annotated[int, Ge(ge=0)], objects: tuple[ObjectDescriptor, ...])¶
- manifest_path: Path¶
- objects: tuple[ObjectDescriptor, ...]¶
- class damicore_normalizer.ObjectDescriptor(*, object_id: str, label: str, relative_path: str, size_bytes: Annotated[int, Ge(ge=0)], sha256: Annotated[str, _PydanticGeneralMetadata(pattern='^[0-9a-f]{64}$')])¶
- class damicore_normalizer.PartitionFile(*, method: Literal['quantile', 'percentile', 'jenks'], parameter: int, end: Literal['high', 'low'], rank_start: Annotated[int, Ge(ge=0)], rank_end: int, size_bytes: Annotated[int, Ge(ge=0)], sha256: Annotated[str, _PydanticGeneralMetadata(pattern='^[0-9a-f]{64}$')])¶
One emitted subset: which partition end it is, which ranks it holds, and its bytes.
The file name and the row count are derived, not persisted: the name is fixed by the naming rule and the count is a difference of two fields, so storing either would be a second copy of a truth already in the record.
- method: PartitionMethod¶
- end: PartitionEnd¶
- class damicore_normalizer.PartitionManifest(*, schema_version: Literal[1], partition_rule: Literal['v1'], input: DelimitedPartitionInput | SpreadsheetPartitionInput, column: str, decimal: Literal['.', ','], row_count: Annotated[int, Ge(ge=2)], files: tuple[PartitionFile, ...])¶
The schema of partition.json: what reproduces the partition and what verifies it.
- schema_version: Literal[1]¶
- partition_rule: Literal['v1']¶
- input: PartitionInput¶
- decimal: Literal['.', ',']¶
- files: tuple[PartitionFile, ...]¶
- class damicore_normalizer.PartitionResult(*, manifest_path: Path, manifest: PartitionManifest)¶
What partition_dataset returns: where the manifest is, and the manifest itself, so the result cannot disagree with the artifact.
- manifest_path: Path¶
- manifest: PartitionManifest¶
- class damicore_normalizer.SpreadsheetSource(*, kind: Literal['xlsx'] = 'xlsx', split: Literal['columns', 'rows'] = 'columns', sheet: str | None = None)¶
Split one worksheet of an .xlsx/.xlsm workbook.
sheetisNoneonly when the workbook holds exactly one worksheet. A workbook with several requires the name, because defaulting to the first would silently decide which data was analyzed. There is no delimiter or encoding: a workbook declares neither.- kind: Literal['xlsx']¶
- split: Literal['columns', 'rows']¶
Errors¶
- class damicore_normalizer.NormalizerError(message: str, *, code: str = 'normalizer_error', **context: object)¶
Base error raised by the standalone normalizer package.
codeis the stable machine-readable identifier a caller branches on; the orchestrator maps it onto its own public class, so it is part of this package’s contract rather than a message detail.Neither the message nor
contextmay carry a cell value, a whole input row, or the contents of an adopted file. This is the package that reads all three, which is what makes the rule worth stating at the class every one of its failures passes through: an error reaches logs and issue trackers that the data itself is not allowed to reach. Say what was wrong and where – a line number, a field count, a type name, a path – never the value.