Proposal: datasets.yml manifest tooling — loader/validator + Croissant interop

​

Context

​

Goal

1. Loads datasets.yml into typed records and reports precise, located errors.
2. Validates the base asset record plus tier-conditional required fields, with clear messages (which entry, which field, why).
3. Ingests a published Croissant JSON-LD to bootstrap a draft entry.
4. Emits a Croissant JSON-LD file per entry (or for the whole registry).
5. Round-trips DataCite citation fields for citability / DOI minting.

​

Design sketch

honest_scholar/dataset/manifest.py        # module in the honest-scholar package

​

Data model

​

Validation rules

• Core (all tiers): id (unique slug), version, tier ∈ {A,B,C}, license (SPDX id or explicit terms + URL), redistributable: bool, access ∈ {open,gated}, files[] each with path + sha256 matching ^sha256:?[0-9a-f]{64}$, datasheet (path/URL required; a "N/A + <reason>" placeholder is accepted structurally by validate but flagged by audit as a datasheet-completeness deficiency, never a clean state).
• Tier-conditional: source/retrieval required for Tier B (and retrieval.kind ∈ {http,doi,openml,git-lfs,manual}); acquisition instructions required for Tier C; sensitivity required if PII/confidential.
• Cross-entry: duplicate id detection; a tier: A entry with redistributable: false or access: gated is a hard error (tier follows the license/access, not convenience — matches the skill’s tier-consistency guardrail).
• Errors accumulate (report all, not first-fail) and are formatted as entry '<id>': <field>: <reason>; validation returns a structured result the audit verb renders as a gap report.

​

Croissant mapping (registry ⇄ JSON-LD, schema.org/Dataset)

​

DataCite

​

API / CLI the skill verbs call

load(path) -> Manifest                       # parse + structural decode; raises with line context
validate(manifest) -> ValidationReport       # all rule violations, per entry
entry_from_croissant(json_ld) -> DatasetEntry # ingest → draft (partial, flags TODOs)
croissant_for(entry) -> dict                  # emit one entry's JSON-LD

honest-scholar dataset validate [datasets.yml]              # register/audit gate; exit!=0 on error
honest-scholar dataset ingest <croissant.json> [--into datasets.yml]
honest-scholar dataset emit <id> [-o <id>.croissant.json]   # or --all

• register calls ingest (if a Croissant is supplied) then validate on the new entry before writing; the human still confirms tier + license.
• audit calls validate across every entry and folds the report into its fixity/presence/completeness output.

​

Dependencies & posture

• Deps: pyyaml + stdlib json only. No jsonschema, no pydantic, no network, no crypto. Fits the plugin’s light posture (asset-registry.md, “Plugin vs. consumer”: the plugin owns the schema + loader/validator).
• Read/write scope: metadata only. Never opens data files, never hashes, never reaches the network — that boundary is what keeps this proposal separable from retrieval/mirror tooling.
• Placement: ships in the plugin (generic engine); consumer projects own the datasets.yml entries and blobs.

​

Open questions

• Package vs. single file: start as one manifest.py, or split croissant.py / schema.py now?
• Should base-record validation be shared with literature (common substrate spine) or kept dataset-local until a second consumer exists?
• Croissant recordSet / RAI metadata: emit a minimal stub, or leave to the human?
• SPDX id validation — ship a small allowlist, or accept any non-empty string plus the explicit-terms+URL alternative?
• DataCite: v4.6 (anchored in the standards digest) vs. v4.7 — does emit need to declare a schema version?

​

Acceptance criteria

• load parses a valid datasets.yml into typed records and raises with the offending line/entry on malformed YAML.
• validate flags every missing/invalid core field and every tier-conditional omission (B: source/retrieval; C: instructions; PII: sensitivity), accumulating all violations with entry '<id>': <field>: <reason> messages; exits non-zero from the CLI on any error.
• Duplicate id, and a tier: A entry with redistributable: false or access: gated, are hard errors.
• emit produces schema.org/Dataset Croissant JSON-LD whose distribution[] carries contentUrl + sha256, validated by a round-trip entry → croissant → entry that preserves core + citation fields.
• ingest bootstraps a draft entry from a published Croissant and marks tier/retrieval/datasheet/sensitivity as human-TODO.
• No dependency beyond pyyaml + stdlib; no network or filesystem access to data files during validate/emit/ingest.
• register and audit invoke honest-scholar dataset …; the skill’s § Tooling note already documents this CLI (the interim manual workaround stays until the module lands).

​

Links

• Skill: ../../../skills/dataset/SKILL.md (Manifest, Tiers; § Tooling)
• Standards digest: ../../../resources/references/dataset-management-standards.md (§2, §3)
• Substrate base record: ../../../resources/substrate/asset-registry.md
• Design: ../03-dataset.md, ../04-substrate-and-contract.md
• ADRs: ../../../decisions/0009-dataset-thin-manifest.md, 0012-shared-substrate.md
• Separate proposal (not this one): retrieval / rclone mirror / fixity tooling