honest-scholar dataset capability (sub-spec 3). · Status: verified-source digest; migrates to the plugin’s resources/references/.
Grounding for a domain-neutral dataset registry + tiered retrieval + private mirror.
1. Named standards
- FAIR — Wilkinson et al. (2016), Scientific Data 3:160018, DOI:10.1038/sdata.2016.18. A1.2 allows an auth step; A2 requires metadata to persist even when data don’t → the citable basis for gated Tier C (metadata/datasheet/citation stay open; blob gated). FAIR ≠ open. Map: F→persistent id/DOI/rich metadata/version; A→retrieval recipe +
access(open/gated) + mirror; I→SPDX license ids, schema.org/Croissant names; R→license/provenance/datasheet. - Datasheets for Datasets — Gebru et al. (2021), CACM 64(12):86–92, DOI:10.1145/3458723. Seven sections. Questions that become mandatory registry fields: Distribution→license & redistribution terms (drives tier), Distribution→access (open/gated), Composition→sensitivity/PII flag, Maintenance→version + maintainer, Collection→source. Rest stays as the linked datasheet doc.
- MLCommons Croissant — Akhtar et al. (2024), NeurIPS D&B, arXiv:2403.19546; workshop DOI:10.1145/3650203.3663326. JSON-LD on schema.org/Dataset +
distribution(files + checksums),recordSet, RAI metadata. Emitted by HF/Kaggle/OpenML/Google Dataset Search. Use as the interop target, not the internal source of truth: model registry fields to map onto Croissant names (name,license,citeAs,version,distribution[].contentUrl,.sha256); ingest a published Croissant to bootstrap an entry; emit one per entry. Keep tier / mirror / resolution as registry-native. NeurIPS D&B now requires a Croissant file — emitting it is venue-mandated, not gold-plating. - Data citation — FORCE11 Joint Declaration (2014): 8 principles; Evidence (a data-reliant claim cites the data) and Specificity & Verifiability (resolves to exact version/subset + fixity → version + checksum, not just a name) map directly. DataCite Metadata Schema (v4.6): 6 mandatory — Identifier(DOI), Creator, Title, Publisher, PublicationYear, ResourceType → the citation core; also the minimum to mint a DOI. DOIs satisfy FAIR-F + JDDCP Unique-ID/Persistence.
- DOI archival vs private mirror — Zenodo / figshare / Dryad mint DataCite DOIs (durable public copies). Decision rule: redistributable → prefer a public DOI archive (record the DOI); non-redistributable → private mirror only. Not mutually exclusive for redistributable data.
- Fixity — checksums (recommend SHA-256; MD5 only if a source publishes only MD5); verify on every materialization + mirror hop. BagIt (RFC 8493, 2018):
data/payload +manifest-<alg>.txt; allow (not require) for multi-file snapshots to the mirror / a Zenodo deposit. - DMP — funder/institutional requirement; the registry is the executable per-dataset instantiation of a DMP’s storage/preservation/access/licensing sections. Treat DMP export as optional downstream, not a core field set.
[funder wording not individually verified]
2. Registry schema (one entry) — names mapped to Croissant/DataCite
Required core rationale:
id+version+checksum = JDDCP Specificity + provenance; license+redistributable+access = compute tier + stay legal; datasheet+citation = FAIR-R/Credit.
3. Licensing → tier
Tier = f(redistribution right, access automation); redistribution dominates “can it go Tier A (committed)?”
Hard rule: a private mirror is storage, not a redistribution license. Tier +
redistributable set by the license, never by mirror presence; refuse to promote to Tier A on mirror existence; never surface mirror contents publicly.
4. Provenance → evidence
Each experiment record copies, at run time,dataset_id + version + checksum (+alg) of every dataset used (+ optionally doi). verify recomputes checksums against the registry before a run counts as valid (records verified-against-registry). Two-way traceability. Aligns with NeurIPS D&B Croissant requirement + reproducibility-checklist norms.
5. One skill, verbs
Onedataset skill with register / fetch(resolve) / verify / mirror / export(cite) — verbs share one artifact (the entry) and one policy (tier + resolution chain); they’re sequential lifecycle stages with heavy state overlap (fixity is not separate from fetch). Keep the generic engine in the plugin; entries + blobs in the consumer. If the SKILL grows, factor reference material (tier table, schema, datasheet template) into supporting files (progressive disclosure), not multiple skills.