honest-scholar sub-spec 4 — Shared substrate & experiment-backend contract

Sub-spec of 00-meta-spec.md. Defines the foundation that sub-specs 2 (literature) and 3 (dataset) build on: the asset-provenance substrate (registry pattern + private mirror + fixity + persistent-ID) and the experiment-backend contract. Governed by the meta-spec’s ⚑ guiding principle (assistants, not researchers) and its light-dependency posture. Grounding: dataset-management-standards, dataset-tooling-mirror.

​

1. Purpose

1. The asset-provenance substrate — the common machinery under literature and dataset: how the plugin keeps a git-tracked registry of externally sourced, persistently identified, license-bearing, mirror-able assets, and how it fetches and verifies durable copies. literature and dataset are distinct front-ends over this one substrate (they share the mechanism, not the file).
2. The experiment-backend contract — the stable interface through which the pipeline skills obtain and cite experimental evidence, so the runner is hot-swappable and no pipeline skill depends on a concrete tool.

​

2. The asset-provenance substrate

​

2.1 What is shared vs. extended

• literature extends it with bibliographic fields (authors/venue/year), a triage sidecar (role, disposition, rationale, seeded links, citation intent), and the PDF as the files[] payload. Its registry format is a standard bibliography (BibTeX / CSL-JSON) + a YAML triage sidecar, not a clone of the dataset manifest (sub-spec 2).
• dataset extends it with tier (A/B/C), a retrieval recipe, datasheet, sensitivity, and data blobs as the files[] payload; its registry is datasets.yml (sub-spec 3).

​

2.2 Persistent-ID & citation layer

​

2.3 Private mirror + fixity (the mechanism)

• Engine: rclone — invoked as a subprocess (a single Go binary, not a Python dependency). Backend-agnostic (Google Drive first; S3/B2/R2/SFTP/WebDAV/… by config). The registry names a logical remote + base path; the credential-bearing rclone.conf is never committed (untracked file via RCLONE_CONFIG, or env-var remotes from CI secrets). A rclone.conf.example (remote name + type only) is committed. rclone obscure is not encryption and is never committed.
• Fixity: two-layer. The authoritative checksum is SHA-256, stored in the registry (integrity == identity == citation-verifiability). rclone’s native per-backend hash (often MD5 on Google Drive/S3) is used only as a transfer check; after any transfer the local bytes are re-hashed against the registry SHA-256 — so verification is backend-independent and the manifest is the single source of truth. (Resolves the meta-spec §10 hash open item: SHA-256 authoritative; MD5/native only as a transport check.)
• Store layout: gitignored cache; content-addressed on the wire (sha256/<hash> mirror keys → dedup, integrity == identity), name-addressed for use (symlink/copy from the content-addressed blob).

​

2.4 Resolution chain

1  LOCAL CACHE      exists and sha256==registry  → return; else discard-if-corrupt
2  PRIVATE MIRROR   (if configured) rclone copy mirror:base/<key> → verify → return; else fall through
3  PUBLIC SOURCE    front-end fetcher (literature: PDF/DOI; dataset Tier A git-LFS, Tier B pooch/http/doi)
                    → assert sha256==registry (hard fail) → populate mirror → return
4  GATED / MANUAL   print acquisition instructions → wait for operator drop → verify → populate mirror → return

​

2.5 License / redistribution rule (shared, non-negotiable)

​

2.6 Plugin vs. consumer

• Plugin: base-record schema + loader/validator; the persistent-ID/citation module; the rclone mirror integration + fixity + content-addressed cache; the resolution chain (steps 1/2/4); the rclone.conf.example generator and secret-hygiene docs. Deps: rclone (subprocess) + pyyaml (+ pooch, owned by the dataset front-end).
• Consumer: the registry entries + SHA-256 checksums; the real rclone remote config (untracked/CI); mirror blobs; the gitignored cache.

​

3. The experiment-backend contract

• A run-ref is the citable unit of evidence. findings.md (hypothesis) and the paper ledger/decision reference run-refs, never raw numbers copied by hand. The tables capability is the only writer of result numbers into docs (managed, regenerable blocks).
• The provenance stamp must carry the dataset fingerprint from §2 (id+version+sha256), so every reported result resolves to exact bytes.
• is-current is what makes selective re-execution honest: a hypothesis or paper can ask “is my evidence stale?” without knowing how the backend computes staleness.
• The contract is agnostic to how experiments run (local, GPU fan-out, cluster, cached). Nothing above mentions a scheduler.

​

3.1 Binding

​

3.2 Implementations

• The plugin bundles no backend — it ships the contract only, so it stays domain-neutral. Each project supplies an implementation of the four capabilities and binds it in papers.md.
• A typical implementation maps the capabilities onto the project’s own tooling: run = its experiment executor (local, GPU-pool, or cluster); evidence = committed results + a provenance sidecar (dataset id+version+sha256, config hash, code provenance); tables = a managed-block write-back into docs; is-current = a provenance/closure hash. A lightweight local runner and a large-scale GPU orchestrator are equally valid.
• Because the contract is abstract, different projects may bind different backends without touching any pipeline skill.

​

3.3 Agency-principle interaction

​

4. Open items

• run-ref format — opaque string vs structured (backend-hash + trial-id); settle when cross-backend interop is first contemplated. Default: opaque string the backend can resolve.
• Croissant/BibTeX export placement — whether the persistent-ID module owns export, or each front-end does; lean toward the shared module for pid/citation and the front-end for its native format (BibTeX for literature, Croissant for dataset).
• Cache directory name — align with the meta-spec’s .honest-scholar/ decision.

​

5. Downstream