hypothesis-testing

• Agency — the verdict is a material decision: the researcher decides it and signs it by name. The experiment backend stamps evidence; it never adjudicates.
• Understanding — a refuted hypothesis is a successful, done result, not a failure. The point is to learn the truth of the claim, whichever way it falls.

​

When to use

• A backlog.md item has just been promoted and needs to be resolved.
• You are resuming an in-flight hypothesis under docs/research/<paper>/hypotheses/<YYYY-MM-DD-slug>/ (pick up at the first unfinished staged doc).
• You need to re-check whether a resolved hypothesis’s evidence has gone stale (is-current on its run-refs). Revising a signed verdict is itself a material decision — it re-triggers the defend guardrail and a new named sign-off, not a silent flip.

​

Staged documents

1. hypothesis.md — the free-form claim, carried over from the promoted backlog item. State the claim, why it matters, and what a confirming vs. refuting outcome would look like. No method yet.
2. strategy.md — the science, before any engineering. This is the heart of the skill:
   • How the claim will be confirmed or refuted — the decisive comparison, the measured quantity, the decision rule.
   • Rival hypotheses + discriminating tests — at least one competing explanation and a result that would separate it from your claim.
   • The rigor-kit choices (below): confirmatory-vs-exploratory tag, preregistered analysis, power/MDE, severity argument, equivalence bounds if the claim is a null.
   • Calls literature position --level hypothesis — an adversarial precedent rapid-review answering “would a reviewer say this is already known?” Cite the precedent that would undercut novelty.
   • Declares the datasets it will use; materialize/verify them via the dataset skill so their fingerprints are pinned.
   • defend fires here on the strategy target — assumptions, entailments, falsifiers, rival explanations, and the methodology it invokes. Resolve surfaced gaps before proceeding to engineering. See ../defend/SKILL.md.

   e.g. claim: “pruning to 50% sparsity doesn’t hurt accuracy.” Strategy — decisive comparison: pruned vs. dense test accuracy; it is a null claim, so TOST against a pre-registered ±1% equivalence bound, powered for it; rival: “the task is too easy to separate them,” discriminated by a harder split; tag: confirmatory.

3. design.md / plan.md — the engineering, delegated. Hand off to the engineering backend bound in .honest-scholar/config.yml (engineering_backend:) via the engineering-delegation contract: design → plan → implement. Store the design and plan under this hypothesis folder (design.md, plan.md); implement builds the experiment harness/code. honest-scholar does not design experiments, write plans, or implement — it composes with the engineering backend for all of that. The experiment backend’s run (below) then executes the built harness to produce the runs.
4. findings.md — the verdict, a material decision.
   • Report results by citing the experiment backend’s run-refs — never hand-copied numbers. Result tables/figures are written by the backend’s tables capability as managed, regenerable blocks.
   • State the verdict: confirmed | refuted | inconclusive, with the reasoning tied to the strategy’s decision rule and severity argument.
   • The verdict is recorded with a named human sign-off + date. The defend guardrail fires before sign-off (findings-verdict checkpoint): stop, surface any gap, offer to examine/teach, record; the human may override, override logged. Not a hard block — but a gap can never be passed silently.

​

How it works

​

Rigor kit

• Confirmatory vs. exploratory / preregistration — fix the analysis before outcomes; curbs HARKing and shrinks the forking-paths garden. Does not forbid exploration, only mislabeling it.
• Rival hypotheses + discriminating tests — a competing explanation plus a result that separates it.
• Severity — passing counts as evidence only if the test would probably have failed were the claim false.
• Power / MDE — set before data; a null from an underpowered test is uninformative, not evidence of no effect.
• TOST / equivalence — a non-significant p is not “no effect”; a null claim needs a pre-set equivalence bound. (The most cargo-culted null claim.)
• Disclosure checklist + file-drawer — report all runs and analyses; selective reporting is what corrupts the record.
• Per-dataset datasheets — Gebru-style, produced via the dataset skill for every dataset the strategy declares.
• Red-team pass — adversarial self-review of the strategy and the findings before the verdict is signed.

​

Composition

• hypothesis-exploration — upstream; a promoted backlog item seeds hypothesis.md.
• literature — position --level hypothesis in the strategy stage (adversarial precedent review).
• dataset — materialize, verify, and datasheet every declared dataset.
• The bound engineering backend — design → plan (and implement) for the design.md/plan.md stage, via the engineering-delegation contract. All engineering lives here (ADR-0002, refined by ADR-0023).
• The bound experiment backend — run / evidence / tables / is-current.
• defend — strategy target (during strategy) and guardrail (before the findings sign-off).
• progress — reads this hypothesis’s status frontmatter; a resolved hypothesis (any verdict) rolls up as done/green.

​

Guardrails

• Science before engineering. No design.md/plan.md until strategy.md is written and examined. No numbers in findings.md until runs exist.
• Never hand-copy results. Cite run-refs and let tables write the result blocks. A number without a run-ref is not evidence.
• The verdict is the human’s. It is a material decision recorded with a named human sign-off; the backend stamps evidence but never adjudicates. The defend guardrail fires before sign-off and any override is logged.
• Refuted is done, not failed. Verdict and readiness are distinct axes; a refuted claim is a successful result. Do not treat it as a red mark or bury it — file-drawer discipline forbids that.
• Don’t do engineering here. Delegate design/plan/implementation to the bound engineering backend; this skill owns only the scientific stages.
• Stay behind the firewall. This skill resolves; it does not generate candidate hypotheses, and it does not adjudicate without the researcher’s recorded decision.
• Follow-ups become issues, not TODOs — a deferred check or a known gap in the record is captured as a self-contained GitHub issue, not left in a doc’s margin.

​

Commit attribution

Generated-with: honest-scholar (https://github.com/davorrunje/honest-scholar)
HonestScholar-Skill: hypothesis-testing