honest-scholar meta-spec §2.2 + the cross-cutting defend skill (sub-spec 1). · Status: verified-source digest; migrates to the plugin’s resources/references/.
Companion to the agency principle. Where agency says the human decides and is accountable, the Understanding principle says the human must understand their own work and the methods they use, well enough to decide and defend — and the plugin examines and teaches to ensure it. Goal: growth, not producing papers/theses the author cannot understand or defend.
The principle
Every material claim, decision, and method must be understood by the author to the standard a good mentor or reviewer would expect. The plugin verifies and builds that understanding through Socratic questioning and teaching; it never substitutes for it. A paper, thesis, or decision may not advance past something the author cannot explain or defend — silently.
Why examination works (and teaches)
Socratic questioning reference: Paul & Elder, The Thinker’s Guide to the Art of Socratic Questioning (2007), ISBN 978-0944583319. (The “Feynman technique” has no peer-reviewed primary source — folklore; lean on IOED + self-explanation.)
What a reviewer/examiner probes (the examination agenda for “your work”)
Original contribution / novelty vs. prior work; methodology justification (“why this method”); limitations & threats to validity (self-awareness is itself what’s tested); real-time reasoning; “what would change your mind / falsify this?”. Bluffing is penalized; “let me think” is rewarded. Sources: Petre & Rugg, The Unwritten Rules of PhD Research, 3e (2020), ISBN 9780335262120; viva guides (UCalgary “Nasty PhD Viva Questions”; Warwick Viva FAQs). Reviewer-rebuttal / self red-teaming is the pre-submission analog (practitioner guidance).Examining the methodology (anti-cargo-cult)
Following rigor rituals without understanding defeats them. Examine the why, not just the how.- Conceptual vs. procedural knowledge — procedures without conceptual anchoring are brittle/misapplied; the two develop iteratively. Hiebert & Lefevre (1986), reprint DOI:10.4324/9780203063538; Rittle-Johnson, Siegler & Alibali (2001), DOI:10.1037/0022-0663.93.2.346; Star (2005), JRME 36(5):404–411. Transfer requires understanding: NRC (2000), How People Learn, DOI:10.17226/9853.
- “Cargo cult science” — all the forms, none of the substance; “you must not fool yourself.” Feynman (1974), Caltech commencement.
- “Mindless statistics” / the null ritual — mechanical NHST replacing statistical thinking. Gigerenzer (2004), J. Socio-Economics 33(5):587–606, DOI:10.1016/j.socec.2004.09.033.
- ASA statement on p-values — misuse from not understanding them. Wasserstein & Lazar (2016), Am. Statistician 70(2):129–133, DOI:10.1080/00031305.2016.1154108; “Beyond p<0.05” (2019), DOI:10.1080/00031305.2019.1583913.
- Researcher degrees of freedom / garden of forking paths — Simmons, Nelson & Simonsohn (2011), Psych. Science 22(11):1359–1366, DOI:10.1177/0956797611417632; Gelman & Loken (2013/2014). Motivation: Ioannidis (2005), PLoS Med 2(8):e124, DOI:10.1371/journal.pmed.0020124.
- Preregistration / confirmatory-vs-exploratory: fixes analysis before outcomes → curbs HARKing + shrinks forking paths; does NOT forbid exploration, only mislabeling it.
- Rival hypotheses + discriminating tests: name a competing explanation + a result that separates it.
- Severity: passing is evidence because the test would probably have failed if the claim were false (Mayo — cite explicitly if used).
- Power / MDE: set before data; a null from an underpowered study is uninformative, not evidence of no effect.
- TOST / equivalence: a non-significant p is NOT “no effect”; equivalence needs a pre-set bound. (The most cargo-culted null claim.)
- Disclosure / file-drawer: selective reporting is what makes p-values unreliable.
- PRISMA: a reconstructable audit trail vs. cherry-picking — not a diagram drawn after an ad-hoc search.
- Explore/confirm firewall (
honest-scholar’s own): it operationalizes forking-paths avoidance — letting exploration set the confirmatory test is the mechanism that invalidates it.
Matt Pocock’s “grill me” — same mechanic, inverted purpose
His skill (marketplace; writeup) interviews a developer to elicit requirements they already hold, one question at a time, and lets the AI recommend answers — the human is the ground truth.honest-scholar’s defend skill verifies understanding of an external truth (claims/methods/literature); it must not supply an answer key for novel claims. Same machinery, opposite epistemics.
Design (as settled in discussion)
A cross-cuttingdefend skill (companion to progress), a Socratic tutor-examiner whose goal is growth.
- Targets:
claim | cited-work | methodology, with stage presets —hypothesis-testing→strategy (assumptions/entailments/falsifiers/rivals);paper-synthesis→positioning (novelty, citation support); thesis defensibility gate→full mock viva. - Loop: probe → detect gap → teach (explain + link references) → re-probe (retrieval loop) → until it holds.
- Two invocation paths: (1) self-invoked on demand; (2) automatic guardrail at material-decision checkpoints (findings verdict, publish decision, thesis defensibility) and when a method is used whose rationale is undemonstrated.
- Guardrail = stop, surface, offer to examine/teach, record — human may override, override logged. Not a hard block (agency: the human drives; EU AI Act human-oversight allows override); but you cannot silently pass a gap. (Optionally the thesis defensibility gate could be made blocking — deferred open item.)
Hard constraints (agency interlock)
- Ask, don’t grade substance. Report “you could not explain X / had no answer for the falsification probe” (observed fact), never “you are wrong about X” (a judgment the AI isn’t entitled to make on a novel claim).
- Teaching is source-grounded, scoped by target. Methodology + cited-work are largely established → teach freely from sources (these digests, the author’s citations, external authoritative refs). Novel claim → teach how to reason/defend, cite, but never assert the claim’s truth.
- Settled vs. contested. Probe hard on settled errors (non-significant p = “no effect”); on contested choices (frequentist vs. Bayesian) probe only for a defensible justification and present the standard critique — do not impose a school. Miscalibrating (treating contested as error) violates agency.
- Calibrate depth to stakes (severity/power logic): hardest on the method carrying the central claim. “Enough” = can state what it buys + its key assumption/limit + answer the one canonical critique. Anti-Goodhart: reward articulable understanding, not checklist completion — don’t let the
defendskill become its own ritual.