← Notebook library

Research Round 12 · 4 October 2026

After the last revision: calibrating release risk

Reasoning & VerificationEvaluation SciencePublished · experiment proposedArchitecture impact: High

Research question

Can the final revised answer be released under a defined selective-risk target?

Summary

Proposes calibrating the terminal answer object rather than the first draft, and tests whether revision and provenance features add useful release coverage. Includes exact synthetic diagnostics; no new model benchmark.

Major findings

  • Exact synthetic diagnostics show how candidate thresholds trade coverage for an upper risk bound at fixed δ = 5% and family size 3. These are worked calculations, not model performance.

Experiments

  • E12-A Exact synthetic calibration diagnostics Executed (diagnostic)
  • E12-B Terminal release calibration study Proposed

Hypotheses

  • H12-1 Terminal evidence-and-revision features improve held-out release coverage by at least five points at a 2% selective-risk target.
  • H12-2 Revision state adds information beyond task difficulty.
  • H12-3 Answer binding catches content changed after scoring.
  • H12-4 Pooled performance conceals predictable regressions in revised and conflicted episodes.

Architecture implications

  • Revision-drift guard. Flag synthesis that omits supported risks or adds untraced claims, rather than presenting it as consensus.
  • Uncalibrated release-risk contract. Return a null risk score until version-bound, externally labelled calibration exists.

Open questions

  • Do terminal evidence and revision features improve useful coverage at identical generation compute?
  • Do revised and conflicted episodes need separate risk calibration?

Connected research

Full experimental notes, prompts, internal architecture details and implementation procedures are kept by Encevra Research.