Encevra Research Round 12

After the Last Revision: Calibrating Release Risk for Collaborative AI

Research paper and proposed experiment | 4 October 2026

Evidence status: Primary-source synthesis, an explicit protocol, and executed exact synthetic diagnostics. No new Encevra benchmark using language models was run. No improvement in Encevra's real-world accuracy, calibration, or compute efficiency is claimed.

Abstract

An AI collaboration can gather evidence, correct a draft, and still lack a defensible reason to release the final answer. This round asks whether a score computed from the completed collaboration can support useful coverage at a stated error limit, compared with scores inherited from individual agents. The statistical object is the whole episode after its last revision, including evidence selection, stopping, synthesis, and any formatting that can change meaning. We propose a version-bound release protocol using externally labeled calibration episodes and a predeclared family of acceptance rules. The primary target is the probability of any material error among released answers, with a separate failure probability over calibration samples. This differs from average unsupported-claim proportion, unconditional wrong-answer frequency, and confidence elicited from a model. Existing selective-prediction and risk-control methods supply the mathematics; Encevra's contribution is a concrete integration and falsifiable evaluation of revision-aware terminal scoring. Exact synthetic diagnostics expose denominator mistakes, insufficient calibration, selection multiplicity, and dependence disguised as sample size. A paired experiment reuses identical model traces across release rules, making generative compute exactly equal. Brain-inspired metacognition motivates one feature ablation, not a claim that an agent architecture copies a brain. The remaining empirical question is whether terminal evidence and revision information improve useful release coverage enough to justify their measurement and labeling cost.

1. Archive audit and carry-forward

The current project was inspected before topic selection: the Research overview, rounds index, full Round 9 and Round 10 page texts, the completed Round 11 manuscript, the publications registry, and the uncertainty and confidence passages in the combined Rounds 4-6 page and Round 8. The archive's latest completed entry was Round 11, expanded revision 2, dated 3 October 2026. Accordingly, this paper is Round 12, correcting the recurring request's stale Round 11 number. No later completed round was indexed at inspection. [A1]

Round 11 requested an instrumentation pilot. This round completes a narrow first slice: statistical calculations and structural checks for the records needed at release. It does not complete the proposed model-based action-value pilot. That unresolved handoff remains open.

2. Research question, novelty, and exclusions

Question: With the collaborative policy and total generation compute held fixed, does a score using final evidence and revision state yield higher useful release coverage at a controlled answer-level selective risk than confidence inherited from agents?

The question has two separable parts. Statistical validity asks whether the calibration procedure controls the specified risk under its assumptions. Empirical utility asks whether a score distinguishes good final answers well enough to release a useful fraction. A valid method can abstain almost always; a useful ranking can lack a valid certificate. Neither substitutes for the other.

The literature already contains conformal abstention for language models, calibration of aggregated reasoning paths, calibrated multi-agent fact filtering, and calibrated communication control. Round 12 claims no first invention of these ideas. [S8, S9, S13, S14] Its proposed contribution to Encevra is the combination of a terminal answer contract, revision-aware scoring, version binding, and a paired evaluation that can show whether the extra state is useful. A negative result would favor a simpler confidence gate.

The initial scope is bounded, source-grounded question answering and structured decisions with an explicit rubric. The first implementation accepts a complete answer or abstains from its conclusion. It does not prune inconvenient claims to improve apparent accuracy. Partial answers require a separately specified loss and coverage contract. The round does not address autonomous external actions, continuous online learning, arbitrary distribution shift, per-person guarantees, or a new verification-allocation policy. Statistical acceptance is also not site deployment or authorization to act.

3. What the literature establishes—and what it does not

The review prioritizes original papers and authoritative statistical documentation. Searches covered confidence elicitation, selective prediction, conformal abstention, multi-agent risk control, final-answer aggregation, and metacognition, with a freshness check through 4 October 2026. This is a targeted review, not an exhaustive systematic review. Section 16 records versions, dates, working links, and inspected portions.

3.1 Competing approaches

Tian's positive findings and Xiong's less optimistic findings concern different tasks, models, prompts, and metrics; treating them as a direct contradiction would be misleading. Likewise, an uncertainty signal that predicts abstention can influence behavior without accurately predicting correctness. Kumaran et al. provide causal evidence for confidence-guided abstention in their studied models, not a certificate for Encevra's outputs. [S6, S7, S16]

3.2 Closest neighboring work

Kostka and Chudziak calibrate multi-agent fact retention using a query-level false-discovery proportion and a deviation-adjusted score. Their labels concern entailment by grounding context and come from a stronger model. That target differs from global truth and from the probability that a released answer contains any material error. The useful carry-forward is the query as a dependence unit, not treating every claim as an independent sample. [S8]

Consilience calibrates one-step communication regret under a surrogate discussion objective. Its appendix explicitly separates that guarantee from final-answer correctness, simultaneous trajectory coverage, and an unchanged score distribution at later rounds. [S9] Round 12 instead evaluates the completed episode. Abbasi-Yadkori et al. already study answer-or-abstain decisions; their defined loss counts answering incorrectly over all queries. Gu et al.'s aggregation guarantee also concerns unconditional confident errors. [S13, S14] Our narrower distinction is the denominator and the final revision state, not the existence of abstention.

3.3 Claim and evidence register

4. Define the object before assigning confidence

Let an episode include the task, evidence snapshot, private and shared agent states, tool results, controller actions, random seeds, stopping decision, final synthesis, and final text. Write F_theta for the frozen procedure producing that episode and A for its terminal answer. The procedure may adapt within an episode. Its adaptation rule, models, prompts, retriever, memory snapshot, and stopping behavior must nevertheless be fixed before calibration.

The primary binary label is E = 1 if the released candidate has any material error under the fixed rubric, and E = 0 otherwise. A material error changes a required conclusion, unit, applicability condition, source attribution, or other consequential assertion. Missing required content is a failure under the same rubric, so omission cannot silently improve correctness. The operational completeness detector may miss omissions; external labeling still counts them as errors.

A score s is an ordering statistic computed after synthesis. It may lie in [0,1] without claiming to equal the probability of correctness. For threshold t, release only when required structural checks pass and s >= t. Denote this event by G_t = 1.

Coverage:             C(t) = P(G_t = 1)
Selective risk:       R(t) = P(E = 1 given G_t = 1)
Wrong-release rate:   W(t) = P(E = 1 and G_t = 1)
Identity:             W(t) = C(t) * R(t), when C(t) > 0

These definitions follow the accepted-example meaning of selective risk. [S1] Our population is explicitly the preregistered task sampler, not all questions a future user could ask.

If 100 of 1,000 tasks are answered and 20 answers are wrong, the wrong-release rate is 2%, coverage is 10%, and selective risk is 20%. Reporting only “2% error” would hide the relevant denominator. If every released answer contains one wrong claim among 100 claims, mean false-discovery proportion is 1% while every answer contains an error. These are constructed identities, not observations about a model.

An aggregate guarantee is also not a guarantee for each subtype. In another fixture, 900 unchanged answers have nine errors and 100 revised answers have 20. Pooled error is 2.9%; revised-answer error is 20%. The revised subgroup needs a separately specified analysis and, if a formal group guarantee is claimed, a separately accounted calibration test.

5. Proposed protocol: release only the calibrated terminal object

5.1 Freeze and identify the procedure

Create a manifest containing model revisions, all prompt templates, decoding settings, routing and stopping code, tool versions, evidence and memory policies, output formatting, score coefficients, label rubric, and population definition. Canonically serialize and hash these artifacts. A model alias alone is insufficient if its provider can change the underlying model without an observable revision.

A frozen procedure may run extra checks on difficult tasks and stop early on easy ones. It must do so under the same rule during calibration and evaluation. Agents may share errors, prompts, and sources within an episode. That affects achievable utility but does not require pretending that their votes are independent observations.

Across episodes, do not update shared memory, score coefficients, or the controller from calibration outcomes. If the intended system learns continually, the current iid binomial construction does not certify it; a different sequential analysis is needed. Merely labeling data “exchangeable” is not enough for the iid binomial calculation used here.

5.2 Obtain terminal features without another round of persuasion

After the last content-changing operation, compute a short feature vector from the observable record. Candidate features include terminal worker confidence, agreement before information exchange, number of independently sourced evidence families, unresolved material contradictions, whether the conclusion changed, whether dependencies were rechecked, and the fraction of required claims with usable receipts. All definitions and missing-value handling are fixed on development data.

Fit a small regularized score on development labels, then freeze it. The principal comparison is between this terminal score and simple inherited-confidence scores. “Three sources” is not automatically three independent sources; provenance features must distinguish copies and shared primary origins. Missing provenance should remain visible. It must not be silently imputed as corroboration.

The score is an empirical predictor, not an entailment proof. Agreement and contradiction flags themselves can be wrong. To isolate the value of terminal information, every main comparison receives the same already-produced episode and the same feature-extraction outputs. No score is allowed an uncharged extra model call.

5.3 Calibrate a fixed candidate family

Use disjoint development, tuning, calibration, and final evaluation sets. Development fits features; tuning fixes thresholds and all score variants. Calibration determines which predeclared rules meet the risk target. Final evaluation measures performance without changing the selected rules. A rule includes the score definition and structural eligibility checks, not just a number.

For the first implementation, use exact binary tests and a transparent Bonferroni family correction. This instantiates the selective-classification use of Learn then Test. [S2] It intentionally avoids importing an exact binomial formula into a fractional claim-level loss.

Let M be the total number of rule candidates whose certification can influence the selection. For each fixed rule, count m accepted calibration episodes and k material-error episodes among those accepted. Choose target risk alpha and calibration failure level delta in advance.

p_t = P(Binomial(m, alpha) <= k)
Certify rule t only if m > 0 and p_t <= delta / M.
From certified rules, select the highest calibration coverage.
If none qualifies, return "no certified release rule".

Equivalently, use a one-sided exact binomial upper bound with tail level delta/M and require it not to exceed alpha. NIST documents binomial-tail inversion; the one-sided form assigns the full tail budget to the upper endpoint. [S12]

The elementary justification for this particular binary implementation is as follows. Conditional on the accepted count of a fixed rule, independently sampled accepted episodes have Bernoulli error rate R(t). If that rate exceeds alpha, an unusually small error count has probability bounded by the lower binomial tail evaluated at alpha. Each test is therefore super-uniform under its unsafe null. A union bound over the fixed family limits the probability of certifying any unsafe member to delta. This remains true when the candidate tests are dependent. Selecting any member of the resulting safe set preserves the simultaneous statement. The score need not itself be a calibrated probability.

The result concerns the fixed deployment rule's population risk with probability at least 1-delta over calibration sampling. It is not a posterior probability that this one answer is correct, nor a guarantee that every finite deployment batch has error at most alpha. Zero accepted samples mean no usable certificate; selective risk is not recorded as zero.

A fixed-sequence alternative can reduce multiplicity cost if the ordering is specified before calibration and testing stops at its first non-rejection. It does not require that true risk be monotone. [S2] The pilot uses Bonferroni because it makes joint searches across scores and thresholds easy to audit. A later study may compare the statistical efficiency of valid alternatives without presenting this conservative choice as universally optimal.

5.4 Bind the decision and forbid silent mutation

Store the population identifier, manifest digests, score identifier, threshold, loss definition, calibration set identity, m, k, alpha, delta, family size, and selection rule. Bind each release event to the exact final-answer digest and evidence snapshot recorded when it was scored.

A content change after scoring sends the answer back through the terminal procedure. Rechecking a digest alone is not recalibration: the relevant question is whether the changed behavior was part of the frozen procedure used to produce calibration examples. A new rewrite instruction, new tool, new memory, or changed model revision is a new configuration. It cannot inherit the old certificate merely because its confidence number is similar.

The release record is an audit artifact, not a badge claiming universal truth. If no rule qualifies, return an abstention or a clearly unresolved conclusion. If later evidence is sought, that continuation must be included in a separately calibrated pipeline. Do not repeatedly generate new versions until one happens to pass a gate calibrated for one attempt.

6. Worked example: a corrected configuration answer

Consider a fictional technical question: “Is feature Q enabled by default in release 4.2, and does an existing configuration override that default?” One agent sees an older manual; another sees a current release note. The draft says “enabled.” A check reveals that 4.2 changed the default to disabled but preserved explicit existing settings. The synthesizer revises the answer and includes both conditions.

A first-draft confidence of 0.96 refers to the wrong conclusion. Averaging it with a reviewer's 0.90 does not produce a probability for the revised text. The terminal record instead notes the version correction, two source origins, the explicit override condition, and dependency revalidation. A development-fitted terminal scorer gives this completed answer 0.97. That score is hypothetical and is not evidence of correctness.

Now consider the executed synthetic calibration fixture below. Its 1,000 labeled episodes and scores were constructed, not produced by models. The family has three thresholds, alpha = 0.02, and delta = 0.05, giving a per-candidate test level of 0.016667.

The rule at 0.95 has the largest calibration coverage among certified candidates: 60%. The fictional score of 0.97 would pass it only if structural and version checks also pass. Nothing in the table establishes the example's truth; that comes from its sources and the external rubric. Nor does the fixture demonstrate 60% coverage for Encevra.

Suppose a final style pass then deletes “explicit existing settings override the default.” The answer has changed materially after scoring. The bound belongs to the earlier procedure and output contract; the system must not release the shortened conclusion on the strength of the old decision. Conversely, a purely deterministic formatting transform can be included in the frozen pipeline and applied before the final digest.

7. Executed diagnostic results

diagnostics.py and diagnostic_results.json accompany this paper. They use exact binomial probabilities and inverse-beta calculations through SciPy 1.17.0. They also exercise a small release-record validator. There is no model sampling, benchmark dataset, randomized performance simulation, or fitted score in these diagnostics.

For the multiplicity diagnostic, each of twenty constructed candidate pipelines has independent binary errors across 300 episodes and across candidates, with true risk 2.01%, just above the 2% target. The uncorrected 5% test accepts a candidate with at most one error; its actual false-positive probability is 1.6183%. Searching all twenty gives 1-(1-0.016183)^20 = 0.278416. With correction, only zero-error candidates pass, and the family probability is 4.4284%. These numbers demonstrate a possible failure, not the actual dependence or risk of real threshold families.

For zero errors, the one-sided bound is 1 - (delta/M)^(1/m). Hence the best-case requirement is m >= ceil(log(delta/M)/log(1-alpha)). Accepted count, not total generated count, determines this calculation. At 30% coverage, even the zero-error requirement of 400 accepted cases suggests roughly 1,334 total calibration episodes in expectation; it does not guarantee that a particular sample reaches that count.

The diagnostics also check two non-monotonicity examples. Retaining a correct low-scored claim and an incorrect high-scored claim gives false-discovery proportion 1/2; rejecting only the correct claim raises it to 1. Likewise, a high-confidence answer subgroup can have greater error than the full set. Rejecting more items makes unconditional wrong-release frequency non-increasing for a fixed answer pool, but does not make either ratio non-increasing. The monotonic-FDP statement in [S8, section 4.4] therefore cannot be taken as a generic property of thresholding. This counterexample alone does not invalidate fixed-sequence testing, which has a separate validity argument.

The validator is intentionally limited. Matching identifiers cannot detect false sources, incorrect labels, an unrecorded provider update, or dependence between nominally different families. Its synthetic zero reasoning-token count is an invented fixture value. A real accounting adapter must record unavailable quantities as unknown rather than report them as zero.

8. Brain-inspired hypothesis with a discriminating test

Fleming and Daw model self-evaluation using a monitoring representation coupled to, but potentially distinct from, the representation supporting a decision. This computational account does not establish a mandatory serial module architecture. [S11] The engineering hypothesis is simply that an answer's production state and its final reliability state may contain different useful information.

Test that idea by giving one scorer only agent confidence and another scorer the same confidence plus the final evidence and revision features. Both receive identical completed answers. The second scorer gets no extra generative inference. If it improves coverage at matched selective risk, terminal information adds predictive value. If removing revision features leaves performance unchanged, the proposed metacognitive interpretation loses its specific support.

An alternative explanation is that the added features are ordinary indicators of task difficulty, response length, or stronger source access. Include those controls and replace revision features with independently sampled values from frozen development records in the same task stratum. A performance gain that survives only because one scorer sees better evidence would support information access, not a specialized monitoring mechanism.

The recent confidence-intervention study in language models strengthens the reason to distinguish a confidence representation from the policy that uses it. Its behavioral and activation experiments do not demonstrate calibrated multi-agent release or a literal equivalence to biological metacognition. [S16] No biological measurement is being reproduced in this proposed Encevra experiment.

9. Experiment proposal for preregistration

This is a proposed protocol, not an already registered or executed model study. The final task sampler, variants, sample size, tests, and decision criteria must be locked before calibration and test labels are accessed.

9.1 Dataset and truth contract

Build a reproducible task sampler with a fixed mixture of version-sensitive document questions, short multi-source decisions, and executable reasoning tasks. Each episode has explicit required answer elements and a frozen evidence packet. Include cases where revision repairs an error, cases where an initially correct answer can be damaged, shared misleading sources, and unresolved contradictions.

Separate the task-generating families used for development from those used for evaluation. For calibration and test, define the same sampling law over held-out families, and use one independently drawn episode per sampled family unit in the primary analysis. Do not count correlated paraphrases as extra iid observations. If iid family sampling cannot be defended, report the results as benchmark estimates and withhold the exact binomial certification claim. Group bootstrapping can describe variability but does not automatically repair that theorem's assumptions.

The proposed initial allocation is 400 development episodes, 400 tuning episodes, 2,000 calibration episodes, and 2,000 untouched test episodes. These are planning values, not a claim of sufficient power. Before accessing calibration labels, use development-based simulations to assess achievable precision and fix the final sample size. If accepted counts are too small, report “insufficient calibration” instead of loosening the risk target after seeing results.

Use executable checks where they actually define the task's answer. For source-grounded claims, use independent human assessment under a written rubric, blinded to release scores and method identity, followed by adjudication. Audit both accepted and abstained answers; labeling only accepted outputs conceals useful coverage and failure modes. A stronger LLM can assist triage, but its agreement is not ground truth. Record ambiguous labels and repeat analyses with unresolved material cases counted as errors. Calibration controls the selected label contract, not facts the labeling process systematically misses.

9.2 Controlled total compute

The primary score comparison runs the upstream collaborative pipeline once per episode and reuses its immutable trace for every candidate release rule. Generation, retrieval, checking, synthesis, and final formatting are therefore exactly identical across these arms. All candidate terminal features are extracted once as a common measured cost. Each rule performs only a bounded deterministic score calculation and threshold comparison. This isolates the value of information already present in the collaboration.

Log input, output, and exposed reasoning tokens; cached and uncached prefill; tool calls; retries; timeouts; retrieval and scoring costs; and wall time. The offline labeling and score-fitting costs are reported separately and amortized at stated deployment volumes. No account-usage target is an experimental budget.

If a later comparison introduces an additional model judge, its entire call must come out of the same fixed total inference budget as generation and verification. Compare it with allocating that budget to the existing pipeline. For cross-architecture claims, use the same model/backend and a measured prefill-plus-decode compute accounting method, alongside cost and latency. Equal output-token limits alone are not equal total compute. When hidden reasoning or provider internals prevent accounting, describe a token- or cost-matched comparison and do not claim exact FLOP matching.

9.3 Baselines and ablations

Fit all learned scores on the same development labels with matched fitting effort. Apply the same statistical calibration family to the score arms. A plain threshold such as “confidence at least 95%” can be reported as a heuristic diagnostic, but it receives no statistical guarantee by its name.

Predeclare twenty thresholds for each of the five scored arms and three feature ablations below: eight score definitions and a family of 160 candidate rules. If the final design has a different number, update M before calibration and record it. Do not test eight separate families at 5% and then select the winner while still claiming a 5% family guarantee. Release-all is an unfiltered descriptive reference; if it becomes eligible for certified selection, include it in the tested family too.

The three scored ablations remove revision features, remove source-origin features, and replace revision features using independent draws from stratum-matched development records. Freeze that replacement distribution before calibration and use independent per-episode randomness; do not permute calibration records together and then assume unchanged iid sampling. These three variants are included in the 160-rule family. A separate post-scoring rewrite stress test measures violation of the terminal contract; it receives no certificate from the pre-rewrite rule. Additional variants require an updated predeclared family or an independently budgeted calibration split.

An eventual architecture comparison should include a strong single agent with the same tools and evidence as well as the collaborative pipeline. This round's paired score experiment cannot establish that multiple agents are better than one.

9.4 Falsifiable hypotheses and decision criteria

H1: Terminal information has practical value. At a predeclared 2% selective-risk target and 5% calibration failure level, the terminal evidence-and-revision score improves held-out release coverage by at least five percentage points over the strongest simple score, while meeting the complete-answer rubric. Five points is a proposed practical threshold, not a literature-derived constant. Report a paired confidence interval for the coverage difference. To support the claimed practical gain, require the lower confidence limit to exceed five percentage points. A positive interval below that margin supports a smaller improvement only; an interval crossing zero is inconclusive. No certified rule means this hypothesis is unsupported at the proposed operating point.

H2: Revision state adds information beyond difficulty. Removing revision features reduces useful coverage or error discrimination after controlling for task category, source access, and answer length. If the ablation and within-stratum feature replacement leave outcomes unchanged within the preregistered precision, prefer the simpler score. This specifically tests the brain-inspired design idea.

H3: Answer binding catches a real evaluation hazard. Content changes inserted after scoring increase missed material changes when binding checks are absent. Measure detection recall and false rejection on a separately labeled mutation set. The current structural fixture verifies only literal recorded mismatches; it does not establish semantic detection performance.

H4: Pooled performance conceals predictable regressions. Revised and shared-source-conflict episodes have different risk-coverage behavior from easy unchanged episodes. Estimate these strata on held-out data. If an explicit subgroup guarantee is a requirement, calibrate it as such with multiplicity and sample size accounted for; a post hoc subgroup plot is not a certificate.

The mathematical procedure is not falsified by a single test-set proportion above 2%: finite test samples fluctuate, and the calibration guarantee allows a specified failure probability. Conversely, a favorable test proportion does not prove the assumptions. Diagnose whether an apparent violation reflects sampling variability, labels, leakage, version drift, or a genuine implementation error before interpreting it.

10. Metrics that expose regressions

For any score not intended as a probability, report ranking metrics directly; fit a probability mapping on development data before using proper probability scores. Fix ECE bins before the final evaluation. Do not rank systems solely by a bin-sensitive ECE difference. [S5-S7]

Report model and task strata even when the aggregate looks good, including unchanged/revised answers, source-family count, contradictory evidence, and missing provenance. For inferential comparisons, preregister the primary contrast and correct additional confirmatory comparisons. Exploratory plots remain labeled exploratory.

11. Alternative explanations and counterfactual checks

Better evidence, not better monitoring. If only the terminal scorer sees new sources, its advantage is confounded. The primary paired design gives all arms the same episode and charges common extraction costs. An evidence-access ablation can then separate information acquisition from scoring.

Difficulty sorting, not revision awareness. A changed-answer flag may simply identify hard tasks. Match or adjust for task type, observed pre-revision performance proxies, and length; compare against a score containing those controls without revision variables. Residual association is predictive evidence, not proof of a causal monitoring mechanism.

A conservative operating point, not improved ranking. Lower risk can come from fewer releases. Compare coverage at the same risk target, full risk-coverage curves, and correct useful releases per task. Never compare a 20%-coverage gate with a 90%-coverage baseline using accuracy alone.

A better judge, not a better system. If a scoring model also supplies calibration labels, agreement can masquerade as truth. Use independent labels and an audit that deliberately includes shared errors and misleading source documents. Grounded entailment and external factual truth must be separately named when both matter.

Selection on calibration luck. Trying feature sets, risk targets, or task filters until something certifies invalidates an unadjusted statement. Preserve failed variants and lock the family before labels. A fresh independent calibration sample is needed after a substantive redesign.

12. Expected failure modes and limits

Low separability: all scores may confidently favor a shared mistake. Calibration can still be statistically valid under its assumptions while returning no useful release rule. The protocol cannot manufacture information absent from the episode.

Label blindness: an evaluator may miss an omitted condition or treat a false grounding document as authoritative. An error bound on that evaluator's labels does not repair the rubric. Adjudication quality and label sensitivity analyses are part of the experiment.

Population drift: changing source freshness, user mix, task stakes, model revisions, tools, or memory can change the error distribution. The first protocol provides no arbitrary-shift guarantee. Detectable manifest changes invalidate the old configuration; undetectable drift requires external auditing. Online shift adaptation is separate research, with explicit feedback assumptions, rather than a promise added to this static gate. [S15]

Across-episode dependence: duplicated problems, common hidden source errors, and memory updates can make calibration observations dependent. A large raw count does not fix this. Predeclare the unit and sampling law; use a valid dependence-aware method if the iid model is unsuitable.

Unseen subgroups: low average risk can coexist with high risk in rare conditions. Distribution-free conditional inference has known limits; its regression impossibility results should not be transplanted as a blanket theorem about every discrete Encevra task. The practical warning is narrower: this paper's population statement does not supply arbitrary per-input or subgroup validity. [S10]

Overconservative family correction: Bonferroni can discard useful rules. That is a cost of this auditable starting point. A preregistered fixed sequence or other valid risk-control method may improve power, but its assumptions and target must be checked rather than inferred from the name.

State and text mismatch: the final prose may not preserve the claims that were checked. The terminal digest prevents accidental reuse only if the manifest, scorer, and application actually enforce it. The included validator is a fixture, not production middleware or a security proof.

Limited empirical reach: none of the synthetic numbers measures an LLM's calibration, validates the proposed feature set, or demonstrates improved collective intelligence. Published neighboring results are not replications. The source review is selective, and exact budget matching outside the paired replay design remains an implementation obligation.

13. What Encevra should carry forward

Adopt a single named risk contract for the first pilot: any material error or required-content failure in a released complete answer, on the preregistered episode population. Preserve wrong-release rate and claim-level factuality as separate diagnostics. Freeze the full adaptive procedure before collecting calibration outcomes; do not equate a frozen policy with a non-adaptive episode.

Build the terminal record before building a complicated confidence model. It must connect the exact answer, evidence, revision history, external labels, and complete cost accounting. Try a simple inherited score first. The proposed revision-aware score earns its place only if the paired experiment shows useful additional coverage without hiding damage or omission.

Do not present an error-control method as proof that collaboration is valuable. That claim requires a separate comparison with a strong single-agent system under the same evidence access and total compute. Round 11's action-value identifiability problem also remains unresolved.

14. Next-round handoff

The next empirical round should implement the full episode logger and run the locked pilot, beginning with a small label-and-accounting audit before buying thousands of model runs. Its deliverables are real terminal traces, independent labels for accepted and abstained answers, a frozen manifest, measured accounting completeness, and the paired release-rule comparison.

The go/no-go question is concrete: does final evidence and revision state improve useful coverage enough to justify maintaining it as a release signal? If not, simplify the gate and return to the upstream source of errors. If yes, test transfer to one separately defined population with fresh calibration before claiming broader applicability. Do not use the next round to restate calibration theory or add an untested brain metaphor.

15. Reproducibility and audit notes

The companion package contains the manuscript, a PDF rendering, diagnostics.py, exact results, and a compact README. Run python3 diagnostics.py with SciPy installed to reproduce the numerical tables. Synthetic fixtures are deterministic. Assertions check the selected threshold, the zero-error formula, the zero-accepted convention, and the ten structural scenarios.

The inspected recent PDFs were obtained directly from their primary repositories when browser extraction failed. Text extraction does not mean every appendix theorem was independently proved or every experiment reproduced. Source hashes recorded during inspection were f58edd5b543faea1ddb84d1dc66f256856817f8e44ac42cdfac82895cd803db4 for the UAI fact-verification PDF and f1c62d1da8754dc0a53b5c4e86635f61962101dcbbf3d2305bc5b165c6e39fae for Consilience. External papers are linked, not bundled as Encevra's own work.

16. Sources and inspection ledger

All links were checked or retrieved on 4 October 2026. Dates below distinguish publication metadata from inspected preprint versions. Where rendered HTML displayed a date inconsistent with its arXiv version metadata, the version metadata is used. “Inspected” names the sections actually used, not a claim to have audited the whole paper.

[A1] Encevra research archive. Project. Research overview, combined rounds index, Round 9, Round 10, Round 11. Inspected current project source via the Lovable connector: overview, index, full texts of 9/10/11, confidence passages in 4-6/8, and publications registry. Round 9: 30 September; Round 10: 1 October; Round 11 expanded revision: 3 October 2026. Supports numbering and program continuity, not external validation of Encevra performance.

[S1] Ran El-Yaniv and Yair Wiener. On the Foundations of Noise-free Selective Classification. JMLR 11, 1605-1641, May 2010. Record; PDF. Inspected definitions 1-2 and the risk-coverage formulation. Supports terminology; its noise-free learning results are not assumed for LLMs.

[S2] Anastasios N. Angelopoulos, Stephen Bates, Emmanuel J. Candès, Michael I. Jordan, and Lihua Lei. Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control. First submitted 3 October 2021; inspected v5, 29 September 2022. Record; full text. Inspected sections 1.1, 2.1-2.3, and 3.2: risk definition, testing family, and selective-classification example. Supplies the established calibration framework; the Encevra integration and diagnostics are not new statistical theorems.

[S3] Anastasios N. Angelopoulos, Stephen Bates, Adam Fisch, Lihua Lei, and Tal Schuster. Conformal Risk Control. First submitted 4 August 2022; inspected v4, 13 June 2025. Record; full text. Inspected introduction, algorithm, theorem conditions, and non-monotone-loss discussion. Supports the distinction between expected-loss control and the high-probability risk statement used here.

[S4] Anastasios N. Angelopoulos. Conformal Risk Control for Non-Monotonic Losses. Preprint v1, 23 February 2026. Record; full text. Inspected introduction, setup, stability definition, and theorem 1. Supports the existence of newer non-monotone extensions; their conditions are not established for Encevra.

[S5] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On Calibration of Modern Neural Networks. ICML 2017, PMLR 70, 1321-1330; conference 6-11 August 2017. Record; PDF. Inspected calibration definitions and temperature-scaling results/discussion. Supports a simple calibrator baseline; it does not establish validity for collaborative language generation.

[S6] Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D. Manning. Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback. EMNLP, 5433-5442, 6-10 December 2023. Record; PDF. Inspected methods, metric definitions, results framing, and limitations. Supports verbal-confidence baselines and the short-answer factual-recall scope caveat.

[S7] Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs. ICLR 2024; preprint first submitted 22 June 2023, revised 17 March 2024. Record; conference PDF. Inspected framework, prompting/aggregation definitions, evaluation metrics, and section 5.1. Supports task-dependent overconfidence and separate calibration/discrimination evaluation.

[S8] Adam Kostka and Jarosław A. Chudziak. Controlling Uncertainty and Hallucination Risk in Multi-Agent Fact Verification. UAI 2026, PMLR 337, 3143-3161; volume published 6 August, conference 17-21 August 2026. Record; primary PDF. Downloaded and inspected sections 3-5, including loss, labels, calibration, and experimental setup. Closest fact-filtering comparison; no result replicated here.

[S9] Abhijith Babu, Ramneet Kaur, Vishal Pramanik, Olivera Kotevska, Nathaniel D. Bastian, Susmit Jha, Sunny Raj, Yanzhao Wu, Sumit Kumar Jha, and Anirban Roy. Consilience: Conformally Calibrated Communication Control for Hidden-Profile Multi-Agent Reasoning. Preprint v1, 20 August 2026. Record; PDF. Downloaded and inspected formulation, state/control definitions, proposition 1, and appendix discussion of guarantee scope and round-specific exchangeability. Supports separating communication regret from terminal correctness.

[S10] Rina Foygel Barber, Emmanuel J. Candès, Aaditya Ramdas, and Ryan J. Tibshirani. The limits of distribution-free conditional predictive inference. First submitted 12 March 2019; inspected preprint v2, 15 April 2020. Record; PDF. Inspected introduction and definitions distinguishing marginal, conditional, and approximate conditional coverage. Used as a scope warning; no direct transfer of its regression theorem to Encevra is asserted.

[S11] Stephen M. Fleming and Nathaniel D. Daw. Self-Evaluation of Decision-Making: A General Bayesian Framework for Metacognitive Computation. Psychological Review 124(1), 91-114, January 2017. DOI: 10.1037/rev0000045. Author-hosted PDF; publication record. Inspected first-/second-order model framing and simulation discussions of confidence/error detection. Supplies a computational analogy, not proof of a neural implementation or an AI advantage.

[S12] NIST/SEMATECH e-Handbook of Statistical Methods. 7.2.4.1. Confidence intervals. Technical reference. Undated page; accessed 4 October 2026. Inspected exact binomial interval construction and tail equations. Supports the bound calculation; the paper explicitly uses a one-sided, family-adjusted endpoint.

[S13] Yasin Abbasi-Yadkori, Ilja Kuzborskij, David Stutz, András György, Adam Fisch, Arnaud Doucet, Iuliya Beloshapka, Wei-Hung Weng, Yao-Yuan Yang, Csaba Szepesvári, Ali Taylan Cemgil, and Nenad Tomasev. Mitigating LLM Hallucinations via Conformal Abstention. Inspected arXiv v1; record lists submission 4 April 2024. Record; full text. Inspected introduction and section 2 loss and abstention definitions. Supports existing whole-answer abstention; its loss denominator must not be confused with selective risk.

[S14] Yu Gu, Zijun Yu, Vahid Partovi Nia, and Masoud Asgharian. Pause and Reflect: Conformal Aggregation for Chain-of-Thought Reasoning. Preprint v1, 13 May 2026. Record; full text. Inspected aggregation framing, section 3.4/theorem 2, and limitations. Supports the comparison to unconditional confident-error control over aggregated paths; no numerical result is imported as Encevra evidence.

[S15] Varun Kotte. When Can Conformal Risk Control Certify LLM Outputs? Bounds, Impossibility, and Adaptation for Structured Generation. Inspected preprint v2, 7 September 2026. Record; retrieved PDF. Downloaded and inspected setup, threshold testing, and sections 3.6-3.8. Relevant negative evidence on score fusion and transfer. The broader theoretical hierarchy and online-monitoring claims were not independently audited and are not adopted by this protocol.

[S16] Dharshan Kumaran, Nathaniel Daw, Simon Osindero, Petar Veličković, and Viorica Patraucean. Causal evidence that language models use confidence to drive behaviour. Nature Machine Intelligence 8, 1411-1425, published 7 September 2026. Publisher full text. Retrieved publisher HTML directly after browser extraction failed; inspected abstract, phase framing, model/activation-intervention description, and discussion. Supports a functional confidence-versus-policy distinction, not calibrated release guarantees or literal brain copying.