Encevra Research Round 14

When the Judges Are Uncertain: Measuring Revision Gains Without an Infallible Evaluator

5 October 2026 · Research paper · Version 1.0

Evidence status: Primary-source synthesis, original mathematical examples, and 30 executed offline diagnostic checks. No new Encevra model benchmark, model inference, expert annotation, production change, or deployment was performed in this round. The live experiment below is a proposal for preregistration, not a registered or completed study.

Abstract

An answer can improve while its evaluation score falls, or deteriorate while its score rises. Encevra's preceding round showed why saved final scores cannot establish whether a correction helped and proposed independently labeled revision experiments. This round examines the next measurement problem: the labels themselves can be missing, ambiguous, or systematically biased toward one experimental arm. We distinguish agreement, correctness under a fixed rubric, and preference within a specified reviewer population. We propose preserving allowed joint labels for initial and revised answers, reporting bounds on paired improvement and repair/damage, protecting a random audit sample, and estimating judge error on the comparison actually being claimed. Exact constructed examples show that 97.5% snapshot-label accuracy can coexist with a reversed system comparison; a pooled correction can repair the overall mean while preserving the wrong ordering; and identical-artifact constraints can narrow an uncertainty interval without resolving either artifact's correctness. A prospective study compares evaluation methods on the same frozen outputs with matched annotation and judge budgets. The contribution is a measurement protocol and reference diagnostics, not evidence that Noetis improves reasoning or that an evaluator has been calibrated.

1. Research question and continuity

Research question: When collaborative revision appears to improve answers, under what evaluation design does that conclusion survive plausible uncertainty and differential bias in the labels used to measure it?

The inspected archive contains completed Rounds 11, 12, and 13. Its next sequential entry is therefore Round 14, correcting the original request's Round 11 designation. The archive audit used project commit cae367dfbf71dadb0bd6aa5d3ecb100a7e3b09a2 and the current Research overview, rounds index, publications registry, and notebook records. The full Round 13 manuscript, including its evidence-lineage extension, was read. Rounds 9 and 10 had been read in full earlier in this research session; their current agenda and hypothesis sections were checked again. The expanded Round 11 and complete Round 12 provide the intervening handoff.

Round 13 also covers missing challenges, circular evidence, source dependence, and distinct kinds of agent independence. Repeating those topics would not close the immediate gap. This paper instead asks whether the proposed measurement process can support its intended conclusion even after the trace is complete. It does not claim to have rerun Round 13's project audit.

The novelty is specific to Encevra's program. Partial identification, measurement-error correction, prediction-powered inference, and disagreement-aware annotation already exist. The proposed contribution combines their lessons at a paired revision boundary, where one shared initial answer, multiple continuations, and unresolved judgments must remain connected. It is not a claim to invent a new general estimator.

2. Scope and the target of evaluation

The primary target is a difference in whole-answer task success between two frozen continuation policies on a declared task distribution. Success requires all mandatory task content, no material false or unsupported additions, and appropriate treatment of insufficient evidence. Requirement-level repair and damage are secondary explanations of that endpoint. A preference for a polished answer is a different target.

For the finite collection actually generated, let Y_iB and Y_iC denote binary success of the baseline and candidate terminal answers under the same rubric. The finite-cohort contrast is:

D_i = Y_iC - Y_iB
Delta_N = (1/N) sum_i D_i

This contrast concerns those realized answers. A claim about future tasks or a policy's average stochastic behavior additionally needs task sampling, model-repeat handling, and uncertainty over independent source families. Auditing every stored output removes audit-sampling uncertainty; it does not remove future-task uncertainty.

Three label targets must remain distinct:

Arithmetic disagreement should prompt recomputation. A disagreement about whether an unstated commercial objective matters may instead reveal an underspecified task. Calling both phenomena “noise” hides the action needed to resolve them.

Excluded are a new answer-generation architecture, an agent-confidence score, a replacement release-risk guarantee, a learned verification router, hidden chain-of-thought collection, and a new production benchmark result. This round does not authorize promotion of any reusable procedure or product policy.

3. Literature map and competing positions

Correct the measurement error. Lee et al. analyze binary judge sensitivity and specificity, including uncertainty from the calibration sample. Their shift result assumes invariant judge behavior conditional on the true label. It does not permit arbitrary changes in response style or arm-specific error rates. They explicitly treat human disagreement as absent or resolved by a predefined protocol. [S1]

Correct a surrogate with representative labels. Prediction-powered inference uses trusted measurements to correct the aggregate error of model predictions. Chen et al. compare this family with confusion-matrix correction and derive efficiency results under their same-distribution, randomly labeled setup. The apparent disagreement with Lee et al. concerns different transport assumptions, not a universal winner. Sampling and the definition of the reference label remain decisive. [S2, S3]

Control the annotation design. Egami et al. make labeling probabilities known and positive by design, allowing imperfect surrogate labels to support downstream inference under stated conditions. This is the reason to retain a protected random audit or a fully recorded probability sample. Reviewing only suspicious examples does not supply a representative residual correction. [S4]

Model evaluators instead of collapsing their votes. Dubois et al. use Bayesian generalized linear models to represent grader, item, and format effects. Such models can expose structured disagreement, but an estimated counterfactual with a bias term removed is not an observed causal intervention. STABLEVAL retains posterior label uncertainty and optimizes ranking stability; its latent-label model assumes conditional independence of annotators and adequate identification. A stable ranking can still reflect a shared mistake. [S5, S6]

Sometimes disagreement belongs to the target. Pavlick and Kwiatkowski find persistent disagreement in natural-language inference, including after collecting more ratings and varying context. That supports retaining judgment distributions where appropriate. It does not make a wrong numerical calculation a legitimate alternative preference. [S7]

Make a subset of ground truth executable. SALT constructs long-form tasks with deterministic target sequences and explicitly chosen evaluation units. It offers a useful anchor for evaluating label methods without an LLM judge. Its exact-string setting does not automatically cover free-form decisions with several equally valid phrasings or unresolved sources. [S8]

Debiasing one feature is insufficient. The September 2026 DIAL preprint separates position effects from alignment with human preferences. Its theory has an explicit comparison model, design conditions, and fixed-weight inference scope. Swapping answer order is a useful control, not a correctness guarantee. A separate controlled study by Sun et al. finds source-label effects on trust judgments; this motivates blinding authorship, with no assumption that its effect size transfers to Noetis. [S9, S12]

Small studies need particular care. The September 2026 evalstats preprint studies judge corrections and small-sample testing through extensive simulations, emphasizing representative human subsets and limits of familiar interval methods. It does not validate every future clustered Encevra design. Lail's September preprint separates uncertainty about reviewer opinions from disagreement within the reviewer population; on its real-data study, a simple fewest-labels-first rule was not beaten. This makes that simple rule an essential comparator, not an optional weak baseline. [S10, S11]

These positions are compatible when their estimands and assumptions are explicit. An efficient estimator relative to a fallible reference is still relative to that reference. A more stable ranking need not identify factual correctness. An executable anchor can validate a narrow class of judgments while leaving semantic assessment open.

4. Claims and evidence

5. Worked example: an accurate judge reverses the comparison

Consider 100 invented paired tasks. The baseline succeeds on the first 80; the candidate succeeds on the first 78. The true difference is therefore −2 percentage points. A judge labels the baseline perfectly but incorrectly accepts five candidate failures, yielding 83 candidate successes and a measured difference of +3 points.

Across the 200 snapshot labels, the judge is correct on 195: 97.5% accuracy. This high agreement does not protect a small comparison from concentrated errors. The example does not require the judge to mislabel most answers or to have poor aggregate accuracy.

Suppose the analyst knows the pooled confusion rates exactly: sensitivity is one, and specificity is 37/42. Applying the same classical correction to both arms gives:

corrected success = (judged success + specificity - 1)
                    / (sensitivity + specificity - 1)

The corrected overall mean is exactly 79%, the true pooled mean. Yet the corrected difference is +3.4054 points, still the wrong direction. There is no contradiction: pooled correction recovers one aggregate under the constructed mixture, while the arms have different conditional error rates. Baseline specificity is one; candidate specificity is 17/22.

More generally, write b_a = E[J_a − Y_a] for arm a. The judge contrast equals the target contrast plus b_C − b_B. A large common bias might cancel; a small differential bias can dominate the claimed gain. The required audit target is the paired residual, not only each judge's pooled agreement score.

This is a stress case, not a claim that any particular current evaluator behaves this way. It motivates checking treatment-by-judge interactions, candidate presentation, and the initial-versus-terminal boundary. It also shows why blindly increasing the number of judge-labeled tasks can sharpen a wrong conclusion.

6. Proposed protocol: preserve uncertainty at the comparison boundary

The proposed protocol has six stages. It uses Round 13's immutable answer snapshots and source bindings but adds an evaluation ledger rather than another reasoning role.

1. Freeze the target. Record the whole-answer endpoint, requirement rubric, task date, admissible source versions, allowed uncertainty, output object, and exclusion rules before judging. Distinguish inability to answer correctly from a missing evaluation label. A failed generation is a task failure under the primary endpoint; a reviewer who has not finished is missing measurement.

2. Preserve individual assessments. Store each reviewer's label, supporting source or executable check, uncertainty reason, review duration, and rubric version. The initial assessment must survive adjudication. Do not replace it with the final consensus. Hide model identity, branch identity, checker recommendation, and prior scores when feasible. Randomize snapshot order before reviewers see initial/terminal pairings.

3. Classify unresolved judgments. Use separate states for missing review, missing evidence, conflicting evidence, unclear rubric, and genuine reviewer-population variation. None automatically means correct, incorrect, or nonmaterial. A supported conditional answer can be successful on an underdetermined task when the frozen rubric requires that qualification.

4. Preserve joint possibilities. For each task, retain an allowed set of triples (Y_initial, Y_B, Y_C) consistent with the currently accepted evaluation evidence. The shared initial answer has one label variable, not independent copies in each arm. If identical answer bytes occur under the same task, sources, time, rubric, and endpoint, the corresponding correctness variables must agree. A matching digest under different conditions is insufficient. Semantic equivalence requires an additional fallible judgment and is not enforced automatically.

5. Protect an audit sample. Select a random core from the frozen cohort before adjudication outcomes are known. Keep additional diagnostic selections in a separate analysis track unless their inclusion probabilities and estimator are explicitly supported. Include judge agreements, successful-looking answers, and abstentions in the eligible population. Maintain sampling seeds and selections as experimental records.

6. Report the decision's sensitivity. Report the naive contrast, representative-audit correction, unresolved-label bounds, and a declared error-budget sensitivity for provisionally resolved labels. Give a promotion, rejection, or inconclusive decision relative to a prespecified practical margin, with uncertainty that matches the claimed population. Do not silently resolve the most influential disputed cases in favor of the candidate.

The ledger is a record of evaluation evidence, not an automatic truth extractor. If two reviewers agree because both overlook an exception, their agreement can produce the same wrong singleton set. Random blind re-audit of agreements, executable anchors, and sensitivity to residual error remain necessary.

7. Bounds on improvement, repair, and damage

Let P_i be the allowed terminal-label pairs projected from a task's joint triple set. With no additional constraints across tasks, the tight lower and upper bounds on the finite-cohort contrast are:

L_N = (1/N) sum_i min_{(b,c) in P_i} (c-b)
U_N = (1/N) sum_i max_{(b,c) in P_i} (c-b)

Each endpoint is attained by choosing the corresponding permitted completion per task. This elementary construction follows the partial-identification principle of retaining all outcomes consistent with the evidence rather than filling gaps with an unearned point estimate. [S15] If tasks share constraints on labels, the minimization must enforce those constraints jointly; the simple formula need not then be sharp.

The interval is an identification interval for the finite cohort, conditional on valid allowed sets. It is not automatically a 95% confidence interval. Sampling variability, reviewer mistakes outside the sets, and future-task generalization are different uncertainties.

Repair and damage additionally depend on the initial label. For a fixed completion and an arm a:

repair_a = count(Y_initial=0 and Y_a=1) / count(Y_initial=0)
damage_a = count(Y_initial=1 and Y_a=0) / count(Y_initial=1)

Compute extrema over permitted joint completions. Do not minimize a numerator and maximize an unrelated denominator, and do not multiply separately estimated marginal correctness probabilities unless the required conditional independence is justified. If an eligible denominator can be zero, retain an explicit undefined possibility. Different componentwise extrema may arise from different completions and should not be presented as a jointly achievable performance profile.

Round 13 already established that aggregate scores do not identify repair and damage. The additional concern here is that uncertain marginal labels cannot replace the missing joint information. For example, initial and terminal correctness of 0.8 are compatible with no changes at all. They are also compatible with repairing all 20 initially wrong items while damaging 20 of the 80 initially correct items. The conditional repair rates are respectively 0% and 100%; damage rates are 0% and 25%. Multiplying the same marginals as independent would instead imply 80% repair and 20% damage. This demonstrates an identification failure, not a new observation of model behavior.

For provisionally resolved labels, separately report the smallest number of snapshot-label flips that changes the conclusion. In Section 5, three adverse flips erase the observed +3-point gain, and four reverse it. One arbitrary task-pair replacement can change the contrast by two units; one binary snapshot-label flip changes it by at most one. These are different error-budget units and must not be interchanged.

8. Worked example: uncertainty can cancel without becoming truth

In a second invented 100-task cohort, 60 pairs are both successful, 10 both unsuccessful, 15 favor the candidate, and five favor the baseline. The remaining ten pairs are unresolved.

Dropping the unresolved pairs reports 10/90 = 11.1111 points of improvement on a selected subset. With no further information, all ten unresolved contrasts range from −1 to +1, so the whole-cohort interval is [0, 20] percentage points. Treating missing labels as ties gives 10 points but conceals the uncertainty.

Now suppose six unresolved pairs contain exactly the same answer artifact under the same evaluation conditions. Their correctness remains unknown, but both sides must receive the same correctness label. Each contributes zero to the contrast whether it is right or wrong. The other four pairs remain unrestricted. The interval becomes [6, 14] points, a 60% reduction in width.

The narrower interval does not contain six newly verified correct answers. It contains six comparisons whose difference is known. This is why a comparison ledger should preserve joint information instead of flattening every answer into a separate probability.

This example supports a software invariant and a possible efficiency mechanism. It does not establish a positive population effect or predict how often identical artifacts will occur. The experiment must measure useful interval reduction on naturally occurring outputs, not inflate it by manufacturing easy duplicate pairs.

9. A protected random audit and a conservative inference boundary

Suppose the frozen judge supplies a contrast prediction q_i between −1 and 1 for every task. Select m of the N tasks uniformly without replacement, retaining both arms for each selected task. With resolved reference contrasts D_i, use the finite-population difference estimator:

Delta_hat = mean_all(q_i) + mean_audit(D_i - q_i)

Conditional on the fixed cohort, reference labels, and judge outputs, the estimator is exactly unbiased over random audit selection. The reason is simple: the audit mean residual has the population mean residual as its expectation. No equal-error assumption across experimental arms is needed. Its design variance is (1 − m/N) S_residual² / m, where S_residual² is the finite-population variance with denominator N−1. The reference script enumerates all 70 size-four audits of an eight-task example and verifies both identities exactly. This is a simple design-based construction related to the surrogate-correction literature, not a new general PPI theorem. [S3, S4]

If audited labels are unresolved, each task yields a lower and upper possible residual. A conservative audit-selection confidence envelope is the mean prediction plus the audit's lower/upper mean residual, expanded on both sides by:

h(m, alpha) = sqrt(8 log(2/alpha) / m)

Intersect the result with [-1,1]. The bound follows from a residual range of [-2,2] and the Hoeffding inequality for sampling without replacement. [S14] For complete auditing, m=N, directly use the full-cohort identification interval with no audit-sampling radius. Empty intersections, if they occur, are reported rather than silently repaired.

The coverage statement is conditional: all allowed sets must contain the reference labels, the cohort and predictions must be frozen, the audit must be genuinely random, and the audit size must not be chosen by repeatedly checking for significance. It says nothing about an erroneous reference standard or future task sampling. If joint label-set coverage itself is only 1−beta, an additional union-bound allowance is needed; ordinary reviewer agreement does not supply beta.

At 95% confidence, the simple range-only half-width is 1.1089 with 24 audits and 0.5544 with 96. It is deliberately conservative and often uninformative. These values are not minimum required sample sizes. They show why this fallback should not be marketed as a way to certify a two-point benefit from a tiny pilot. Variance-sensitive finite-population bounds or validated PPI methods may be more useful, but their full assumptions and tuning uncertainty must be retained. [S2, S10, S14]

A diagnostic selection is not interchangeable with that random core. In Section 5's example, the judge marks only three tasks as favoring the candidate. Auditing just those and treating their residual mean as representative yields −97 points, while the target is −2. Two hidden candidate errors occur on judge-agreement pairs. The problem is selection bias, not a failure of the random-audit estimator. Probability sampling can deliberately oversample difficult strata, but a suitable weighted estimator and positive coverage of the population must be designed first.

10. One bounded brain-inspired design idea

Yu and Dayan's computational account distinguishes expected unreliability within a context from unexpected evidence of a changed context. Their proposed neuromodulatory interpretation is a hypothesis in computational neuroscience. It is not a demonstration that an LLM, evaluator, or multi-agent system implements those biological mechanisms. [S13]

The useful engineering analogy is to distinguish uncertainty within a fixed evaluation specification from evidence that the specification or evaluator behavior has changed. The former may warrant another independent assessment. The latter may require freezing the comparison, revising the rubric or judge version, and restarting the affected measurement on a clearly marked development set.

This distinction is not identical to the epistemic/aleatoric decomposition in [S11], and neither is automatically observable. A testable implementation would retain explicit uncertainty-reason fields plus randomized invariant controls. Compare it with a single undifferentiated uncertainty flag at the same reviewer-time budget. Measure whether it identifies rubric/version faults earlier without increasing unjustified suspensions or merely classifying difficult cases as permanently ambiguous. If the added fields do not improve those outcomes, omit the analogy-driven machinery.

11. Falsifiable hypotheses

The following are prospective, task-specific hypotheses. Proposed thresholds are design choices to freeze after feasibility work, not observed results or universal standards.

The secondary analogy ablation in Section 10 is optional and should be preregistered separately if funded. Passing a structural identity check is a software acceptance requirement, not evidence that one of these empirical hypotheses is true.

12. Proposed experiment with fixed generation and evaluation budgets

Stage A: finish the Round 13 collection gates. Do not claim to analyze a pilot that has not run. Its 24-task smoke test and subsequent 96-task development phase remain the starting plan. Keep initial answers, both continuation arms, complete attempt accounting, source bindings, and externally visible prompts. If that collection is unavailable, run executable evaluation fixtures and label-protocol development only, as in this round.

Stage B: freeze a paired evaluation corpus. Use the ordinary-revision and receipt-revision arms, with the shared initial answer. For this label-method pilot, select one of each arm's repeated continuations using a seed fixed before labels or scores are seen. The pilot then has 120 task triples, or 360 snapshots, if all planned tasks complete. Retain failed generations. This target is the selected realized continuation corpus; a separate analysis can average all repeats within task to estimate policy behavior.

Before label collection, specify required-item truth where executable, sources and tolerances, completeness conditions, material added-claim rules, and uncertainty allowances. Use source-family splits between development and later confirmation. Add a separate sentinel set with checked arithmetic, negation, scope changes, unchanged-answer controls, label swaps, order swaps, and irrelevant formatting. Sentinels diagnose evaluation; they do not change the primary task-distribution denominator.

Stage C: construct the reference ledger. Two qualified reviewers independently assess all 360 snapshots, giving 720 first-pass snapshot assessments. This count describes a proposed workload, not labor already performed. Reveal pairing only after those records are fixed. A third reviewer adjudicates disagreements and a prespecified random subset of agreements, with a fixed time cap and unresolved outcomes retained. Estimate minutes per assessment in the smoke test before committing the development phase's total labor budget. One reviewer's repeated judgments do not count as independent experts.

The resulting ledger is a strong reference, not infallible truth. For executable tasks, compare it to independently verified oracles. For semantic tasks, preserve original labels, disagreement reasons, adjudication, and uncertainty sets. A rubric change creates a new evaluation version and a fresh analysis; it cannot be used opportunistically to improve the preferred arm.

Stage D: compare evaluation methods by masked replay. All methods receive exactly the same frozen generated answers and source access. Predetermine allowed judge calls and expose the same audit budget to each method. The full reference ledger is hidden except for the labels a method is permitted to use; it is used afterward to assess decisions. This is an evaluation-method comparison, not another model-generation contest.

Asymptotic methods such as PPI++ are secondary comparators only after their assumptions, fitting split, and small-sample behavior are validated for the proposed design. Do not add a fitted correction that uses the hidden evaluation labels and then call the comparison held out.

Controlled total compute and cost. Answer-generation compute is identical because every method reuses the same answers. For the eventual comparison of continuation policies, retain Round 13's identical source access and post-fork caps, charge the common prefix to each hypothetical policy, and report the actual branched collection bill separately. No one is credited with free evidence collection.

Evaluation has its own budget: judge input/output/reasoning usage without double counting, failures, retries, tool work, elapsed time, and human minutes. Count calibration, prompt tuning, order swaps, and referee calls. A method using fewer resources can retain that saving; do not pad it with useless calls. Compare cost ceilings and cost-quality frontiers rather than asserting equality from token counts across unlike models. Unknown costs remain unknown, and missing usage blocks a strong matched-cost claim.

For masked replay, audit sizes of 24, 48, and 96 task triples are a planning grid, not three opportunities to stop when significance appears. Freeze one primary budget for confirmation or allocate error across the declared family. Source families are the units for generalization uncertainty; requirements and repeated answers within a family are not independent new tasks.

13. Analysis, regressions, and decision rules

Report the measurement funnel first: attempted tasks, captured snapshots, sampled audits, completed reviews, unresolved cases by reason, adjudication time, oracle discrepancies, and deviations from blinding or sampling. Then report the answer outcomes. A beautiful effect estimate with unknown label coverage is not an adequate result.

For simulation development, use null effects and effects above and below the practical margin, class-conditional and arm-dependent judge errors, clustered failures, missingness related to difficulty, and shared reviewer mistakes. Prespecify a 5% false-promotion target and an informativeness floor: at effects at least two points beyond the practical margin, correct decisive outcomes must not fall more than five points below the strongest equal-budget baseline. Evaluate both criteria with uncertainty across independent simulation replications. These are proposed gates, not gates passed in this round.

For the later semantic corpus, a “wrong decision” means disagreement with the declared reference analysis; it is not automatically disagreement with objective truth. Run sensitivity to mistakes in that reference, including mistakes shared by both primary reviewers. For task-family sampling, prefer a method validated for the actual number and dependence structure of families. A generic bootstrap over individual requirements is invalid for that target. The small pilot should primarily establish measurement feasibility and effect variability; it is not sized to settle Round 11's small improvement and damage margins.

A future promotion requires both the prespecified useful-improvement condition and the harm condition. A result that fails to establish benefit is not evidence of equivalence. A label-sensitive result should be called inconclusive until more informative evaluation evidence is obtained. Release-risk calibration from Round 12 remains a separate subsequent stage on untouched data and a frozen policy.

14. Executed results and reproducibility

The delivered standard-library script, diagnostics.py, ran successfully and wrote diagnostic_results.json. It executes 30 checks, including seven exhaustive cross-checks of the label-flip dynamic program against all completions of a small example. These are tests of stated calculations and selected input behavior, not 30 model-performance experiments.

The code preserves zero-denominator flags and rejects selected malformed inputs. It does not perform semantic annotation, authenticate evidence, verify that an audit was randomly selected, implement every cross-task dependency, or provide a production evaluation service. Its dynamic programs are intended for small diagnostic examples; scaling and schema enforcement require separate engineering work.

To reproduce, use Python 3.10 or later and run python3 diagnostics.py from the supplied package. No model keys, external data, network, or third-party Python packages are required. The JSON distinguishes exact fractions from their decimal display. The package includes the manuscript, source ledger, executed results, and a manifest of its own files. External source PDFs and private project files are not redistributed.

15. Alternative explanations and failure modes

A real improvement can coincide with evaluator bias. Finding a format effect does not prove that the candidate is worse. Estimate the task outcome and bias separately. Length normalization can erase useful evidence as well as remove superficial cues, so compare minimally altered controls and inspect their semantic equivalence.

Better instructions can change the target. A rubric clarification may improve agreement by deciding a previously unspecified issue. That is valuable development, but not a retroactive validation of an old benchmark. Preserve both versions and test the new one on fresh cases.

Consensus can be confidently wrong. Two humans and several LLMs may share assumptions, access the same incomplete source, or follow the same misleading hint. Reviewer count alone does not identify independence. Random agreement audits and executable sentinels help expose some shared errors; they do not bound every one.

The reference can leak. A reviewer may infer the treatment from answer structure, or a judge may see adjudicator notes. Record suspected unblinding; exclude leaked gold from model inputs; retain deviations. Removing all informative structure merely to make arms look alike would change the product endpoint.

Partial identification can become permanent indecision. Broad label sets protect against unsupported claims but may be useless. Report decisive correct outcomes and cost together with interval coverage. A proposed method loses practical value if a simpler random human audit gives equally reliable decisions faster.

Coupling can be overclaimed. Identical text under different task dates can have different correctness. Similar phrasing is not proof of identical meaning. Erroneous equality constraints create narrow, wrong bounds. The default should preserve uncertainty when identity conditions cannot be established.

Audit allocation can compromise inference. A protected random sample stops being protected if investigators replace difficult cases, adapt its size after seeing results, or use all calibration labels to tune the judge and evaluate it. Separate exploratory diagnosis, fitting, and final evaluation.

Repair-rate bounds can mislead through denominators. Ambiguity about the initial answer changes who is eligible for repair or damage. Retain the joint possibilities and report undefined rates. A small known-error subset is not an adequate substitute for all initially wrong requirements.

Simulation success can be circular. A latent-label method tested only on data generated from its own model can look reliable by construction. Include conditional dependence, asymmetric missingness, rubric drift, and reference errors outside that model. Even successful stress tests do not establish performance on actual Encevra tasks.

16. Limitations and next-round handoff

This is a focused review as of 5 October 2026, not a systematic survey. Recent preprints have not been independently reproduced here. Full primary text was inspected for the cited methods at the depths recorded below; theorem proofs and experiments were not universally reimplemented. The archive review establishes continuity but supplies no newly collected answer transitions or external labels.

The executed examples demonstrate what can go wrong and verify a small reference implementation. They cannot establish the prevalence of differential judge bias, the accuracy of human adjudication, the performance of the proposed protocol, or a useful evaluation-budget optimum. No new Encevra benchmark was run. No statement in this paper certifies a Noetis release or establishes a multi-agent advantage.

The next round should collect evidence rather than add another evaluation architecture. First complete the trace and accounting gates from Round 13. Then run its 24-task smoke test with this round's preserved reviewer records, shared initial labels, unresolved states, and random audit selections. Report actual review times and failure categories before funding the larger phase. If no qualified independent labels can be obtained, continue with closed, executable tasks and state the restricted scope.

The handoff artifact should contain frozen answer triples; source/rubric bindings; individual and adjudicated labels; allowed joint sets with reasons; audit selections and probabilities; all judge calls and usage; and separate records of generation failure, missing measurement, and genuine task ambiguity. The first useful result may be that the intended endpoint cannot yet be labeled consistently enough to compare small gains. That is a measurement finding, not a reason to manufacture consensus.

Only after an effect survives the chosen evaluation assumptions should it inform the Round 11 controller or the Round 9 procedure store. Only after the policy is frozen and the error reference is defensible should Round 12's separate release-calibration study proceed. Round 14 makes those dependencies explicit.

17. Full sources and inspection ledger

All links were checked or their linked primary PDFs retrieved on 5 October 2026. Publication dates below refer to the cited edition or venue, not search-engine crawl dates. “Inspected” describes the actual access depth; it does not mean all proofs or reported experiments were independently reproduced.

[S1] Chungpa Lee, Thomas Zeng, Jongwon Jeong, Jy-Yong Sohn, and Kangwook Lee. How to Correctly Report LLM-as-a-Judge Evaluations. ICML 2026, PMLR 306:65231–65252; conference 6–11 July 2026. Inspected the official record and PDF setup, correction, interval construction, Section 8 shift assumption, and Appendix A limitations. Supports the distinction between sensitivity/specificity correction and arbitrary distribution-shift robustness; human reference judgments are assumed resolved. Record · Primary PDF.

[S2] Yiqun T. Chen, Sizhu Lu, Sijia Li, Moran Guo, and Shengyi Li. Efficient Inference for Noisy LLM-as-a-Judge Evaluation. ICML 2026, PMLR 306:15836–15860; 6–11 July 2026. Inspected official record and PDF Sections 2–4 and discussion. Supports randomly labeled same-distribution setup, residual correction, and conditional efficiency comparisons. Asymptotic claims are not treated as small-pilot guarantees. Record · Primary PDF.

[S3] Anastasios N. Angelopoulos, Stephen Bates, Clara Fannjiang, Michael I. Jordan, and Tijana Zrnic. Prediction-Powered Inference. Science 382:669–674, 10 November 2023; inspected arXiv:2301.09633v4, 9 November 2023. Inspected the general principle and rectifier construction. Supports correcting imperfect predictions using a suitable reference sample; does not turn a fallible human reference into objective truth. Inspected manuscript · Journal DOI.

[S4] Naoki Egami, Musashi Hinck, Brandon M. Stewart, and Hanying Wei. Using Imperfect Surrogates for Downstream Inference: Design-based Supervised Learning for Social Science Applications of Large Language Models. NeurIPS 2023, volume 36:68589–68601. Inspected official PDF Sections 1–3, sampling Assumption 1, and pseudo-outcome construction. Supports known positive labeling probabilities and the separation of prediction quality from inferential validity. Record · Paper.

[S5] Magda Dubois, Harry Coppock, Mario Giulianelli, Ole Jorgensen, Timo Flesch, Lennart Luettgau, and Cozmin Ududec. Quantifying Biases in LLM-as-a-Judge Evaluations. ICML 2026, PMLR 306:27120–27135; 6–11 July 2026. Inspected record, PDF grader and interaction models, and MT-Bench case-study setup. Supports modeling several evaluator effects jointly. Model-based removal of an estimated effect is distinguished from an executed intervention. Record · Paper.

[S6] Akash Bonagiri, Gerard Janno Anderias, Saee Patil, Angelina Lai, Devang Borkar, Gezheng Kang, Ishant Gandhi, Setareh Rafatirad, and Houman Homayoun. STABLEVAL: Disagreement-Aware and Stable Evaluation of AI Systems. ICML 2026, PMLR 306:9039–9057; 6–11 July 2026. Inspected record and PDF Sections 3–5 and 7.5. Supports retaining posterior label distributions and treating ranking stability as a distinct objective; conditional independence and identifiability are explicit limits. Its theoretical claims were not re-proved. Record · Paper.

[S7] Ellie Pavlick and Tom Kwiatkowski. Inherent Disagreements in Human Textual Inferences. TACL 7:677–694, November 2019; journal posting 14 November 2019. Inspected primary PDF introduction, task definition, and context-manipulation methods. Supports the possibility of persistent, meaningful disagreement in NLI. No claim that all Encevra labels are inherently subjective follows. Record · Paper.

[S8] Ido Amit, Ido Galil, and Ran El-Yaniv. Evaluating LLM Uncertainty in Long-Form Generation Using Deterministic Ground Truth. ICML 2026, PMLR 306:2371–2430; 6–11 July 2026. Inspected primary PDF introduction, task setup, unit definitions, and strict-match criterion. Supports executable evaluation anchors and granularity discipline. Reported model results and benchmark code were not reproduced. Record · Paper.

[S9] Zesheng Cai, Yingqi Fan, Sichang Chen, and Jin-Hong Du. DIAL: Position-Debiased LLM Judges with Adaptive Human Preference Calibration. arXiv:2609.31215v1, 25 September 2026, preprint. Inspected PDF introduction, comparison design, order-effect model, and declared theoretical scope. Supports separating position debiasing from human-preference alignment. The full theoretical appendix was not audited and no performance result is imported. Paper.

[S10] Ian Arawjo. How to Run Statistics over LLM Judges and Trust the Results: Calibrated Inference for Small-Sample AI Evaluation with evalstats. arXiv:2609.35815v1, 20 September 2026, preprint. Inspected PDF Sections 1 and 6, including simulation design, random-labeling assumptions, and binary-test scope. Supports examining finite-sample behavior and calibration selection. Software and the extensive simulation suite were not run here. Paper.

[S11] Ryan Lail. Decomposing LLM-Judge Uncertainty to Target Expert Labels. arXiv:2609.06444v1, 6 September 2026, preprint. Inspected PDF main text, Bayesian model, both experiments, and limitations. Supports distinguishing uncertainty about a reviewer distribution from disagreement within it; the simple count-based baseline ties the proposed method in the real-data setting. No Encevra transfer is established. Paper.

[S12] Xin Sun, Di Wu, Sijing Qin, Isao Echizen, Abdallah El Ali, and Saku Sugawara. Label Effects: Shared Heuristic Reliance in Trust Assessment by Humans and LLM-as-a-Judge. ACL 2026, pages 32378–32392, July 2026. Inspected official record and PDF source-label manipulation, judge prompts, placebo-label control, and trust-rating results. Supports blinding provenance labels; trust ratings in this study are not factual correctness on Encevra tasks. Record · Paper.

[S13] Angela J. Yu and Peter Dayan. Uncertainty, Neuromodulation, and Attention. Neuron 46:681–692, 19 May 2005. Inspected author-hosted PDF summary and introduction defining expected and unexpected uncertainty. Used only to motivate a falsifiable separation of evaluation states. No literal brain-copying or validated biological implementation claim is made. Author-hosted paper · DOI.

[S14] Rémi Bardenet and Odalric-Ambrym Maillard. Concentration Inequalities for Sampling Without Replacement. Bernoulli 21(3):1361–1385, August 2015; inspected arXiv:1309.4029v2, 27 July 2015. Inspected setup, Lemma 1.1, Proposition 1.2, and discussion of conservative versus finite-population bounds. Supports the conservative audit-selection envelope; the script does not implement all improved inequalities in the paper. Inspected paper · DOI.

[S15] Charles F. Manski. Nonparametric Bounds on Treatment Effects. American Economic Review 80(2):319–323, May 1990. Inspected scanned primary-paper pages 319–320, including the identification setup and bounding argument. Supplies the partial-identification perspective, not the specific Encevra implementation or a statistical confidence guarantee. Primary-paper scan.

Internal archive sources. Encevra Research Rounds 9, 10, 11, 12, and 13, plus the Research overview, numbered index, publications and public notebook registry. Canonical links identify the archive pages; the project source was the inspected authority because preview changes may precede public deployment. Round 13's audit findings are carried forward as its findings, not newly replicated results. No internal notebook database or private reviewer record was inspected.

Access note. The browser retrieval service could not display several arXiv HTML editions or PMLR PDFs served as generic binary content. Their primary PDF files were successfully retrieved and read locally. The source ledger records those actual PDFs rather than claiming full-text access from a search snippet. An attempted ICLR source titled Noisy but Valid could not be retrieved in full and is not used as evidence or included among the numbered references.