Encevra Research Round 13

From final scores to measurable corrections

An intervention-ready audit of Noetis

4 October 2026 • Research memorandum • Expanded edition • Version 1.0

Round 13 examines whether the available Noetis records can support the research program established in Rounds 10-12: selective correction, defensible cost comparisons, and evaluation of the exact answer eventually released.

The immediate result is a measurement finding. The inspected exports preserve final outputs and heuristic scores, but do not preserve the paired answer states, independent labels, and complete accounting needed to estimate repair, damage, or matched-cost advantage. Executed tests also identify scoring behavior that is sensitive to answer packaging and a provider-parser path that represents unknown output usage as zero.

This round therefore supplies three things: a reproducible audit of existing artifacts, an offline reference trace validator with adversarial checks, and a concrete pilot protocol and implementation handoff. It does not report fresh language-model performance, completed human adjudication, or a calibrated release guarantee.

Recommended next action: preserve the pre-repair answer and checker result; record every provider attempt; replace heuristic headline evaluation with a fixed, independently labeled rubric; then run a small instrumentation pilot before collecting a larger comparison set.

Scope: Encevra project snapshot. Findings describe the inspected files and exports, not an uninspected production database or later snapshots. The original project was not modified. The expanded pass adds evidence-lineage research, a reproduced challenge-omission path, and exact dependence examples.

1. Decision and principal findings

The proposed correction mechanism remains plausible. The current evidence cannot establish its benefit. This distinction matters because a system can obtain a higher final score by finding more errors, by producing more rubric keywords, by attaching a longer findings appendix, or by spending more computation. Those explanations require different decisions.

First, retain the existing records as diagnostic evidence. The exports explicitly identify their scoring as a keyword and quote-overlap diagnostic rather than adjudicated quality. Replaying the current scorer reproduces all 119 nonempty stored scores. The audit does not discover a general mismatch between saved and current scores. It discovers that the reproduced measure cannot carry the stronger interpretation the research program needs.

Second, treat answer packaging as part of the evaluation specification. Adding the full source as a labeled appendix removes all detected-issue credit in every one of the 114 outputs that initially had positive detection credit. Removing the appended approved findings changes detection counts in 44 of 59 nonempty panel outputs. Neither operation by itself tells us whether the answer is correct. Together they establish that the measured endpoint must be defined and the evaluator repaired before score movement becomes architectural evidence.

Third, preserve transitions, not just terminal artifacts. Current panel code performs a final check and permits a bounded repair. The returned result retains the final brief and latest check, but not the full pair of before/after briefs and before/after checker findings. A repair-attempt flag cannot distinguish a useful correction from a newly introduced error.

Fourth, make incomplete usage visibly incomplete. The inspected exports contain panel usage for 19 cases and baseline usage for none. An executed provider-parser fixture turns input-only reported usage into a result with output tokens equal to zero and estimated=false. Its live frequency is unknown. These are reasons to repair accounting, not estimates of the system’s true cost.

Fifth, separate the next two experiments. A local revision experiment can test whether evidence packaging changes outcomes at a fixed state. A later end-to-end experiment can test whether Noetis outperforms a strong single agent with comparable evidence access and measured resources. Success in the first experiment would not establish success in the second.

The strongest justified product decision is to improve observability and evaluation now. The strongest justified research decision is to reserve performance claims until those measurements exist.

2. Continuity, scope, and evidence status

Round 10 made selective, evidence-conditioned correction and a strong matched-cost single-agent baseline central. Round 11 moved toward choosing among checking actions using their expected benefit, damage, and cost. Round 12 focused on risk attached to the exact terminal output and explicitly called for a small logging and labeling pilot before scaling evaluation. Round 13 follows that handoff rather than introducing another untested architectural metaphor.

The work here has four distinct evidence classes. Keeping them separate prevents a software test or thought experiment from becoming an accidental performance claim.

There were no new model calls, no provider billing reconciliation, no production database inspection, no independent human labeling campaign, and no deployment. The report does not infer that every live Noetis run has the same omissions as these exports. The application has richer internal result structures and a persistence path; those are discussed where relevant.

The inspected project files were obtained at one commit. Original benchmark source text was matched to current fixtures by case ID and character count. All 60 character counts agree, but equal length is not proof of identical historical bytes. The replay therefore establishes current-code behavior on saved answers and pinned fixtures. A future experiment must persist the source digest at run time.

The source review prioritizes primary papers, first-party research, and official instrumentation documentation. Access depth is disclosed in the source ledger. The conceptual recommendations are inferences from those sources and the local audit, not claims that an external benchmark has validated Noetis.

The reproducibility package contains newly written research code, audit results, and protocols. It does not redistribute the original project repository. Replaying the repository-specific audit requires the authorized snapshot whose hashes are supplied.

3. What the newer literature changes

Recent process-evaluation research reinforces the need to observe interventions and intermediate artifacts, while also showing why richer explanations should not be confused with causal evidence.

CHIVE evaluates proposed explanations through counterfactual prompt edits and repeated model samples. Its behavioral labels are distinguished from open-ended explanatory hypotheses. In the tested settings, activation-based interpretation tools did not improve over a transcript baseline. The useful transfer is methodological: make a proposed cause predict an observable change, then test it. Its repeated sampling design is not a universal sample-size prescription for Noetis. [S1]

TraceElephant emphasizes complete multi-agent interaction traces, including inputs, outputs, and metadata, and studies dynamic attribution through actual reruns. Its attribution results do not make every rerun a clean causal intervention; replacement quality and downstream behavior still matter. For Noetis, this supports capturing enough state to execute a controlled branch rather than merely asking another model to explain a failure. [S2]

AgentProcessBench supplies human process labels that distinguish correct, neutral, and harmful steps. This is relevant because an exploratory action can be unproductive without being a factual error. Process labels describe a trajectory; they do not by themselves estimate what would have happened under a different action. [S3]

GroundEval uses executable contracts over observable agent state, including access, timing, and evidence provenance. The transfer is to encode the constraints that are actually checkable. An observed tool trace need not expose private internal reasoning. A contract can reject a wrong source version while remaining unable to judge a subtle argument. [S4]

These sources motivate a division of labor. Deterministic validators should enforce identity, ordering, scope, budget, and provenance. Independent task labels should assess correctness and completeness. Controlled continuations should estimate the effect of an intervention. A single self-reported “confidence” field cannot substitute for all three.

The immediate gap is therefore operational. Noetis already has roles, findings, a synthesis, and a repair path. The next useful advance is to turn that path into an observable experiment with a clearly specified endpoint. Adding another role before this gap is closed would make attribution and accounting more difficult.

4. Attribution is not yet intervention evidence

The Who&When benchmark frames failure attribution around an agent and a decisive step, with expert annotation. That is a useful localization task, but an attribution label is not an empirical estimate of the probability that a repair will succeed when executed repeatedly. [S5]

A2P uses abductive explanation and predicted counterfactual continuations to improve attribution. Its predicted continuations are model-generated hypotheses rather than live execution of the original system. Such a method could suggest candidate repairs for Noetis, but the candidates would still need actual evaluation. [S6]

MAST supplies a failure taxonomy that can make incident records more consistent. A taxonomy organizes what went wrong; it does not show which intervention would have prevented it. Use a failure category as a diagnostic tag, not as a measured action value. [S7]

The causal distinction is older than agent systems. Counterfactual learning requires assumptions about interventions and the data-collection process. Partial-feedback policy evaluation additionally depends on action support and assignment probabilities. Recorded outcomes from an adaptive checker cannot simply be compared with outcomes from cases where checking was skipped. [S8, S9]

First-party faithfulness research also cautions against treating an answer’s stated rationale as a complete causal account of its behavior. For this program, the practical response is to log returned messages, actual tool artifacts, and interventions. Hidden chain-of-thought is neither required nor proposed. [S10]

BudgetVerify remains relevant to the comparison set because it considers different levels of verification effort. This round could inspect its abstract but could not retrieve its full methods. No numerical result or reproduction claim from that paper is used here. [S11]

The resulting research rule is simple: an explanation can propose an experiment; a trace can make it possible; an executed, well-controlled comparison can supply evidence. These are different stages. The proposed pilot deliberately stops short of claiming a general causal model of the whole agent team.

A “brain-inspired” contribution would be stronger if it made this distinction concrete. Assigning credit to evidence that changed a specific answer requirement is an operational hypothesis. Naming components after brain regions does not provide additional evidence. Round 13 therefore focuses on measurable credit assignment at the revision boundary.

5. Audit corpus and reproducibility

The audit uses three saved benchmark runs and the scorer and fixture definitions at the pinned commit. The run headers identify a medium-effort openai/gpt-5.6-sol baseline, a shared baseline-prompt digest, and concurrency of three. The third run also identifies the JEV_NOETIS execution mode.

There are 60 distinct case IDs. The runs use disjoint case sets. Within a case, the panel and baseline are paired. Across these three runs, an average-score difference is also a case-mix difference. It cannot establish an improvement caused by an architecture version.

There are 120 output slots: two per case. One panel output is empty and has a recorded error on b4-mfa-partial-rollout. This failure belongs in episode-level reporting. It should not disappear because the nonempty-output scorer analysis uses a denominator of 119.

The audit script extracts the original pure scorer functions and imports the fixture definitions using Node’s TypeScript stripping facility. It does not rewrite the scoring logic. It checks SHA-256 hashes against an included manifest before processing the source files, so a different snapshot fails explicitly rather than being silently labeled as the inspected commit.

For each output it records the saved score, current score, saved detection count, current detection count, score after adding a source appendix, and score after isolating the synthesis. Per-record results are in results/artifact_audit.json. The same file contains the run headers and source hashes.

A separate fixture executes the inspected provider parser with an in-memory mock stream and a mocked credential value. It makes no network request and does not read a live provider key. These details are part of the reproduction boundary: this is an implementation audit, not a rerun of the historical models.

The original exports’ diagnostic-scoring disclosure is retained. This round recommends a stronger evaluator for stronger research claims; it does not retroactively present the historical exports as a blinded scientific study.

6. What the saved records can and cannot answer

The three exports preserve enough information to reproduce a narrow diagnostic endpoint. They do not preserve enough to estimate the mechanism targeted by Rounds 10-12.

These counts refer to the inspected export schema, not all possible fields in live storage. panel-runs.server.ts has a persistence path for the full panel result, material, reports, and usage. That is a useful integration point. The live database was not examined, and the report makes no claim about records that were not retrieved.

The result and trace types nevertheless reveal a more specific limitation. PanelTraceEntry includes role, requested and actual model, fallback status, latency, and usage. It does not itself preserve the complete message inputs and outputs required to recreate a state. A trace containing model names and timings is operationally useful but insufficient for exact experimental branching.

Similarly, the code’s completed or incomplete release status describes runtime checks. It is not the calibrated release condition developed in Round 12. Treating these as separate fields avoids an accidental claim that a completed execution has met a statistical error-risk threshold.

The absence of paired intermediate answers also blocks a tempting shortcut. It is not valid to call every correct terminal answer a successful repair. Some were already correct. Nor can the number of repair attempts estimate the number of repairs. A checker can flag a harmless issue, fail to change the answer, correct one requirement while breaking another, or induce an unnecessary abstention.

The next export should preserve the experiment’s unit of analysis directly: one episode with stable source versions, every attempted call, identified answer snapshots, and the labels attached to those exact snapshots.

7. The repair path exists, but its transition is lost

Inspection of panel-core.server.ts shows a concrete bounded-repair mechanism. A synthesis is produced, a final check is run, and a repair is attempted when the check returns issues. A successful repair replaces the brief and is checked again. The result returns the final brief, latest checker result, and a repair-attempt indicator.

This is an important distinction from a system with no repair mechanism. The proposed instrumentation should preserve the existing behavior while recording what it currently overwrites.

At minimum, persist the original synthesis before assigning a repaired value, the first checker result before replacing it, the repair request, the repair response, and the post-repair checker result. Each should have a content digest and a stable relation to the episode and provider attempt. A revision record should connect the two answer snapshots and list the requirements the revision intended to change.

The analysis must still use external labels. The checker’s own list of issues is a model output. It can be useful for localization, but it is not independent proof that the original answer was wrong or that the revised answer fixed it.

decision-persist.server.ts contains a status-based confidence mapping for findings: supported maps to 0.9, narrowed to 0.6, unsupported to 0.1, and other status to 0.4. These values are categorical proxies. They should not be relabeled as probabilities elicited from a model or used as the Round 12 confidence baseline without disclosing their construction.

The integration should preserve three separate concepts:

• Checker judgment: what a checker claimed about an answer.
• Independent label: whether a required item and the complete answer are correct under the fixed rubric.
• Release statistic: a quantity derived from a frozen, separately evaluated policy.

There is also a source-access issue. A finding may be supported by the approved packet while missing a constraint from the original task. The terminal evaluator must inspect all required task dimensions, not merely consistency with the packet selected by the system itself.

No architecture change is necessary to begin collecting these records. The highest-value first patch is to stop overwriting the evidence needed to judge the architecture already present.

8. Scorer failure: a source appendix erases detections

The scorer includes a global copied-text guard. When an answer appears to reproduce too much of the source, the guard suppresses planted-issue detection. The intended negative control is reasonable: pasting a source should not count as analyzing it. Its current scope also suppresses valid analytical text when that text is accompanied by the source.

The executed transformation leaves each saved answer intact and appends a labeled source appendix. Among the 119 nonempty answers, 114 initially have at least one detected issue. All 114 lose every detected issue after the source is appended.

This is a deterministic property of the inspected scorer and these texts. It is not a 100% failure rate for the underlying models. It is also not a claim that adding an appendix must leave every quality dimension unchanged. An appendix can affect length, usability, or instruction compliance. The defect is specifically that previously credited analysis is erased by a global overlap decision.

A second fixture confirms that pure source copying still receives zero issue detections. The fix should preserve this negative control. Simply deleting the copied-text guard would remove the demonstrated false-negative mechanism but could reintroduce false-positive credit for pasted material.

A better design separates source blocks from analytical claims, then evaluates the claims against the task rubric. Reproduced material can establish provenance or provide reference context. It does not itself count as identifying an inconsistency. Explicitly marked quotation boundaries help, but a model should not be able to grant itself analytical credit merely by changing markup.

The first regression gate is therefore a pair: a valid analytical answer with an optional source appendix should retain its supported issue labels, while a source-only answer should not gain them. The report does not supply a universal semantic grader that solves this distinction. It supplies a reproducible failure and the required behavior for an adjudicated replacement.

Until then, the existing numeric score should remain a development diagnostic rather than the principal research outcome.

9. Scorer failure: wording and packaging change the endpoint

A constructed answer for the renewal-math case states: “There is no mismatch: the committed total of $648,000 is correct.” The scorer credits the planted issue that the committed total does not reconcile with the fee schedule. The answer denies the error; the keyword-based detector nevertheless counts it as found.

For that case, the specified schedule gives 16,000 × 12 × (1 + 1.07 + 1.07²) = 617,260.80. The arithmetic establishes the fixture’s contradiction. It does not validate the scorer’s broader correctness judgments.

The quote-overlap component has a different vulnerability. One supported quotation and one invented quotation yield a grounding fraction of 0.5. Repeating the supported quotation nine times while retaining the same invented quotation yields 0.9. The unsupported assertion is unchanged. The metric is sensitive to quote occurrences rather than the reliability of the answer’s unique claims.

The panel endpoint is also a bundle. panelOutputText combines the synthesis with appended supported or narrowed findings. When the audit isolates the synthesis, detection counts change in 44 of 59 nonempty panel outputs: 43 decrease and one increases.

These are scorer outputs, not independent correctness percentages. A full findings bundle can be a legitimate product deliverable. The problem is ambiguity about which object the research claim concerns.

The next benchmark should specify and score two endpoints separately: the exact user-facing brief and the complete report package, when the product actually delivers both. Baselines should be allowed comparable output structure, source access, and required content. It would be misleading to compare a short baseline brief with a panel brief plus an unbounded appendix without accounting for that choice.

A fixed rubric should reward correct identification, penalize incorrect assertions and material omissions, and prevent repetition from diluting unsupported content. Deterministic arithmetic checks are useful where applicable. Open-ended claims still require an adjudication process whose limitations are recorded.

10. Accounting: missing is not zero

The code and exports expose three different accounting gaps. They should be tracked separately because they require different fixes.

Export asymmetry. Panel aggregate usage is present in 19 saved records, all from successful cases in the third run. Baseline usage is absent throughout. The inspected baseline path uses a text-only generation return, so the benchmark does not retain detailed usage for that side. Equal model labels or equal effort settings do not establish equal tokens, latency, or cost.

Partial provider reports. An executed fixture supplies the inspected Anthropic stream parser with reported input usage of 100 tokens and no reported output count. The parser returns 100 input tokens, zero output tokens, and estimated=false. A positive control that includes an output count of seven returns seven. This demonstrates a missing-data representation problem in that path. It does not establish its frequency in real provider traffic.

Fallback attempts. The inspected roster wrapper records the returned fallback call and a fallback reason after a preferred call fails. The failed preferred attempt is not returned as a separate usage-bearing trace entry through this path. A timeout or failed attempt may have consumed resources. A final successful response is therefore not necessarily the full episode bill.

The replacement accounting contract should preserve, for every attempted call, requested model, actual model if known, start/finish status, provider request identity where available, and field-level usage provenance. A missing output count should remain null. An estimated count should carry its method. A known zero should mean that the relevant source actually reported zero.

Official OpenTelemetry conventions provide useful naming for request/response models, tool inputs and outputs, and token usage. Their guidance treats reasoning tokens as part of output usage rather than an extra quantity to add again. The GenAI conventions are evolving, so the implementation should pin the version it follows. [S12]

Maintain separate views of provider-reported quantities, estimates, and reconciled billing. Token totals are not dollars; a model-based price estimate is not an invoice. Wall-clock latency, serial critical-path time, concurrency, and aggregate work also answer different questions.

The included reference aggregator reports a known sum and the number of unknown attempts. It returns a null total when the total cannot be completed. This preserves useful partial information without silently turning missing usage into free computation.

11. A minimal episode contract

The production record should make one episode reconstructible without storing private internal reasoning. The necessary material is the observable task, returned messages, tool artifacts, configuration, and revision history. Large or sensitive payloads can live in an access-controlled artifact store, with the trace carrying references and digests.

Use stable IDs for calls, branches, requirements, sources, and artifacts. Preserve source lineage as well as source count. Ten citations to copies of one release note are not ten independent sources. An authority label should describe provenance and task scope, not assert that a source is infallible.

Source identity and temporal availability are separate. A document can exist but be unavailable to a branch, or be published but not yet effective for the task date. Store those distinctions explicitly. The reference validator checks declared source observations and sequence constraints; a production collector must additionally verify retrieved bytes and enforce actual access boundaries.

The event format uses a sequence number, previous-event hash, and current-event hash. Structured values are hashed using documented canonical JSON; source text uses raw UTF-8 bytes without normalization. These hashes detect divergence relative to a trusted record. They do not authenticate an author or prevent someone from rewriting and rehashing the entire file.

The provided Python implementation is a single-writer offline reference. It writes a JSONL representation, validates a selected set of structural invariants, and supports isolated in-memory state copies. It is not a durable database, a concurrent event service, an authentication layer, or a sandbox for arbitrary tools. Production work should preserve this distinction rather than treating a passing fixture as a completed observability platform.

12. Evidence receipts and revision boundaries

An evidence receipt is a structured record explaining what an observed artifact bears on. It should point to verifiable material instead of merely saying that another agent disagrees.

A proposed receipt contains a requirement ID, the answer span under review, a source ID and version, an exact location or executable result, the claimed relation between evidence and requirement, and the checker’s proposed correction. The relation may be support, contradiction, missing support, scope mismatch, or unresolved conflict. The receipt also records the producing attempt and whether its interpretation has been independently checked.

The receipt is itself fallible. Correct quotation does not establish a correct inference. The source can have the wrong jurisdiction, date, population, units, or contractual scope. A receipt should therefore separate the raw evidence from the checker’s interpretation and proposed answer. This separation also creates a usable experimental manipulation: the continuation can see the same evidence with or without the peer’s verdict.

A revision should preserve enough structure to detect collateral changes. Record the intended targets before revision. After revision, compare all rubric requirements, not just those targets. If the checker flags a total and the reviser fixes it while changing a correct payment deadline, the episode contains both a repair and damage.

Do not let the system define its own denominator by deleting claims. Required task content is fixed before generation. Added material claims receive separate review. An answer that removes every difficult statement should lose completion credit rather than appear maximally safe.

A replay state must include the answer, externally visible conversation, versioned source set, relevant tool state, memory version, configuration, and remaining budget. It must also identify artifacts produced by earlier actions. A transcript alone may omit a tool cache or memory write that changes a later result.

The reference freeze_state and fork_state functions require these state categories and copy them into isolated local branches. They verify identity at the moment of branching. They cannot make a remote search index reproducible, reverse an external side effect, or force a provider to reproduce a stochastic sample. For the pilot, use immutable document snapshots and read-only or simulated tools. Record residual nondeterminism instead of claiming exact replay.

This contract makes evidence-conditioned revision testable. It does not assume that a receipt is superior to ordinary reconsideration; that is the hypothesis the experiment must evaluate.

13. Define the estimand before collecting branches

The first intervention question is local: given a frozen state at the revision boundary, how does the representation of already collected evidence affect the next answer? This is narrower than the question of whether the entire Noetis pipeline is worth its cost.

Let S denote an eligible frozen state, a a continuation protocol, and Y(a) the independently judged success of its terminal answer. The primary local contrast is the average of Y(receipt) − Y(ordinary) over the prespecified pilot task distribution and the continuation randomness. A second contrast is Y(receipt + verdict) − Y(receipt). It tests the additional effect of exposing the peer’s recommendation.

The distribution matters. If states are collected only when the current checker flags an issue, the result applies to that selected population. It does not establish benefit on all tasks or validate the checker’s selection policy. The pilot should retain all sampled states and tag checker status so that selected and unselected subsets are visible.

For each fixed required item, obtain an initial and terminal correctness label. Then report:

• Repair rate: initially wrong required items that become correct, divided by initially wrong required items.
• Damage rate: initially correct required items that become wrong, divided by initially correct required items.
• Task success: all required content satisfied, with no material unsupported or incorrect added claim.
• Release and abstention outcomes: reported separately under a fixed policy, without removing failures from the denominator.

A zero eligible denominator produces an undefined rate, recorded as null, rather than a perfect score. Missing labels remain missing and must be counted. Requirements nested inside one task are correlated; they do not become independent trials because they have separate IDs.

A useful decomposition is that the change in the number of correct required items equals repairs minus damage. That identity does not identify either component from the net change alone. It also does not account for new unsupported claims unless those are evaluated separately.

These definitions deliberately prevent several shortcuts: equating checker agreement with correctness, calling every final success a repair, treating all added text as harmless, or interpreting more self-reported confidence as less error. The task-level endpoint remains primary when deciding whether the user receives a better answer.

14. Two exact examples of what the old endpoint hides

The included diagnostics compute two constructed examples. Every count and probability in this section is stipulated. These are logical demonstrations, not measurements of Noetis or any model.

In the first example, both systems begin with 75 correct and 25 wrong required items. Both finish with 80 correct items. Their revision behavior is very different.

Even knowing both initial and final aggregate accuracy would not distinguish these worlds. Paired labels are necessary. The inspected exports do not even contain the initial answers, making a retrospective repair-rate estimate still less supportable.

The second example concerns action selection. Suppose checking succeeds on 95% of easy tasks and 60% of hard tasks, while stopping succeeds on 90% of easy tasks and 30% of hard tasks. Checking is beneficial in both strata. But imagine the observed checking group contains 10% easy and 90% hard tasks, while the stopping group has the reverse mixture.

The reversal is caused by task selection, not by checking becoming harmful. A real adaptive controller is likely to select difficult or uncertain cases for extra work. Logs of that policy therefore need assignment information and appropriate identification assumptions before they can support action-value estimates.

The pilot’s controlled branches avoid this particular between-case comparison by comparing continuations from the same frozen state. That still leaves task sampling, stochastic variation, branch contamination, and grading error. Pairing solves one problem; it does not solve all of them.

These examples are reproduced with exact rational arithmetic where probabilities are involved. Their purpose is to establish which information must be collected, not to predict the magnitude of a future improvement.

15. The next pilot: scope and collection

The next live study should be an instrumentation and labeling pilot. Its purpose is to find collection failures, measure adjudication effort, estimate variability, and verify that the intended interventions are actually delivered. It should not be presented as a confirmatory demonstration of a small accuracy gain.

Start with 24 smoke-test tasks, six in each of four strata. After repairing collection defects and freezing the protocol, collect 96 additional tasks, 24 per stratum. All 120 are developmental data. The later confirmatory set must be fresh and must not reuse task families that informed prompt or rubric changes.

These strata are a proposed stress-test allocation, not an estimate of production prevalence. Report each stratum separately. A population-wide result requires a documented target distribution and appropriate sampling or weighting.

For each task, prepare an independent rubric before model generation. Collect one initial answer and the common evidence-gathering prefix. Freeze the revision boundary before revealing branch outcomes. Include states whether or not the checker reports a problem, with that status recorded. This permits a descriptive examination of unnecessary revision as well as attempted correction.

The active branches use the same backbone, model policy, source corpus, tool permissions, and post-fork resource cap. Use two independent continuation draws per active arm for the pilot. This exposes some instability but is not enough to estimate a precise treatment effect at each individual state.

At 120 tasks, three active arms and two draws imply 720 continuation calls if each continuation is exactly one model call, plus initial generation and any evidence/checker calls. Real totals may be higher. Measure the 24-task stage before fixing the larger stage’s budget and concurrency. This report neither launches these calls nor commits provider spend.

Record collection failures and unresolved labels as outcomes of the pilot. Do not silently replace difficult or failed episodes until a clean-looking dataset remains.

16. Interventions, shared costs, and fairness

The local experiment begins after a common evidence-collection prefix. This timing is deliberate: the pilot tests revision behavior conditional on evidence having been collected. It does not estimate the value of choosing to collect that evidence in the first place.

Receipt construction must occur without access to hidden gold labels. Raw evidence, source versions, and tool results are shared across the active arms. If the checker discovers a new document, all active arms must receive the same raw document or the experiment becomes a comparison of information access as well as presentation. The structured interpretation is the intended treatment in C; any further information differences must be documented.

The common prefix includes the evidence/checker work used to build the receipts. Charge that shared prefix to each arm’s hypothetical end-to-end episode total, and show incremental continuation cost separately. In the actual pilot collection bill, the shared prefix is incurred once per task; do not multiply it as if it were physically rerun for each branch. Branching creates two valid accounting views, not one interchangeable total.

Arm A at this boundary also inherits the already incurred prefix cost. An earlier stop before checking is a different policy. The later end-to-end study must compare that policy and a strong single-agent workflow using their actual chosen work. A local evidence-packaging advantage does not establish that the shared checker was worth paying for.

Randomize branch execution order within task, isolate memory and caches where they can affect outputs, and record the execution window. Fix retry and timeout policies in advance. A failed continuation should remain a failure under the primary endpoint, with a secondary analysis describing recoveries if retries are part of the policy.

Identical random seeds, when a provider exposes them, do not guarantee identical hidden computation. Treat repeated samples as stochastic continuations. Archive exact requests and returned outputs so that the delivered treatment can be audited even when exact model replay is impossible.

17. Labeling without letting the system grade itself

Build the task rubric before generating the initial answer. It should identify required claims or actions, their supporting evidence, acceptable uncertainty, and errors that make the answer materially misleading. The rubric should not be reverse-engineered from whichever answer a favored arm produces.

For numerical requirements, preserve the inputs, units, formula, rounding rule, and tolerance. For source questions, preserve scope and effective dates. For underdetermined tasks, specify what can be concluded and what remains unknown. A refusal to invent an answer can be correct, while an unnecessary refusal to answer a fully supported question can be incomplete.

Two annotators should independently label the initial and terminal snapshots in randomized order, blinded to branch identity, model identity, scorer result, and checker verdict where practical. Perfect blinding may be impossible when formats differ; record suspected unblinding and use normalized presentation only if it does not remove task-relevant content.

After independent labels are saved, disagreements should be adjudicated by a separate reviewer or a documented consensus procedure. Report initial agreement, disagreement counts, adjudication reasons, and unresolved items. Agreement is a reliability measure, not proof that the rubric is true. If only one qualified analyst is available, label the study as single-annotator exploratory work.

The binary transition function in the reference kit accepts only resolved item labels. It raises an error when a required item lacks a label. The production analysis must retain unresolved-label counts and reasons rather than coercing them to correctness or quietly dropping them.

Keep checker proposals and external annotation in separate stores until the blinding stage is complete. Any gold answer, adjudicator note, or future source accidentally exposed to a branch is a protocol deviation. Flag the affected episode and report the deviation; do not quietly rerun it until it looks ordinary.

Automated judging can assist triage and consistency checks. It should not provide both the proposed repair and the supposedly independent ground truth without a disclosed validation procedure.

18. Analysis and decision rules

For the pilot, report completion of the measurement process before reporting model outcomes. The first table should count attempted episodes, valid terminal bindings, captured provider attempts, usage fields with known provenance, labeled snapshots, unresolved labels, and protocol deviations.

Then report task-level outcomes by arm and stratum. Average the repeated continuations within task before computing a paired arm contrast. Show repair and damage numerators with their eligible denominators, newly introduced material errors, abstention, completion failures, latency, and both measured and unresolved cost. Do not average the old diagnostic score into a new “quality” headline.

If tasks share source families or templates, uncertainty analysis should resample or otherwise account for the family, not treat every continuation or requirement as independent. A paired, family-cluster bootstrap is a reasonable exploratory summary when enough independent families exist. A sparse pilot with only a few families should be described primarily rather than given falsely precise confidence intervals.

The confirmatory study needs a fresh sample, a frozen endpoint, a frozen controller, and a prespecified contrast. Choose its size using the pilot’s task-level discordance and clustering, not an independence assumption over hundreds of internal claims. The small improvement and damage margins proposed in Round 11 require a dedicated power analysis; 120 developmental tasks do not establish that such margins are resolved.

Cost comparisons should expose at least three quantities: incremental continuation cost, hypothetical per-policy episode cost including the common prefix, and the actual bill for collecting the branched experiment. An unknown total stays unknown. Report cost coverage alongside averages; do not average only the cheapest-looking fully observed episodes and imply a complete total.

The local receipt hypothesis is weakened if C does not improve task success over B under comparable post-fork resources, if apparent gains consist only of length or keyword changes, or if improvements come with unacceptable damage. A positive result must survive the prespecified output endpoint and independent labels.

The peer-verdict hypothesis is separate. If D changes answers more often but does not improve correctness, the verdict may be inducing conformity. If C helps and D does not, keep the evidence packaging while reconsidering opinion exposure.

Only after the policy and endpoint are frozen should the Round 12 release-risk procedure be applied on its own untouched calibration data. An internal completed flag, status-based confidence proxy, or pilot success percentage is not a replacement for that procedure.

19. Implementation order and acceptance gates

The following order makes each change independently reviewable. It preserves the existing product path while making its scientific interpretation less ambiguous. The complete engineering handoff is included separately in the kit.

There are three gates before the 96-task stage. First, the collector must retain every started provider attempt and every answer state required for the chosen contrast. Second, terminal identity and source provenance checks must pass or the episode must be explicitly marked invalid for that analysis. Third, the labeling trial must show that reviewers can apply the rubric and record disagreements without consulting branch outcomes during task construction.

These are development gates, not statistical certifications. A null token count can be a valid representation of an incomplete provider report. The gate is that the null is retained and surfaced, not that every provider must magically supply all fields. Matched-cost analysis remains blocked until adequate accounting exists.

Do not ship a broad scoring rewrite merely because a keyword counterexample is easy to fix. Repairing negation with a short blacklist would leave paraphrase, scope, omission, and quotation attribution unsolved. Preserve the diagnostic if it remains useful for development, name it accurately, and add the independently adjudicated endpoint required by the research claim.

The acceptance suite should protect actual research invariants, not mirror implementation details. A code refactor that preserves those invariants should not require rewriting the scientific protocol.

20. What the reference kit actually implements

The package contains a pinned-source audit, a standard-library Python event reference, and executable diagnostics. It is designed to make the findings inspectable and to provide a small integration target, without implying that production instrumentation has been installed.

The 26 checks cover valid and tampered records; source observation and temporal availability; terminal and revision bindings; missing target, model, and input fields; invalid assignment probability; orphan and unfinished calls; negative and partial usage; reasoning-token double-count prevention; missing manifests and episode closure; post-closure changes; unknown usage; branch prefix identity and isolation; missing labels; zero transition denominators; and inclusion of failed-attempt cost.

All 26 passed in the execution environment. This verifies the tested contract behavior for these fixtures. It does not show that every malformed input is handled, that provider adapters comply, that logs are authenticated, or that answers improve.

The constructed usage example contains 100 input tokens, 20 output tokens, and five reasoning tokens. Output remains 20, not 25. The constructed retry example totals 0.001 plus 0.002 dollars as 0.003 dollars. These are stipulated amounts used to test accounting behavior, not observed provider charges.

The scorer audit also has an important control: every nonempty saved numeric score reproduces. The one numeric mismatch is the empty failed panel result, stored as zero by the runner while the pure scorer produces 12.5 on empty text. Detection counts reproduce throughout. This is explained as an error-path convention, not unexplained scorer drift.

The kit intentionally leaves unresolved semantic grading unresolved. A structural validator that accepts an episode can still be accepting a perfectly recorded wrong answer.

21. Threats to interpretation

Snapshot and selection limits. Three saved runs are a convenience sample. They contain 60 distinct tasks but do not establish a production task distribution. The current scorer and fixtures may differ from historical runtime code even though the outputs’ diagnostic scores reproduce. Character counts do not replace missing run-time source hashes.

Measurement limits. The copied-text and negation counterexamples concern a heuristic measure. They do not establish that the model outputs are generally poor, that the panel loses to the baseline, or that every high diagnostic score is wrong. Conversely, perfect replay of the heuristic does not validate it as a truth measure.

Endpoint limits. A synthesis and a report bundle serve different product purposes. The audit demonstrates the need to declare the endpoint; it does not assume that appended findings are illegitimate. A future comparison must make the intended user experience explicit.

Implementation limits. Static code inspection identifies paths and retained fields, not live incidence. The provider parser’s missing-usage behavior is reproduced with a mocked stream. The failed-attempt accounting concern is a code-path finding, not a measured dollar loss. The live persistence database was not inspected.

Prototype limits. The event logger checks selected structural relationships. It does not verify all payload types, retrieve referenced artifacts, enforce remote source permissions, provide transactional durability, or authenticate the hash chain. The local branch copy is not a sandbox for real tools.

Intervention limits. A receipt can change information salience, length, formatting, and interpretation simultaneously. If the goal becomes isolating one of those mechanisms, add a prespecified ablation after the basic pilot works. Do not claim that C versus B identifies a unique cognitive mechanism merely because the average outcome differs.

Statistical limits. Repeated continuations share tasks, sources, and prompts. More branches do not automatically mean more independent evidence. Developmental tuning, selective failure replacement, and rubric changes can invalidate a later confirmatory interpretation.

Research-transfer limits. External attribution and process benchmarks differ from Encevra’s tasks and deployment constraints. Their useful contribution is a set of methods and cautions. They do not supply a performance prior strong enough to replace direct measurement. The source ledger identifies the one abstract-only item and the manuscript versions actually inspected.

These limits narrow the conclusions, but they do not erase the actionable result: the current exported measurement is insufficient for the stronger claims, and the report identifies concrete changes that make those claims testable.

22. What would change the research direction

The next round should be driven by the instrumentation pilot’s evidence. The following decision table keeps unfavorable results useful rather than turning every outcome into an argument for more agents.

Do not use this table to select a favorable analysis after seeing results. The pilot is exploratory, so it can guide the next protocol. The confirmatory question and analysis must then be fixed on a fresh sample.

A modest next milestone would be valuable: a complete, independently labeled set of episodes in which each correction can be traced from an observed source to a before/after answer, with failed calls and unknown costs visible. That would support research questions the current saved outputs cannot answer.

The sequence after that milestone is constrained by the earlier rounds. First validate the local intervention and its harm profile. Then test the end-to-end policy against a strong alternative under measured resources. Only then apply terminal-output release calibration to the frozen policy. None of these stages is interchangeable with a high heuristic score or an attractive failure explanation.

Round 13’s contribution is the audit and executable measurement contract needed to begin that sequence. It preserves the ambition of evidence-conditioned collective reasoning while making the next claim smaller, falsifiable, and reviewable.

23. Extension: preserve the safeguards already present

The additional pass examines citations.ts, evidence-audit.ts, and evidence-audit.server.ts, together with their tests and the panel’s approval functions at the same pinned commit. Thirteen executed cases reproduce the current behavior using invented inputs. They include working controls as well as gaps.

The most important positive finding is that quote location does not automatically approve a claim. In the fixture, a statement says a service begins on 1 May while quoting a real passage saying 1 April. The location check returns verified, but support remains unreviewed and isApproved returns false. An invented quotation is rejected. A review batch missing an expected finding ID is rejected. A narrowed verdict without replacement wording becomes unsupported. These controls should survive the next patch.

ALCE’s evaluation separates answer correctness from citation support and evaluates whether cited passages support the generated claim. That distinction is directly relevant here. An entailment evaluator is still a fallible measurement system, so its agreement with independent labels must be checked on the intended tasks. [S15]

Two additional quotation fixtures identify narrower issues. ASCII ellipsis between two valid passages is accepted, but the equivalent Unicode ellipsis is rejected because cleanup removes that separator before segmentation. Separately, case-normalized matching locates a quotation with a changed case-sensitive key. The returned source span preserves the original key, while the supplied quotation differs from it.

The second behavior is documented normalization, not a demonstrated end-to-end false approval. It nevertheless means that “located after normalization” and “byte-exact quote” are different labels. A production receipt should preserve the supplied text, the exact source slice, and the normalization method. The reviewer and user should be able to see the source’s actual spelling and case.

The semantic reviewer already receives full material for scope checks, not just a short quote. The extension therefore does not recommend rebuilding that existing safeguard. It recommends preserving its evidence and measuring whether it works.

The independent first-pass audit also withholds the draft and other specialists’ conclusions. That is a useful protection against answer exposure. Its observations are subsequently inserted into the audit prompt without passing through the same quote-location function, however. Treat those observations as unverified proposals until their source and support checks are recorded. Independence of presentation does not make a proposal true.

24. Extension: a declared challenge can disappear

The evidence-use audit asks the model to return materiality assessments that include challenges_draft. It separately asks the recommendation record to list challenging finding IDs and dispositions. The mapping code retains the materiality flag and reason but drops challenges_draft.

The later validator checks whether a supported, material finding appears in the recommendation’s challenge list or dispositions. If the recommendation omits it from both places, the earlier declaration that it challenges the draft is no longer available to enforce coverage.

The executed fixture makes this concrete. A supported finding prohibits cancellation before 30 September. The draft recommends proceeding and canceling freely in June. The materiality response identifies that finding as a challenge, but the recommendation lists no challenge. The extracted mapping and validator return no structural problems. The local auditRelease function returns no conditional flag. This is not a test of the full production release path or a measured live wrong-answer rate.

The repair is a coverage contract. Require exactly one assessment for every expected finding ID and reject unknown or duplicate IDs. Preserve an explicit unassessed state rather than converting absence to nonmaterial. Carry the declared challenge designation into the stored record. Require one disposition for each supported material challenge, regardless of whether a second list repeats it.

This does not require assuming that every material finding is a challenge. A cost fact may support the draft. It requires consistency between the assessment the system actually produced and the obligations enforced downstream. The model can still fail to recognize a challenge; that is a semantic failure requiring independent labels, not something structural completeness alone can solve.

The existing applyReviewBatch function already follows an exact expected-ID coverage pattern for claim review. Reusing that design is preferable to introducing an unrelated mechanism. The included reference challenge_coverage function demonstrates the proposed invariants without modifying the product.

25. Extension: evidence lineage needs more than a citation count

A citation can answer several different questions. Where did the text occur? What claim did it support? Which computations used it? Was the same underlying observation repeated elsewhere? These relationships should not be compressed into a single support count.

Provenance-semiring research distinguishes knowing which inputs contributed from knowing how they contributed. Its formal setting is database queries, not open-ended reasoning. The useful design transfer is to retain alternative and joint dependencies rather than flattening every supporting source into an undifferentiated set. [S17]

For example, a quoted unit price and a quoted quantity jointly support a computed total. Either of two independently sufficient signed notices might support a start date. These are different structures. Removing one price input breaks the calculation; removing one redundant notice might not break the date claim.

The reference extension represents declared support as alternatives of jointly required parents. It computes minimal source-ID sets, rejects missing dependencies and cycles, and retains source versions. Source origins are grouped separately, allowing mirrors of the same notice to remain visibly related. Unknown origins remain unknown. These operations trace declared dependencies; they do not prove that a natural-language inference is valid.

The local current-code audit also accepts a challenge that names itself as resolving evidence, provided the finding has supported status. Its uncertainty validator accepts the recommendation ID as evidence resolving an uncertainty attached to that same recommendation. In the constructed contradictory fixture, these are circular records. No semantic model or complete release was executed.

A proposed guard rejects the affected claim and descendants that depend on it as fresh resolving evidence. A valid resolution should point to an applicable observation or computation with its own checked inputs. This does not mean that every repeated reference is forbidden. Reusing a source is legitimate; presenting a conclusion’s repetition as new support for that conclusion is the problem.

A source-lineage graph alone is insufficient for computational diversity. Two algorithms can recompute the same total from the same source. Preserve both computation records and their implementations even though they share source roots. Their agreement may be useful, but its value depends on shared bugs and assumptions, which must be measured rather than inferred from two node IDs.

This extension adds an auditable explanation of where apparent corroboration comes from. It deliberately supplies no automatic conversion from source count to probability.

26. Extension: four kinds of independence

The phrase “independent agents” should be decomposed before it appears in an experiment or product claim.

Kim et al. measure substantial shared errors across models. Their approximately 60% figure on one leaderboard is agreement conditional on both models being wrong. It is not a universal Pearson correlation coefficient and should not be inserted as such into an ensemble variance formula. Their findings motivate measuring dependence on the target task, not assuming a fixed correlation for Noetis. [S14]

A constructed voting example shows why this distinction matters. Suppose seven agents each have accuracy 0.7. If their correctness indicators are independent, majority accuracy is exactly 0.873964. Now suppose that half the time all seven share one correctness draw and half the time they draw independently. Individual accuracy remains 0.7, pairwise correctness correlation becomes 0.5, and majority accuracy falls to 0.786982.

For exchangeable, equal-variance indicators, the average’s variance corresponds to n / [1 + (n − 1)ρ] independent observations. In this constructed seven-agent case it is 1.75. That is a variance comparison, not the accuracy of a literal 1.75-agent committee and not a measured property of Noetis.

A second exact example concerns duplicated evidence. With prior probability 0.5 and one symmetric signal that is correct with probability 0.8, a positive signal gives posterior probability 0.8. Three copies of that same signal still give 0.8. Incorrectly treating the copies as independent yields 4³ / (1 + 4³) = 0.984615.

Three genuinely conditionally independent signals could justify the larger update under those stipulated assumptions. Three agents repeating one supplied message cannot. The next experiment should record which situation occurred rather than using the number of agreeing messages as a substitute.

27. Extension: read the theoretical assumptions literally

InfoDelphi studies evidence partitioning for multi-agent forecasting. Its covariance proposition assumes a linear public/private error decomposition, zero cross-covariances, and symmetric private noise. Those assumptions make the stated correlation identity tractable; partitioning documents alone does not establish them. The paper also invokes data processing to argue against information gain from exchanging only numeric forecasts. [S13]

These are separate claims. A covariance identity can be correct under its assumptions while a broader interpretation of communication remains too strong. In particular, a recipient that receives another agent’s private signal does not evolve only from its own previous estimate. The incoming message must appear in the information model.

The exact counterexample on the next page shows that numeric probability messages can carry useful private evidence. It does not refute the conditional covariance calculation or disprove the paper’s empirical findings. It shows why Noetis should not treat natural-language rationales as mathematically necessary for all useful communication.

The practical question is task-dependent. A scalar can be sufficient in a simple probabilistic model with known reliability. A contract decision may need the actual clause, scope, exceptions, units, and calculation. The advantage of a structured receipt is inspectability and preservation of relevant information, not a universal theorem that numbers cannot communicate knowledge.

Demystifying Multi-Agent Debate provides another transfer warning. Its diverse initialization samples extra candidates, and its confidence mechanism includes training agents to express and use confidence through reinforcement learning. Adding a confidence number to an unchanged prompt is not a reproduction of that method. Candidate-generation and training costs belong in the comparison. [S16]

There is a further distinction between information and computation. If the input already contains a complete arithmetic problem, the correct result is a function of that input. A better algorithm can improve answer accuracy without adding a new external fact. The existence of shared source material therefore does not prove that multiple computations are useless.

The resulting design stance is neither “debate always helps” nor “closed-context reasoning cannot help.” Additional work needs to contribute a useful observation, computation, error check, or representation under a measured budget. The pilot should determine which contribution actually changes the answer and whether it creates collateral errors.

28. Extension: an exact numeric-message counterexample

Consider an unknown bit Y, equally likely to be zero or one. Two agents observe X₁ and X₂, each formed by independently flipping Y with probability 0.1. Each agent’s calibrated posterior is 0.9 when its observation is one and 0.1 when its observation is zero.

A numeric probability message therefore preserves all information in that agent’s binary observation. After agent one receives agent two’s probability, Bayes’ rule gives the following posterior.

With one signal, conditional entropy is 0.468996 bits. With both signals, it is 0.257914 bits. The additional numeric message supplies 0.211081 bits of conditional information. Expected Brier loss decreases from 0.09 to 0.054878. The included script enumerates all eight combinations of target and noise bits using exact probabilities, then computes the entropy values.

No language-model behavior is assumed. These are constructed distributions with known reliability and conditional independence. Their purpose is to disprove the general inference that numeric-only communication cannot introduce useful private information. They do not establish that model-generated confidence estimates are reliable, independent, or sufficient for Noetis tasks.

A valid data-processing argument can bound information in an output by the information in the complete input state that produced it. It cannot omit a newly received peer signal and then conclude that the recipient learned nothing. Nor should mutual information between error variables be substituted without justification for information a message provides about the task target.

There is a related measurement lesson: information improvement need not be identical to top-one accuracy improvement. With two equally reliable symmetric signals, disagreement can create a tie, so a forced binary decision with symmetric tie-breaking need not improve accuracy even as the posterior and Brier loss improve. Choose the endpoint that corresponds to the product decision.

For Noetis, the relevant endpoint remains a complete, correct, supported answer. Structured evidence should be tested against ordinary revision on that endpoint. This example strengthens the experimental discipline without selecting a winning architecture in advance.

29. Extension: stress tests for repetition and counterevidence

Add an offline stress suite before spending on the live pilot. Its fixtures should state the source facts and intended invariant explicitly. The current-code reproductions and proposed reference controls supply the first cases.

A later model-based stress study can hold the raw source packet and initial answer fixed while varying receipt repetition, order, and declared provenance. Compare one receipt with exact duplicated copies that are explicitly identified as copies. Then compare genuinely additional evidence, when the task supplies it. Do not label all agreeing independently computed analyses as literal duplicates.

Track whether revisions become more confident or more frequent without becoming more correct. If duplicated content changes outcomes, examine context length, position, and attention effects before calling the effect social conformity. A length-matched neutral appendix and randomized ordering help separate these explanations. The interventions themselves still consume tokens and belong in the accounting.

For the current challenge omission, use paired fixtures where the same challenges_draft=true assessment is either repeated or omitted from the recommendation’s second list. The structural result should not depend on that repetition. Separately, assess whether models recognize the challenge in the first place. Those are different failure rates.

Do not add every stress manipulation to every pilot task immediately. Establish collector and labeling reliability first. Then select a small, prespecified mechanism study using fresh task families and an explicit continuation budget. Its results remain exploratory until a later frozen test.

The live study should not expose hidden gold labels through “helpful” receipts. A receipt-producing model may see the same approved sources as the other arms, but gold correctness and adjudicator notes remain outside the episode. Otherwise a test of evidence exchange quietly becomes a test of answer leakage.

30. Extension: integration and executed deliverables

The expansion adds two immediate implementation priorities to the earlier handoff. Preserve challenge designation through the materiality mapping, and make the difference between normalized quote location and exact source text explicit. Neither requires adding another reasoning role.

The new audit_evidence.mjs executes 13 cases against pinned project functions and mapping code. Five are working-control cases, five reproduce gaps, two demonstrate structural limits around circular resolution, and one documents the limit of case-normalized location. All expected current behaviors reproduced. “Passing” these reproduction assertions does not mean the product passed a safety or quality evaluation.

The new evidence_contract.py and dependence_diagnostics.py execute 14 checks of the proposed reference controls, including challenge preservation, missing and duplicate assessments, source mirrors, joint versus alternative support, unknown provenance, missing dependencies, and cycles. They also compute the three exact dependence examples. All 14 checks passed. As with the earlier 26 checks, these are offline software and mathematical diagnostics.

The reference graph stores declared support, not discovered truth. A syntactically valid but irrelevant source remains a semantic error. The materiality validator prevents a recognized challenge from being dropped; it cannot force a model to recognize every challenge. These boundaries should remain visible in the API and evaluation reports.

The next live pilot should collect outcomes for these failure categories in addition to repair and damage. If the first 24 tasks reveal repeated materiality omissions or unsupported independent observations, fix that path before expanding collection. Increasing sample count while a record silently loses counterevidence would make the dataset larger without making the central comparison interpretable.

31. Primary-source ledger: process and attribution

Sources were checked on 4 October 2026. This is a focused review for the instrumentation question, not a systematic review or a claim of exhaustive coverage. The identifiers below match the in-text citations. Manuscript versions are specified where methods were inspected.

[S1] Karvonen, Ong, Kantamneni, and Marks. Would this change your answer? Evaluating Explanations of LLM Behavior In The Wild with Counterfactual Experiments. arXiv:2608.16747v1, 17 August 2026. Methods and results inspected, with the first-party CHIVE article as supplementary context. Supports the distinction between explanation hypotheses and tested behavioral predictions. No Noetis performance transfer is assumed. Paper • First-party article.

[S2] Chen et al. Seeing the Whole Elephant: A Benchmark for Failure Attribution in LLM-based Multi-Agent Systems. arXiv:2604.22708v1, 24 April 2026; the abstract page reports ACL 2026 acceptance. Full-trace methods and dynamic-attribution discussion inspected. Supports the need to preserve actual inputs and state for useful diagnosis. It does not establish that every localized repair is causally sufficient. Paper.

[S3] Fan et al. AgentProcessBench: Diagnosing Step-Level Process Quality in Tool-Using Agents. arXiv:2603.14465v1, 15 March 2026. Labeling methods and appendix inspected in v1; a later v2 dated 1 June 2026 is listed in the metadata. Used here for the distinction between correct, neutral, and harmful process steps, not for transferring benchmark scores. Inspected version • Version history.

[S4] Flynt. GroundEval: A Deterministic Replacement for LLM-as-Judge in Stateful Agent Evaluation. arXiv:2606.22737v2, 2 July 2026. Contract definitions and methods inspected. Used for observable source, access, and time constraints. The report does not adopt the title’s strongest implication as a claim that deterministic contracts can replace all semantic adjudication. Paper.

[S5] Zhang et al. Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems. arXiv:2505.00212, 2025. Task definition and annotation methodology inspected in the PDF. Used to distinguish failure localization from repeated, executed repair evaluation. Paper.

[S6] West et al. Abduct, Act, Predict: Scaffolding Causal Inference for Automated Failure Attribution in Multi-Agent Systems. arXiv:2509.10401v2, 23 September 2025. Method and experimental setup inspected. Predicted counterfactual continuations are treated as hypotheses, not observations of rerun systems. Paper.

32. Primary-source ledger: evaluation and accounting

[S7] Cemri et al. Why Do Multi-Agent LLM Systems Fail? 2025; arXiv:2503.13657v3 and the NeurIPS 2025 Datasets and Benchmarks record. Taxonomy construction and failure categories inspected. Used to motivate consistent descriptive incident tags, not to estimate the benefit of an intervention. Manuscript • Proceedings record.

[S8] Bottou et al. Counterfactual Reasoning and Learning Systems: The Example of Computational Advertising. Journal of Machine Learning Research 14(101):3207-3260, 2013. Intervention, randomization, and evaluation discussion inspected. Supplies general causal-evaluation foundations; it is not an LLM experiment. Journal page • Paper.

[S9] Dudík, Langford, and Li. Doubly Robust Policy Evaluation and Learning. arXiv:1103.4601, 2011. Partial-feedback evaluation setup inspected. Supports the need for action support and assignment information. No claim is made that a bandit estimator directly identifies arbitrary multi-step Noetis policies from the available logs. Paper.

[S10] Anthropic. Reasoning models don't always say what they think. First-party research article, 3 April 2025. Experimental discussion inspected. Used as a caution about treating stated rationales as complete behavioral explanations. It does not imply that all explanations are useless or that hidden reasoning should be collected. Article.

[S11] Jenq and Shen. BudgetVerify: Budget-Tiered Verification for Financial QA. arXiv:2609.33052, 27 September 2026. Abstract only: retrieved through primary arXiv search results; direct full-text retrieval failed. Included only as context for verification-tier baselines. No numerical claim, detailed method assumption, or reproduction rests on it. Abstract.

[S12] OpenTelemetry. Generative AI semantic conventions. Official repository and attribute documentation, inspected 4 October 2026. Used for request/response and token-accounting vocabulary, including reasoning tokens as a subset of output tokens. The conventions have moved to a dedicated repository; an integration must pin its schema version. Repository • Attribute documentation.

Internal continuity sources. Encevra Research Round 10; Encevra Round 11, Expanded, revision 2; Encevra Round 12, version 1; and the project’s src/content/research-round-12.md. These establish the research handoff and prior proposed targets. Their synthetic calculations and prospective protocols are not reclassified here as live model results.

33. Primary-source ledger: extended evidence research

[S13] Li et al. Diverse Evidence, Better Forecasts: Multi-Agent Deliberation Under Information Asymmetry. arXiv:2607.01661v1, 2 July 2026. Theory, methods, limitations, and appendices C-E inspected. Used as a candidate evidence-partitioning method. The report distinguishes its conditional covariance proposition from a stronger numeric-communication interpretation and supplies an original counterexample to the latter. Paper.

[S14] Kim, Garg, Peng, and Garg. Correlated Errors in Large Language Models. ICML 2025, PMLR 267:30038-30066; arXiv:2506.07962v1. Data definitions, methods, and judge analysis inspected. Used to distinguish conditional agreement on wrong answers from other correlation measures. It does not supply a universal dependence coefficient for Noetis. Proceedings • Inspected manuscript.

[S15] Gao, Yen, Yu, and Chen. Enabling Large Language Models to Generate Text with Citations. EMNLP 2023, pages 6465-6488. Citation metrics and human-evaluation methods inspected. Used for the separation of correctness, citation support, and citation coverage. The older tested models’ performance is not treated as a current-model estimate. Paper • Record.

[S16] Zhu et al. Demystifying Multi-Agent Debate: The Role of Confidence and Diversity. arXiv:2601.19921v1, 2026. Initialization and confidence-training methods inspected. Used to distinguish additional candidate sampling and learned confidence interaction from a prompt-only confidence field. No reproduced performance result is claimed. Paper.

[S17] Green, Karvounarakis, and Tannen. Provenance Semirings. PODS 2007. Definitions and provenance examples inspected in the author-hosted paper. Used as a conceptual basis for preserving how inputs contribute to outputs. The kit’s small declared-support graph is not a full semiring engine or a semantic theorem prover. Author-hosted paper.

Access and interpretation. All five extended sources were inspected beyond their abstracts. External reported performance was not reimplemented. The new mathematical examples are original constructed diagnostics. The evidence-code findings come from the pinned Encevra snapshot, not from those external papers.

34. Reproduction and handoff record

The report and kit are self-contained for the offline reference diagnostics. The expanded pass adds python3 kit/dependence_diagnostics.py (14 reference checks) and node kit/audit_evidence.mjs /path/to/authorized/snapshot results/evidence_audit.json (13 pinned-code cases). The repository audit additionally requires the authorized source snapshot. No private repository files or live credentials are included in the kit.

Runtime used

Node 24.19.0 for the scorer/parser audit; Python 3.12 for the reference diagnostics. The Python research code uses the standard library. The PDF was generated separately with ReportLab.

Reproduce the offline contract checks

python3 kit/diagnostics.py

Expected result: 26 checks pass; constructed transition and selection examples match results/diagnostic_results.json; the scripted episode is written to results/constructed_episode.jsonl.

Reproduce the project audit

node kit/audit_existing.mjs /path/to/authorized/snapshot results/artifact_audit.json

Expected result: 60 case records, 119 nonempty outputs, 114 of 114 eligible appendix transformations losing detection credit, zero nonempty saved-score mismatches, and the documented parser fixture. All required source-file hashes are checked before use. Node may emit a warning that its TypeScript stripping API is experimental.

Artifact interpretation

The JSON results are the exact executed evidence. The constructed episode is invented test data. The protocol is prospective. The handoff identifies production changes to make; it is not a deployment receipt. The source ledger records access limits. File hashes in the package manifest identify the delivered bytes, not an external authenticity guarantee.

Priority handoff

1. Preserve before/after repair artifacts and carry recognized challenges through the audit mapping.
2. Capture symmetric baseline/panel usage and every provider attempt.
3. Specify brief and bundle endpoints and add independent task labels.
4. Pass collection and labeling gates on 24 smoke-test tasks.
5. Freeze the protocol for 96 further developmental tasks.
6. Design a fresh matched-cost confirmatory comparison using pilot variance and failure data.

The unresolved empirical question remains whether evidence-conditioned revision improves complete answers enough to justify its cost and damage risk. Round 13 does not answer it with a proxy. It supplies the measured defects, reference machinery, and collection plan needed to answer it directly.