{
  "round": 14,
  "access_date": "2026-10-05",
  "scope": "Focused review; source PDFs are not redistributed. See each entry for actual inspection depth.",
  "sources": [
    {
      "id": "S1",
      "record": "Chungpa Lee, Thomas Zeng, Jongwon Jeong, Jy-Yong Sohn, and Kangwook Lee. _How to Correctly Report LLM-as-a-Judge Evaluations._** ICML 2026, PMLR 306:65231\u201365252; conference 6\u201311 July 2026. Inspected the official record and PDF setup, correction, interval construction, Section 8 shift assumption, and Appendix A limitations. Supports the distinction between sensitivity/specificity correction and arbitrary distribution-shift robustness; human reference judgments are assumed resolved. [Record](https://proceedings.mlr.press/v306/lee26bl.html) \u00b7 [Primary PDF](https://raw.githubusercontent.com/mlresearch/v306/main/assets/lee26bl/lee26bl.pdf).",
      "links": [
        "https://proceedings.mlr.press/v306/lee26bl.html",
        "https://raw.githubusercontent.com/mlresearch/v306/main/assets/lee26bl/lee26bl.pdf"
      ]
    },
    {
      "id": "S2",
      "record": "Yiqun T. Chen, Sizhu Lu, Sijia Li, Moran Guo, and Shengyi Li. _Efficient Inference for Noisy LLM-as-a-Judge Evaluation._** ICML 2026, PMLR 306:15836\u201315860; 6\u201311 July 2026. Inspected official record and PDF Sections 2\u20134 and discussion. Supports randomly labeled same-distribution setup, residual correction, and conditional efficiency comparisons. Asymptotic claims are not treated as small-pilot guarantees. [Record](https://proceedings.mlr.press/v306/chen26cq.html) \u00b7 [Primary PDF](https://raw.githubusercontent.com/mlresearch/v306/main/assets/chen26cq/chen26cq.pdf).",
      "links": [
        "https://proceedings.mlr.press/v306/chen26cq.html",
        "https://raw.githubusercontent.com/mlresearch/v306/main/assets/chen26cq/chen26cq.pdf"
      ]
    },
    {
      "id": "S3",
      "record": "Anastasios N. Angelopoulos, Stephen Bates, Clara Fannjiang, Michael I. Jordan, and Tijana Zrnic. _Prediction-Powered Inference._** Science 382:669\u2013674, 10 November 2023; inspected arXiv:2301.09633v4, 9 November 2023. Inspected the general principle and rectifier construction. Supports correcting imperfect predictions using a suitable reference sample; does not turn a fallible human reference into objective truth. [Inspected manuscript](https://arxiv.org/pdf/2301.09633v4) \u00b7 [Journal DOI](https://doi.org/10.1126/science.adi6000).",
      "links": [
        "https://arxiv.org/pdf/2301.09633v4",
        "https://doi.org/10.1126/science.adi6000"
      ]
    },
    {
      "id": "S4",
      "record": "Naoki Egami, Musashi Hinck, Brandon M. Stewart, and Hanying Wei. _Using Imperfect Surrogates for Downstream Inference: Design-based Supervised Learning for Social Science Applications of Large Language Models._** NeurIPS 2023, volume 36:68589\u201368601. Inspected official PDF Sections 1\u20133, sampling Assumption 1, and pseudo-outcome construction. Supports known positive labeling probabilities and the separation of prediction quality from inferential validity. [Record](https://proceedings.neurips.cc/paper_files/paper/2023/hash/d862f7f5445255090de13b825b880d59-Abstract-Conference.html) \u00b7 [Paper](https://proceedings.neurips.cc/paper_files/paper/2023/file/d862f7f5445255090de13b825b880d59-Paper-Conference.pdf).",
      "links": [
        "https://proceedings.neurips.cc/paper_files/paper/2023/hash/d862f7f5445255090de13b825b880d59-Abstract-Conference.html",
        "https://proceedings.neurips.cc/paper_files/paper/2023/file/d862f7f5445255090de13b825b880d59-Paper-Conference.pdf"
      ]
    },
    {
      "id": "S5",
      "record": "Magda Dubois, Harry Coppock, Mario Giulianelli, Ole Jorgensen, Timo Flesch, Lennart Luettgau, and Cozmin Ududec. _Quantifying Biases in LLM-as-a-Judge Evaluations._** ICML 2026, PMLR 306:27120\u201327135; 6\u201311 July 2026. Inspected record, PDF grader and interaction models, and MT-Bench case-study setup. Supports modeling several evaluator effects jointly. Model-based removal of an estimated effect is distinguished from an executed intervention. [Record](https://proceedings.mlr.press/v306/dubois26a.html) \u00b7 [Paper](https://raw.githubusercontent.com/mlresearch/v306/main/assets/dubois26a/dubois26a.pdf).",
      "links": [
        "https://proceedings.mlr.press/v306/dubois26a.html",
        "https://raw.githubusercontent.com/mlresearch/v306/main/assets/dubois26a/dubois26a.pdf"
      ]
    },
    {
      "id": "S6",
      "record": "Akash Bonagiri, Gerard Janno Anderias, Saee Patil, Angelina Lai, Devang Borkar, Gezheng Kang, Ishant Gandhi, Setareh Rafatirad, and Houman Homayoun. _STABLEVAL: Disagreement-Aware and Stable Evaluation of AI Systems._** ICML 2026, PMLR 306:9039\u20139057; 6\u201311 July 2026. Inspected record and PDF Sections 3\u20135 and 7.5. Supports retaining posterior label distributions and treating ranking stability as a distinct objective; conditional independence and identifiability are explicit limits. Its theoretical claims were not re-proved. [Record](https://proceedings.mlr.press/v306/bonagiri26a.html) \u00b7 [Paper](https://raw.githubusercontent.com/mlresearch/v306/main/assets/bonagiri26a/bonagiri26a.pdf).",
      "links": [
        "https://proceedings.mlr.press/v306/bonagiri26a.html",
        "https://raw.githubusercontent.com/mlresearch/v306/main/assets/bonagiri26a/bonagiri26a.pdf"
      ]
    },
    {
      "id": "S7",
      "record": "Ellie Pavlick and Tom Kwiatkowski. _Inherent Disagreements in Human Textual Inferences._** TACL 7:677\u2013694, November 2019; journal posting 14 November 2019. Inspected primary PDF introduction, task definition, and context-manipulation methods. Supports the possibility of persistent, meaningful disagreement in NLI. No claim that all Encevra labels are inherently subjective follows. [Record](https://aclanthology.org/Q19-1043/) \u00b7 [Paper](https://aclanthology.org/Q19-1043.pdf).",
      "links": [
        "https://aclanthology.org/Q19-1043/",
        "https://aclanthology.org/Q19-1043.pdf"
      ]
    },
    {
      "id": "S8",
      "record": "Ido Amit, Ido Galil, and Ran El-Yaniv. _Evaluating LLM Uncertainty in Long-Form Generation Using Deterministic Ground Truth._** ICML 2026, PMLR 306:2371\u20132430; 6\u201311 July 2026. Inspected primary PDF introduction, task setup, unit definitions, and strict-match criterion. Supports executable evaluation anchors and granularity discipline. Reported model results and benchmark code were not reproduced. [Record](https://proceedings.mlr.press/v306/amit26a.html) \u00b7 [Paper](https://raw.githubusercontent.com/mlresearch/v306/main/assets/amit26a/amit26a.pdf).",
      "links": [
        "https://proceedings.mlr.press/v306/amit26a.html",
        "https://raw.githubusercontent.com/mlresearch/v306/main/assets/amit26a/amit26a.pdf"
      ]
    },
    {
      "id": "S9",
      "record": "Zesheng Cai, Yingqi Fan, Sichang Chen, and Jin-Hong Du. _DIAL: Position-Debiased LLM Judges with Adaptive Human Preference Calibration._** arXiv:2609.31215v1, 25 September 2026, preprint. Inspected PDF introduction, comparison design, order-effect model, and declared theoretical scope. Supports separating position debiasing from human-preference alignment. The full theoretical appendix was not audited and no performance result is imported. [Paper](https://arxiv.org/pdf/2609.31215v1).",
      "links": [
        "https://arxiv.org/pdf/2609.31215v1"
      ]
    },
    {
      "id": "S10",
      "record": "Ian Arawjo. _How to Run Statistics over LLM Judges and Trust the Results: Calibrated Inference for Small-Sample AI Evaluation with evalstats._** arXiv:2609.35815v1, 20 September 2026, preprint. Inspected PDF Sections 1 and 6, including simulation design, random-labeling assumptions, and binary-test scope. Supports examining finite-sample behavior and calibration selection. Software and the extensive simulation suite were not run here. [Paper](https://arxiv.org/pdf/2609.35815v1).",
      "links": [
        "https://arxiv.org/pdf/2609.35815v1"
      ]
    },
    {
      "id": "S11",
      "record": "Ryan Lail. _Decomposing LLM-Judge Uncertainty to Target Expert Labels._** arXiv:2609.06444v1, 6 September 2026, preprint. Inspected PDF main text, Bayesian model, both experiments, and limitations. Supports distinguishing uncertainty about a reviewer distribution from disagreement within it; the simple count-based baseline ties the proposed method in the real-data setting. No Encevra transfer is established. [Paper](https://arxiv.org/pdf/2609.06444v1).",
      "links": [
        "https://arxiv.org/pdf/2609.06444v1"
      ]
    },
    {
      "id": "S12",
      "record": "Xin Sun, Di Wu, Sijing Qin, Isao Echizen, Abdallah El Ali, and Saku Sugawara. _Label Effects: Shared Heuristic Reliance in Trust Assessment by Humans and LLM-as-a-Judge._** ACL 2026, pages 32378\u201332392, July 2026. Inspected official record and PDF source-label manipulation, judge prompts, placebo-label control, and trust-rating results. Supports blinding provenance labels; trust ratings in this study are not factual correctness on Encevra tasks. [Record](https://aclanthology.org/2026.acl-long.1495/) \u00b7 [Paper](https://aclanthology.org/2026.acl-long.1495.pdf).",
      "links": [
        "https://aclanthology.org/2026.acl-long.1495/",
        "https://aclanthology.org/2026.acl-long.1495.pdf"
      ]
    },
    {
      "id": "S13",
      "record": "Angela J. Yu and Peter Dayan. _Uncertainty, Neuromodulation, and Attention._** Neuron 46:681\u2013692, 19 May 2005. Inspected author-hosted PDF summary and introduction defining expected and unexpected uncertainty. Used only to motivate a falsifiable separation of evaluation states. No literal brain-copying or validated biological implementation claim is made. [Author-hosted paper](https://www.gatsby.ucl.ac.uk/~dayan/papers/yud2005.pdf) \u00b7 [DOI](https://doi.org/10.1016/j.neuron.2005.04.026).",
      "links": [
        "https://www.gatsby.ucl.ac.uk/~dayan/papers/yud2005.pdf",
        "https://doi.org/10.1016/j.neuron.2005.04.026"
      ]
    },
    {
      "id": "S14",
      "record": "R\u00e9mi Bardenet and Odalric-Ambrym Maillard. _Concentration Inequalities for Sampling Without Replacement._** Bernoulli 21(3):1361\u20131385, August 2015; inspected arXiv:1309.4029v2, 27 July 2015. Inspected setup, Lemma 1.1, Proposition 1.2, and discussion of conservative versus finite-population bounds. Supports the conservative audit-selection envelope; the script does not implement all improved inequalities in the paper. [Inspected paper](https://arxiv.org/pdf/1309.4029v2) \u00b7 [DOI](https://doi.org/10.3150/14-BEJ605).",
      "links": [
        "https://arxiv.org/pdf/1309.4029v2",
        "https://doi.org/10.3150/14-BEJ605"
      ]
    },
    {
      "id": "S15",
      "record": "Charles F. Manski. _Nonparametric Bounds on Treatment Effects._** American Economic Review 80(2):319\u2013323, May 1990. Inspected scanned primary-paper pages 319\u2013320, including the identification setup and bounding argument. Supplies the partial-identification perspective, not the specific Encevra implementation or a statistical confidence guarantee. [Primary-paper scan](https://ipcid.org/evaluation/outros_temas/Manski%20-%20Nonparametric%20Bounds.pdf).",
      "links": [
        "https://ipcid.org/evaluation/outros_temas/Manski%20-%20Nonparametric%20Bounds.pdf"
      ]
    }
  ],
  "retrieved_pdf_digests": [
    {
      "local_source_id": "amit26a",
      "sha256": "49c716c180e2bab95125c50e18cacd2c27db87003cd8f36761e17d4cec468533",
      "bytes": 14773873
    },
    {
      "local_source_id": "bardenet",
      "sha256": "e9e6cda02b46aaa708c1775453640b874ecfcb4cd0ecff132a15f87a37cb6cec",
      "bytes": 607949
    },
    {
      "local_source_id": "bonagiri26a",
      "sha256": "47eca97c48674762d7f8ed1f8a72a5ffad3ac17fcd216b7b7b6f73773c04df71",
      "bytes": 1546267
    },
    {
      "local_source_id": "chen26cq",
      "sha256": "8b28698cc644ee702cde29dcd4943fabcda8ddebec7d0047cdc6a7475e3e86a9",
      "bytes": 890389
    },
    {
      "local_source_id": "dial",
      "sha256": "2f6f30fbf6c9086c431d6823ce726d5bb6339ac7763635ddeb973ebdf1321682",
      "bytes": 5604577
    },
    {
      "local_source_id": "dubois26a",
      "sha256": "0357be63db1b4729d2c4195444853d03f25dfe47b78aa3606e22fa87ee990920",
      "bytes": 3861395
    },
    {
      "local_source_id": "egami",
      "sha256": "758ff9088d48eeedbee295e9fd25ab0b109334f623bcd116ba2575f443f9f8fa",
      "bytes": 1162108
    },
    {
      "local_source_id": "evalstats",
      "sha256": "4ba184d40265373fa878e2a35d23235c6205ce0ebf2c7514ae2ade97f0ef0e1c",
      "bytes": 7991413
    },
    {
      "local_source_id": "judge_uncertainty",
      "sha256": "d9b63d330b9cfbde4d6f61da2bad4f2277cfa0d27500f64ecebbd31377d87a6a",
      "bytes": 484665
    },
    {
      "local_source_id": "label_effects",
      "sha256": "35e3307d3326375bd3e4b388d304fe107fbf219977764909d7ebe0b13876748f",
      "bytes": 22743717
    },
    {
      "local_source_id": "lee26bl",
      "sha256": "c1645e31f0f1b6a183ef15ad201b123fb137447897adcea861420f213faa7752",
      "bytes": 3427451
    },
    {
      "local_source_id": "pavlick",
      "sha256": "6145dd60a513f3426d50718a2301f4189fce10735935ae86d8c01c826f6a6933",
      "bytes": 1509900
    },
    {
      "local_source_id": "ppi",
      "sha256": "a16bf2c108fea9f559d00708c731d29d673c1dcce5d4acfd65e64a694b7ae17a",
      "bytes": 4833991
    },
    {
      "local_source_id": "yudayan",
      "sha256": "f5db3f1173017afc5bee5e656049abe63ab95732cab7c61bf10b8b95b62666a5",
      "bytes": 439116
    }
  ]
}