PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Checklists to Reduce Lottery Effects in Scientific Peer Review

Xuanyou Liu, Ziyang Guo, Jason Hartline, Jessica Hullman

arXiv:2610.28666v1cs.DLcs.AIstat.AP

TL;DR

Reviewer lottery error arises when a small set of reviews attends to only some issues that a larger pool of eligible reviewers would raise. The paper learns coarsened checklists from historical reviews and selects one using correlated agreement; in LLM-simulated reviews, the learned checklist reduces error on both datasets.

  • Problem

    A small set of peer reviews may attend to only some issues that a larger pool of eligible reviewers would raise, contributing to reviewer lottery error.

    Source: §1 · p. 1

  • Method

    The approach groups paper-resolvable atomic claims from historical reviews into a hierarchy of checklist items and selects a coarsening that maximizes correlated agreement.

    Source: §1 · p. 2

  • Results

    58% on NeurIPS 2021 and 21% on ICLR 2022: the learned checklist reduces review error for a typical three-reviewer committee in simulated review.

    Source: §1 · p. 2

  • Takeaways & Limitations

    For committees of three or more reviewers on both datasets, the checklist with the highest correlated agreement also has the lowest error.

    Source: §1 · p. 2

    A shared representation can introduce bias by causing reviewers to underweight their private information.

    Source: §1 · p. 2

Abstract

from arXiv · show

Reviewer lottery error describes error due to a small set of peer reviews attending to only some of the issues with a paper that a larger pool of eligible reviewers would raise. We show this error can be reduced by surfacing a checklist of criticisms learned from a historical review corpus. Our approach generates a tree of checklists representing coarsenings of the atomic review claim space, then selects the coarsening that maximizes a theoretically-motivated measure of correlated agreement among reviewers. We realize this approach with an LLM-assisted pipeline and test it on NeurIPS 2021 and ICLR 2022 review data. In simulated review with LLM reviewers, the learned checklist reduces the mean squared error (MSE) of a three reviewer committee by 58% on NeurIPS and 21% on ICLR, and on both venues the checklist with the highest correlated agreement has the lowest error for committees of three or more. In a human study with 20 participants, the learned NeurIPS checklist lowers the MSE of a paper ranking, measured against the NeurIPS 2021 order, from 0.48 to 0.22.

1 Introduction

Reviewer lottery error can arise because individual reviewers cover only part of a paper’s evaluation space. The paper learns checklist representations from historical reviews, selects among them using correlated agreement, and evaluates their effect on review error.

1 Introduction

  • Reviewer lottery variance can arise because any single reviewer covers only a small portion of a paper’s evaluation space.A different assignment of qualified reviewers may therefore produce a different outcome.Source: §1 · p. 1
  • Higher human-review loads associated with increased scientific production from LLMs are described as increasing noise and contributing to a review death spiral.The passage links this spiral with greater incentives for authors to submit weak papers.Source: §1 · p. 1
  • A shared checklist can provide common baseline checks while reviewers retain judgment and expertise, but shared representations risk causing them to underweight private information.The paper frames checklist design as balancing reproducible variance reduction against added shared bias.Source: §1 · pp. 1–2
  • The design objective selects a coarsened review-claim representation that maximizes correlated agreement as a proxy for reducing lottery variance without adding shared bias.Correlated agreement compares agreement between independent reviews of the same paper with expected agreement between reviews of different papers; historical data cannot estimate decisions under checklists reviewers did not observe.Source: §1 · p. 2
  • The LLM-assisted pipeline decomposes historical reviews into paper-resolvable atomic claims, groups them into named checklist items, and selects their granularity.Claims are labeled by the source of information that would resolve them, and are required to be settled by the paper or its supplementary material.Source: §1 · p. 2
  • 58% on NeurIPS 2021 and 21% on ICLR 2022: the learned checklist reduces review error for a typical three-reviewer committee.On NeurIPS 2021, it also has lower error than the official checklist at every committee size; on both datasets, the highest-correlated-agreement checklist has the lowest error for committees of three or more.Source: §1 · p. 2

2 Related Work

Prior work documents reviewer-lottery effects across fields and explores computational interventions, while AI review research examines both verifiable feedback and risks such as hallucination and bias.

2 Related Work

  • Lottery effects have been documented in grant funding and machine-learning peer review, while proposed computational interventions have tended to target assignment, calibration, and dishonest behavior rather than reviewer attention.The cited NeurIPS 2021 experiment found substantial variation in accepted papers across two non-communicating committees reviewing the same 882 papers (Beygelzimer et al., 2023).Source: §2 · p. 2
  • AI review studies report overlap with human comments and strong performance on targeted verifiable questions, while newer systems ground comments in quoted passages or other evidence.Because large-scale human peer-review studies are infeasible, evaluation approaches include recall of implanted defects and LLM simulations of aggregate feedback and outcomes (Jin et al., 2024).Source: §2 · p. 2
  • AI reviewers can be gameable, hallucinate, and attend unevenly across review dimensions; consistent summary representations may also lead reviewers to undervalue private knowledge.The cited examples include over-weighting technical validity and under-weighting novelty (Shin et al., 2025).Source: §2 · p. 3

3 Problem: Learning a Checklist to Reduce Lottery Variance

The section frames checklist design as coarsening verifiable review claims to reduce reviewer lottery variance without adding enough shared bias to raise error. It motivates this design with evidence of limited issue overlap and refuted criticisms in peer review.

3.1 Reviewer Lottery Variance

  • The NeurIPS 2021 dual-committee analysis uses overlap between issues raised by independent committees to estimate claim-level lottery variance and separately examines incorrect criticisms.The authors describe this as a novel claim-level analysis.Source: §3.1 · p. 3
  • One reviewer raised 6.4% of distinct issues further reviewers from the same pool would raise; only 34.3% of claims overlapped across committees, 81% of issues had one reviewer, and the paper refuted 18.2% of judged concrete claims.The coverage estimate used Chao2 richness. The respective 95% confidence intervals were [5.9, 6.9] for coverage, [32.8, 35.8] for overlap, and [16.3, 20.2] for refutation.Source: §3.1 · pp. 3–4

3.2 Reducing Variance By Surfacing a Common LLM-Completed Checklist

  • A shared checklist surfaces binary claims resolvable from paper materials, which can reduce reviewer variance but may shift assessments through shared attention or weighting.Checklist-assisted review improves finite-sample estimation when its variance reduction exceeds the added squared bias.Source: §3.2 · pp. 4–5

3.3 Checklist Selection

  • Because historical reviews do not reveal counterfactual checklist assessments, the method selects coarsened clusters of verifiable claims using within-paper agreement minus across-paper agreement.The objective rewards reproducible overlap among reviews of the same paper while subtracting overlap across papers.Source: §3.3 · pp. 5–6
  • Under monotone, submodular MSE reduction, positive stand-alone benefits for ECA-contributing signals, and limited signal redundancy, the selected checklist approximates the optimal checklist.The approximation ratio is α = (1 − κ)/ρ and approaches optimality as ρ approaches 1 and κ approaches 0.Source: §3.3 · p. 6

4 An LLM-assisted Pipeline for Learning Checklists

The pipeline learns a hierarchy of paper-checkable checklist items from historical reviews and selects a granularity by maximizing expected correlated agreement (ECA). It applies this process to NeurIPS 2021 and ICLR 2022 review corpora.

4 An LLM-assisted Pipeline for Learning Checklists

  • The pipeline maximizes expected correlated agreement, Asame − Adiff, to select a cut through the checklist hierarchy.Cutting near the root yields fewer, higher-level items; cutting near the leaves yields more specific items.Source: §4 · pp. 6–7
  • LLMs split reviews into atomic claims, label their type and resolving source, filter for paper-settled facts or requests, and help merge and name checklist items.The remaining claims are embedded, merged when judged exactly the same, and clustered with a Ward tree.Source: §4 · pp. 6–7
  • The pipeline is demonstrated on the NeurIPS 2021 dual-committee corpus and the ICLR 2022 review dataset.The NeurIPS subset contains 298 papers with two independent committees per paper; the ICLR dataset contains 300 selected papers.Source: §4 · p. 8
  • ECA is highest at k=100 on both corpora (+10.2 points on NeurIPS and +12.9 on ICLR), so the authors select that granularity.Coverage, recall, and Adiff decrease as k decreases, consistent with lower-k checklists being more general.Source: §4 · p. 8

5 Evaluation: Do the Selected Checklists Reduce Reviewer Error?

The evaluation tests whether learned checklists reduce review error in LLM-simulated committees and a small human ranking study. Simulated results show lower error for small committees, while the human study also finds lower ranking MSE with a checklist.

5 Evaluation: Do the Selected Checklists Reduce Reviewer Error?

  • A stable human-review population target would require dozens of independent reviewers, and no existing dataset met the study’s data requirements.The authors therefore use a larger LLM-simulated study and a smaller human validation experiment.Source: §5 · p. 8

5.1 Simulation Experiment

  • The simulation uses held-out papers from both venues, with LLM reviewers assigned limited coverage of concerns and scores compared against an estimated population target.The no-checklist condition is unbiased by construction, while checklist error combines variance and squared bias; checklist effects are tested across several granularities.Source: §5.1 · pp. 8–9

5.2 Simulation Results

  • The k=100 learned checklist lowers single-reviewer MSE from 0.82 to 0.21 on NeurIPS and 0.84 to 0.34 on ICLR, halving NeurIPS committee-of-four MSE (0.10 vs. 0.21).On both venues, k=100 has the lowest MSE for every committee of three or more reviewers.Source: §5.2 · p. 9
  • A checklist lowers single-reviewer MSE on NeurIPS from 0.82 to 0.21 and halves it for four reviewers (0.10 vs. 0.21), with the no-checklist committee catching up at about N=10.The gain comes from lower variance at the price of a small bias; on ICLR, the no-checklist committee catches up at N=5.Source: §5.2 · p. 9, Figure 3 · p. 10

5.3 Human Study

  • The human study uses rankings of two sets of three papers, with topic and checklist condition counterbalanced; pairwise correctness against the NeurIPS order defines the error measure.The authors describe this ranking target as less contestable than a small-sample average of human scores.Source: §5.3 · p. 10
  • 0.48 to 0.22: the checklist lowers ranking MSE in a human study with 20 participants; for a committee of three, MSE is 0.11 with the checklist and 0.32 without.The ranking target is the NeurIPS 2021 paper order.Source: §5.3 · p. 10

6 Conclusion

The paper formalizes checklist design as choosing a coarsened representation of review claims that reduces lottery variance without adding more shared bias than it removes. It builds an LLM-assisted pipeline and reports lower simulated and human ranking error with the selected checklist.

6 Conclusion

  • A shared checklist helps a small committee only when it removes more lottery variance than the bias it adds.The paper frames checklist design as selecting a coarsening of the review-claim space.Source: §6 · p. 10
  • The LLM-assisted pipeline turns historical reviews into paper-checkable checklist items and chooses granularity by maximizing ECA.ECA measures how much more two independent reviews of the same paper agree than reviews of different papers.Source: §6 · p. 11
  • In simulated review, the selected checklist reduces three-reviewer committee error by 58% on NeurIPS and 21% on ICLR; in a 20-participant human study, ranking MSE falls from 0.48 to 0.22.On both venues, the highest-ECA checklist has the lowest error for committees of three or more.Source: §6 · p. 11

A Full assumptions and proof for Theorem 3.1

The theorem selects a coarsening of verifiable review claims by maximizing correlated agreement, then bounds its MSE benefit under assumptions about reviewer behavior and signal redundancy. The guarantee weakens with variation in signal value and redundancy, and an empirical estimate of agreement yields a related conditional bound.

Full assumptions and proof for Theorem 3.1

  • The selection objective maximizes same-paper agreement over independently drawn-paper agreement across signals in a candidate checklist.Each candidate checklist partitions verifiable atomic claims into signals; the objective is Δ(g) = Asame(g) − Adiff(g).Source: A Full assumptions and proof for… · pp. 14–15
  • The theorem assumes representative historical reviews and nonnegative signal contributions to correlated agreement, with positive contributions linked to positive stand-alone MSE benefits.Under representative sampling, the agreement objective decomposes across signals.Source: A Full assumptions and proof for… · p. 15
  • It also assumes adding checklist signals cannot increase MSE, marginal benefits diminish, and the selected checklist retains enough stand-alone benefit on average.The limited-redundancy condition constrains average loss, not every signal’s individual marginal contribution.Source: A Full assumptions and proof for… · pp. 15–16
  • Under these conditions, the selected checklist achieves at least (1 − κ)/ρ of the best attainable MSE reduction; ρ = 1.25 and κ = 0.2 yield a 0.64 fraction.ρ captures relative variation in stand-alone MSE benefit per unit of agreement, while κ bounds average diminishing value.Source: A Full assumptions and proof for… · pp. 15–17
  • When ρ = 1 and κ = 0, maximizing correlated agreement is MSE-optimal within the candidate checklist family.More generally, the bound stays positive when ρ is bounded and average redundancy stays below one.Source: A Full assumptions and proof for… · p. 17
  • A sharper finite-size bound improves the main factor, while estimated agreement within ε of population agreement gives a 2ε objective-loss bound under the selected-checklist redundancy condition.The finite-size result uses the maximum candidate checklist size; the empirical bound requires A4 for the selected checklist.Source: A Full assumptions and proof for… · pp. 17–18
  • The guarantee depends on actual reviewer behavior: different reasoning, attention overload, or systematic misinterpretation may violate its conditions.The theorem does not derive these conditions from reviewer rationality.Source: A Full assumptions and proof for… · pp. 16, 18

B Claim-level analysis of the NeurIPS 2021 reviews

The NeurIPS 2021 analysis estimates how much review issues individual reviewers and committees surface, and how much separate committees’ claims overlap. It finds substantial unseen issue coverage and limited cross-committee agreement.

B.1 Data and definitions

  • The corpus contains 298 papers reviewed by two noncommunicating committees, and LLMs split reviews and decision notes into atomic claims labeled by statement type, resolution source, and missing-paper assumptions.The analysis includes 25,868 claims: 23,920 reviewer claims and 1,948 area-chair claims; concrete claims are paper-checkable, conceptual claims are evaluations, and out-of-scope claims require other work or the world.Source: §B.1 · p. 18
  • Strict matches merge repeated claims within a reviewer, while loose matches across reviewers group different descriptions of the same issue, with LLM review splitting groups that combine distinct issues.Loose matching can connect claims across layers, including a concrete claim and a conceptual claim.Source: §B.1 · pp. 18–19

B.2 What share of a paper’s issues does a single reviewer raise?

  • One reviewer raises about 20% of observed issues, while all reviewers together raise 31% of the estimated issue pool; a committee of three or four covers a little over half the observed set.The Chao2 estimate is about 90 issues for a typical paper; the mean of 124 is pulled upward by a few papers.Source: Table 2 · p. 19, §B.2 · p. 19

B.3 Do two committees raise the same issues?

  • One claim in three is raised at some level of detail by the other committee, while exact restatements cover one in eight claims.Loose overlap is 40.4% for conceptual claims and 30.9% for concrete claims.Source: §B.3 · p. 19

B.4 Are the criticisms right?

The manuscript check counts a claim as refuted only when the paper supplies quoted evidence against it, while its upheld verdict requires a searched-for absence. On the checked papers, refuted claims were common, and the camera-ready version limits what can be inferred about the submissions reviewers saw.

B.4 Are the criticisms right?

  • A claim is refuted only with quoted evidence from the manuscript; an upheld verdict requires searching at least three terms without finding the claimed material.Partial, unsettleable, and unclear verdicts are not counted as wrong, making the refuted rate conservative under this rule.Source: §B.4 · pp. 19–20
  • The check used camera-ready PDFs rather than the submissions reviewers read, and 89.1% of 1,164 refuted claims assumed something was missing.Because accepted papers could be revised after review and submitted PDFs are unavailable, the authors cannot determine which effect is larger: conservative counting or later additions.Source: §B.4 · p. 20
  • 18.2% of judged concrete claims were refuted by the paper; among clear verdicts, reviewer and AC claims were refuted at similar rates, 34.2% and 34.3%.Claims also raised by the other committee were upheld and refuted at rates within 0.8 points of claims it did not raise, with overlapping intervals.Source: §B.4 · p. 20
  • The checker matched its answer key on 48 of 50 verdicts (96%, Wilson 95% CI [86.5, 98.9]).The check used 25 sentences turned into omission criticisms, tested against the source paper and an unrelated paper.Source: §B.4 · p. 20

B.5 What does the area chair keep?

Area chairs’ notes retain issues more often when multiple reviewers raise them, regardless of whether the manuscript check upholds, partially supports, or refutes them. The appendix also describes the LLM pipeline’s claim extraction, matching, checking, hierarchy construction, and recall evaluation.

B.5 What does the area chair keep?

  • Just over half of AC claims trace to a review (54.6% loose, 22.9% strict); traced claims are refuted at 16%, versus 23% of AC additions, with overlapping intervals.Most traced claims match one reviewer (59%), while 29% match two and 12% match three or more.Source: §B.5 · p. 21
  • 56.5% of issues raised by three or more reviewers reach the AC note, compared with 5.7% of issues raised by one reviewer; correctness makes no difference.The respective confidence intervals are [51.2, 61.7] and [5.1, 6.4]; upheld, partial, and refuted issues are each kept at similar rates within reviewer-count groups.Source: §B.5 · p. 21

C.1 Claim extraction

  • The pipeline splits reviews into spans, rewrites checkable claims as statements about the paper, and labels their type, what would settle them, and assumed absences.The extraction and labeling steps use multiple passes; claims settled only by literature or the world are out of scope for the paper-checkable layers.Source: §C.1 · pp. 21–22

C.2 Same-issue pair labelling

  • Within a reviewer and layer, only exact duplicates are merged; across reviewers, loose matches count as the same issue, with oversized linked groups split when they combine distinct issues.Loose overlap includes exact matches and one claim being more specific than another.Source: §C.2 · p. 23

C.3 Manuscript check

  • The manuscript checker requires an eight-word verbatim quote to refute a claim and at least three search terms to uphold an alleged absence.It distinguishes refuted, partial, valid, out-of-scope, and unclear claims; positive errors such as incorrect table entries receive no upheld label.Source: §C.3 · pp. 23–24

C.4 Building the hierarchy

  • The hierarchy is built from held-out concrete claims: exact duplicates are merged, nearest-neighbor pairs are judged, and Ward clustering forms a tree whose cuts define checklist granularity.An LLM names checklist items at each cut as short questions based on the cluster and sampled claims.Source: §C.4 · p. 24

C.5 Recall against verified issues

  • Recall measures how often checklist items surface 32 upheld human criticisms from 16 papers, with each checklist cut tested in a separate run.A target counts as found when the LLM flags a gap on the checklist item containing it.Source: §C.5 · p. 25

C.6 The scoring study

The scoring study compares checklist-assisted and unaided simulated reviewers against a population score estimated from all unaided reviewers. It also describes the held-out human-study papers and additional analyses of target stability and score spread.

C.6 The scoring study

  • Each simulated reviewer writes criticisms before scoring, and the two conditions use the same scoring prompt apart from the condition name.Separating criticism writing from scoring is intended to keep the model from choosing a score first and writing a review to match.Source: §C.6 · pp. 25–26
  • Without a checklist, reviewers receive separate nonoverlapping branches; with one, eight reviewers per paper receive the same checklist in different random orders.The no-checklist condition models first-round review and skips branches about rebuttal and revision.Source: §C.6 · p. 25
  • For each paper and condition, variance, bias, and MSE are computed from observed scores, then averaged equally across 30 papers per venue, with squared paper-level bias.Source: §C.6 · p. 26

C.7 Human study papers

  • The six human-study papers are three NeurIPS 2021 pruning papers and three adversarial-robustness papers, all held out when the checklist tree was grown.Source: §C.7 · p. 26

D Additional results of the scoring study

D.1 Stability of the population target

  • The population target is each paper’s average score across all unaided simulated reviewers, used to assess how closely committees of different sizes approach it.Source: §D.1 · p. 26

D.2 ICLR results

On ICLR 2022, the k=100 checklist reduces single-reviewer variance, but its biases vary more across papers than on NeurIPS and unaided committees catch up sooner.

D.2 ICLR results

  • 0.84 to 0.18: the k=100 checklist cuts single-reviewer variance on ICLR, with mean bias of −0.06.Source: §D.2 · p. 26
  • 0.16 versus 0.07: ICLR checklist mean squared bias exceeds NeurIPS, reflecting wider per-paper bias spread.Source: §D.2 · p. 26
  • N=5: the no-checklist committee catches up sooner on ICLR than on NeurIPS; at N=4, the checklist is still marginally ahead, 0.205 against 0.209.Source: §D.2 · p. 26

D.3 Simulated and human reviewer variance

The simulated reviewers reproduce human score spread reasonably well at the venue level, though they do not reproduce which papers split reviewers. Committee-average scores also move closer to the all-reviewer target as committee size grows.

D.3 Simulated and human reviewer variance

  • Typical committee-mean deviations from the all-reviewer target narrow from about ±1.1 points for one reviewer to ±0.55 for four and ±0.17 for all reviewers.The all-reviewer value reflects remaining uncertainty in the target itself.Source: Figure 4 · p. 27
  • On NeurIPS, simulated score variance (0.84) is close to the variance within one human committee (0.93), and the simulated curve stays inside the one-committee band at every committee size.Pooling the two human committees raises variance to 1.50; the simulated curve’s gap from the one-committee band is small next to the drop to the k=100 checklist curve.Source: §D.3 · pp. 27–28, Figure 6 · p. 28
  • On ICLR, simulated variance is 0.86 versus human variance of 1.44, but the ratio is 1.69 [0.87, 2.70] and the simulated curve stays within the human band at every committee size.Human scores use a six-value scale, whereas simulated reviewers use a 1-to-10 scale; the human interval is wide because papers have about four reviewers.Source: Figure 6 · p. 28, §D.3 · p. 28
  • Per-paper human and simulated variances are uncorrelated, while their mean ratings track paper ordering with Spearman correlations of 0.79 on NeurIPS and 0.77 on ICLR.The two NeurIPS human committees also have weak per-paper variance correlation (0.09).Source: §D.3 · p. 28
  • A single human rating has MSE 2.123 on NeurIPS and 2.874 on ICLR against the LLM-defined target, a comparison that does not establish human bias.Mean human ratings are 0.38 points below the target on NeurIPS and 0.33 points above it on ICLR.Source: §D.3 · p. 29

Related papers

Loading 2610.28666v1…