Peer Reviews of Peer Reviews: A Randomized Controlled Trial and Other Experiments

1 Introduction

• Reliable assessments are needed to make review-quality incentives fair and useful, and to evaluate peer-review policy interventions that use quality scores as outcome measures.Some proposed mechanisms rely on authors accurately reporting review quality; intervention studies often treat reviewer or editor assessments as a gold standard.Source: §1 · p. 2
• The study examines review-quality evaluation at NeurIPS 2022 and uses a randomized controlled trial to test whether longer reviews receive higher perceived-quality scores.Participants included paper authors, reviewers, and meta-reviewers.Source: §1 · p. 2
• Review evaluations show inconsistency, miscalibration, subjectivity, and bias toward irrelevant information, echoing problems in reviews of papers.The authors caution that these effects matter when using evaluation scores to incentivize reviewing or measure policy changes.Source: §1 · p. 2

2 Related work

• A prior CVPR study found a weak positive association between review length and authors’ helpfulness ratings, but could not separate content quality from a length effect.The present study uses a randomized trial that lengthens reviews without adding useful information to measure the causal effect on perceived quality.Source: §2 · p. 3
• Earlier author ratings linked acceptance recommendations with review evaluations, but potential quality confounders remained; this study uses non-author ratings to control for them.The passage also notes an algorithm proposed to de-bias author-provided evaluations [WSWS21].Source: §2 · p. 3
• Studies in conference and medical-journal settings assess review quality using meta-reviewers, editors, or authors, while this work examines evaluation reliability at a large computer-science conference.The cited medical-journal work reports editor agreement and an association between author satisfaction and acceptance recommendations.Source: §2 · p. 3
• Funding agencies also assess review quality and use those assessments to discard low-quality reviews or inform future reviewer records.A survey of 30 funding organizations found that over 60% evaluated all reviews as standard practice.Source: §2 · p. 3
• Length bias in human evaluations of scientific reviews parallels reported length bias in evaluations of LLM responses, which may affect RLHF fine-tuning [DLH22] [SGXD23] [DGLH24] [MXC24].The passage states that longer-response preferences can bias models fine-tuned using human feedback toward longer responses.Source: §2 · pp. 3–4

3 Experimental setup

• 7,740 opt-in evaluators rated 9,870 reviews, completing 24,638 evaluations; the participants included meta-reviewers, paper reviewers, authors, and external reviewers.Reviews were assigned to maximize overlap across evaluators, enabling comparisons of multiple ratings of the same reviews.Source: §3 · p. 4
• Evaluators rated overall review quality on a 7-point scale and assessed understanding, coverage, substantiation, and constructiveness on 5-point Likert scales.The criteria were based on proposed Review Quality Indicators and tailored for concise, machine-learning-domain evaluations.Source: §3 · p. 4
• Overall-score distributions were symmetric around a median of 4, while criterion-score distributions were left-skewed toward positive ratings.Figure 1 presents marginal score distributions by evaluator type.Source: §3 · p. 4, Figure 1 · p. 5

A Randomized Control Trial

• 458 external reviewers and meta-reviewers were randomly assigned to evaluate either original reviews or lengthened versions, without being told the experiment’s specific goal.They were eligible because they had not seen the original reviews; the RCT data was not used in the paper’s other analyses.Source: A Randomized Control Trial · pp. 5–6
• For ten selected reviews, researchers added non-informative text while aiming not to improve review quality; mean length rose from 268 to 755 words.The additions included repeated summaries, text-box headers, multiple-choice responses, and paper abstracts; Figure 2 illustrates the procedure and Figure 3 shows word counts.Source: A Randomized Control Trial · pp. 5–6, Figure 2 · p. 5, Figure 3 · p. 6

4 Main results

• The results examine whether uselessly elongated reviews receive biased quality evaluations.This is one of the five analysis dimensions listed for the main results.Source: §4 · p. 6
• The results also analyze authors’ outcome-induced bias and disagreement between evaluators rating the same review.The section list refers to author outcome bias and inter-evaluator agreement or disagreement.Source: §4 · p. 6
• The remaining analyses address evaluator miscalibration and subjectivity in review-quality scores.The section list identifies these as Sections 4.4 and 4.5.Source: §4 · p. 6

4.1.1 Methods

• 458 NeurIPS reviewers and meta-reviewers were randomly assigned to evaluate either the short or elongated version of one of 10 reviews.Each review’s two versions had the same underlying content.Source: §4.1.1 · p. 6

4.1.2 Results

• 4.29 versus 3.73 mean overall scores for long versus short reviews; both reviewer and meta-reviewer groups rated elongated reviews significantly higher.The long and short versions contained the same underlying content.Source: §4.1.1 · p. 6, Table 1 · p. 7, §4.1.2 · p. 7
• All criteria scores increased with length after Holm–Bonferroni correction, with the strongest effect on Coverage.The overall score used a 7-point scale, while criteria scores used a 5-point scale.Source: Table 2 · p. 7, §4.1.2 · p. 8

4.1.1 Methods

• The Mann–Whitney statistic τ compares long- and short-version evaluations within each paper and represents the probability that a long review scores higher, with ties split evenly.Under no effect, τ = 0.5; values above 0.5 indicate a positive length bias.Source: §4.1.1 · p. 7

4.2.2 Results

• τ = 0.82 (p < 0.0001) for overall quality scores: authors rated accept reviews higher than reject reviews in matched pairs.On the 7-point evaluation scale, accept reviews averaged 1.406 points higher.Source: Table 3 · p. 9, §4.2.2 · p. 9

4.2.1 Methods

• The 418 matched pairs held author and paper fixed, limited the review-length ratio to 1.5×, and required at least two non-author ratings within 1 point.The non-author ratings were used to match reviews on perceived quality.Source: §4.2.1 · pp. 8–9

4.2.2 Results

• Accept reviews also scored roughly 1 point higher on the 5-point Understanding, Coverage, Substantiation, and Constructiveness criteria; all differences survived Holm–Bonferroni correction.The authors note that participants had no explicit incentive to rate accept reviews higher.Source: §4.2.2 · p. 9

4.3 Inter-evaluator (dis)agreement

• Consistency is one component of evaluation reliability, but identical median scores could reflect evaluator laziness rather than useful assessment.The authors use consistency to ask whether multiple evaluators give similar review-quality evaluations.Source: §4.3 · p. 9

4.3.1 Methods

• The method compares paired evaluators’ rankings of the same two reviews, discarding ties and counting whether their rankings agree or disagree.The analysis included 25,346 paired-score quadruples for overall scores and 18,193–19,870 for the criteria; 0.5 is the random-baseline agreement rate.Source: §4.3.1 · p. 10

4.3.2 Results

• Disagreement rates were approximately 0.3 for overall and criterion scores, versus 0.25–0.3 for paper-review scores at NeurIPS 2016.The authors note that the domains differ, so the comparison suggests similar agreement rates rather than identical evaluation settings.Source: §4.3.2 · p. 10, Figure 5 · p. 10

4.4.1 Methods

• The linear model represents each review score as objective review quality plus an evaluator offset and evaluation-specific idiosyncrasy.Objective quality is modeled as a normal variable; evaluator offsets are modeled with separate zero-mean distributions for meta-reviewers, reviewers, authors, and opt-in reviewers.Source: §4.4.1 · p. 11
• The authors fit the model by maximum likelihood and use the evaluator-offset variance αb,g to estimate miscalibration by evaluator type.The model has six variance hyperparameters; a larger αb,g indicates greater likelihood of miscalibration for that type.Source: §4.4.1 · pp. 10–11

4.4.2 Results

• Author-offset variance was 0.780, versus approximately 0.45 for other evaluator types; the model comparison suggests review-evaluation miscalibration may be at least as high as paper-evaluation miscalibration.For NeurIPS 2014 paper reviews, the reported normalized evaluator-offset variance was 0.19.Source: §4.4.2 · p. 11, Table 4 · p. 11

4.5 Subjectivity

• Evaluators may differ in the importance they assign to criteria such as coverage, substantiation, and constructiveness when forming an overall score.The authors describe this variation as a potential source of arbitrariness in review evaluation.Source: §4.5 · pp. 11–12

4.5.1 Methods

• Subjectivity is estimated as the absolute difference between overall scores and a learned monotone mapping from criteria scores to overall scores.The authors use mean L(1, 1) loss; higher loss indicates greater variability in how evaluators map criteria to overall scores.Source: §4.5.1 · p. 12
• Using separate mapping functions for each evaluator type did not improve test loss over a combined-evaluator function.Across five 75%–25% train-test splits, combined-function train/test losses were 0.456/0.457, versus 0.448/0.465 for per-type functions.Source: §4.5.1 · p. 12

4.5.2 Results

• Evaluators’ mean loss was approximately 0.45, compared with 0.402 for paper reviews after re-normalizing for the different score scales.The adjustment assumes a linear mapping from the 10-point paper-review scale to the 7-point review-evaluation scale.Source: §4.5.2 · p. 12

5 Discussion and limitations

• Review-quality evaluations show the same inconsistency, bias, miscalibration, and subjectivity as paper reviews, requiring careful interpretation of their scores.Source: §5 · p. 12

5.1 Limitations

• Limitations include possible Hawthorne effects, within-role opt-in selection, a length experiment using only non-original reviewers and meta-reviewers, and different review and paper rating scales.Source: §5.1 · pp. 12–13
• The study could not examine dishonest behavior, which may become a problem if review-quality evaluations carry high stakes such as reviewer awards.Source: §5.1 · p. 13

5.2 Open problems

• For incentives and policy experiments, the authors advise caution with human evaluation scores and identify semi- or fully automated review-quality evaluation as a possible avenue.Source: §5.2 · pp. 13–14
• Review length may be a cheap signal or a genuine signal of quality, so incentive designers need to account for both possibilities.Source: §5.2 · p. 13
• LLM-review comparisons using authors’ ratings should account for recommendation-related bias and the possibility that filler-heavy, longer reviews receive inflated scores.Source: §5.2 · pp. 13–14

• Overall review quality is rated on a seven-point scale from very low to exceptional, with examples at both ends.Source: A Questionnaire for participants · pp. 16–17
• Four criteria assess whether a review demonstrates adequate understanding, covers required aspects, supports its evaluations, and provides constructive feedback.Source: A Questionnaire for participants · p. 17
• Criterion prompts include paper-specific detail, coverage of strengths and weaknesses, grounded arguments, actionable criticism, and polite, unbiased writing.Source: A Questionnaire for participants · p. 17

• The appendix reproduces original and uselessly elongated reviews for two of the 10 RCT papers; both were accepted and publicly available.The other papers and their reviews were not public.Source: B Original and Extended Reviews · p. 17
• The GDRO review praises constraining the DRO uncertainty set to the data manifold and reports convergence and bounded-error rates alongside simulation and real-world evidence.Source: B Original and Extended Reviews · p. 19
• The GDRO reviewer questions whether a training-set manifold covers unseen distributions, potentially limiting out-of-distribution generalization.Source: B Original and Extended Reviews · p. 19
• The review flags per-iteration graph estimation as a possible large-dataset efficiency burden and asks for comparisons with general OOD methods.Source: B Original and Extended Reviews · p. 19
• The second sample describes an invariance-learning method that avoids architectural changes and bilevel optimization, while challenging its claim to recover true invariance.Source: B Original and Extended Reviews · p. 20
• That review says performance trails fixed augmentation, robustness to invariance attacks is not systematically evaluated, and inference requires multiple samples without a cost-accuracy analysis.Source: B Original and Extended Reviews · pp. 20–21