The AI Review Lottery: Widespread AI-Assisted Peer Reviews Boost Paper Scores and Acceptance Rates

Study 1: Prevalence of AI-assisted reviews

• 15.8% of ICLR 2024 reviews were estimated to be AI-assisted, a lower bound because the estimate was not corrected for GPTZero false negatives.The estimate was based on correcting the detector’s 2024 predictions using its false-positive rate in pre-ChatGPT years.Source: RESULTS · p. 3, Figure 2 · p. 3

Study 2: Effect of AI-assisted reviews on paper scores

• AI-assisted reviews scored 0.14 points higher than human reviews on average (95% CI [0.08, 0.19]), and scored higher in 53.4% of differing-score pairs (p = 0.002; relative probability difference: +14.4%).The score-pair comparison uses reviews assigned to the same paper; the figure plots score differences against human reference scores.Source: RESULTS · pp. 3–4, Figure 3 · p. 4

Study 3: Effect of AI-assisted reviews on acceptance rate

• In the matched-sample analysis, submissions receiving AI-assisted reviews had 13.8% higher odds of acceptance (p = 0.024), or 3.1 percentage points higher chances (p = 0.024).The comparison matched submissions on review counts, human-review scores, and semantic similarity; the regressions adjusted for the matching controls.Source: RESULTS · p. 4
• 4.9 percentage points higher acceptance (p = 0.024) was observed for submissions in the borderline [5, 6) bin receiving an AI-assisted review, corresponding to a 31.1% increase in odds (p = 0.031).The [5, 6) bin contained 20.7% of submissions in the matched sample; the 4.9-point increase was a 6.5% relative increase in acceptance chance.Source: RESULTS · p. 5

DISCUSSION

• AI-assisted reviews were associated with higher scores and acceptance, with acceptance effects especially pronounced for borderline submissions.For submissions averaging 5–6 on human reviews, acceptance increased by 4.9 percentage points; the overall increase was 3.1 percentage points.Source: DISCUSSION · p. 5, Figure 4 · p. 5
• The findings raise concern that AI-assisted reviews may reduce trust in peer review and science, while motivating ongoing measurement as LLMs and their use evolve.The discussion also notes possible heightened risk for work on societal norms and values, given known LLM biases.Source: DISCUSSION · p. 5
• Among submissions receiving only human reviews in the [5, 6) human-score bin, 73.6% were accepted; that bin contained 20.7% of submissions.Source: DISCUSSION · p. 5
• The study measures outcomes rather than review quality, which could remain unchanged even if AI assistance changes how reviews are delivered.The authors describe assessment of review quality as an avenue for future research.Source: DISCUSSION · p. 6
• The findings provide limited insight into whether reviewers use AI to edit self-written reviews or copy LLM-generated reviews.The authors say investigating these usage modes may require interviews, focus groups, and surveys.Source: DISCUSSION · p. 6
• The discussion says LLMs may also help reviewers improve clarity, identify flawed critiques, or contextualize findings, while calling for community participation in guidance.Possible uses also include automated tests related to reproducibility.Source: DISCUSSION · p. 6

MATERIALS AND METHODS

• The dataset contains ICLR submissions and reviews from 2018–2024, with review ratings, textual explanations, and confidence scores.The ratings are ordinal, taking values 1, 3, 5, 6, 8, or 10.Source: MATERIALS AND METHODS · p. 6
• Reviews are labeled AI-assisted when GPTZero assigns a probability of human generation below 0.5, and prevalence is estimated by subtracting an estimated false-positive ratio of 1.7%.The adjustment assumes the false-positive ratio estimated on pre-ChatGPT reviews also applies to 2024 reviews; the authors note this assumption may be biased because GPTZero was not trained on such reviews as human text.Source: MATERIALS AND METHODS · pp. 6–7
• A proportional odds model is used for ordinal review scores, avoiding the assumption that rating categories are equally spaced.The model includes an indicator for AI assistance and estimates an odds ratio.Source: MATERIALS AND METHODS · p. 7
• The score-model odds ratio is expressed as the probability that an AI-assisted review scores higher than a human review in a same-submission pair with different scores.The reported probability is derived from the odds ratio as x = κ/(1 + κ), where κ is the paired-review odds.Source: MATERIALS AND METHODS · p. 7
• Acceptance effects are estimated with two-step matching, pairing submissions with one AI-assisted review to candidates with the same review count and no AI-assisted reviews.Content-based matching uses Sentence-BERT embeddings of abstracts and associated reviews, keeps the highest-cosine-similarity candidate above 0.1, and matches 98.5% of potential pairs.Source: MATERIALS AND METHODS · p. 7
• The authors check matching quality against randomly matched submissions and assess GPTZero robustness through word-level checks and alternative labeling thresholds.They report higher keyword and abstract-embedding similarity for matched submissions, and say results remain robust across threshold values.Source: MATERIALS AND METHODS · p. 7

Appendix A: Dataset

• The collection contains 23,959 main-conference submissions from 46,257 authors and 1,263 institutions, gathered for ICLR submissions from 2018–2024.The table reports submissions and reviews per year, including GPTZero-detected AI-assisted review counts before false-positive correction.Source: Appendix A · p. 10, Table I · p. 10
• The 86,690 reviews include textual evaluations, confidence scores, and overall ratings on the discrete scale 1, 3, 5, 6, 8, and 10.Confidence scores use a consistent one-to-five scale.Source: Appendix A · p. 10
• Ratings 1 and 3 indicate rejection, 5 and 6 mark the acceptance boundary, and 8 and 10 indicate acceptance or strong acceptance.A score of 5 is marginally below the acceptance threshold, while 6 is marginally above it.Source: Appendix A · p. 10

B: On the Robustness of the AI-assisted Labeling Threshold

• Results for review prevalence, score differences, and acceptance remain robust when the AI-assisted labeling threshold is varied from 0.5 to lower values.The authors choose lower thresholds to limit false positives that could arise from classifying human reviews as AI-assisted.Source: B: On the Robustness of the… · p. 11, Figure 5 · p. 11
• Even at a stringent threshold of 0.05, more than 7% of reviews are estimated to be AI-assisted.The prevalence is corrected for the false-positive rate, as in the primary analysis.Source: B: On the Robustness of the… · p. 11, Figure 5 · p. 11
• The score-difference and acceptance analyses show stable effects across alternative labeling thresholds.The appendix reproduces the score-difference and acceptance figures using the varied thresholds.Source: B: On the Robustness of the… · p. 11, Figure 6 · p. 12, Figure 7 · p. 12

• Table II reports ordinal regression coefficients and a model summary for Eq. 3.Source: Table II · p. 13

• Matching requires exact agreement on review count and scores, followed by review–abstract embedding cosine similarity above 0.1.Source: D: On the Effect of LLMs on… · p. 14
• Keyword hits occurred in 19.6% of matched pairs versus 1.0% of unmatched pairs.The unmatched comparisons randomly paired an AI-assisted submission with another ICLR 2024 submission.Source: D: On the Effect of LLMs on… · p. 14
• Matched abstracts had an average BERTScore of 0.836 versus 0.822 for unmatched pairs (p<0.001 in a K-S test).The authors use this comparison to assess the robustness of content-based matching.Source: D: On the Effect of LLMs on… · p. 14
• Tables III and IV report regression coefficients and model summaries, while Tables V and VI report stratified linear and logistic regression coefficients.Source: D: On the Effect of LLMs on… · p. 14, Table V · p. 16, Table VI · p. 17
• The analysis assumes no confounder affects both receipt of an AI-assisted review and acceptance; for borderline papers, Γ is 1.07.Within matched pairs, the reported probability range is 0.48 to 0.52 without changing rejection of the no-effect null at p = 0.05.Source: D: On the Effect of LLMs on… · p. 14
• Across matching thresholds, the estimated acceptance effect for borderline papers remains qualitatively similar to the main analysis.Figure 8 shows the stratified effects; the text reports no statistical difference across thresholds.Source: D: On the Effect of LLMs on… · p. 14, Figure 8 · p. 15

• For each word, the analysis estimates the probability that a review containing it is classified as AI-assisted; Table VII lists the 30 most predictive words.The word dictionary is filtered to nouns, verbs, and adjectives.Source: E: Prevalence Checks · p. 18, Table VII · p. 18

• The analysis collects volunteer names from the official pages of AISTATS, ICLR, ICML, and NeurIPS.Source: F: Overlap of Conference Reviewers and… · p. 19
• Tables VIII and IX report Jaccard similarity across conferences for reviewers and for area chairs and meta reviewers, respectively.The captions define similarity values of 1 as full overlap and 0 as no overlap.Source: F: Overlap of Conference Reviewers and… · p. 19, Table VIII · p. 19, Table IX · p. 19

G: Examples of Matched Submissions

• The table compares abstracts from papers with AI-assisted reviews against abstracts from papers receiving only human reviews.Its examples are randomly sampled from the matches identified by the study’s procedure.Source: Table X · p. 20, G: Examples of Matched Submissions · p. 20