PaperlayerView exact revision on arXiv

Source-linked AI summary

Are We There Yet? Revealing the Risks of Utilizing Large Language Models in Scholarly Peer Review

Rui Ye, Xianghe Pang, Jingyi Chai, Jiaao Chen, Zhenfei Yin, Zhen Xiang, Xiaowen Dong, Jing Shao, Siheng Chen

arXiv:2412.01708v1cs.CLcs.AIcs.HCcs.LG

TL;DR

Peer review is under workload pressure as LLM use grows, yet risks to review reliability remain a concern. This study tests manipulation and inherent flaws in LLM reviews, finding vulnerabilities that argue against relying on LLMs as primary reviewers.

  • Problem

    Peer review faces increasing submissions and substantial labor demands as LLM use grows, while the risks of relying on LLM-generated reviews need scrutiny.

    Source: §1 · p. 2, §4 · p. 13

  • Method

    The study evaluates explicit and implicit manipulation and inherent flaws in LLM reviews, using open-access ICLR 2024 reviews and measures of review consistency and paper ratings.

    Source: §2 · p. 3, §4 · p. 13

  • Results

    Injected content reached 90% agreement; disclosed limitations matched LLM reviews 4.5× more consistently than human reviews, while title-only papers rated higher or comparably than 42% of full papers.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    The authors conclude that LLMs are insufficiently robust to replace human reviewers and recommend limited, supervised use until risks are addressed.

    Source: §4 · pp. 13–14

    Implicit manipulation is difficult to detect or hold authors accountable for.

    Source: §2.2 · p. 8

Abstract

from arXiv · show

Scholarly peer review is a cornerstone of scientific advancement, but the system is under strain due to increasing manuscript submissions and the labor-intensive nature of the process. Recent advancements in large language models (LLMs) have led to their integration into peer review, with promising results such as substantial overlaps between LLM- and human-generated reviews. However, the unchecked adoption of LLMs poses significant risks to the integrity of the peer review system. In this study, we comprehensively analyze the vulnerabilities of LLM-generated reviews by focusing on manipulation and inherent flaws. Our experiments show that injecting covert deliberate content into manuscripts allows authors to explicitly manipulate LLM reviews, leading to inflated ratings and reduced alignment with human reviews. In a simulation, we find that manipulating 5% of the reviews could potentially cause 12% of the papers to lose their position in the top 30% rankings. Implicit manipulation, where authors strategically highlight minor limitations in their papers, further demonstrates LLMs' susceptibility compared to human reviewers, with a 4.5 times higher consistency with disclosed limitations. Additionally, LLMs exhibit inherent flaws, such as potentially assigning higher ratings to incomplete papers compared to full papers and favoring well-known authors in single-blind review process. These findings highlight the risks of over-reliance on LLMs in peer review, underscoring that we are not yet ready for widespread adoption and emphasizing the need for robust safeguards.

1 Introduction

Peer review faces growing workload as LLM use expands, but its risks remain insufficiently understood. Experiments identify manipulation and inherent flaws that challenge LLMs’ reliability as reviewers.

1 Introduction

  • Rising submissions, reviewer shortages, and the labor-intensive process strain peer review, while LLM use has grown alongside studies reporting overlap with human reviews.One cited study reported over 30% overlap on Nature journals.Source: §1 · p. 2
  • The study examines explicit and implicit manipulation alongside inherent flaws, building on three review pipelines previously shown to align substantially with human reviewers.Explicit manipulation embeds nearly invisible white-font content in manuscript PDFs that can be read by PDF parsers; implicit manipulation involves authors highlighting insignificant limitations.Source: §1 · p. 2
  • Injected content produced 90% agreement with LLM reviews, while human alignment fell from 53% to 16%; disclosed limitations aligned with LLM reviews 4.5× more consistently than human reviews.The rating model’s average scores rose from 5.34 to 7.99 after explicit manipulation.Source: §1 · p. 2
  • Title-only papers received higher or comparable ratings than 42% of full papers on average, and well-known authors received more favorable review content in single-blind review.Source: §1 · p. 3
  • The authors conclude that LLMs are not robust enough to serve as primary reviewers and call for safeguards and scrutiny before broader integration.They recommend treating LLM feedback as supplementary to human judgment.Source: §1 · p. 3

2 Results

Using ICLR 2024 reviews, the study tests LLM review reliability through review consistency and 1–10 ratings. It finds vulnerabilities to manipulation and flaws across the examined review systems.

2.1 Explicit Manipulation

  • After injection, review–injection overlap rose from 3.05% to 92.49%, LLM–human consistency fell from 53.29 to 15.91, and estimated ratings rose from 5.37 to nearly 8.The rating estimate was based on a model trained to map reviews to ratings.Source: Table 2 · p. 4, §2.1 · pp. 4–5, Figure 2 · p. 5
  • Replacing 5% of reviews with manipulated LLM reviews could cause 12% of papers to lose their top-30% ranking positions.Ranking shifts became more pronounced as the replacement ratio increased.Source: Figure 3 · p. 5, §2.1 · p. 6

2.2 Implicit Manipulation

  • LLM reviews were 4.5× more consistent with authors’ disclosed limitations than human reviews, making strategically stated weaknesses a potential route to indirectly guide generated feedback.The authors note that authors could prepare responses to disclosed weaknesses before rebuttal.Source: §2.2 · pp. 6–7, Figure 7 · p. 8

2.3 Inherent Flaws

  • Positive ratings increased monotonically with paper length; replacing authors’ affiliations with those of well-known universities, companies, or researchers raised the positive-rating ratio from 36.8% to 40.8%, 41.6%, and 41.2%, respectively.The authorship comparison was conducted in a single-blind setting.Source: §2.3 · p. 9, Figure 10 · p. 9, Figure 9 · p. 9

2.4 Behaviors of Different LLMs

  • LLM–human review consistency correlated positively with Chatbot Arena Score; GPT-4o had the highest consistency and was identified as the most appropriate reviewer candidate.Consistency ranking: GPT-4o > Qwen-2.5-72B-Instruct > DeepSeek-V2.5 > Llama-3.1-70B-Instruct.Source: §2.4 · pp. 10–11, Figure 12 · p. 10
  • Across four LLMs, all were more likely to reiterate disclosed limitations; introduction-content ratings exceeded 95% of full-content ratings, and Llama-3.1-70B-Instruct rated introduction content higher than full content.GPT-4o-0806 and DeepSeek-V2.5 showed a more stepped pattern across title, introduction, and full-paper inputs.Source: §2.4 · pp. 10–11, Figure 11 · p. 10

3 Methods

The methods examine LLM reviewing pipelines through experiments on explicit and implicit manipulation and inherent flaws. They test how manipulated content and incomplete inputs, paper length, and authorship relate to review outputs.

3.1 Leveraging LLMs in Scholarly Peer Review

  • The review system parses papers into key sections, then generates structured feedback on novelty, acceptance, rejection, and suggested improvements.The described pipeline uses ScienceBeam to extract the paper content before prompting the LLM.Source: §3.1 · p. 11

3.2 Explicit Manipulation

  • Explicit manipulation inserts text that tells the LLM to emphasize strengths, minimize weaknesses, and advocate acceptance.The text is placed at the paper’s end in tiny white font, where humans cannot readily see it but the review process extracts it.Source: §3.2 · p. 11

3.3 Implicit Manipulation

  • Implicit manipulation tests whether authors’ disclosed limitations can steer reviews; strategically framing minor or rebuttable weaknesses may pre-empt reviewer concerns.The comparison is that human reviewers assess weaknesses holistically, whereas LLMs may repeat disclosed limitations and let minor issues dominate, skewing reviews toward acceptance.Source: §3.3 · pp. 11–12

3.4 Inherent Flaws

  • The inherent-flaws experiments probe hallucinations using title-only inputs or a title, abstract, and introduction, including whether feedback is coherent but unsupported.The paper also describes manually simulating these incomplete-input scenarios.Source: §3.4 · p. 12
  • The paper-length experiment examines whether LLM reviews favor longer papers by comparing outcomes across papers of varying token lengths.The question is motivated by LLMs’ reported preference for longer responses.Source: §3.4 · p. 12
  • Authorship-bias experiments examine prestige effects in single-blind review by varying affiliations and replacing selected authors with well-known researchers.The affiliation test includes leading universities and AI research organizations; the author-name test uses three Turing Award laureates.Source: §3.4 · pp. 12–13

4 Discussions

The discussion argues that LLMs’ current risks make them insufficiently robust as primary peer reviewers. It calls for safeguards and scrutiny, while allowing a limited, supervised supplementary role.

Discussions

  • The paper identifies explicit manipulation, implicit manipulation, and inherent flaws as risks to address before widespread adoption.It notes that LLMs can produce plausible reviews even when given an empty paper.Source: §4 · p. 13
  • LLMs are not sufficiently robust to replace human reviewers or serve as the sole or primary means of assessment, so the authors advocate a moratorium until risks are better understood and safeguards are in place.The proposed pause would allow further research, development, and policy implementation.Source: §4 · p. 13
  • Organizers are urged to develop detection tools, accountability measures, and penalties addressing author manipulation and reviewers’ replacement of human judgment with LLM-generated content.The proposed measures target both malicious author behavior and reviewer reliance on generated reviews.Source: §4 · p. 13
  • LLMs may support reviewers with supplementary feedback, but should not replace experts’ judgment.The paper cites suggestions to reviewers at ICLR 2025 as an example of supplementary use.Source: §4 · p. 13
  • The authors call for content validation and debiasing algorithms to flag suspicious manipulation and mitigate improper preference.Their conclusion favors limited and supervised use until manipulation risks and inherent flaws are adequately addressed.Source: §4 · pp. 13–14

Ethical Statement

The authors frame the study as an effort to understand LLM-review vulnerabilities and support stronger safeguards and ethical frameworks.

Ethical Statement

  • The study aims to advance understanding of LLM peer-review vulnerabilities, not provide actionable exploitation methods.The authors connect this aim to developing stronger safeguards and ethical frameworks for responsible use.Source: Ethical Statement · p. 14

A Consistency Metric

The consistency metric compares reviews through LLM-based summarization and semantic matching of their criticisms. It expresses matched points as a proportion of points in one review.

A Consistency Metric

  • The metric first uses an LLM to extract each review’s criticisms as key points in JSON; human evaluation of this summarization achieved F1 0.968.The prompt for summarizing a review takes the paper title and review as input and returns key points in JSON.Source: A Consistency Metric · p. 19, Figure 14 · p. 19
  • The matching stage compares JSON key points semantically, retaining matches with similarity scores of 7 or higher; human assessment yielded F1 0.824.The matching output includes paired point IDs, similarity scores on a 5-to-10 scale, and explanations.Source: A Consistency Metric · p. 19
  • Consistency is the number of matched points divided by the total points in one review, measuring how much one review addresses the other’s points.Both stages use GPT-4o.Source: A Consistency Metric · p. 19

B.1 Training The Review2rating LLM

The authors train Review2Rating to translate textual reviews into ratings, then evaluate its ability to reproduce human rating and acceptance preferences. Its rating error against humans is lower than the disagreement between human reviewers.

B.1 Training The Review2rating LLM

  • Review2Rating infers ratings from textual reviews on a 1–10 scale; on the hold-out testing set, it produced similar rating and acceptance preferences, supporting its effectiveness in translating reviews to ratings.Source: §B.1 · pp. 19–20
  • 28,028 ICLR 2024 review samples, each containing strengths, weaknesses, and a rating, are split 9:1 into training and testing sets.The training data are used to adapt Llama-3-8B through supervised fine-tuning; samples with common ratings 3, 5, and 6 are downsampled.Source: §B.1 · pp. 19–20
  • Review2Rating’s error against human ratings is lower than the inter-human rating error.The authors use this comparison to assess whether the model is sufficiently effective at translating reviews into ratings.Source: §B.1 · p. 20, Table 5 · p. 20

B.2 Calculation of Human Rating Discrepancy as Reference

The authors estimate human rating discrepancy by averaging absolute rating differences between pairs of reviewers on the same papers. This average provides a baseline for comparing model performance.

B.2 Calculation of Human Rating Discrepancy as Reference

  • For each paper, the method considers every unique pair of human ratings and computes the pair’s absolute difference.For ratings r_p,i and r_p,j, the difference is |r_p,i − r_p,j|.Source: §B.2 · p. 21
  • The number of unique rating pairs for a paper with N_p ratings is N_p(N_p−1)/2.This is the pair count used when calculating that paper’s discrepancy.Source: §B.2 · p. 21
  • The overall human discrepancy averages the rating differences across reviewer pairs and papers.This provides a baseline against which model performance can be compared.Source: §B.2 · p. 21

C Relationship Between Paper Ranking and Decision Outcome

The authors model the relationship between normalized paper ranking and accept/reject decisions using logistic regression. Higher-ranked papers are more likely to be accepted, and ranking discriminates strongly between decisions.

C Relationship Between Paper Ranking and Decision Outcome

  • The analysis relates normalized paper rankings to binary accept/reject outcomes through a logistic-regression fit with a shaded confidence interval.The figure’s points represent individual papers.Source: C Relationship Between Paper Ranking… · p. 21
  • McFadden’s R2 is 0.5597 and AUC-ROC is 0.9463 for logistic regression using paper ranking to predict accept/reject decisions.The authors interpret these results as showing strong goodness of fit and discriminative power.Source: C Relationship Between Paper Ranking… · p. 21
  • Higher paper rankings correspond to a higher probability of acceptance, with the fitted logistic-regression curve rising steeply at higher rankings.In Figure 18, each point represents a paper; ranking is on the x-axis and accept/reject outcome is on the y-axis.Source: C Relationship Between Paper Ranking… · p. 21

D Case Study

The case studies illustrate implicit manipulation through authors’ disclosed limitations and explicit manipulation through injected manuscript text. The section also points to a relationship between paper ranking and final decisions.

D Case Study

  • Figure 18 depicts the relationship between paper ranking and final decision.The supplied caption identifies the relationship but does not state a result for this case-study section.Source: Figure 18 · p. 23
  • Figures 19–21 present implicit-manipulation cases in which LLMs tend to reiterate limitations disclosed by the authors.The figure captions describe the same tendency across the three cases.Source: D Case Study · p. 22, Figure 19 · p. 23, Figure 20 · p. 24
  • Figures 22–24 provide case studies of explicit manipulation in LLM Review, AI-Scientist, and AgentReview.The cases compare reviews with and without explicit manipulation.Source: D Case Study · p. 22, Figure 22 · p. 25, Figure 23 · p. 26

Related papers

Loading 2412.01708v1…