When Your Reviewer is an LLM: Biases, Divergence, and Prompt Injection Risks in Peer Review

1 INTRODUCTION

• The work addresses concerns about overly positive feedback, missed research significance, confidentiality, and covert instructions embedded in submitted documents.The stated questions cover model performance, differences in evaluative focus, and whether embedded prompts manipulate reviews.Source: §1 · p. 2
• The study compares human reviews, GPT-5-mini reviews, and reviews with injected prompts across 991 ICLR 2023 and 450 NeurIPS 2022 papers.Source: §1 · p. 2
• GPT-5-mini systematically inflates ratings for papers rated lower by humans, while giving more calibrated evaluations for highly rated papers.Its ratings are also sensitive to reviewer-provided prompts.Source: §1 · p. 2
• Humans and LLMs diverge moderately in identifying strengths and weaknesses: humans emphasize study novelty and clarity, while LLMs focus on empirical rigor and implementation.Source: §1 · p. 2
• Overarching embedded instructions modestly inflate ratings and leave strengths and weaknesses largely intact, while field-specific instructions can substantially bias outputs.Source: §1 · pp. 2–3
• The authors frame the findings as informing design and policy for responsible LLM integration, with LLMs serving as calibrated assistants rather than judges.Source: §1 · p. 3

2.1 Peer Review Practices and Challenges

• Rising submission volumes, reviewer-recruitment difficulties, limited time, and expertise mismatches strain peer review and its quality.Source: §2.1 · p. 3

2.2 LLMs in Peer Reviewing

• Prior work reports helpful LLM review outputs but also rating inflation and technical biases; a comprehensive benchmark of reviewing performance remains missing.Source: §2.2 · p. 4

2.3 Prompt Injection Attacks

• Document-based prompt injection can exploit hidden instructions in submissions, but systematic analysis of this vulnerability in peer review remains unexplored.The study assesses whether embedded prompts manipulate review outcomes and which aspects are most vulnerable.Source: §2.3 · p. 4

3 METHODS

• The framework compared official human reviews, LLM reviews, and injection-conditioned LLM reviews, then used bottom-up topic clustering to compare their content.Source: §3 · pp. 4–5

3.1 Set 1 Construction: Human Review Collection

• A stratified sample of 1,441 ICLR 2023 and NeurIPS 2022 papers supplied official reviews with ratings, strengths, and weaknesses.Sampling was based on meta-review decisions; paper topic areas were also extracted for fine-grained analysis.Source: §3.1 · pp. 5–6

3.2 Set 2 Construction: LLM Review Generation

• GPT-5-mini generated structured reviews using venue-specific rating exemplars drawn from papers whose human reviewers unanimously agreed on the score.The model was instructed to assign a score only when the target matched or exceeded the corresponding reference paper.Source: §3.2 · pp. 6–7

3.3 Set 3 Construction: LLM Review with Prompt Injection

• Review Set 3 tested overarching and field-specific instructions embedded through TrueType glyph remapping that camouflaged prompts to humans while preserving their intended text for LLMs.Field-specific instructions targeted review elements such as ratings or the number of reported weaknesses.Source: §3.3 · p. 7
• The overarching injection instructed the model to override prior instructions, give a positive review, and omit negatives.Source: §3.3 · p. 7
• To test location effects, the same prompt was placed at four PDF positions: the first page, quarter point, midpoint, and last page.Source: §3.3 · pp. 7–8
• To test repetition, prompt frequency varied across r = 3, 5, 7, and 9 at each location, examining reinforcement alongside detection risk.Source: §3.3 · p. 8
• Field-specific injections targeted two review fields: assign the highest possible rating (10/10) or limit the review to one weakness.These prompts tested whether embedded instructions could inflate ratings or suppress critical feedback.Source: §3.3 · p. 8

3.4 Topic Clustering

• Review strengths and weaknesses were condensed into one-to-five-word labels, then modeled with BERTopic using sentence embeddings, UMAP, and K-Means.The elbow method selected 80 clusters, and c-TF-IDF identified representative keywords and documents.Source: §3.4 · pp. 8–9
• Two authors manually organized the 80 clusters into eight high-level themes, reaching 100% agreement before co-author consensus.Source: §3.4 · p. 9
• The unified theme scheme was applied to both strengths and weaknesses to support consistent comparisons across review groups.Source: §3.4 · pp. 9–10

3.5 Ethics Statement

• The study excluded reviewer identifiers from public OpenReview data and focused on aggregated patterns, making attribution to individual reviewers infeasible.The authors state that this approach aligns with common IRB interpretations exempting such research from formal human-subjects review.Source: §3.5 · p. 10

4.1 Performance of LLMs in Reviewing Research Papers

• Across 1,441 papers, LLMs averaged 6.86 versus 5.70 for human reviewers, a 1.16-point higher rating.The authors describe this as an inflationary tendency in LLM evaluations.Source: §4.1 · p. 10, Figure 2 · p. 11
• For ICLR papers rated 3.5 by humans, 97.67% received higher LLM ratings (+2.48 on average), versus 65.62% and −0.34 for papers rated 7.5.The authors conclude that LLMs overestimate weaker papers but give more calibrated evaluations of stronger ones.Source: §4.1 · p. 11
• LLMs correctly classified 89/100 notable-top-25% and 294/300 poster papers, but only 62/91 notable-top-5% papers; 479/500 rejected papers were rated acceptable.Using LLM ratings as acceptance decisions would have classified 95.8% of originally rejected papers as acceptable at least at poster level.Source: §4.1 · p. 11
• More than 50% of papers in General Machine Learning, Reinforcement Learning, and Applications fell within [–1, 1), while Theory, Optimization, and Social Aspects showed wider positive rating gaps.These track-level differences indicate closer alignment in some areas and greater LLM overrating in others.Source: Figure 3 · p. 12, §4.1 · p. 12
• Without reference papers, only 2% of submissions received rating 5 versus 15% with references; under tougher instructions, about 73% received rating 6.The authors report that small instruction changes drastically shifted rating distributions, raising concerns about robustness and reliability.Source: Table 2 · p. 12, §4.1 · p. 13

4.2 Comparative Analysis of Review Content

• The review-content analysis compares Review Set 1 and Review Set 2 using two complementary analyses.Source: §4.2 · p. 13
• Both reviewer types treated novelty of study design as a central strength and technical implementation as the most frequent weakness; novelty appeared in under 5% of weakness comments.This pattern positions novelty mainly as a positive differentiator rather than a perceived deficiency.Source: §4.2 · p. 14
• Figure 4 shows humans more often highlighted novelty of study design and presentation clarity as strengths, while LLMs more often focused on quality of technical implementation.The reported shares were 29% vs. 17% for novelty and 15% vs. 5% for clarity (Human-Strength vs. LLM-Strength); technical implementation was 16% vs. 11%.Source: §4.2 · p. 14
• JSD between LLM and human reviewers was 0.031 for strengths and 0.043 for weaknesses, indicating moderate divergence overall and greater divergence in weaknesses, consistent with humans emphasizing clarity.LLMs cited quality of technical implementation in 31% of reviews, versus 24% in human reviews.Source: §4.2 · p. 14
• 69.6% of papers had mid-range (0.3–0.6) topic similarity for strengths and 64.5% for weaknesses, indicating moderate paper-level overlap.Aggregate topic alignment therefore did not ensure that LLMs captured the same aspects as humans on individual papers.Source: §4.2 · pp. 14–15, Figure 5 · p. 14
• Human reviews used a broader, negatively skewed lexicon, whereas LLM reviews relied on a narrower, more polarized vocabulary of praise and recurring technical critiques.Human negative terms ranged from “unclear” and “issue” to “comparison” and “baseline”; LLM critiques included “assumption,” “hyperparameters,” and “computational.”Source: §4.2 · p. 16

4.3 Impact of Prompt Injection

• 7.21 was the average rating under overarching injection, approximately 0.35 above baseline, while strength and weakness JSDs were only 0.0003 and 0.0010.The injected reviews also retained lexical patterns and sentiment structure close to those of the base model.Source: Figure 6 · p. 15, §4.3 · p. 16
• 62.7% of papers received a rating of 8 with first-page injection, versus 61.1% with last-page and 59.2% with middle placement.First-page placement yielded 903 papers rated 8, compared with 879 for last-page placement; the reported pattern was U-shaped.Source: §4.3 · pp. 16–17, Figure 7 · p. 17
• Average scores stayed near 7.25–7.29 as prompt repetition varied from 3 to 9, and counts of papers rated 8 varied only from 908 to 935.The reported average scores for repetition counts 3, 5, 7, and 9 were 7.29, 7.27, 7.26, and 7.25, respectively.Source: §4.3 · p. 17, Figure 7 · p. 17
• Once an injection influenced the model, further repetitions did not substantially increase its effect, suggesting the model accepted or rejected instructions rather than responding incrementally.The passage presents this as a possible explanation of the repetition results.Source: §4.3 · p. 17
• Overarching prompts caused only modest rating inflation and left strengths and weaknesses largely unchanged at topic and textual levels.The prompts also did not induce extreme ratings such as 10/10.Source: §4.3 · p. 17
• GPT-5-mini assigned 10/10 in 30% of cases and reduced reviews to a single weakness in 30% under field-specific prompts.Both prompts were inserted three times per paper.Source: §4.3 · p. 18
• GPT-4o-mini assigned 10/10 to nearly 57% of papers and reported only one weakness for up to 81%, making it more susceptible than GPT-5-mini.The comparison indicates that newer versions show some resistance, but it remains incomplete.Source: §4.3 · p. 18

5.1 Reconciling Utility and Vulnerability in LLM Review

• LLMs can generate structured strengths and weaknesses and approximate human ratings on stronger submissions, but systematically inflate ratings on weaker papers.Source: §5.1 · p. 18
• Human reviews emphasize novelty and clarity while LLM reviews prioritize empirical rigor and technical detail, making LLMs better suited as supplements than judges.Source: §5.1 · p. 18
• Field-specific prompt injections can coerce extreme ratings or suppress weaknesses despite modest effects from overarching instructions, while GPT-5-mini resists attacks better than GPT-4o-mini.Source: §5.1 · p. 19

5.2 Implications for Policy and Design

• Prompt injection forced a perfect 10/10 score in 30% of cases, motivating policies that classify hidden instructions as misconduct and require platform defenses.Source: §5.2 · p. 19
• Because outputs are unstable and topical overlap makes them a poor proxy for expert judgment, the authors propose AI-use disclosure to keep human reviewers accountable.Source: §5.2 · p. 19
• A review co-pilot could assign repetitive technical checks to an LLM and reserve novelty and impact judgments for humans, but must address automation bias and clarify AI behavior.Source: §5.2 · p. 20
• A bias-calibration dashboard could display calibrated score ranges and flag generic keywords, but may anchor Area Chairs on scores over flaws in model reasoning.Source: §5.2 · p. 20

5.3 Limitations and Future Work

• The experiments primarily examined GPT-5-mini, so comparisons across more current and earlier models are needed to clarify review biases and how injection robustness changes.Source: §5.3 · p. 20
• The dataset covers only computer-science venues (ICLR and NeurIPS), leaving performance in fields with different evaluation criteria untested; future work could broaden models and domains.This could provide a fuller understanding of LLM reviewing capabilities, biases, and robustness to prompt injection risks.Source: §5.3 · p. 20

A Prompt Design for LLM-Based Reviewing

• The evaluation uses 1,441 papers from ICLR 2023 and NeurIPS 2022 to compare human reviews, LLM-generated reviews, and injection-conditioned LLM reviews.The review prompt compares each target paper with uploaded references scored 1, 3, 5, 6, 8, and 10.Source: §6 · p. 20, A Prompt Design for LLM-Based Reviewing · p. 24

6 CONCLUSIONS

• LLMs inflate ratings for weaker papers but align more closely with human reviewers on stronger ones.Content analysis also finds moderate divergence: humans emphasize novelty and clarity, whereas LLMs prioritize empirical rigor and implementation.Source: §6 · pp. 20–21
• Field-specific embedded instructions can coerce extreme ratings and suppress weaknesses, while overarching prompts cannot restructure generated reviews.The conclusion recommends safeguards and calibrated use of LLMs as reviewer aids rather than replacements.Source: §6 · p. 21

B Review Comment Labeling Prompt

• The review-generation prompt requests structured fields for strengths and weaknesses, along with a rating and confidence.The labeling prompt summarizes each strength or weakness in 1–5 words for topic clustering.Source: A Prompt Design for LLM-Based Reviewing · p. 24, B Review Comment Labeling Prompt · p. 25

C Topic Modeling Results

• Strength-comment clusters include themes such as novel framing and interpretation, problem reformulation, clear motivation, and novel, well-motivated ideas.Table 3 presents topic IDs, representations, and annotated topics; the supplied rows include several problem-framing and motivation clusters.Source: Table 3 · p. 25, C Topic Modeling Results · p. 26

2. Novelty of Study Design

• Study-design novelty clusters cover models and architectures, training, datasets, privacy and security, theory, and methodology.The section defines novelty as the novelty of proposed models, architectures, theoretical concepts, or technical approaches.Source: §2 · p. 26

3. Rigor of Empirical Evaluation

• Empirical-evaluation themes include robustness, efficiency, baselines, experimental validation, ablations, reproducibility, and evaluation breadth.The section frames rigor in terms of the thoroughness, breadth, and strength of experimental evidence used to validate claims.Source: §3 · p. 26

4. Quality of Theoretical Analysis

• Theoretical-analysis themes include rigorous proofs, convergence guarantees, theoretical grounding, and substantial or practical theoretical contributions.The section defines this dimension by the soundness, depth, and significance of formal proofs, theoretical grounding, and analytical contributions.Source: §4 · p. 27

5. Quality of Technical Implementation

• Technical implementation themes include interpretability, methodology, algorithms, reproducibility, diagnostics, and theoretical or algorithmic contributions.Source: §5 · p. 27

6. Significance of Potential Impact

• Impact themes emphasize actionable insights and broad applicability or usefulness.Source: §6 · p. 27

7. Presentation Clarity

• Presentation themes include clarity, visualization, readability, writing quality, and transparent, reproducible research.Source: §7 · p. 27

2. Novelty of Study Design

• Novelty feedback ranges from incremental contributions to limited or absent novelty.Source: §2 · p. 28

• Reported concerns include strong or restrictive assumptions, modeling assumptions, and limits on applicability and scope.Source: §3 · p. 28
• Other themes flag missing proofs or limited theoretical justification, analysis, explanation, and grounding.Source: §3 · p. 28
• The topic list also includes approximation error, limited guarantees, and convergence-scope concerns.Source: §3 · p. 28

• Evaluation concerns include insufficient experiments, comparisons, and ablation analysis.Source: §4 · p. 28
• Other concerns include missing baselines, limited metrics and human evaluation, and insufficient real-world validation.Source: §4 · pp. 28–29
• The themes flag limited dataset diversity, task and domain generalization, evaluation breadth, and experimental scale.Source: §4 · pp. 28–29

5. Quality of Technical Implementation

• Technical concerns include computational cost and scalability, hyperparameter sensitivity, robustness, and limited failure-mode analysis.Source: §5 · p. 29

6. Significance of Potential Impact

• Impact-related themes question practical applicability and the work's usefulness.Source: §6 · p. 29

7. Presentation Clarity

• Presentation concerns include notation and methodological clarity, writing, figures, reproducibility, and insufficient experimental detail.Source: §7 · pp. 29–30

8. Rigor of Ethical Evaluation

• Ethical-evaluation themes identify limited analysis of bias and fairness, alongside missing privacy evaluation.Source: §8 · p. 30