AgentReview: Exploring Peer Review Dynamics with LLM Agents

Motivation

• Existing analyses struggle to isolate the effects of intertwined review factors and latent reviewer biases, while sensitive data raises privacy concerns.The introduction also identifies variable review quality, unclear reviewer motives, and imperfect mechanisms as persistent challenges.Source: §1 · p. 1

Key findings

• 27.2% lower rating standard deviation after reviewers align with peers, alongside an 18.7% decline in commitment when one reviewer is under-committed.The findings are presented as social influence and altruism fatigue or peer effects, respectively.Source: §1 · p. 2
• Biased reviewers can amplify one another’s negative opinions, while decisions change by 27.7% when reviewers know author identities for only 10% of papers.The authors associate these results with groupthink and echo-chamber effects, and with authority bias and halo effects.Source: §1 · p. 2
• The rebuttal phase has a less significant effect on final outcomes, potentially because reviewers rely heavily on their initial impressions.The authors describe this possible mechanism as anchoring bias.Source: §1 · p. 2

Contributions

• AGENTREVIEW is presented as the first framework to use LLM agents to simulate the entire peer-review process and disentangle the influences of its participants and mechanisms.The framework is described as open and flexible, with customizable reviewer, author, area-chair, and review-mechanism variables.Source: §1 · pp. 1–2, Figure 1 · p. 2
• The simulations produced more than 53,800 reviews, rebuttals, updated reviews, meta-reviews, and decisions, forming a dataset intended to support future peer-review research.The authors also say their findings align with sociological theories and could inform future research.Source: §1 · p. 2

2.1 Framework Overview

• Reviewers, authors, and area chairs are LLM agents; reviewer characteristics include commitment, intention, and knowledgeability, each represented through fixed prompt-based categories.The framework also models author-identity visibility and three area-chair styles: authoritarian, conformist, and inclusive.Source: §2.1 · p. 3

2.2 Review Process Design

• The five-phase pipeline runs from independent reviewer assessments through author rebuttals, reviewer–area-chair discussion, meta-review compilation, and final decisions.Each reviewer initially sees only the manuscript and their own assessment; the simulation fixes acceptance at 32%, with three to four acceptances per batch of ten papers.Source: §2.2 · pp. 3–4

2.3 Data Selection

• Using stratified sampling, the study selected submissions spanning 2020–2023 across oral, spotlight, poster, and rejection categories, yielding a diverse dataset of 350 rejected papers, 125 posters, 29 spotlights, and 19 orals.Source: §2.3 · p. 4

2.4 Baseline Setting

• Because real reviewer variation makes direct rating comparisons potentially inappropriate, the study uses a setting without specified agent characteristics as its baseline for measuring changes in one variable.Across settings, it generates 10,460 reviews and rebuttals, 23,535 reviewer–AC discussions, 9,414 meta-reviews, and 9,414 paper decisions.Source: §2.4 · p. 4

3.1 The Role of Reviewers

• Reviewer experiments varied commitment by replacing normal reviewers with responsible or irresponsible reviewers and reported social-influence, echo-chamber, and halo effects.Source: §3.1 · pp. 4–5

3.1.1 Overview

• Reviewer-rating standard deviation declined after Reviewer-AC discussion, especially when knowledgeable or responsible reviewers dominated; responsible, knowledgeable, and benign reviewers generally gave higher scores.Final ratings improved in most settings following discussions, which the authors describe as addressing reviewer concerns.Source: §3.1.1 · p. 5

3.1.2 Reviewer Commitment

• 18.7%: average review length fell from 195.5 to 159.0 words after reviewer–AC discussion when one irresponsible reviewer was present.Replacing two normal reviewers with irresponsible ones lowered post-discussion average ratings by 0.25, from 5.256 to 5.005; baseline ratings rose 0.06.Source: §3.1.2 · pp. 5–6

3.1.3 Reviewer Intention

• 5.11 to 3.35: average ratings fell as malicious reviewers increased from zero to three; one and two malicious reviewers corresponded to 0.14 and 0.10 declines among normal reviewers.Lack-of-novelty mentions by malicious reviewers were 10.4%, a 182.9% increase over the 3.69% reported for benign reviewers.Source: §3.1.3 · p. 6

3.1.4 Reviewer Knowledgeability

• 24%: less knowledgeable reviewers were more likely to mention insufficient discussion of limitations, while experts gave 6.8% more critiques of experimental validation.Source: §3.1.4 · p. 6

3.2 Involvements of Area Chairs

• Inclusive area chairs aligned most closely with baseline final decisions; authoritarian chairs had Cohen’s Kappa of 0.266 and 69.8% agreement.Conformist chairs showed high semantic overlap with reviewer evaluations, which the authors say might indicate a lack of independent judgment.Source: §3.2 · p. 7

3.3 Impacts of Author Anonymity

• 0.008 Jaccard index: for lower-quality papers, acceptance decisions diverged most when three reviewers knew the authors’ renowned identities.The indices were 0.364, 0.154, and 0.008 for 1, 2, and 3 informed reviewers; changes were more influenced by the number of informed reviewers than the percentage of papers with known identities.Source: Figure 6 · p. 8, §3.3 · p. 8

3.4 Effects of Peer Review Mechanisms

• 0.20 Jaccard index in accepted papers: removing numeric ratings yields minimal overlap with baseline, while removing rebuttals leaves final decisions close to baseline.The paper suggests anchoring and similar quality improvements across submissions as possible explanations for rebuttal removal’s minimal impact.Source: §3.4 · p. 8

Analysis of Peer Review Systems

• Prior studies examine peer-review bias and review quality [Stelmakh et al., 2021; Ugarov, 2023; Liu et al., 2023a], conflicts of interest [McIntosh and Vitale, 2023], fairness [Stephen, 2024], assignments [Jovanovic and Bagheri, 2023; Saveski et al., 2024], and rebuttals [Huang et al., 2023].Source: §4 · p. 8
• These studies primarily analyze real-world review data, making it difficult to isolate how specific factors affect outcomes amid peer review’s complexity and variability.Source: §4 · p. 8

LLMs as Agents

• LLM-agent research includes agent-based modeling systems such as ChatArena, ChatEval [Chan et al., 2023], Lumos [Yin et al., 2023], and MPA [Zhu et al., 2024].Source: §4 · p. 9

• AGENTREVIEW is presented as the first LLM-based framework for simulating the peer review process.Source: §5 · p. 9
• The framework disentangles intertwined factors that affect review outcomes while preserving reviewer privacy.Source: §5 · p. 9
• The authors describe the work as a foundation for more equitable and transparent review mechanisms and suggest studying interactions among variables in future work.Source: §5 · p. 9

• AGENTREVIEW cannot dynamically incorporate or adjust experimental results in response to reviewer comments during the Reviewer-Author Discussion phase because LLMs cannot generate new empirical data.Source: Limitation · p. 9
• The analysis mainly isolates individual variables, although real-world peer reviews involve multiple interacting dimensions.Source: Limitation · p. 9
• The authors did not directly compare simulated outcomes with actual reviews, citing difficulty establishing a consistent baseline amid variation in human reviewer characteristics and review outcomes.Reviewer commitment, intention, and knowledgeability can vary across papers, topics, and time periods.Source: Limitation · p. 9

• The sensitivity and scarcity of real-world review data complicate comprehensive study because of ethical and confidentiality constraints.Source: Ethical Consideration · p. 9
• Reviewer knowledgeability, good intention, and commitment are described as important to accurate, objective, and comprehensive evaluations.Source: Ethical Consideration · p. 9
• Because review is unpaid and time-consuming, reviewers may conduct cursory or superficial evaluations.Source: Ethical Consideration · p. 9
• The authors advise against replacing human reviewers with LLMs, citing their imperfections and the need for human oversight to ensure fair, valuable assessments.Source: Ethical Consideration · p. 9

A.1 Review Categorization

• GPT-4 classifies each line of generated-review acceptance and rejection reasons, creating a new category when an entry does not fit the predefined categories.Source: §A.1 · p. 13
• The categorization yields five distinct reasons for acceptance and seven for rejection.Source: §A.1 · p. 13
• Table 4 is identified as presenting dataset statistics, but the supplied passage does not state any values.Source: Table 4 · p. 13
• Figure 7 shows the distributions of initial and final ratings as the number of reviewers aware of the authors’ prestigious identity varies.Source: Figure 7 · p. 13

A.2 Experimental Costs

• GPT-4-1106-preview was used throughout the experiments for consistent evaluation.The paper says it was selected for its language understanding and generation capabilities.Source: §A.2 · p. 13
• A baseline without specified role personalities provides a consistent reference for testing individual-variable changes.Reusing baseline reviews and rebuttals where applicable reduces run-to-run variability and API costs.Source: §A.2 · p. 13
• Approximately $2780 was spent on API usage across all tests.Source: §A.2 · p. 13

A.4 Behavioral Analysis of LLM Agents

• The example simulated review for “Image as Set of Points” substantially overlaps with the corresponding human review.The paper presents the simulated and real reviews in Tables 6 and 7.Source: §A.4 · p. 14, Table 6 · p. 17, Table 7 · p. 19
• In 90% / 77% / 39% of sampled papers, at least 2 / 3 / 4 of four LLM-generated review points aligned with human reviewers.The 100-paper analysis also reports that LLMs surfaced some insights overlooked by human reviewers.Source: §A.4 · p. 14

A.5 Additional Results and Statistics

• Appendix Table 4 summarizes generated-document word and character counts, while Table 5 reports average ratings as normal reviewers are replaced by malicious reviewers.Source: §A.5 · p. 14
• Figure 7 reports consistently higher average reviewer ratings as more reviewers know prestigious authors’ identities, and after rebuttals.Source: §A.5 · p. 14
• Most simulated reviewer ratings fall in [5, 5.25], while irresponsible and malicious settings show a second peak at [4.25, 4.5].Source: §A.5 · p. 14

A.6 Future Works

• The authors identify LLM limits in generating novel empirical data and capturing nuanced human judgment, and note an initial focus on English-centric venues.They cite English-language prevalence, data availability, and current models’ stronger English performance as reasons for the initial focus.Source: §A.6 · pp. 14–15

A.5 Additional Results and Statistics

• Figure 8 compares Cohen’s Kappa against baseline across paper ratios with known author identities; smaller values indicate lower correlation with baseline.Source: §A.5 · p. 14, Figure 8 · p. 15

A.6 Future Works

• Proposed future work includes broader out-of-distribution and adversarial testing, more ablations, clearer methodological explanations, and fuller discussion of limitations.Source: §A.6 · p. 18