Source-linked AI summary
Can LLM feedback enhance review quality? A randomized study of 20K reviews at ICLR 2025
Nitya Thakkar, Mert Yuksekgonul, Jake Silberg, Animesh Garg, Nanyun Peng, Fei Sha, Rose Yu, Carl Vondrick, James Zou
TL;DR
Rapidly rising AI-conference submissions have strained peer review and raised concerns about review quality. The paper evaluates Review Feedback Agent, a guarded multi-LLM system that provides optional feedback on problematic review comments in a randomized ICLR 2025 study. Feedback recipients updated reviews, produced clearer and more informative reviews, and engaged more during rebuttals, although the targeted categories may not cover every aspect of review quality.
Problem
Rapidly increasing submissions have strained peer review, motivating evidence on whether automated feedback can improve review quality.
Method
Review Feedback Agent uses multiple LLMs and reliability tests to provide optional feedback on vague comments, misunderstandings, and unprofessional remarks.
Results
26.6% of reviewers who received feedback updated their reviews, incorporating 12,222 suggestions, while revised reviews were judged clearer and more informative.
Takeaways & Limitations
The intervention was associated with longer reviews and more engaged rebuttal discussions, supporting its potential to improve review specificity and engagement at scale.
Takeaways & Limitations
The feedback categories may not capture all aspects of review quality, and the authors identify expanding them as future work.
Abstract
from arXiv · showhide
Peer review at AI conferences is stressed by rapidly rising submission volumes, leading to deteriorating review quality and increased author dissatisfaction. To address these issues, we developed Review Feedback Agent, a system leveraging multiple large language models (LLMs) to improve review clarity and actionability by providing automated feedback on vague comments, content misunderstandings, and unprofessional remarks to reviewers. Implemented at ICLR 2025 as a large randomized control study, our system provided optional feedback to more than 20,000 randomly selected reviews. To ensure high-quality feedback for reviewers at this scale, we also developed a suite of automated reliability tests powered by LLMs that acted as guardrails to ensure feedback quality, with feedback only being sent to reviewers if it passed all the tests. The results show that 27% of reviewers who received feedback updated their reviews, and over 12,000 feedback suggestions from the agent were incorporated by those reviewers. This suggests that many reviewers found the AI-generated feedback sufficiently helpful to merit updating their reviews. Incorporating AI feedback led to significantly longer reviews (an average increase of 80 words among those who updated after receiving feedback) and more informative reviews, as evaluated by blinded researchers. Moreover, reviewers who were selected to receive AI feedback were also more engaged during paper rebuttals, as seen in longer author-reviewer discussions. This work demonstrates that carefully designed LLM-generated review feedback can enhance peer review quality by making reviews more specific and actionable while increasing engagement between reviewers and authors. The Review Feedback Agent is publicly available at https://github.com/zou-group/review_feedback_agent.
1 Introduction
Rapidly increasing AI-conference submissions have strained peer review, with vague and inconsistent reviews contributing to author dissatisfaction. This study introduces a randomized ICLR 2025 deployment of Review Feedback Agent, finding that optional LLM feedback improved review informativeness and engagement.
- 47% and 61% year-over-year submission increases at ICLR in 2024 and 2025 intensified challenges in obtaining constructive peer reviews.
- 12.9% of reviews at the 2023 ACL meeting were flagged for poor quality, primarily because of vague, surface-level criticisms.
- Review Feedback Agent uses multiple LLMs to provide feedback targeting vague or unjustified comments, content misunderstandings, and unprofessional remarks.
- 26.6% of reviewers who received feedback updated their reviews, incorporating 12,222 suggestions, while blinded researchers judged revised reviews more informative and clearer.
- Feedback recipients produced reviews averaging 80 additional words and longer author-reviewer rebuttal discussions, indicating greater engagement.
- The study presents a large-scale deployment intended to make reviews more actionable and informative and promote a more constructive scientific process.
2 Methods
The ICLR 2025 experiment randomly varied review access to an optional feedback system that identifies specific review problems and generates guarded, reviewer-facing suggestions. Its five-LLM pipeline processed review and paper text, with feedback posted only after reliability checks passed.
- Feedback targets: The agent targeted vague or generic critiques, misunderstandings addressable from the paper, and unprofessional statements.It encouraged specificity, highlighted relevant paper sections, or requested rephrasing, respectively.
- Experiment design: The experiment randomly assigned papers to groups in which none, half, or all reviews could receive feedback.Feedback was triggered after initial submission and delayed by one hour, while reviewers retained the option to revise their reviews.
- Feedback delivery: Feedback was visible only to the reviewer and ICLR program chairs, and reviewers could ignore it or revise their review without automated changes.
- Experiment design: 18,946 reviews received feedback over four weeks, while 2,692 selected reviews needed no feedback and 829 failed reliability tests.Each review took roughly one minute to process and cost around 50 cents.
- System architecture: The system used paper text and review sections as input, excluding reviewers’ initial scores, and generated comment-feedback pairs through a five-LLM pipeline.
- System architecture: Two Actors generated feedback, followed by an Aggregator, Critic, and Formatter; the final output had to pass reliability tests before posting.The pipeline allowed at most two attempts and returned an error after a second failed attempt.
3 Results
The randomized ICLR 2025 study found that feedback increased review updates and length, and incorporated suggestions were associated with clearer, more informative reviews and greater rebuttal engagement. Feedback primarily targeted vague or underspecified comments, while acceptance rates did not change significantly.
- Study design: Randomization assigned reviews to feedback or control groups, with updates defined using an edit-distance threshold greater than 5.The feedback group included reviews selected to receive feedback, including those that did not ultimately receive it.
- Review updates: 26.6% of reviews that successfully received feedback were updated, compared with 9.4% in the control group, a difference of roughly 17 percentage points.The feedback group included 18,946 successfully delivered reviews, of which 5,031 were updated.
- Review changes: Being selected for feedback increased average review length by about 14 words, while reviewers who updated after feedback added 80 words compared with 2 words for non-updaters.Review length covered the summary, strengths, weaknesses, and questions sections; the 14-word estimate is diluted by non-delivery and non-updating.
- Feedback incorporation: 89% of reviewers who updated after receiving feedback incorporated at least one suggestion, and reviewers incorporated 12,222 feedback items overall.Incorporation was assessed with an LLM-based pipeline applied to updated reviews.
- Review quality: Blinded researchers preferred modified reviews 89% of the time in a 100-example evaluation of updated reviews with incorporated feedback.The evaluation focused on reviews receiving 3–4 feedback items where incorporated feedback exceeded 0.60.
- Feedback content: Feedback mainly addressed vague comments and requests for specific analyses, baselines, or references; incorporation rates ranged from 14% to 18% across clusters without significant differences.Overall, 17.7% of feedback items were incorporated, and the agent rarely addressed content misunderstandings because it required high confidence and direct evidence.
4 Related Works
Prior work shows growing use of LLMs across peer review, but this study addresses the previously unexamined use of LLMs to provide feedback directly on reviews at scale.
- 10.6% of reviewers at ICLR 2024 reportedly used LLMs to assist with drafting peer reviews.
- 57.4% of surveyed researchers found GPT-4-generated paper feedback helpful, while 82.4% considered it more useful than feedback from at least some human reviewers.
- Human evaluators of peer reviews were highly susceptible to bias from review length and paper score, indicating substantial subjectivity in assessment.
- No prior study had specifically examined LLM-generated feedback targeting vague comments, content misunderstandings, and unprofessional remarks in peer reviews.
5 Discussion
The study reports that optional LLM feedback improved review specificity, clarity, and actionability while increasing reviewer engagement, although the evaluated feedback categories do not cover every aspect of review quality.
- 27% of reviewers updated their reviews after receiving feedback, and blinded AI researchers judged updated reviews clearer and more informative.
- The feedback intervention was designed to preserve reviewer control because feedback was optional and reviewers retained authority over final reviews and scores.
- The feedback categories focused on specificity, misunderstandings, and professionalism, so they may not capture all aspects of review quality.
- The randomized controlled study was motivated by the limited rigorous quantification of earlier machine-learning applications in peer review.
A Agent Prompts
The agent prompts instruct LLMs to identify only substantive review problems and provide targeted, evidence-based suggestions that make comments more specific, actionable, and professionally phrased.
- The prompts instruct the agent to evaluate each comment in the context of the complete review and focus on weaknesses or reviewer questions rather than summaries or strengths.
- For suspected content misunderstandings, the agent should quote the relevant paper section exactly and ask whether it addresses the reviewer’s concern.
- Actionable review comments should identify where an issue occurs, state the concern explicitly, explain why it matters, and suggest concrete examples or remedies.
- The prompts tell the agent not to nitpick and to avoid feedback on comments that are already specific and actionable.
- Questions that are already clear and specific generally receive no feedback, because reviewer confusion need not itself be actionable.
- The agent also addresses vague claims, such as brittleness or poor writing, by requesting specific examples, explanations, or proposed remedies.
B Reliability tests
The Review Feedback Agent used real-time LLM reliability tests to prevent feedback that merely praised or restated reviews, addressed authors instead of reviewers, or had formatting errors. These guardrails filtered feedback toward actionable suggestions for improving review comments.
- Reliability test design: The reliability suite screened generated feedback in real time before delivery to reviewers.The tests were designed to identify feedback that failed quality criteria.
- Reliability test design: Feedback had to provide critical suggestions rather than simply praise the review.The test rejected praise that did not identify ways to improve the reviewer’s comment.
- Reliability test design: The system verified that feedback addressed reviewers with suggestions for improving their reviews, not authors with advice about improving papers.This distinction kept the feedback focused on review quality rather than manuscript revision.
- Reliability test design: The system rejected feedback that merely restated a reviewer’s comment without adding meaningful, unique suggestions.This test targeted feedback that sounded constructive but supplied no new actionable guidance.
- Reliability test design: A formatting test protected against malformed feedback caused by pipeline errors.
C Average score changes during review and rebuttal periods
During both the review and rebuttal periods, reviewers selected for feedback did not change their scores more than control reviewers. Within the feedback group, reviewers who updated their reviews showed distinct score changes, especially during rebuttal.
- Review period: No significant difference in score updating appeared between feedback and control groups during the review period.The comparison covered soundness, presentation, contribution, rating, and confidence between initial and modified reviews.
- Review period: Among feedback recipients, reviewers who updated their reviews were more likely to decrease soundness and increase confidence scores.
- Rebuttal period: No significant difference in score updating appeared between feedback and control groups during the rebuttal period.
- Rebuttal period: During rebuttal, reviewers who updated their reviews significantly increased every score except confidence compared with non-updaters.The increases covered soundness, presentation, contribution, and rating.
D Incorporation model validation
The incorporation model compared original reviews, modified reviews, and individual feedback items to determine whether suggestions were incorporated. Its validation showed high agreement with human labels, while the score-change figure contextualized how feedback recipients’ reviews evolved.
- Score-change context: During the review period, feedback and control groups showed no significant difference in score updating, while feedback-group updaters differed from non-updaters in soundness and confidence changes.
- Incorporation model: The incorporation model used original review text, modified review text, and an individual feedback item to classify whether the suggestion was incorporated.
- Model validation: 92% of test feedback items were labeled successfully by the incorporation model against human-annotated ground truth.The figure reports that most false positives and all false negatives reflected human annotation errors caught by the model.