Source-linked AI summary
Insights from the ICLR Peer Review and Rebuttal Process
Amir Hossein Kargaran, Nafiseh Nikeghbal, Jing Yang, Nedjma Ousidhoum
TL;DR
As conference submissions grow, the effect of rebuttals on peer-review outcomes remains important to understand, especially given prior evidence that author responses have only marginal impact on final scores. This study analyzes ICLR 2024 and 2025 reviews and rebuttals, finding that changes are concentrated among borderline papers and are strongly predicted by initial and co-reviewer scores.
Problem
The study examines how rebuttals and reviewer dynamics relate to score changes, against prior research suggesting author responses generally have only a marginal impact on final scores.
Source: §1 · pp. 1–2Method
Results
Rebuttals primarily affect borderline papers, while initial scores and co-reviewer ratings are the strongest predictors of score changes; roughly one in five accepted submissions likely benefit from rebuttal-driven score increases.
Source: §1 · p. 2, §6 · p. 11Takeaways & Limitations
Evidence-backed clarifications, rather than generic or vague defenses, are more likely to result in positive score changes.
Source: §1 · p. 2The findings are specific to ICLR, the venue on which the study focuses.
Source: Limitations · p. 11
Abstract
from arXiv · showhide
Peer review is a cornerstone of scientific publishing, including at premier machine learning conferences such as ICLR. As submission volumes increase, understanding the nature and dynamics of the review process is crucial for improving its efficiency, effectiveness, and the quality of published papers. We present a large-scale analysis of the ICLR 2024 and 2025 peer review processes, focusing on before- and after-rebuttal scores and reviewer-author interactions. We examine review scores, author-reviewer engagement, temporal patterns in review submissions, and co-reviewer influence effects. Combining quantitative analyses with LLM-based categorization of review texts and rebuttal discussions, we identify common strengths and weaknesses for each rating group, as well as trends in rebuttal strategies that are most strongly associated with score changes. Our findings show that initial scores and the ratings of co-reviewers are the strongest predictors of score changes during the rebuttal, pointing to a degree of reviewer influence. Rebuttals play a valuable role in improving outcomes for borderline papers, where thoughtful author responses can meaningfully shift reviewer perspectives. More broadly, our study offers evidence-based insights to improve the peer review process, guiding authors on effective rebuttal strategies and helping the community design fairer and more efficient review processes. Our code and score changes data are available at https://github.com/papercopilot/iclr-insights.
1 Introduction
The paper examines how rebuttals and reviewer dynamics relate to score changes in ICLR peer review. Its analyses find that rebuttal effects are concentrated among borderline papers, with initial and co-reviewer ratings strongly predicting changes.
1 Introduction
- Rising submission numbers and the effort rebuttals require motivate examining their effectiveness in conference peer review.Prior research cited by the authors suggests author responses generally have only a marginal impact on final scores.Source: §1 · p. 1
- The study investigates how often rebuttal score changes affect papers, when rebuttals occur, how reviewer comments relate to ratings, and which author responses are associated with score changes.The authors analyze ICLR 2024 and 2025 data using statistical methods and LLM-based techniques.Source: §1 · p. 2
- Rebuttals primarily affect borderline papers, and initial scores and co-reviewer ratings are the strongest predictors of rebuttal score changes.The authors estimate that roughly one in five top papers benefit through rank improvements; evidence-backed clarifications are more likely to accompany positive score changes than generic or vague defenses.Source: §1 · p. 2
- Reviewer disagreements decrease by roughly 9–10% after rebuttals, with strongest convergence for oral or spotlight papers and minimal effects for low-scoring or rejected papers.Rebuttals submitted in the middle of the period may be most effective, while very early or last-minute responses have less impact on engagement and score changes.Source: §1 · p. 2
- Low-rated papers tend to draw criticism for writing, experimental, and methodological flaws, while high-rated papers are recognized for novelty and methodological soundness.The analyses also report that late reviewer submissions may correlate with slightly higher scores, though the effect is minimal.Source: §1 · p. 2
2 Corpus
The corpus combines publicly accessible ICLR reviews, author responses, and decisions with archived scores from before and after rebuttals. It covers the full review cycle for ICLR 2024 and 2025, with some pre-rebuttal score records missing.
2 Corpus
- The corpus draws on a venue where reviews, author responses, and final decisions are publicly accessible, including for rejected papers.The authors note this transparency avoids survivorship bias in peer-review studies using ICLR data.Source: §2 · p. 2
- ICLR reviews include written feedback and ratings, followed by author rebuttals and reviewer discussion before final decisions.Submissions are typically reviewed by three to five reviewers; overall ratings in ICLR 2024 and 2025 use {1, 3, 5, 6, 8, 10}.Source: §2 · p. 2
- Figure 1 describes submitted-paper distributions by average overall rating and final decision, while Figure 2 describes top-threshold displacement before and after rebuttals in 2024 and 2025.Source: Figure 1 · p. 3, Figure 2 · p. 3
- The corpus contains over 19,000 papers and over 74,000 reviews; fewer than 5% of papers lacked recorded pre-rebuttal scores because of release-time technical issues.The authors archived data at initial review release and again after the rebuttal period because OpenReview displays final versions that overwrite preliminary reviews.Source: §2 · p. 3
3 Empirical Analysis of Review and Rebuttal Dynamics
Rebuttals mostly leave review scores unchanged, but score shifts can alter rankings and are concentrated in borderline cases. Review activity follows deadlines, while reviewer disagreement generally decreases after rebuttal.
3.1 Effect of Rebuttals
- Most review scores stayed unchanged (81% in 2024; 75% in 2025), while increases were more common than decreases and concentrated in borderline cases.Papers with increased scores were more likely to be accepted than papers with unchanged or decreased scores, though many still were not accepted.Source: §3.1 · pp. 3–4
- Over 40% of papers in the pre-rebuttal top 5% were displaced after score updates in both years, making rebuttal-related shifts most evident among top-ranked papers.Nearly 20% of papers lost a position in the top 30%; rankings are only a proxy for acceptance and omit meta-review factors.Source: §3.1 · p. 4
3.2 Temporal Dynamics of Reviews and Rebuttals
- Review scores varied by submission order: later-submitted reviewers tended to assign higher scores, while papers consistently received both high and low scores.The pattern was reported for ICLR 2024 and 2025; most papers had four reviews.Source: §3.2 · p. 5
- Reviewer activity peaked just before review deadlines, then rose again near the rebuttal deadline alongside author activity.The activity pattern was broadly similar in ICLR 2024 and 2025, with reviewers especially active on the final day in 2024.Source: §3.2 · p. 5
- Nearly one-third of first author messages submitted November 18–24 were followed by a reviewer score increase; very early and late messages were less often associated with increases.The same broad timing pattern also appeared in ICLR 2024, with some differences.Source: §3.2 · pp. 5–6
3.3 Co-Reviewers Influence
- The analysis measures co-reviewer influence as shrinking reviewer-score disagreement, but without unexposed controls it cannot attribute convergence solely to that influence.Disagreement is the absolute score difference for reviewer pairs, normalized by the score range when comparing before and after rebuttal.Source: §3.3 · p. 6
- Many review-score changes, mostly increases, occurred within the first hour after reviews were released.The analysis treats review modification timestamps at release as the starting point.Source: §3.3 · p. 6
- Average reviewer disagreement fell 9–10% overall, with larger reductions for spotlight and oral submissions and generally below 7% for rejected and withdrawn submissions.Reductions were 29% and 26% for spotlight submissions and 41% and 48% for oral submissions in 2024 and 2025, respectively.Source: §3.3 · pp. 6–7, Table 3 · p. 7
4 LLM-Based Analysis of Review and Rebuttal Dynamics
The analyses relate review content and ratings to score patterns, then examine which rebuttal strategies are associated with score changes. Evidence-backed clarification is associated with increases, while vague or evasive responses tend to accompany unchanged scores.
4.1 Reviewer Evaluation Patterns
- Higher ratings brought fewer perceived weaknesses and more recognized strengths; low ratings emphasized writing and experimental flaws, while high ratings emphasized novelty and methodology.Mid-range reviews balanced strengths and weaknesses, often describing promising but underdeveloped contributions.Source: §4.1 · p. 7
- GPT-4o labeled review strengths and weaknesses using an expert-refined taxonomy, with analysis sampled across 4,000 papers from ICLR 2024 and 2025.The taxonomy covered review aspects including novelty, motivation, methodology, experiments, results, data, and presentation.Source: §4.1 · pp. 7–8
- Using all features yielded macro-F1 0.49 for rating classification, compared with 0.43 using word counts alone.The model grouped scores 1 and 3 as Low, 8 and 10 as High, and treated borderline scores 5 and 6 separately.Source: §4.1 · p. 8
- Overall ratings correlated most strongly with contribution, followed by soundness; they correlated positively with strengths-section length and negatively with weaknesses-section length.Higher-confidence reviews tended to have longer weaknesses sections and lower overall ratings.Source: §4.1 · p. 8, Figure 8 · p. 8
4.2 Common Rebuttal Strategies and Outcomes
- The three-class score-change model reached macro F1 0.52, and evidence-backed clarification was associated with increases while vague or evasive defenses tended to accompany unchanged scores.Initial ratings and co-reviewer scores were also important features; reviewer engagement could accompany either an increase or a decrease.Source: §4.2 · p. 9, Table 7 · p. 10, Table 8 · p. 10
5 Related Work
Prior research has studied review content, score prediction, and author-reviewer interaction. This work categorizes review strengths and weaknesses by rating group and examines how reviewer attributes and rebuttal strategies relate to scores and score changes.
5 Related Work
- Prior review studies examined review quality, tone, deficiencies, usefulness, disagreement, and score or acceptance prediction from textual features.Examples include measures of review length and quality, politeness, and score prediction (Kang et al., 2018; Fernandes & Vaz-de Melo, 2022; 2024).Source: §5 · p. 10
- This work categorizes review strengths and weaknesses by rating group and models scoring patterns, extending prior work with a larger ICLR corpus and analysis of timing and rebuttal outcomes.Its interpretability is used to identify review attributes associated with reviewer decisions.Source: §5 · pp. 10–11
- Gao et al. (2019) analyzed ACL 2018 score changes using a corpus with consent-related coverage concerns, while Huang et al. (2023) studied ICLR 2022 rebuttals without releasing data or score changes.The passage reports that 69% of authors withheld consent to share responses in the Gao et al. corpus.Source: §5 · p. 11
6 Conclusion
Analyzing ICLR 2024 and 2025 reviews with statistical methods and LLM-based text categorization, the study finds that rebuttals mainly affect borderline papers. Initial scores, co-reviewer ratings, and response strategies are associated with score changes.
6 Conclusion
- Approximately 20% of accepted submissions likely benefited from rebuttal-driven score increases, which primarily affected borderline papers.Initial reviewer scores and co-reviewer ratings were the strongest predictors of rebuttal score changes.Source: §6 · p. 11
- Reviewer disagreements narrowed after rebuttals for high-scoring submissions, while low-scoring papers remained largely unaffected.Source: §6 · p. 11
- Evidence-backed clarifications and precise responses were most strongly associated with positive score changes; vague or defensive rebuttals had little effect.Source: §6 · p. 11
- The findings offer authors guidance on rebuttals and inform program chairs considering fairness and efficiency in review-process design.Source: §6 · p. 11
Limitations
The study’s findings are descriptive and correlational, and their scope is limited by venue, data, and analysis choices. The authors identify several boundaries while noting steps taken to mitigate some limitations.
Limitations
- The reported relationships are correlational and descriptive, not causal.The analyses cannot disentangle causal effects from underlying confounding factors.Source: Limitations · p. 11
- Findings are specific to ICLR and are tailored to that venue.The study does not establish that the same patterns hold at other venues.Source: Limitations · p. 11
- A small subset of discussions or score-change records may be missing, though the authors judge these gaps unlikely to meaningfully affect findings.Examples include unavailable discussions referenced in meta-reviews and records lost to technical issues.Source: Limitations · p. 11
- LLM-based categorization can be imperfect, although categories and prompts were refined with expert input and showed relatively high agreement with human annotations.Source: Limitations · p. 11
- Score and outcome prediction was used to identify correlated features, not to benchmark predictive performance, so results are exploratory.The authors used an interpretable multinomial logistic regression model.Source: Limitations · p. 11
- The analysis covers review and rebuttal stages but excludes meta-reviews, which may affect borderline papers.Meta-review analysis is left for future work.Source: Limitations · p. 11
- The study analyzes score dynamics rather than acceptance decisions, unlike concurrent work focused on acceptance decisions.The authors state this choice avoids potential incentives for gaming or reverse-engineering acceptance factors.Source: Limitations · p. 11
Reproducibility Statement
The authors provide code and derived score-change data while relying on the public OpenReview record for underlying data. They also report the model configuration and compute cost.
Reproducibility Statement
- Experimental code and the ICLR 2024 and 2025 score changes are shared, while public OpenReview data are not republished.The derived score-change data are shared under CC BY 4.0; prompts are provided in the Appendix and will also be shared with the code.Source: Reproducibility Statement · p. 12
- The experiments used GPT-4o model gpt-4o-2024-08-06 with top_p=0 and temperature=0.Source: Reproducibility Statement · p. 12
- The reported total computation cost was $500 in OpenAI credits.Source: Reproducibility Statement · p. 12
Ethics statement
The study uses publicly accessible ICLR peer-review data and reports aggregate patterns rather than attributing findings to individuals. It frames textual analysis as identifying general patterns and cautions against causal interpretations.
Ethics statement
- The study uses publicly accessible OpenReview data and did not collect private or confidential data.Reviews, rebuttals, and decisions are accessible under ICLR’s transparent peer-review policy.Source: Ethics statement · p. 12
- Results are analyzed in aggregate, without attributing them to specific reviewers or authors.The analysis treats reviews as research artifacts for understanding process trends, not for evaluating individuals.Source: Ethics statement · p. 12
- LLM text analysis is used to identify general patterns, with human experts guiding annotation categories despite acknowledged limitations and biases.Source: Ethics statement · p. 12
- The study’s observational analyses do not disentangle causal effects from confounding factors, so reported relationships are correlational rather than causal.Source: Ethics statement · p. 12
A.1 Additional Related Work
Related NLP work on peer review spans review and rebuttal analysis as well as several adjacent tasks. Recent LLM-review research and debate have raised concerns about quality and fairness and prompted proposals for conference-process changes.
A.1 Additional Related Work
- NLP peer-review research includes reviewer assignment, review and meta-review generation, guided skimming, and citation prediction.The passage cites work across these task areas.Source: §A.1 · p. 18
- Recent work on using LLMs for reviewing has raised concerns about review quality and fairness.Source: §A.1 · p. 18
- Debates about an “AI conference peer review crisis” have prompted proposals such as dedicated tracks for refutations and critiques at ML conferences.Source: §A.1 · p. 18
A.2 Design of Prompts
The section describes how the authors built and validated taxonomies and GPT-4o prompts for classifying reviewer praise, criticism, and author responses, then reports rating-linked patterns in those categories.
A.2 Design of Prompts
- Two academic experts refined weakness and strength categories by annotating reviews, reconciling overlapping or new categories, and checking an additional 10 papers for stability.Source: §A.2 · p. 18
- GPT-4o classified review statements using prompts selected against manual labels, with the chosen design achieving 84% pairwise human agreement.Source: §A.2 · p. 18
- Low-rated papers are often criticized for unclear writing, weak baselines, limited datasets, or poor algorithmic description, while high-rated papers are praised for novelty, methodological soundness, efficiency, and clear presentation.Source: §A.2 · p. 19, Figure 9 · p. 20, Figure 10 · p. 21
- The response taxonomy separates coverage, author stance, and response strategy; its selected GPT-4o prompt achieved 81% pairwise human agreement with manual annotations.Source: §A.2 · p. 19
- The dialogue prompt first records whether an author addressed each reviewer concern, then labels agreement or disagreement and the response strategy.Source: §A.2 · pp. 19, 24