Source-linked AI summary
What makes a successful rebuttal in computer science conferences? : A perspective on social interaction
Junjie Huang, Win-bin Huang, Yi Bu, Qi Cao, Huawei Shen, Xueqi Cheng
TL;DR
The paper asks what makes rebuttals successful in CS conference peer review, where empirical evidence on their impact and contributing factors is limited. It analyzes ICLR2022 review data through score comparisons, signed social network analysis, rebuttal-strategy analysis, and prediction models. Score increases were associated with higher acceptance rates, and balanced network structure increased after rebuttal.
Problem
Few studies had empirically investigated what makes a successful rebuttal, despite rebuttals offering authors a chance to clarify reviewer misunderstandings and potentially improve scores.
Source: §1 · p. 2Method
The authors analyze ICLR2022 rebuttals using review-score comparisons, signed social network analysis, quantified author strategies, and machine-learning prediction.
Source: §5 · p. 9, §6 · p. 10, §8.1 · p. 13Results
43.25% of papers had increased scores, and their acceptance rate was approximately 58.38%, versus 13.77% for papers with decreased scores and 13.26% for unchanged scores; balanced network structure also increased after rebuttal.
Source: §4 · p. 6, §8.1 · p. 13Takeaways & Limitations
The findings identify reviewer social effects and author rebuttal strategies as factors associated with rebuttal outcomes and provide an empirical basis for studying rebuttals through social network analysis.
Source: §8.1 · p. 13The study could not assess reviewer–Area Chair interactions, and its strategy tests used t-tests rather than causal analysis.
Source: §8.2 · pp. 13–14
Abstract
from arXiv · showhide
With an exponential increase in submissions to top-tier Computer Science (CS) conferences, more and more conferences have introduced a rebuttal stage to the conference peer review process. The rebuttal stage can be modeled as social interactions between authors and reviewers. A successful rebuttal often results in an increased review score after the rebuttal stage. In this paper, we conduct an empirical study to determine the factors contributing to a successful rebuttal using over 3,000 papers and 13,000 reviews from ICLR2022, one of the most prestigious computer science conferences. First, we observe a significant difference in review scores before and after the rebuttal stage, which is crucial for paper acceptance. Furthermore, we investigate factors from the reviewer's perspective using signed social network analysis. A notable finding is the increase in balanced network structure after the rebuttal stage. Subsequently, we evaluate several quantifiable author rebuttal strategies and their effects on review scores. These strategies can help authors in improving their review scores. Finally, we used machine learning models to predict rebuttal success and validated the impact of potential factors analyzed in this paper. Our experiments demonstrate that the utilization of all features proposed in this study can aid in predicting the success of the rebuttal. In summary, this work presents a study on the impact factors of successful rebuttals from both reviewers' and authors' perspectives and lays the foundation for analyzing rebuttals with social network analysis.
1. Introduction
The paper studies rebuttals as social interactions in CS conference peer review, addressing the limited empirical evidence on what makes them successful. It focuses on score changes, reviewer influence, author strategies, and predicting score revision.
1. Introduction
- The study is motivated by concerns that CS conference peer review can be arbitrary amid scarce reviewers and difficulties judging groundbreaking work.The introduction also notes concerns about reproducibility and unethical behavior under “publish or perish” pressure.Source: §1 · p. 1
- Few studies had empirically measured what makes a successful rebuttal, motivating analysis from both reviewers’ and authors’ perspectives.The paper frames rebuttals as a chance to clarify misunderstandings and potentially increase review scores and acceptance chances.Source: §1 · p. 2
- The study examines score changes between the initial and final reviews and considers reviewer social pressure and author rebuttal strategy.The rebuttal stage includes author responses and subsequent reviewer discussion.Source: §1 · pp. 2–3
- Four research questions ask whether rebuttals change review outcomes, whether peer effects influence scores, which author strategies are effective, and whether score revisions can be predicted.The questions correspond to score changes, signed social network analysis, strategy analysis, and machine-learning prediction.Source: §1 · p. 3
2. Related Work
Related work situates rebuttals within the high-stakes, conference-centered CS review system, describes open-review datasets, and motivates an interpretable analysis of reviewer peer effects.
2.1. The Significance of Conference Proceedings in Computer Science
- Prestigious CS conference publications are linked to researchers’ reputation, funding, and career outcomes, while rebuttals aim to reduce arbitrariness in a competitive, low-acceptance setting.The paper distinguishes conference rebuttals from journal responses, which are described as aimed at gaining greater recognition for a paper.Source: §2.1 · p. 3
2.2. Open Peer Review Dataset
- Prior review datasets and models largely focused on NLP tasks, while this study targets social interactions between reviewers and authors.The study examines reviewer social pressure and author rebuttal strategies, building on open peer review resources.Source: §2.2 · pp. 3–4
2.3. Peer Effect in Peer Review
- The study uses Signed Social Network Analysis to model reviewer peer effects with positive and negative links, offering an interpretable approach to opinion influence.Prior work found balanced structures increased after rebuttal, but reviewer identities and privacy make author-referee signed bipartite networks difficult to obtain.Source: §2.3 · p. 4
3. Dataset
The dataset captures ICLR2022 reviews before and after rebuttal, enabling analysis of paper outcomes, reviewer activity, and author and paper characteristics.
3.1. Data Collection
- The crawled ICLR2022 dataset contains 3,338 papers, 13,021 reviews, and 37,478 replies; anonymized reviewer identities prevent constructing a signed bipartite graph.The researchers crawled OpenReview at two timestamps because live review results overwrite preliminary versions.Source: §3.1 · p. 4
3.2. Data Description
- Of 3,338 submissions, 54 received Oral, 176 Spotlight, and 865 Poster presentations; 2,243 were rejected, including 667 Desk rejection/Withdrawn cases.Among reviews, scores of 6, 8, and 10 were treated as positive, yielding a positive ratio of 0.449.Source: §3.2 · p. 5
- Papers receiving additional reviewers had a 52.5% score-increase rate, above the reported 43.59% overall post-rebuttal increase rate.139 papers received additional reviewers; 123 had one, 13 had two, and three had three.Source: §3.2 · p. 5
3.3.1. Paper Analysis
- Accepted papers had significantly more authors than rejected papers (4.80 vs 4.27); title and abstract length and supplementary-material upload showed no significant differences.The analysis also found significant differences in keyword counts and completion of a one-sentence summary.Source: §3.3.1 · p. 6
- Accepted papers more often had skilled authors (0.80 vs 0.60) and significantly more pages and revisions than rejected papers.The paper defines skilled authors by prior ICLR publication or inclusion among top Computer Science Scientists.Source: §3.3.1 · p. 6
3.3.2. Review Analysis
- Positive reviews scored higher on correctness, novelty, significance, and sentiment, and had more discussion replies than negative reviews (1.55 vs 1.35).Negative reviews had higher word counts overall, although strong-acceptance reviews had the highest word counts.Source: §3.3.2 · p. 6
4. Rebuttal Results
In ICLR2022, papers whose average scores increased after rebuttal had higher acceptance rates than papers with decreased or unchanged scores. Score changes were concentrated among borderline papers, while most reviewers kept their initial scores.
4. Rebuttal Results
- 43.25% of papers increased average scores (1,444/3,338), and their acceptance rate was 58.38%, versus 13.77% for decreased-score and 13.26% for unchanged papers.Only 17.95% (2,310/12,863) of reviews showed an increase.Source: §4 · p. 6, Table 3 · p. 7
4.1. Paper Perspective
- Most accepted papers gained scores; the initial acceptance cutoff was between 5.0 and 6.0, the final cutoff was 6.0, and rebuttals primarily benefited borderline papers.Across score groups, most papers maintained or increased their average scores; increases were especially common above 6.0.Source: §4.1 · pp. 7–8
4.2. Review Perspective
- Most reviewers kept their scores; increases clustered around 5.0, while at 8.0 decreases exceeded increases but remained less common than KEEP.Recommendation scores correlated positively with detailed scores and negatively with confidence; final review scores correlated positively with initial scores.Source: §4.2 · pp. 8–9
5. Signed Social Network Analysis
The authors use signed social network analysis to examine reviewer interactions during rebuttal. Across three conference datasets, balanced motifs increased and unbalanced motifs decreased after rebuttal.
5. Signed Social Network Analysis
- The authors argue that peer-score summary statistics used in prior work cannot reflect reviewer counts and lack interpretability.The cited prior approach used score summaries such as maximum, minimum, mean, median, and standard deviation.Source: §5 · p. 9
- SSNA represents reviewer-paper opinions as signed motifs, classifying two patterns as unbalanced and two as balanced under balance theory.The authors describe unbalanced patterns as potentially exerting negative peer influence that may prompt a reviewer to change their score.Source: §5 · pp. 9–10
- The analysis assigns signs using conference-specific scoring scales and compares balanced-motif proportions before and after rebuttal across three datasets.The authors use a paired t-test to assess whether the before-and-after changes are significant.Source: §5 · p. 10
- Across ICLR2022, ACL2018, and TCSC, balanced motif counts and per-paper proportions rose after rebuttal while unbalanced motifs fell; per-paper proportions rose significantly.The reported example of motif counts changes from 11,720 to 13,329 balanced motifs and from 7,208 to 6,087 unbalanced motifs.Source: §5 · p. 10, Table 4 · p. 10
6. Strategy Analysis
The strategy analysis compares quantifiable rebuttal approaches with review outcomes, treating review-score increases after rebuttal as successful. Strategy-adopting groups had higher success rates than groups not adopting the strategies.
6. Strategy Analysis
- INC (a review-score increase after rebuttal) counted as successful, while KEEP and DEC counted as non-successful; the overall success ratio was 17.95%.Source: §6 · pp. 10–11
- The analysis operationalized effort through reply counts and word counts, politeness through positive patterns, and concern coverage through review-response similarity.The authors compared top and bottom groups for these measures.Source: §6 · p. 11
- The authors also compared whether responses included references and whether they mentioned other reviewers’ IDs.These measures represented the add-references and make-consensus strategies.Source: §6 · p. 11
- For each strategy, the authors split responded reviews into two groups and used analyses and t-tests to test outcome differences.Source: §6 · p. 11
- G1 groups adopting rebuttal strategies had significantly higher success rates than non-adopting groups; unresponded reviews still had revised comments 2.82% of the time.The authors interpret the make-consensus strategy as an example of introducing peer effects into responses.Source: §6 · p. 11
7. Rebuttal Success Prediction
The section formulates rebuttal success prediction as an imbalanced binary classification task, combines review, paper, peer-effect, and author-strategy features, and evaluates the resulting models. The all-feature model performed best, with peer-effect features strongest among individual feature types.
7.3. Results
- All-feature models performed best; peer-effect features led individual feature types, meta-information performed poorly, and machine-learning models achieved AUC > 0.5.Source: Table 6 · p. 12, §7.3 · p. 13
7.1. Problem Definition
- Success prediction asks whether the final review score exceeds the initial score; score increases are positive cases, while KEEP and DEC are negative.Because the task is imbalanced, the authors use Macro-F1 and AUC to measure model performance.Source: §7.1 · p. 12
7.2. Methods
- Metadata, review/rebuttal text, peer-effect, and author-strategy features feed a two-layer ReLU MLP, trained with 80/10/10 data splits.Source: §7.2 · pp. 12–13
8. Conclusion
The conclusion reports increased balanced network structures after rebuttals and discusses author strategies, while noting limits in measuring social influence and strategy effects. It also identifies future directions for analyzing negative links and comparing peer review across conferences or years.
8.1. Summary and Discussion
- Balanced network structure increased after rebuttals across three top-conference datasets; the authors suggest social pressure may influence reviewers’ score changes.They also discuss building consensus, politeness, and continued engagement as possible response strategies.Source: §8.1 · p. 13
8.2. Limitation and Future Work
- The study did not assess reviewer–Area Chair interactions; its strategy t-tests analyze mean variance, and many strategies are difficult to quantify.Source: §8.2 · pp. 13–14
- Future work includes examining reasons behind negative peer-review links and comparing multiple conferences or one conference across several years.Source: §8.2 · p. 14