Source-linked AI summary
Prior and Prejudice: The Novice Reviewers' Bias against Resubmissions in Conference Peer Review
Ivan Stelmakh, Nihar B. Shah, Aarti Singh, Hal Daumé III
TL;DR
As conferences consider disclosing prior submission histories, this paper tests whether a prior-rejection signal affects novice reviewers’ evaluations. In a randomized controlled trial, reviewers who received the signal gave lower overall scores than controls, with quality most affected among the criteria described.
Problem
As conferences consider revealing prior submission histories, the paper asks whether learning that a submission was previously rejected changes reviewers’ evaluations.
Source: §1 · pp. 1–2Method
Results
Reviewers given the resubmission signal scored papers 0.78 points lower on the 10-point Likert overall score (95% CI = [−1.30, −0.24]); quality showed the largest bias among criteria.
Source: §4.2 · p. 8Takeaways & Limitations
The findings provide evidence that can inform conference organizers’ decisions about review-system design, while whether and how to reuse past reviews remains an open question.
Source: §5 · p. 12Because the study omits more experienced reviewers, it is unknown whether its findings extend to the general reviewer population.
Source: §5 · p. 11
Abstract
from arXiv · showhide
Modern machine learning and computer science conferences are experiencing a surge in the number of submissions that challenges the quality of peer review as the number of competent reviewers is growing at a much slower rate. To curb this trend and reduce the burden on reviewers, several conferences have started encouraging or even requiring authors to declare the previous submission history of their papers. Such initiatives have been met with skepticism among authors, who raise the concern about a potential bias in reviewers' recommendations induced by this information. In this work, we investigate whether reviewers exhibit a bias caused by the knowledge that the submission under review was previously rejected at a similar venue, focusing on a population of novice reviewers who constitute a large fraction of the reviewer pool in leading machine learning and computer science conferences. We design and conduct a randomized controlled trial closely replicating the relevant components of the peer-review pipeline with $133$ reviewers (master's, junior PhD students, and recent graduates of top US universities) writing reviews for $19$ papers. The analysis reveals that reviewers indeed become negatively biased when they receive a signal about paper being a resubmission, giving almost 1 point lower overall score on a 10-point Likert item ($\Delta = -0.78, \ 95\% \ \text{CI} = [-1.30, -0.24]$) than reviewers who do not receive such a signal. Looking at specific criteria scores (originality, quality, clarity and significance), we observe that novice reviewers tend to underrate quality the most.
1 Introduction
Conference submission growth is straining peer review, while venues increasingly disclose or reuse prior submission histories and reviews. This study tests whether telling novice reviewers that a paper was previously rejected changes their evaluations.
1 Introduction
- Submission growth at leading ML and CS conferences is outpacing growth in qualified reviewers, challenging peer-review sustainability.Conferences also offer shorter turnaround times than top journals, allowing authors to resubmit quickly after rejection.Source: §1 · p. 1
- Several conferences have begun requesting or requiring disclosure of prior submissions or reviews, partly to reduce reviewer burden, but authors have expressed skepticism.Practices vary from making histories available to senior committee members to requiring authors to include past reviews.Source: §1 · pp. 1–2
- The paper calls changed evaluations prompted by a prior-rejection signal “resubmission bias,” hypothesizing that reviewers may judge a paper differently after learning it was rejected.The direction is not assumed: the signal could produce negative or positive bias.Source: §1 · pp. 2–3
- Because reviews of the same manuscript often disagree, the authors argue that resubmission bias could add to randomness and disadvantage previously rejected papers.They connect this concern to possible effects on researchers’ career trajectories.Source: §1 · p. 2
- The study tests for this bias among novice reviewers, a population that constitutes a large fraction of leading ML and CS conference reviewers.The authors caution that findings should not be generalized to the whole reviewer population.Source: §1 · pp. 2–3
- The study does not aim to endorse or oppose review reuse; its findings may inform when, if at all, prior histories become visible and how novice reviewers are trained.If bias is present, organizers may prefer to hide submission history during initial reviewing.Source: §1 · p. 3
2 Related Literature
Related work places resubmission bias among several influences on peer review, distinguished in part by when reviewers receive the relevant information. The paper also situates its contribution within empirical and computational efforts to understand and improve peer review.
2 Related Literature
- An experiment with medical-school faculty found that 47% of reviewers changed scores after seeing external scores, usually toward those scores, though the authors note this may reflect consensus-seeking rather than biased judgment.The present experiment removes group deliberation to measure reviewers’ attitude toward a submission.Source: §2 · p. 3
- The file drawer problem describes more favorable judgments of sound experimental work when results reject the null hypothesis than when otherwise equal results are nonsignificant.The cited work distinguishes it from resubmission bias by the information reviewers receive during reviewing rather than beforehand.Source: §2 · p. 4
- Resubmission bias differs from social influence and the file drawer problem in that reviewers receive its stimulus before reviewing submissions.Social-influence signals arrive after reviewing, whereas file-drawer information is observed during it.Source: §2 · p. 4
- Mahoney’s experiment found harsher evaluations when reviewers saw results contradicting their theoretical perspective than when results supported it.The paper later draws connections between this confirmatory bias and resubmission bias.Source: §2 · p. 4
- Research on single-blind review has examined gender, affiliation, and fame biases; evidence of fame and affiliation biases preceded WSDM’s switch to double-blind reviewing.The cited work also reports a meta-analysis finding bias against female-authored submissions.Source: §2 · p. 4
- Other computer-science work develops methods for issues including miscalibration, strategic behavior, demographic bias, and reviewer assignment; the authors envision combining policy guidelines with computational techniques to mitigate biases.The paper frames its work as part of research aimed at understanding and improving conference peer review.Source: §2 · p. 4
3 Experimental Setup
The authors ran a controlled experiment modeled on conference peer review, using novice reviewers and anonymized papers. Reviewers were randomly assigned to see or not see a prior-rejection signal before evaluating a paper.
3 Experimental Setup
- The experiment replicated relevant components of machine-learning conference peer review while controlling whether reviewers received a resubmission signal.The study used 19 anonymized preprints from varied machine-learning subareas, with venue identifiers removed.Source: §3 · pp. 4–5
- The study focused on novice and junior reviewers, estimated to comprise about 35% of ICML 2020 reviewers, while noting that more experienced reviewers were not represented.Each ICML 2020 submission was assigned at least one reviewer from this subset.Source: §3 · p. 5
- Recruitment targeted master’s and junior PhD students in ML-related fields at five top US universities; participants were not told the study’s dual goal of measuring the signal’s effect.The researchers received 200 candidates, over 90% of whom were students or recent graduates of the named universities.Source: §3 · p. 5
- After bidding on papers, participants were assigned one paper each, with assignments designed to satisfy bids and ensure at least eight reviewers per paper.Thirteen participants who did not bid were removed from the pool.Source: §3 · p. 5
- Reviewers were randomly allocated to control or test conditions; the test checklist stated that the paper had been rejected at NeurIPS 2019, while the control checklist omitted that signal.The checklist answers were “Yes” for all papers in both conditions.Source: §3 · p. 6, Figure 1 · p. 6
4 Analysis of the Experiment
The experiment compares novice reviewers’ evaluations across resubmission-signal and control conditions. Test reviewers gave significantly harsher ratings, with the strongest bias in quality scores; the reported bootstrap intervals may be less accurate than expected.
4 Analysis of the Experiment
- Of 187 participants assigned papers, 134 submitted reviews (71.7%); one further reviewer was excluded for a one-line review rejecting a paper over its page limit.Table 1 compares test and control demographics; none of the differences was significant at α = 0.1.Source: §4 · p. 6, Table 1 · p. 6
4.1 Testing procedure and effect size
- The modified permutation test shuffled reviewers between conditions within each paper, preserving condition counts, and used 10,000 permutations for P-values.Bootstrapped confidence intervals were computed over 5,000 iterations at the paper level.Source: §4.1 · pp. 7–8
- The analysis reports raw mean-score differences, a scaled adaptation of Cohen’s d, and stochastic superiority based on paired test/control reviews.The superiority measure counts ties as half and compares reviews of the same paper.Source: §4.1 · pp. 7–8
4.2 Results
- Δ = −0.78 points on the 10-point overall-score Likert item (95% CI [−1.30, −0.24]): test reviewers scored lower than controls.Table 2 compares evaluations from the two conditions.Source: Table 2 · p. 8, §4.2 · p. 8
- Among originality, quality, clarity, and significance, the bias was strongest for quality based on unit-independent effect sizes.The paper notes that quality is known to be important for overall submission evaluation.Source: §4.2 · p. 8
- The resubmission signal did not seem to affect reviewer confidence, though self-reported confidence may be unreliable; 122 of 133 reviewers selected level 3 or 4.The authors suggest participants may have avoided extreme confidence reports because they hoped to join the ICML reviewer pool.Source: §4.2 · p. 9
- The bootstrap confidence intervals may be less accurate than expected because each paper had few reviewers, although permutation testing controls false-alarm probability under small samples.The experiment’s 133 subjects were split into test and control groups across 19 papers.Source: §4.2 · p. 9
5 Discussion
The discussion interprets the experiment as evidence of resubmission bias among novice reviewers, while emphasizing the effect’s context, possible explanations, and limits on generalization. It leaves the desirability of reusing past reviews to conference organizers and the research community.
5 Discussion
- The controlled comparison identifies resubmission bias: novice reviewers were significantly harsher when told submissions had been rejected at similar venues.The design obtained reviews for the same submission in test and control conditions.Source: §5 · p. 9
Strength of the effect
- At ICML 2012, mean scores of 2.67 versus 2.33 corresponded to sixfold acceptance likelihood, though the evidence is observational.The authors describe the measured resubmission-bias effect on overall scores as small, but note that small score shifts may matter in competitive conferences.Source: §5 · p. 9
5 Discussion
- Subjective reviewer scores can make a between-subject effect less prominent than it might appear in a within-subject design.The authors attribute this possibility to additional variance in evaluations.Source: §5 · p. 9
- Experimental reviewers’ area-chair quality ratings averaged 0.18 higher than ICML reviewers’ (P < .001), while test and control invitation rates did not differ significantly (∆= 0.16, P = 0.075).These comparisons are observational and may reflect differences in review forms, workload, or assignment procedures; the authors characterize the comparison as indirect.Source: §5 · pp. 10–11, Figure 2 · p. 10
Cognitive mechanism of the resubmission bias
- The study does not identify the bias’s cognitive mechanism; the rejection signal may anchor judgments or invite deference to a more experienced reviewer group.The authors present these as possible explanations, not established mechanisms.Source: §5 · p. 11
- Confirmation, hindsight, and outcome biases offer possible parallels, although the authors note that analytical review could reduce reliance on such heuristics.They distinguish these related biases from the resubmission bias itself, while reporting that the latter appeared despite reviewers’ effort and analytical work.Source: §5 · pp. 11–12
On general idea of reusing reviews
- Reusing past reviews may focus effort on unresolved problems, but rejected strong or novel work may be disadvantaged; the study does not judge disclosure desirable.The authors leave that trade-off for community discussion and provide evidence intended to inform conference design.Source: §5 · p. 12
5 Discussion
- The outcome-only stimulus omitted prior reviews and author explanations; the sample also leaves effects by paper quality and among senior reviewers unresolved.The authors identify these as scope limits and directions for future research.Source: §5 · pp. 9, 11, 12