PaperlayerView exact revision on arXiv

Source-linked AI summary

Testing for Reviewer Anchoring in Peer Review: A Randomized Controlled Trial

Ryan Liu, Steven Jecmen, Vincent Conitzer, Fei Fang, Nihar B. Shah

arXiv:2307.05443v1cs.HCcs.DL

TL;DR

Rebuttals invite reviewers to update scores, but evidence about their effects is mixed and anchoring had not been studied in conference peer review. A randomized experiment compared revised scores after initially seeing a weaker paper with control scores after seeing the corrected paper from the outset; it found no significant evidence of anchoring.

  • Problem

    Rebuttals are widely used, but evidence about their usefulness is mixed, and anchoring had not been studied in conference peer review despite being a possible explanation for limited score changes.

    Source: §1 · p. 2

  • Method

    A randomized controlled experiment compared reviewers’ revised scores after they first saw a weaker paper with control scores from reviewers who saw the corrected paper from the outset.

    Source: §1 · pp. 2–3

  • Results

    A=0.50, p=0.61 for Overall and A=0.49, p=0.61 for Evaluation; revised scores showed no significant evidence of anchoring versus control scores.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The main analysis found no evidence of reviewer anchoring; limited score changes in rebuttal phases may have other explanations, including limited reviewer participation.

    Source: §5 · p. 13

    The sample size may have provided insufficient power because score variance was higher than in the conference data used for planning.

    Source: §5 · p. 13

Abstract

from arXiv · show

Peer review frequently follows a process where reviewers first provide initial reviews, authors respond to these reviews, then reviewers update their reviews based on the authors' response. There is mixed evidence regarding whether this process is useful, including frequent anecdotal complaints that reviewers insufficiently update their scores. In this study, we aim to investigate whether reviewers anchor to their original scores when updating their reviews, which serves as a potential explanation for the lack of updates in reviewer scores. We design a novel randomized controlled trial to test if reviewers exhibit anchoring. In the experimental condition, participants initially see a flawed version of a paper that is later corrected, while in the control condition, participants only see the correct version. We take various measures to ensure that in the absence of anchoring, reviewers in the experimental group should revise their scores to be identically distributed to the scores from the control group. Furthermore, we construct the reviewed paper to maximize the difference between the flawed and corrected versions, and employ deception to hide the true experiment purpose. Our randomized controlled trial consists of 108 researchers as participants. First, we find that our intervention was successful at creating a difference in perceived paper quality between the flawed and corrected versions: Using a permutation test with the Mann-Whitney U statistic, we find that the experimental group's initial scores are lower than the control group's scores in both the Evaluation category (Vargha-Delaney A=0.64, p=0.0096) and Overall score (A=0.59, p=0.058). Next, we test for anchoring by comparing the experimental group's revised scores with the control group's scores. We find no significant evidence of anchoring in either the Overall (A=0.50, p=0.61) or Evaluation category (A=0.49, p=0.61).

1 Introduction

This study tests whether reviewers anchor to their initial scores during rebuttal, against a backdrop of mixed evidence about rebuttals’ usefulness and limited score changes. The intervention lowered initial ratings, but revised ratings showed no significant evidence of anchoring relative to control ratings.

1 Introduction

  • Rebuttals have mixed evidence of usefulness, and anchoring had not been studied in conference rebuttals despite being a possible explanation for limited score changes.The introduction reports that only 9.8% of NeurIPS 2016 reviews subsequently changed score and that anchoring has been studied in other settings.Source: §1 · p. 2
  • The study asks whether reviewers who revise after receiving additional evidence give lower scores than reviewers who saw that evidence from the outset.Scenario A provides evidence after initial scoring; Scenario B presents the paper and evidence together, making control scores the counterfactual comparison.Source: §1 · pp. 2–3
  • The randomized mock-review study recruited 108 researchers and used a frozen weak-results figure before revealing the corrected animated figure to experimental reviewers under a technical-error cover story.The frozen figure’s rightmost result was 2.23 versus a 2.21 baseline; the animated figure’s rightmost result was 2.63.Source: §1 · p. 3, Figure 1 · p. 4
  • A=0.64, p=0.0096 for Evaluation and A=0.59, p=0.058 for Overall; initial scores were lower than control scores, indicating the manipulation shifted perceived paper quality.The study used a one-sided permutation test with the Mann-Whitney U statistic and Vargha-Delaney A effect sizes.Source: §1 · p. 3
  • A=0.50, p=0.61 for Overall and A=0.49, p=0.61 for Evaluation; revised scores showed no significant evidence of anchoring versus control scores.The comparison used a one-sided permutation test with the Mann-Whitney U statistic.Source: §1 · p. 3
  • The authors describe this as the first randomized controlled trial of anchoring in peer review and suggest relevance to reviewer discussions and longer-term author feedback.They frame it as a first step because expertise and knowledge may interact differently with biases in peer review.Source: §1 · p. 3

2 Related work

Related work situates the study within research on conference reviewing, rebuttals, and cognitive bias. Prior evidence describes limited score changes after rebuttals, while the authors identify their study as the first randomized controlled trial of anchoring in peer review.

2.1 Conference peer review

  • Studies of conference reviewing address reviewer assignment, misconduct, calibration, subjectivity, and biases including herding; discussion studies also report that reviewers sometimes update scores after new information.The present work focuses specifically on anchoring during author rebuttals rather than reviewer discussions.Source: §2.1 · p. 5

2.2 Rebuttal processes

  • Conference reports find that rebuttals often produce little score movement, although surveys show strong author support for having an opportunity to respond.Reported examples include score changes affecting only a small fraction of submissions and 96% of PLDI 2015 survey respondents favoring rebuttal opportunities.Source: §2.2 · pp. 5–6

2.3 Anchoring bias

  • Anchoring describes estimates that are adjusted insufficiently from an initial value; the authors report that their study is the first randomized controlled trial testing this behavior in peer review.Prior work has documented anchoring across varied domains and applications.Source: §2.3 · p. 6

3 Methods

The methods use a simulated conference review to compare ratings after reviewers see either a weaker paper result followed by its correction or the correct result from the outset. The design aims to make the correction objective and conceal the study’s anchoring purpose, then compares experimental and control score distributions.

3 Methods

  • The study protocol was preregistered and received approval from Carnegie Mellon University’s Institutional Review Board.The methods section covers the experimental design, recruitment and data collection, and analysis.Source: §3 · p. 6

3.1 Experiment design

  • The simulated experiment was designed to test reviewer anchoring while controlling paper quality, rebuttal strength, and reviewers’ awareness of the study purpose.The authors state that paper quality and rebuttal strength cannot be controlled in a real conference setting.Source: §3.1.1 · p. 6

3.1.1 Challenges for the design

  • The design targets four challenges: an objective, detectable improvement; separating anchoring from penalties for author mistakes; equivalent control and corrected-paper experiences; and concealing the study’s purpose.Absent anchoring, the authors expected control and experimental ratings to be equivalent.Source: §3.1.1 · pp. 6–7

3.1.2 Experimental procedure

  • In 30-minute individual Zoom sessions, participants reviewed the same application-focused conference paper, then experimental-group reviewers saw the animation work and were asked to revise their scores and comments.Participants were told the study concerned the effect of new media on reviews; the experimental group initially saw a frozen first frame with a weaker result.Source: §3.1.2 · pp. 7–8

3.1.3 Design justification

  • The paper’s numerical results were changed to make the quality difference clear, while the frozen GIF was presented as a technical error and a media-focused cover story concealed the anchoring hypothesis.The correction matched the control paper, and participants were debriefed about the deception after the study.Source: §3.1.3 · pp. 8–9

3.2 Participation and data collection

  • The study recruited 108 randomly assigned researchers from computer science fields and recorded control scores, experimental initial and revised scores, and review information.Participants were PhD students or PhDs with a recent computer science-related publication; Overall scores used a 1–10 scale and category scores a 1–4 scale.Source: §3.2 · p. 9

3.3 Analysis

  • One-sided permutation tests with Mann–Whitney U compared control scores against experimental initial scores for setup validity and revised scores for anchoring, with false-discovery correction and Vargha–Delaney A effect sizes.The analysis differed from the preregistration, which specified mean differences and Overall scores only; Table 2 reports the initial- and revised-score comparisons.Source: §3.3 · p. 10, Table 2 · p. 11

4 Results

The main tests found no evidence of anchoring after the experiment created a perceived difference in paper quality. Supplemental analyses examined other review categories, reviewer confidence and seniority, and score changes; the authors characterize these analyses as exploratory and data-dependent.

4.1 Main results

  • A=0.6375 for Evaluation (adjusted p=0.0096) and A=0.5857 for Overall (adjusted p=0.0575) indicated perceived quality differed between groups.The Evaluation comparison was significant at α=0.01 and Overall at α=0.1; the difference may have been less reflected in Overall scores.Source: §4.1 · p. 11
  • A=0.5048 for Overall and A=0.4863 for Evaluation, both adjusted p=0.6064; the study found no evidence of anchoring.The comparisons of revised experimental-group scores with control scores failed to reject the null hypothesis that reviewers do not anchor.Source: §4.1 · p. 11

4.2 Supplemental results

  • The authors present supplemental observations as motivation for future work, not as support for statistically significant conclusions.Source: §4.2 · p. 11
  • Revised-versus-control comparisons did not indicate anchoring in the review categories beyond Evaluation.These additional category analyses were exploratory and data-dependent, so the authors present them as motivation for future work rather than statistically significant conclusions.Source: §4.2 · p. 11, Table 3 · p. 12
  • Confident reviewers had equal mean revised and control Overall scores, while unconfident reviewers had generally lower revised scores (A=0.63), with substantial uncertainty.There were 41 confident and 13 unconfident reviewers in each group; the supplemental analyses were exploratory and data-dependent.Source: §4.2 · pp. 11–12, Table 4 · p. 12
  • Junior and senior reviewers had similar results, though effect-size uncertainty was large; more than 50% changed neither Overall nor category scores.The seniority comparison and score-change breakdown were among the exploratory, data-dependent supplemental analyses.Source: §4.2 · pp. 11–12, Table 5 · p. 13, Table 6 · p. 13

5 Conclusion and discussion

The randomized experiment found no evidence of reviewer anchoring in conference peer review. The authors discuss limits on interpreting that result, including statistical power and how closely the study reproduced conference conditions.

5 Conclusion and discussion

  • The study used a randomized controlled experiment designed to test reviewer anchoring in conference peer review.The design used animated media, deception, and a cover story to address challenges and confounders.Source: §5 · p. 13
  • The main analysis found no evidence of reviewer anchoring effects in peer review.The authors note that limited score changes in conference rebuttals may instead reflect weak rebuttal impact, unclear initial statements, or limited reviewer participation.Source: §5 · p. 13
  • Higher-than-expected score variance may have left the sample underpowered, and participants may not have shared conference reviewers’ subfield norms or context.The authors suggest recruiting participants with expertise in one subfield to improve calibration and make paper changes more interpretable.Source: §5 · p. 13
  • The experiment omitted real-conference social dynamics, which could lead reviewers to defend initial positions or be influenced by other reviews.The authors left these secondary social effects for future work because they could also introduce confounds.Source: §5 · p. 13
  • A supplemental analysis suggests the answer to the research question may differ across reviewers with different confidence levels.The authors propose testing effects within subpopulations in future studies.Source: §5 · p. 14

A.1 Detailed deception and revision process

This section outlines the experimental procedure from participants’ initial review through submission of their revised review, then describes deviations and responses.

A.1 Detailed deception and revision process

  • The section covers the deception and revision process from completion of the initial review through submission of the revised review, followed by procedural deviations and how they were addressed.Source: §A.1 · p. 17

A.1.1 Deception and Revision Process

Experimental participants first reviewed a paper with a weaker frozen figure, then saw the corrected animated result and could revise their reviews. The procedure used a separate review form and concealed the figure change as an experiment intervention.

A.1.1 Deception and Revision Process

  • The experimental group initially saw a frozen GIF showing a substantially weaker result, with surrounding text kept vague and no specific result values stated outside the figure.The full animation later displayed the corrected result.Source: §A.1.1 · p. 17
  • After the initial review, the experimenter replaced the static figure with the working GIF and prompted participants to reload the paper by feigning confusion over their “N/A” animation response.The review questions were on a separate Google form; the experimenter downloaded each original response before the participant submitted edits.Source: §A.1.1 · p. 17
  • Participants were asked to revise their initial responses to mirror rebuttal-stage score revision and could edit any part of their review.The authors note that conference reviews are often available as a reference or for direct editing during revision.Source: §A.1.1 · p. 17
  • Table 8 compares ICLR 2022 intra-paper Overall score variance with the variance of initial, revised, and control Overall scores, all on a 10-point scale.Source: Table 8 · p. 18

A.1.2 Deviations to the expected procedure and prepared solutions

The researchers prepared responses to procedural deviations identified during planning and piloting, including confusion about whether participants had seen an animated figure.

A.1.2 Deviations to the expected procedure and prepared solutions

  • The prepared solutions addressed deviations anticipated during planning and observed in the pilot study.Source: §A.1.2 · p. 18
  • When participants mistook the static figure for the animated one, the experimenter asked them to elaborate, then used their description to explain the reload request.Source: §A.1.2 · p. 18
  • When participants asked whether they were supposed to see animation, experimenters avoided a yes-or-no answer and claimed they also did not know until review submission.They later said participants were supposed to see animation and attempted to answer questions identically across conditions.Source: §A.1.2 · p. 18

A.2 Power analysis

Power analysis targeted at least 100 participants to detect a 0.25 difference in means, but observed variances were much higher than estimated. The estimate relied on post-rebuttal scores and participants whose backgrounds may have been less homogeneous than typical reviewers'.

A.2 Power analysis

  • Power calculations assumed normally distributed control and revised Overall scores, with variances sampled from individual ICLR 2022 papers.Separate variance values were sampled for control and revised scores because participants in the two groups might perceive the paper differently.Source: §A.2 · p. 18
  • The study targeted at least 100 participants to detect a 0.25 difference in means (α = 0.05, β = 0.2).Source: §A.2 · p. 18
  • Observed variances were much higher than estimated, and the estimate used post-rebuttal scores because pre-rebuttal scores were unavailable.Post-rebuttal scores may be closer than pre-rebuttal scores because of rebuttals or influence from other reviewers.Source: §A.2 · p. 18
  • Participants came from different computer science subfields, potentially making them less homogeneous than typical reviewers and increasing score variance.Source: §A.2 · pp. 18–19

A.3 Participant recruitment

Participants met doctoral, publication, age, and residency requirements. Recruitment used several outreach channels at Carnegie Mellon University and eight other research universities.

A.3 Participant recruitment

  • Participants had to be PhD students or PhD holders, have a recent computer science-related publication, be over 18, and reside in the United States.The publication had to be within the five years preceding the study.Source: §A.3 · p. 19
  • Recruitment used posters, emails, social media, course announcements, office visits, and word of mouth at Carnegie Mellon University and eight other research universities.Participants selected a 30-minute meeting slot through a sign-up calendar.Source: §A.3 · p. 19
Loading 2307.05443v1…