PaperlayerView exact revision on arXiv

Source-linked AI summary

Has the Machine Learning Review Process Become More Arbitrary as the Field Has Grown? The NeurIPS 2021 Consistency Experiment

Alina Beygelzimer, Yann N. Dauphin, Percy Liang, Jennifer Wortman Vaughan

arXiv:2306.03262v1cs.LGcs.DL

TL;DR

As NeurIPS grew, its chairs tested whether review decisions had become less consistent by having independent committees review duplicate submissions. About half of accepted-paper outcomes would change in a rerun, while the analysis found no evidence that review had become noisier with the conference’s growth.

  • Problem

    As NeurIPS submissions and its reviewer pool grew, the chairs sought to measure whether consistency in reviewers’ decisions had changed.

    Source: §1 · p. 1

  • Method

    The chairs randomly duplicated 10% of submissions and assigned each copy to an independent committee for review.

    Source: §1 · p. 1, §3 · p. 3

  • Results

    About half of the accepted-paper list would change in an independent rerun, and the analysis found no evidence that review had become noisier as NeurIPS grew.

    Source: §1 · p. 1, §5 · p. 8

  • Takeaways & Limitations

    Making NeurIPS significantly more selective may increase arbitrariness, while increasing its acceptance rate may not appreciably reduce it.

    Source: §5 · p. 8

    In 50 duplicated-paper pairs, authors commented only on the copy with the higher average score, but only two such papers were ultimately accepted.

    Source: §5.1 · p. 9

Abstract

from arXiv · show

We present the NeurIPS 2021 consistency experiment, a larger-scale variant of the 2014 NeurIPS experiment in which 10% of conference submissions were reviewed by two independent committees to quantify the randomness in the review process. We observe that the two committees disagree on their accept/reject recommendations for 23% of the papers and that, consistent with the results from 2014, approximately half of the list of accepted papers would change if the review process were randomly rerun. Our analysis suggests that making the conference more selective would increase the arbitrariness of the process. Taken together with previous research, our results highlight the inherent difficulty of objectively measuring the quality of research, and suggest that authors should not be excessively discouraged by rejected work.

1 Introduction

The paper revisits the 2014 NeurIPS consistency experiment as the conference and its reviewer pool have grown. In 2021, independent committees again produced substantial disagreement, underscoring how difficult it is to measure research quality consistently.

1 Introduction

  • Peer review is used to assess research quality, but reviewer judgments are noisy and often disagree across fields.The 2014 NeurIPS experiment likewise found substantial inconsistency between reviewers.Source: §1 · p. 1
  • NeurIPS received more than five times as many annual submissions as at the time of the 2014 experiment, with over 9,000 submissions in 2020, 2021, and 2022.More than 9,000 program committee members played a role in a given year.Source: §1 · p. 1
  • The 2021 chairs repeated the consistency experiment by duplicating 10% of submissions and comparing recommendations from independent committees.The experiment was designed to measure whether reviewer-decision consistency had changed as NeurIPS grew.Source: §1 · p. 1
  • About half of the accepted-paper list would change if the review process were independently rerun, consistent with the 2014 finding.The authors conclude that objectively measuring research merits is inherently difficult.Source: §1 · pp. 1–2

2 Background on the NeurIPS 2021 Review Process

The section outlines NeurIPS 2021’s review procedures and summarizes a separate survey of authors’ perceptions. That survey found frequent disagreement about the relative merits of papers, including among co-authors.

2 Background on the NeurIPS 2021 Review Process

  • The review ran from May 28 to September 28, 2021, was conducted on OpenReview, and was double-blind and confidential.Accepted-paper reviews were released publicly; rejected-paper reviews remained private unless authors opted in.Source: §2 · p. 2
  • Reviewers evaluated submissions, area chairs coordinated reviews and recommendations, and senior area chairs oversaw area chairs and helped calibrate decisions.The committee also included ethics reviewers for submissions flagged for potential ethical concerns.Source: §2 · p. 2
  • Authors had a week to respond after initial reviews, with additional rolling discussion possible if new reviews were added.By the release of initial reviews, 99.7% of submissions had at least three reviews available.Source: §2 · p. 2
  • The conference received 9,122 submissions and accepted 2,334, for an acceptance rate of 25.6%.Accepted papers comprised 55 oral presentations, 260 spotlights, and 2,019 posters.Source: §2 · p. 2
  • Authors predicted acceptance at a median of about 70% when the rate was 25.8%, and authors and co-authors disagreed about paper merits about a third of the time.When an author had one accepted and one rejected paper, the rejected paper was ranked as higher quality about a third of the time; the survey results appear in a separate report [15].Source: §2 · pp. 2–3

3 Methods

The experiment randomly duplicated 10% of submissions and assigned each copy to an independently staffed committee. Its analysis focused on 882 duplicated papers still under review when initial reviews were released, with special acceptance rules for the duplicated set.

3 Methods

  • The team randomly selected 10% of papers for duplication, assigning two area chairs and twice the usual number of reviewers to each paper’s two copies.The committees for the two copies were split at random, with their area chairs assigned to different senior area chairs.Source: §3 · p. 3
  • Unlike the 2014 experiment’s conference-wide committee split, the 2021 implementation effectively made a separate random split for each paper.This was intended to allow more flexibility in reviewer assignment.Source: §3 · p. 3
  • Committees were kept unaware that papers were duplicated, while authors were notified before initial reviews and asked to respond independently to each set.Reused paper IDs helped avoid revealing duplicated papers to committees.Source: §3 · p. 3
  • The duplicate-acceptance rule resulted in 92 papers being accepted that otherwise would not have been, raising the conference acceptance rate from 24.6% to 25.6%.Papers were accepted if at least one copy was recommended for acceptance and no fatal flaw was found.Source: §3 · p. 3
  • The analysis covered 882 duplicated papers still under review when initial reviews were released, excluding papers desk rejected or withdrawn beforehand.It included 118 papers withdrawn after initial reviews; the authors note the withdrawal rate after reviews was 45% higher outside the experiment.Source: §3 · p. 3
  • An Institutional Review Board approved the experiment, and access to personally identifiable review data was restricted to program chairs, the workflow manager, and OpenReview staff.Reviews for accepted duplicated papers and rejected duplicated papers whose authors opted in were publicly available.Source: §3 · p. 3

4 Results

The results show substantial disagreement between independent committees, both in accept/reject recommendations and in which accepted papers would survive a rerun. Disagreement also varies with acceptance thresholds, while ethics flags and follow-up feedback reveal additional sources and forms of inconsistency.

4.1 Inconsistent Decisions

  • 23.0% of duplicated papers received inconsistent accept/reject recommendations; this was in line with 2014, with no obvious increase or decrease in inconsistency.At the 22.7% acceptance rate, the 2021 disagreement rate was a 35% relative improvement over the random baseline.Source: §4.1 · pp. 4–5
  • Disagreement rates changed with the acceptance threshold: bumping down candidate posters yielded 20.3%, while bumping up candidate rejects yielded 29.4%.The respective relative improvements over random were 25% at a 16.0% acceptance rate and 35% at a 34.4% acceptance rate.Source: §4.1 · pp. 4–5, Figure 2 · p. 6

4 Results

  • Average scores from the two committees correlated at 0.575 for initial scores and 0.586 for final scores.Table 1 summarizes recommendations for 882 duplicated papers still under review when initial reviews were released.Source: §4 · p. 4, Table 1 · p. 4, Figure 1 · p. 5

4.2 Accept Precision

  • 50.6% of accepted papers were rejected by the other committee, a 35% improvement over the random baseline and close to the 49.5% reported for 2014.The figure applies across the two committees; the reported values were 51.9% for one and 49.2% for the other.Source: §4.2 · p. 6
  • Accepting only papers recommended as orals or spotlights gave an 8% relative improvement over random, with 88.8% rejected by the other committee; bumping down all posters marked as candidates for bumping down gave 25% and 63.2%, respectively.Choosing a more selective acceptance rate leads to lower accept precision.Source: §4.2 · p. 6

4.3 Ethics Flags

  • Only 3 ethics flags overlapped between committees, despite 23 papers being flagged by one and 22 by the other.The cited NeurIPS ethics-review discussion identifies uncertainty over which papers to flag as a challenge.Source: §4.3 · p. 7

4.4 Feedback from ACs and SACs

  • Among 203 papers recommended for acceptance by only one committee, feedback was received on 99; 48% of multiple-choice responses called the paper borderline, and other cases involved contribution or limitation disagreements and issues such as prior work, proofs, and methodological flaws.Both committees provided feedback for only 18 papers, limiting the analysis.Source: §4.4 · pp. 7–8

5 Discussion

The authors find no evidence that review inconsistency increased as NeurIPS grew, but emphasize that substantial subjectivity remains and that greater selectivity may worsen arbitrariness. They discuss limits from unequal author responses and calibration records, as well as the workload and acceptance-rate effects of the experiment.

5 Discussion

  • The analysis finds no evidence that review became noisier with conference growth, while significant noise and subjectivity remain and some randomness may be unavoidable.The authors connect this difficulty to disagreement over paper quality among reviewers, authors, and even co-authors.Source: §5 · p. 8
  • The authors encourage rejected-paper authors not to be excessively discouraged, noting that 2014 chairs found more than a third later published in high-quality venues.They expect the same of many papers rejected from NeurIPS 2021.Source: §5 · p. 8
  • More selective decisions may increase arbitrariness on both committee disagreement and re-review measures, while higher acceptance may not appreciably reduce it.The authors ask future program chairs to consider this trade-off when setting selectivity.Source: §5 · p. 8
  • The discussion invites debate about alternatives, including reader-led quality judgments or review limited to technical correctness rather than subjective characteristics such as perceived importance.The authors state that they value peer review while welcoming proposals to improve it.Source: §5 · p. 8

5.1 Limitations

  • Some authors gave more detailed responses to the copy they thought likelier to be accepted; 50 pairs received comments only on the higher-scoring copy, and only two were ultimately accepted.This behavior may have skewed the experiment's results.Source: §5.1 · pp. 8–9
  • Calibration records sometimes retained incompatible bump-up or bump-down flags after decisions changed, potentially causing minor inaccuracies in shifted-threshold analyses.The authors cite nine duplicated papers moved from poster to reject while still marked “can be bumped down.”Source: §5.1 · p. 9

5.2 Potential Negative Impact on the Community

  • The experiment increased program-committee workload by approximately 10% and led to 92 additional acceptances, raising the overall acceptance rate from 24.6% to 25.6%.The authors do not believe this meaningfully affected the conference program's overall quality.Source: §5.2 · p. 9

A NeurIPS Paper Checklist

The paper applies the NeurIPS checklist to promote responsible research practices and reports constraints on reproducing the experiment because its data include personally identifiable information.

A NeurIPS Paper Checklist

  • The authors applied the NeurIPS paper checklist to promote responsible research practices.Source: A NeurIPS Paper Checklist · p. 10
  • The experiment data are not publicly accessible because they include personally identifiable information, though reviews for accepted duplicates and opted-in rejected duplicates are public on OpenReview.Source: A NeurIPS Paper Checklist · p. 11
  • The authors report that meaningful error bars were not straightforward to derive for Figure 3.Source: A NeurIPS Paper Checklist · p. 11

B Review Forms

The appendixed forms show what reviewers and area chairs were asked to assess, including paper quality, review confidence, decisions, and ethics-related concerns.

B Review Forms

  • The forms included ethics-related fields for potential concerns, whether ethics review was needed, and the relevant area of expertise.Source: B Review Forms · p. 12
  • The reviewer form also asked about time spent reviewing and responsible-research concerns including potential applications, privacy, legal compliance, and research integrity.Source: B Review Forms · p. 12
  • Reviewer forms requested a rating, confidence score, summary, and full review addressing originality, quality, clarity, and significance.Source: B Review Forms · pp. 11–12
  • Area-chair forms requested a recommendation, confidence qualification, and meta-review explaining the decision in light of reviews, author response, and discussion.Source: B Review Forms · p. 13

C Disclosure to Authors

The email informed authors whose papers were duplicated for the consistency experiment and explained how to handle the two independent review sets. It also set confidentiality, decision, and review-release expectations.

C Disclosure to Authors

  • 10% of submissions were randomly chosen to receive two independent sets of reviews as part of a variant of the 2014 experiment.The experiment was repeated to examine how review-process quality had changed over time.Source: C Disclosure to Authors · pp. 13–14
  • Authors were asked to keep the experiment confidential until the review process ended, when it would be publicly announced.Only authors whose papers were duplicated were informed at that time.Source: C Disclosure to Authors · p. 14
  • Authors were instructed to respond independently to each review set and to the two subsequent rolling discussions.Reviewers and area chairs assigned to each paper copy did not know another copy was under review.Source: C Disclosure to Authors · p. 14
  • When both reviewer sets agreed, their recommendation stood; if only one recommended acceptance, the paper could be accepted unless the other set found a fatal flaw.Program Chairs made the final call in split cases.Source: C Disclosure to Authors · p. 14
  • Both review sets would be released for accepted papers and for rejected papers whose authors opted into public release.Author responses and follow-up discussion with the assigned program committee would also be released.Source: C Disclosure to Authors · p. 14

Related papers

Loading 2306.03262v1…