Design and Analysis of the NIPS 2016 Review Process

• The analysis lacked a ground-truth paper ranking, was post hoc, and focused on ratings and rankings rather than review comments.Source: §1 · p. 2
• Graph-theoretic techniques can ensure a good reviewer assignment, while reviewers and area chairs made very few positive bids.Source: §1 · p. 2
• Review scores showed significant miscalibration, invited and volunteer reviewers had comparable biases and variance, and junior reviewers reported lower confidence.Source: §1 · p. 2
• Reviewers made little change to scores after rebuttals, and accepted papers showed no observable bias toward any research area.Source: §1 · p. 2
• Reviewer disagreement was lower in NIPS 2016 than in NIPS 2015.Source: §1 · p. 2
• A significant fraction of reviewer scores were tied, and ordinal rankings can ameliorate this issue.Source: §1 · p. 2
• Ordinal rankings can also support automated identification of inconsistencies in reviews.Source: §1 · p. 2

2.1 Selecting area chairs and reviewers

• Area chairs and invited senior and volunteer author-reviewers formed the reviewer pool; 100 area chairs were recruited through nominations and subject-area considerations.Source: §2.1 · pp. 2–3

2.2 Assignment of papers to reviewers and area chairs

• Assignments combined bids, TPMS affinity, and subject similarity; area chairs manually assigned one senior reviewer and checked for at least three competent reviewers per paper.Source: §2.2 · p. 3

2.3 Review criteria and scores

• Reviewers rated four paper features on a 1-to-5 scale, separating evaluation across criteria from area chairs’ decision-making role.Source: §2.3 · p. 4

2.4 Discussions and rebuttals

• Authors could rebut reviews before area chairs and reviewers discussed papers; area chairs received flags for low-confidence, high-variance, or missing reviews.Source: §2.4 · p. 4

2.5 Decision procedure

• Buddy pairs made accept/reject decisions jointly, while program chairs selected oral presentations from nominated candidates.Source: §2.5 · p. 4
• 568 papers were accepted, and 45 were selected for oral presentations.Source: §2.5 · p. 5

2.6 Experimental ordinal reviews

• After the review process, 2,189 reviewers provided ordinal rankings, which were collected before final decisions but excluded from the decision procedure.Source: §2.6 · p. 5

3.1 Reviewer bids

• 278 papers received at most 2 positive bids and 816 received at most 5, while 50% of reviewers made 90% of positive bids.Additionally, 148 reviewers made no bids and 1,201 made at most 2 positive bids.Source: §3.1 · pp. 5–6
• The authors recommend making unbid papers more visible and explaining how bidding helps reviewers receive more relevant assignments.They identify incentivizing positive bids and combining bids with paper-content and subject similarity as open problems.Source: §3.1 · p. 6

3.2 Reviewer assignment

• The NIPS 2015 cluster's low connectivity was associated with reinforcement learning, and such graph connectivity issues can increase noise or bias in decisions.The authors recommend checking reviewer-assignment graphs globally before reviews are assigned.Source: §3.2 · p. 8
• NIPS 2015 had a poorly connected reviewer-and-paper cluster, largely associated with reinforcement learning; no such separated cluster was found in NIPS 2016.The NIPS 2015 cluster contained 50 of 78 reviewers identifying reinforcement learning as their primary area.Source: §3.2 · p. 8

3.3 Review-score distribution and mismatches in calibration

• Nearly 60% of scores were 3 or higher, although reviewers were asked to use scores of 3 or higher only for papers in the top 30%.The table compares review-score distributions across four features with the guidelines provided to reviewers.Source: §3.3 · pp. 9–10, Table 1 · p. 9
• High scores may reflect many high-quality submissions or reviewer miscalibration; either possibility left area chairs reducing roughly 60% good papers to 23% accepted.The authors connect this mismatch with frustration among authors whose well-scored papers were rejected.Source: §3.3 · p. 9
• 32% of papers were flagged as having a fatal flaw by at least one reviewer.Source: §3.3 · p. 10
• The authors suggest avoiding numbered rating scales or showing authors score percentiles, and recommend expert input on elicitation design.They identify review calibration and response-interface design as open problems.Source: §3.3 · p. 10

Mean scores

• Only clarity differed significantly between invited and volunteer reviewers after multiple-testing correction; no significant differences appeared among senior, junior, and student groups.For invited versus volunteer reviewers, clarity had p=1.91 × 10−04 and d=0.0788 before correction.Source: §3.4 · p. 10

Self-reported confidence

• Self-reported confidence correlated with seniority, with significant differences between senior, junior, and student reviewers and between the two reviewer pools.The senior-versus-student comparison had d=0.3577; invited-versus-volunteer reviewers had d=0.2943.Source: §3.4 · p. 11

Consistency and discussion participation

• The analysis found no conclusive difference in within-pool agreement between invited and volunteer reviewers; senior reviewers participated more actively in discussions than junior researchers.The agreement tests were not statistically significant across the four features.Source: §3.4 · p. 12

Behavior of different pools of reviewers

• Volunteer reviewers offer scalability and transparency, with no observable bias and inter-reviewer agreement similar to invited reviewers.The paper recommends continuing to include volunteers.Source: §3.4 · p. 12

3.5 Rebuttals and discussions

• 2188 papers received rebuttals; 4180 of 13674 reviewer-paper pairs participated in discussions, producing 5255 posts.The authors suggest requiring reviewers to change or confirm scores after discussion, while noting unresolved questions about discussion incentives and alternatives such as appeals.Source: §3.5 · p. 12, §3.5.2 · p. 14

3.5.1 Who participates in discussions?

• Pool 1 reviewers posted more per reviewer-paper pair than pool 2, and non-student reviewers more than student reviewers; both differences were statistically significant, with small effects.For pool 1 versus pool 2, effect sizes were d=0.0704 for all papers and d=0.0894 for the top 2k. For non-student versus student reviewers, they were d=0.0695 and d=0.0929, respectively.Source: §3.5.1 · p. 13

3.5.2 How do discussions change the scores?

• 1193 of 12154 pre-rebuttal reviews changed scores, and the mean score change was less than 0.1 on a 1-to-5 scale.The changed reviews were distributed across 886 papers.Source: §3.5 · p. 12, §3.5.2 · p. 13, Figure 11 · p. 14
• Final decisions correlated positively with participating reviewers’ scores, including within each reviewer pool, but the analysis cannot determine whether discussion caused the correlation.The overall association had p = 1.6 × 10−6 with d= 0.13.Source: §3.5.2 · p. 14

3.6 Distribution across subject areas

• The submitted-versus-accepted subject-area distributions show no systematic bias toward or against any particular area.The distributions also have a heavy tail, consistent with diversity of topics in the NIPS community.Source: §3.6 · p. 14, Figure 12 · p. 15

3.7.1 Messy middle model

• The messy-middle analysis defines the middle by removing top and bottom fractions of papers ranked by mean score, then finds the largest window with reviewer agreement near chance.The authors provide values to support comparisons across conferences, but note that the post hoc analysis is not strictly comparable to NIPS 2014’s controlled experiment because it lacks a true ranking or counterfactual.Source: §3.7.1 · p. 16
• The messy middle measured 30% in NIPS 2016 versus 45% in NIPS 2015, an improvement on this metric.The NIPS 2016 window was identified by removing the bottom 70% of papers; the NIPS 2015 window by removing 15% from the top and 40% from the bottom.Source: §3.7.1 · p. 16

3.7.2 A bootstrapped analysis

• Bootstrapping found near-zero decision variance for many oral-accepted and rejected papers, but variance near its maximum for a notable fraction of poster-accepted and rejected papers.The analysis resampled each paper’s reviewers and repeated the acceptance decision 1000 times.Source: §3.7.2 · p. 17

3.8 Ordinal data collection

• Ordinal rankings provide comparative feedback and avoid ties, while cardinal ratings support feature-specific evaluation; the paper proposes a hybrid collection method.The NIPS 2016 ordinal rankings were collected after normal review submission and were not used in conference decisions.Source: §3.8 · p. 18, §3.8.3 · p. 20

3.8.1 Tie breaks

• More than 30% of reviewer-score pairs were tied for each feature, with tie fractions about 10%–15% higher in the top 2k papers.The calculation covered all submitted papers and had a total sample size of 26106.Source: §3.8.1 · p. 18

3.8.2 Consistency of ordinal ranking data

• Ordinal rankings had consistency similar to cardinal scores, no significant agreement difference between reviewer pools, and about 90% agreement with median cardinal scores.Across all papers, rankings had roughly five agreements per disagreement with final acceptance decisions; top-2k disagreement rose to 27–28% from 16–17%.Source: Figure 14 · p. 18, §3.8.2 · p. 19, Figure 15 · p. 19

3.8.3 Detecting anomalies

• The authors report that ordinal rankings can help detect review inconsistencies, and identify controlled tests and combination algorithms as open problems.They recommend collecting feature-level cardinal ratings alongside an overall ordinal ranking.Source: §3.8.3 · p. 20

3.9 Checking inconsistencies

• The proposed checks flag cases where a reviewer ranks a paper lower despite rating it higher on all four features, or ranks a paper above one without a fatal flaw.The section quantifies both types of inconsistency in the NIPS 2016 review data.Source: §3.9 · p. 20
• The fatal-flaw mismatch rate was similar among volunteer and invited reviewers, and delayed ordinal surveys alone were not considered a sufficient explanation for the discrepancies.Ordinal data was described as consistent with cardinal data; the authors therefore did not think delay-related noise alone explained the fatal-flaw cases.Source: §3.9 · p. 20
• The authors recommend checking glaring inconsistencies during review and asking the reviewers involved to explain their reasoning.They suggest such mismatches may reflect inadequate reviewing or features that fail to express reviewers’ opinions; reviewers were not contacted because the analysis followed the process.Source: §3.9 · p. 20
• 349 fatal-flaw/ranking mismatches occurred across 176 reviewers; 55 pairs also had feature ratings favoring a paper that the reviewer ranked lower.Among the top 2k papers, there were 55 fatal-flaw mismatches across 33 reviewers and 10 feature-rating mismatches across 10 reviewers.Source: §3.9 · pp. 20–21
• The section leaves open what other inconsistencies could be checked automatically.The recommendation to contact reviewers is intended to help understand the reasoning behind flagged cases.Source: §3.9 · pp. 20–21

4 Discussion and conclusions

• The analysis used tools to detect potential artifacts or biases and statistical tests to validate hypotheses, including comparisons of topic distributions, reviewer pools, and score noise.A larger-than-guideline fraction of papers appeared in the score histogram, prompting a suggestion to redesign score elicitation and author feedback.Source: §4 · p. 21
• The analysis found no obvious bias, but the authors caution that this does not establish that bias is absent.Selection bias in recruiting reviewers and area chairs is difficult to address at this scale.Source: §4 · p. 21
• The AC buddy system scales with more papers but is harder to calibrate because buddy pairs are disjoint; larger AC conferences can also be dominated by self-confident or opinionated ACs.Volunteer author-reviewers were introduced to increase reviewer-pool diversity and were described as less prone to selection bias than invitation based primarily on AC recommendations.Source: §4 · p. 21
• The authors report that invited and volunteer reviewer pools did not make significantly different decisions, while more senior reviewers appeared to contribute more detailed reviews and participate more in rebuttals and discussions.They suggest encouraging and possibly educating junior reviewers to participate in these activities.Source: §4 · pp. 21–22
• The authors identify rebuttal and discussion arbitration as difficult to monitor with scores alone, and suggest confirming scores because reviewers may not update them consistently afterward.They also suggest that mixing ordinal and cardinal scores may reduce calibration, tie-breaking, and anomaly-identification problems.Source: §4 · p. 22
• At this scale, no single person controls every part of the review process, so the authors emphasize continued experiments, analysis, and open discussion.They describe program chairs as unable to control every individual decision or reviewer’s quality, leaving area chairs and reviewers central to the process.Source: §4 · p. 22

APPENDIX

• The appendix presents additional details about the experiments.Source: APPENDIX · p. 24

A Subject areas

• The appendix lists numbered subject areas, including learning theory, reinforcement learning, and ranking and preference learning.The passage supplies names for numbered categories but does not provide further analysis of them.Source: A Subject areas · p. 24

B Messy middle details

• Figure 16 reports agreement ratio r and overlapping paper-reviewer pair count m for every (t, b), ignoring Step 2.3's sample-size restriction.The ratio is agreements divided by agreements plus disagreements; m is the sum of agreements and disagreements.Source: B Messy middle details · p. 24, Figure 16 · p. 25
• Each Figure 16 cell is color-coded by the size of its 95% confidence interval on a log scale.Source: B Messy middle details · p. 24, Figure 16 · p. 25