PaperlayerView exact revision on arXiv

Source-linked AI summary

When Reviewers Lock Horn: Finding Disagreement in Scientific Peer Reviews

Sandeep Kumar, Tirthankar Ghosal, Asif Ekbal

arXiv:2310.18685v1cs.CL

TL;DR

Editors face a growing burden mediating reviewer disagreements, and this paper introduces ContraSciView and a baseline for detecting contradictions in review pairs. RoBERTa large achieves an F1 score of 71.14, while the study focuses on explicit rather than implicit contradictions.

  • Problem

    Editors must mediate reviewer disagreements amid rising submissions, and the paper identifies automatic contradiction detection in peer reviews as a previously unstudied task.

    Source: §1 · pp. 1–2

  • Method

    ContraSciView labels around 28,000 review pairs, annotates 50,303 review-pair comment pairs, and combines the MIMLLN framework with an NLI model for its baseline.

    Source: §3.2 · p. 3, §7 · p. 5

  • Results

    RoBERTa large achieves an F1 score of 71.14 for this task.

    Source: §7 · p. 5

  • Takeaways & Limitations

    The paper establishes a baseline method as a reference point for further research on identifying disagreement within peer reviews.

    Source: §1 · p. 2

    The study focuses on explicit contradictions and does not investigate implicit contradictions, which can be subtle and subjective.

    Source: Limitations · p. 6

Abstract

from arXiv · show

To this date, the efficacy of the scientific publishing enterprise fundamentally rests on the strength of the peer review process. The journal editor or the conference chair primarily relies on the expert reviewers' assessment, identify points of agreement and disagreement and try to reach a consensus to make a fair and informed decision on whether to accept or reject a paper. However, with the escalating number of submissions requiring review, especially in top-tier Artificial Intelligence (AI) conferences, the editor/chair, among many other works, invests a significant, sometimes stressful effort to mitigate reviewer disagreements. Here in this work, we introduce a novel task of automatically identifying contradictions among reviewers on a given article. To this end, we introduce ContraSciView, a comprehensive review-pair contradiction dataset on around 8.5k papers (with around 28k review pairs containing nearly 50k review pair comments) from the open review-based ICLR and NeurIPS conferences. We further propose a baseline model that detects contradictory statements from the review pairs. To the best of our knowledge, we make the first attempt to identify disagreements among peer reviewers automatically. We make our dataset and code public for further investigations.

1 Introduction

Peer review faces concerns about its reliability, while rising submission volumes add to editors’ responsibilities. The paper introduces a task, dataset, and baseline for identifying contradictions in reviewer comments.

1 Introduction

  • Peer review has been criticized for lack of transparency, bias, arbitrariness, inconsistency, and failure to recognize influential papers.The passage also describes peer review as a poorly defined task.Source: §1 · p. 1
  • Rising submissions add to editors’ workload, which includes mediating reviewer disagreements and making informed decisions.Conflicting reviewer views can arise from the complexity of scholarly discourse and subjectivity in interpreting research.Source: §1 · p. 1
  • Reviewer disagreements can confuse authors and editors as they try to address conflicting feedback, which is challenging to identify manually.The proposed investigation aims to help editors pinpoint discrepancies and make informed decisions.Source: §1 · pp. 1–2
  • The system is intended as a supplementary aid; editors should continue manual review because it may miss contradictions.The passage advises decisions based on careful analysis beyond system recommendations.Source: §1 · p. 2
  • The paper introduces contradiction detection in peer reviews, a labeled dataset of around 8.5k papers and 25k reviews, and a baseline method.The baseline is presented as a reference point for further research.Source: §1 · p. 2

2 Related Work

Prior work applies AI to several peer-review tasks and studies contradiction detection in other settings. This paper addresses the stated gap of contradiction detection specifically in peer reviews.

2 Related Work

  • Prior AI applications to peer review include decision and rating prediction, sentiment analysis, argument mining, and review summarization.The passage also lists contradiction detection in scientific claims, reviews, rumors, and other sentence pairs.Source: §2 · p. 2
  • Prior contradiction-detection approaches use linguistic features, review polarity, sentiment analysis, textual similarity, specialized embeddings, or recurrent networks.The cited examples concern scientific claims, reviews, Amazon reviews, rumors, and text pairs.Source: §2 · p. 2
  • Natural language inference classifies text pairs as entailment, neutral, or contradiction, and transformer models such as BERT and its variants have achieved state-of-the-art performance.The passage cites Chen et al. (2017), Gong et al. (2018), and Devlin et al. (2019).Source: §2 · p. 2
  • The authors state that contradiction detection in peer reviews had not been studied, and describe it as complex and requiring subject-domain knowledge.They note that reviewers’ different writing styles and commenting approaches make contradictions non-straightforward to detect.Source: §2 · p. 2

3 Dataset

The dataset uses ICLR and NeurIPS reviews from ASAP-Review and organizes them into review pairs and comment pairs. Pairs with shared aspects and opposing sentiments are selected for human annotation.

3.1 Dataset Collection

  • The collection uses 8,582 of 8,877 ASAP-Review papers, with ICLR reviews from 2017–2020 and NeurIPS reviews from 2016–2019.Reviews have eight aspect categories; all except Summary also receive Positive or Negative sentiment labels.Source: §3.1 · p. 2

3.2 Dataset Pre-processing

  • A review is represented as a list of comments or sentences, and a review pair contains one such list from each of two reviewers.A review-pair comment selects one comment from each reviewer.Source: §3.2 · p. 2
  • The preprocessing creates around 28,000 review pairs and uses shared aspect plus opposing sentiment to identify candidate contradictions for annotation.The process labels 17,095 pairs as having no contradiction comments when no comments share an aspect with opposing sentiments, then annotates 50,303 comment pairs from the remaining review pairs.Source: §3.2 · p. 3

4 Annotating for Contradiction

The authors define contradiction for annotation, train and monitor annotators, and report annotation agreement and aspect-level patterns. The process also describes annotator compensation and work-time limits.

4.1 Annotation Guidelines

  • Annotators judge whether two contemporaneous review passages are very unlikely to both be true, using examples across aspect categories as guidance.The annotation procedure is represented as a flowchart in Figure 4.Source: §4.1 · p. 3

4.2 Annotator Training

  • Six doctoral-student annotators received training based on expert-annotated review pairs, followed by feedback and a second training round.In the second round, most annotators correctly annotated at least 70% of contradictory cases.Source: §4.2 · p. 3

4.3 Annotation Process

  • The team monitored annotations, used iterative feedback, and consulted experts to resolve conflicts; average interannotator agreement was Cohen’s kappa 0.62.The passage characterizes this score as substantial consensus.Source: §4.3 · p. 3

4.4 Annotator’s Pay

  • Annotators were paid according to standard salaries in India based on hours worked, with a daily work limit of six hours.The authors say per-pair pay could compromise quality because review complexity and annotator familiarity affect the time required.Source: §4.4 · p. 3

4.5 Final Dataset

  • Most reviewer disagreements concern clarity, while discrepancies about replicability and meaningful comparison are notably fewer.The authors suggest the latter topics may receive fewer comments; Figure 2 compares contradiction and non-contradiction comment counts by aspect category.Source: §4.5 · pp. 3–4, Figure 2 · p. 4

5 Baseline System Description

The baseline first identifies review-comment pairs with the same aspect but differing sentiments, then classifies those pairs as contradictory or not. It combines aspect-and-sentiment modeling with a contradiction classifier adapted from natural language inference.

5 Baseline System Description

  • The pipeline sends review pairs through aspect-and-sentiment classification, forms same-aspect pairs with differing sentiments, then classifies those pairs as contradictory or not.The intermediate pairs are termed Sentiment Disparity Aspect Pairs (SDAPs).Source: §5 · p. 4
  • MIMLLN jointly performs aspect-category detection and aspect-category sentiment analysis, aggregating key word instances to estimate sentence sentiment for each aspect.It is trained on the human-annotated ASAP-Review dataset.Source: §5 · p. 4
  • The disagreement classifier reframes sentence-pair NLI as a two-label task: contradiction versus non-contradiction, combining entailment and neutral.It targets contradiction detection from Sentence Dependency Pairs.Source: §5 · p. 4

6 Results and Analysis

On the contradiction-classification task, RoBERTa large performs best among the reported models, and models trained on the authors’ dataset outperform those trained on existing NLI datasets. Across the full process, the baseline also exceeds the reported zero-shot language-model results.

6 Results and Analysis

  • 71.14 F1 with RoBERTa Large is the highest score; it outperforms XLNet large by 0.7 points, RoBERTa base by 4.1, XLNet by 4.5, and DistilBERT by 7.5.Models trained on the authors’ dataset outperform those trained on existing NLI datasets; the authors attribute the gap to differences in review writing and content.Source: §6 · pp. 4–5, Table 2 · p. 5
  • 88.60% accuracy for the Aspect Sentiment Classifier and 74.25 F1 for the Reviewer Disagreement Classifier are reported across the full process; ChatGPT scores 64.67 F1, 9 points lower than the baseline.The authors suggest ChatGPT’s lack of explicit training for this downstream task may explain the difference.Source: §6 · p. 5
  • 61.35 F1 for Google BARD is 12 points below the baseline, and BARD registers more false positives than ChatGPT.The authors suggest BARD may require more specialized training for this task.Source: §6 · p. 5

7 Conclusion and Future Work

The paper introduces peer-review contradiction detection, a review-pair dataset, and a baseline combining aspect-sentiment modeling with NLI. The authors identify full-review context, broader domains, and contradiction severity as directions for future work.

7 Conclusion and Future Work

  • The proposed baseline combines MIMLLN and an NLI model, with RoBERTa large performing best at 71.14 F1.The work introduces the task and develops the ContraSciView review-pair contradiction dataset.Source: §7 · p. 5
  • The framework omits full-review context; future work will expand the dataset to other domains, examine NLI’s significance, and categorize contradictions by severity.The authors propose severity levels of high, medium, or low.Source: §7 · p. 5

Limitations

The study focuses on explicit contradictions and excludes implicit contradictions and information from the papers under review. The authors describe implicit cases as harder to detect and potentially requiring deeper contextual knowledge.

Limitations

  • The study mainly focuses on explicit contradictions, such as reviewers directly disagreeing about whether a figure shows a relationship.The examples contrast clear, direct statements about the same claim.Source: Limitations · p. 5
  • Implicit contradictions are not addressed; they can be subjective and difficult to detect, and may require annotators to read the paper and related work.The authors distinguish indirect differences in perception from direct contradiction.Source: Limitations · p. 6
  • The study does not incorporate the reviewed papers, which span varied topics and would require experts from multiple domains.Source: Limitations · p. 6

Ethics Statement

The system is intended as an internal aid for editors, not a substitute for their judgment. The guidelines cover how contradictions are identified and labeled, while the implementation uses aspect-sentiment analysis and a contradiction classifier.

Ethics Statement

  • The system is for editors’ internal use and can generate incorrect contradictions or overlook real ones, so editors are advised to verify its recommendations.Source: Ethics Statement · p. 6

A Annotation Guidelines

  • The annotation process begins with the paper’s title and abstract, then asks annotators to read the reviews and note reviewers’ viewpoints, arguments, and highlighted aspects.Source: A Annotation Guidelines · p. 9
  • Annotators label comments C when they express opposing viewpoints, N when they do not contradict or are unrelated, and CON when the judgment is difficult.Source: A Annotation Guidelines · p. 9
  • The guidelines treat a broad claim of originality as contradicted by a specific critique of one part, while directing annotators to compare comments within the assigned aspect.Source: A Annotation Guidelines · pp. 9–10

B More details on Aspect Sentiment Model

C Implementation Details

D Result of Aspect Sentiment Classifier

Aspect and sentiment performance varies by category and model. The ChatGPT comparison and error analysis illustrate both baseline strengths and failures tied to review context, comment complexity, and upstream aspect classification.

D Result of Aspect Sentiment Classifier

E Comparsion with ChatGPT

  • The ChatGPT prompt processes one review pair at a time, with placeholders for the two reviews.Figure 5 shows the input format.Source: E Comparsion with ChatGPT · p. 11, Figure 5 · p. 12
  • ChatGPT sometimes mistakes comments about different aspects for contradictions, while the baseline reduces these errors and correctly rejects a biological-plausibility false positive.The example comments concern clarity versus substance; another example concerns biological plausibility.Source: E Comparsion with ChatGPT · p. 11

F.1 Lack of Complete Review Context:

  • The baseline can mispredict contradictions when a comment depends on preceding comments in the same review, including when reviewers discuss different algorithms.The authors identify learning from previous discussions as a potential future direction.Source: §F.1 · p. 12

F.2 Issues with Lengthy Comments:

  • Long sentences and technically complex comments containing many terms or mathematical symbols can also confuse the baseline.Source: §F.2 · p. 12

F.3 Error Propagation

An aspect-classification error can propagate to the contradiction classifier, causing it to miss a disagreement that reviewers expressed about motivation.

F.3 Error Propagation

  • For one comment, the aspect-sentiment model predicted Soundness but missed Motivation, although the comment also addressed motivation.Source: §F.3 · p. 12
  • The other reviewer’s comment was classified as Motivation, so the two comments expressed contrasting views on that aspect.Source: §F.3 · p. 12
  • Because the first model missed the aspect, the second model did not flag the discrepancy as a contradiction.Source: §F.3 · p. 12

Related papers

Loading 2310.18685v1…