PaperlayerView exact revision on arXiv

Source-linked AI summary

Can We Automate Scientific Reviewing?

Weizhe Yuan, Pengfei Liu, Graham Neubig

arXiv:2102.00176v1cs.CL

TL;DR

Growing scientific publication volume makes expert peer review laborious, while defining a good review is difficult. The paper develops an aspect-based review-generation approach using annotated machine-learning reviews and targeted summarization models. Generated reviews cover more aspects than human reviews, but can make non-factual assessments and exhibit reviewer biases.

  • Problem

    The growing volume of papers makes expert peer review laborious, and the subjectivity of review criteria makes it difficult to define and evaluate a good review.

    Source: §1 · pp. 1–2

  • Method

    The authors frame review generation as aspect-based summarization, annotate machine-learning-domain reviews by aspect, and train summarization models to generate reviews from papers.

    Source: §1 · p. 2

  • Results

    Generated reviews cover more aspects and can summarize core ideas, but may make non-factual assessments; human and automatic reviewers also exhibit biases.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors see review generation as a possible aid to human reviewers, not a replacement for them at this time.

    Source: §7.3 · p. 15

    The authors say the system cannot yet replace human reviewers and may instead help as part of a machine-assisted review process.

    Source: §7.3 · p. 15

Abstract

from arXiv · show

The rapid development of science and technology has been accompanied by an exponential growth in peer-reviewed scientific publications. At the same time, the review of each paper is a laborious process that must be carried out by subject matter experts. Thus, providing high-quality reviews of this growing number of papers is a significant challenge. In this work, we ask the question "can we automate scientific reviewing?", discussing the possibility of using state-of-the-art natural language processing (NLP) models to generate first-pass peer reviews for scientific papers. Arguably the most difficult part of this is defining what a "good" review is in the first place, so we first discuss possible evaluation measures for such reviews. We then collect a dataset of papers in the machine learning domain, annotate them with different aspects of content covered in each review, and train targeted summarization models that take in papers to generate reviews. Comprehensive experimental results show that system-generated reviews tend to touch upon more aspects of the paper than human-written reviews, but the generated text can suffer from lower constructiveness for all aspects except the explanation of the core ideas of the papers, which are largely factually correct. We finally summarize eight challenges in the pursuit of a good review generation system together with potential solutions, which, hopefully, will inspire more future research on this subject. We make all code, and the dataset publicly available: https://github.com/neulab/ReviewAdvisor, as well as a ReviewAdvisor system: http://review.nlpedia.ai/.

1 Introduction

The paper addresses the underexplored task of automating scientific peer review, where defining a good review is itself difficult. It frames review generation as aspect-based summarization, builds an annotated review dataset, and finds both broader aspect coverage and important weaknesses in generated assessments.

1 Introduction

  • Automatic review generation remains relatively underexplored despite the labor involved in reviewing the growing volume of scientific papers.Earlier systems are noted, but the cited approaches did not extensively evaluate generated-review quality or features.Source: §1 · pp. 1–2
  • The paper treats review generation as aspect-based summarization and evaluates reviews for summary quality, factual correctness, fairness, diverse comments, and informative evidence.Aspects include novelty and potential impact.Source: §1 · p. 2
  • ASAP-Review provides machine-learning-domain reviews annotated with aspect information for richer evaluation of generated reviews.The dataset is named as ASpect-enhAnced Peer Review.Source: §1 · p. 2
  • The work trains state-of-the-art summarization models on the dataset to generate reviews from scientific papers and evaluates different ReviewAdvisor designs.The paper presents this as an initial answer to how well NLP models can meet the proposed review desiderata.Source: §1 · p. 2
  • Generated reviews cover more aspects than human reviews and can summarize core ideas, but may make non-factual assessments and show bias.The paper notes that native-English papers tend to receive higher human Clarity scores, while automatic reviewers are harsher on Originality for non-native English speakers; both reviewer types favor non-anonymous papers.Source: §1 · p. 2
  • The authors conclude that review generation cannot yet replace human reviewers, though it may help as part of a machine-assisted process.They also outline eight challenges and potential future directions for review generation or assistance.Source: §1 · p. 3

2 What Makes a Good Peer Review?

The section frames peer review as difficult to assess because its objectives and judgments can be subjective. It proposes evaluating reviews across multiple desiderata, including decisiveness, comprehensiveness, justification, accuracy, and similarity to human reviews, while leaving computational measurement of kindness to future work.

2.1 Peer Review for Scientific Research

  • A good review should be decisive, comprehensive, justified, accurate, and kind, but peer-review effectiveness and its objectives remain contested.Comprehensiveness entails an organized summary and assessment across aspects; justifications can give authors hints for improvement.Source: §2.1 · p. 3

2.2.1 D1: Decisiveness

  • Review recommendation is compared with the paper’s final decision, while the evaluation framework summarizes metrics for different review desiderata.Recommendation Accuracy is higher when the review more decisively and accurately recommends acceptance or rejection.Source: §2.2 · p. 3, §2.2.1 · p. 4

2.2.2 D2: Comprehensiveness

  • Aspect Coverage counts predefined aspects mentioned in a review, while Aspect Recall measures aspect and polarity agreement with the meta-review and may favor reviews considered in writing it.The authors caution that Aspect Recall should be used alongside other metrics.Source: §2.2.2 · p. 4

2.2.3 D3: Justification

  • Informativeness measures how many negative review comments have corresponding evidence, with the score set to 1 when no negative aspects are mentioned.The evidence-support judgment is conducted manually.Source: §2.2.3 · p. 4

2.3 D4: Accuracy

  • Summary Accuracy manually scores a review’s summary as incorrect or absent, partially correct, or correct; Aspect-level Constructiveness judges whether evidence validly supports negative assessments.When no negative aspects have evidence, Aspect-level Constructiveness is set to 1; the paper notes this may favor reviews that provide little evidence, while their Informativeness will be low.Source: §2.3 · p. 4

2.4 D5: Kindness

  • The paper leaves computational measurement of kindness to future work because nuanced peer-review language is difficult to assess accurately.Source: §2.4 · p. 5

2.5 Similarity to Human Reviews

  • ROUGE measures surface-level word match and BERTScore embedding-space distance; with multiple reference reviews, the paper aggregates each by taking the maximum.Source: §2.5 · p. 5

3 Dataset

The paper builds ASAP-Review, a machine-learning review dataset with fine-grained aspect annotations, to support system training and richer evaluation. It defines eight review aspects and combines human annotation with automated tagging and quality checks.

3.1 Data Collection

  • ASAP-Review collects ICLR papers from 2017–2020 and NeurIPS papers from 2016–2019, retaining review types, decisions, and other available metadata.NeurIPS publicly provides reviews only for accepted papers.Source: §3.1 · p. 5, Table 2 · p. 5

3.2 Aspect-enhanced Review Dataset

  • The dataset adds fine-grained aspect annotation because review structure and the aspects covered are not directly available, yet matter for evaluation.The annotation process follows a four-step pipeline.Source: §3.2 · p. 5, §3.2.2 · p. 6

3.2.1 Aspect Typology and Polarity

  • The eight-aspect typology covers summary, motivation/impact, originality, soundness/correctness, substance, replicability, meaningful comparison, and clarity.Each aspect except summary is also marked positive or negative.Source: §3.2.1 · p. 6

3.2.2 Aspect Annotation

  • Because annotating more than 20,000 reviews manually was unrealistic, the authors trained a BERT-based aspect tagger on human-labeled reviews and refined its predictions with seven heuristic rules.The 1,000 human-annotated reviews had a lowest pair-wise Cohen kappa of 0.653.Source: §3.2.2 · p. 6
  • The automated annotations achieved 92.75% aspect precision and 85.19% aspect recall, with low recall for positive replicability because mentions were rare.The authors describe their evaluation criterion as strict and the resulting metrics as a lower bound.Source: §3.2.2 · p. 7

4 Scientific Review Generation

The paper frames scientific review generation as reviewer-view summarization that must cover paper aspects and provide expert comments. Its system uses extract-then-generate for long papers and adds aspect prediction to encourage aspect-diverse reviews.

4.1 Task Formulation

  • The authors note that imperfect recall for negative aspects slightly raises Aspect Score, while the overall trend remains unchanged.They report that over 85% of the imperfect recall reflects partial recognition rather than failure to detect any negative aspect.Source: §4.1 · p. 7
  • Reviewer-view summarization extends author- and reader-view summaries with domain experts’ critical comments on different aspects of a paper.The authors identify potential uses for authors, reviewers, and readers.Source: §4.1 · p. 7

4.2 System Design

  • Because BART’s standard input limit is 1,024 tokens, the system uses an extract-then-generate design: select salient paper text, then generate a review summary from it.The paper also reports that its dataset has fewer training samples than other benchmark summarization datasets.Source: §4.2 · pp. 7–8, §4.2.1 · p. 8

4.2.1 Two-stage Systems for Long Documents

  • The system explores oracle, introduction-only, cross-entropy, and hybrid extraction; cross-entropy selection favors both paper beginnings and endings, as does the oracle.The cross-entropy method selects diverse content by maximizing unigram entropy; the hybrid combines the abstract with cross-entropy extraction.Source: §4.2.1 · p. 8

4.2.2 Aspect-aware Summarization

  • An auxiliary sequence-labeling task predicts aspects in generated text, using annotated aspects as additional information to support aspect-diverse generation and interpretable review structure.Its loss combines sequence-to-sequence and sequence-labeling losses, with α = 0.1 tuned for development-set aspect coverage.Source: §4.2.2 · pp. 8–9

5 Experiment

The experiments find that generated reviews can cover more aspects and summarize core ideas well, but often lack constructive, reliable assessment and exhibit group disparities. The paper compares model settings, studies review bias, and examines aspect-aware generation.

5.1 Settings

  • Experiments compare three extraction strategies and two generation frameworks on ASAP-Review, using paper full texts and BART-based models.The dataset contains 8,742 papers and 25,986 paper-review pairs; reviews were restricted to 100–1,024 words for training.Source: §5.1 · p. 9

5.2.1 Weaknesses

  • Against reference reviews, the smallest system–human gaps were 41.86% for Recommendation Accuracy and 31.89% for Aspect-level Constructiveness; generated reviews averaged 0.32 questions per review versus 2.04 in references.Source: §5.2.1 · pp. 10–11

5.2 What are Systems Good and Bad at?

  • 14.11% and 2.03% are the maximum system advantages over human reviewers in Aspect Coverage and Informativeness; 4 of 6 systems exceed 80% Summary Accuracy.The top three systems’ Summary Accuracy gaps from human reviewers are not statistically significant.Source: §5.2.2 · p. 11

5.2.3 System Comparisons

  • “Extractive + abstractive” methods outperform pure extraction on ROUGE and BERTScore, full-paper extraction improves aspect coverage over introduction-only input, and higher informativeness does not resolve constructiveness weaknesses.The comparison reports minimum gaps of 2.69, 2.57, 12.44, and 9.72 for R-1, R-1, R-L, and BERTScore, respectively.Source: §5.2.3 · p. 11

5.2.4 Case Study

  • The aspect-enhanced model produces fluent, aspect-aware text with appropriate polarity, though aspects may be short and alignment imperfect.The case study uses the model with cross-entropy extraction and inspects generated tokens associated with aspects.Source: §5.2.4 · p. 11, Table 6 · p. 12

5.3.1 Measuring Bias in Reviews

  • Bias analysis compares positive aspect scores and disparity across review groups, partitioned by author nativeness and paper anonymity.A positive disparity difference means generated reviews favor G0 more than reference reviews, and vice versa.Source: §5.3.1 · p. 12

5.3.2 Nativeness Analysis

  • For nativeness, the system narrows the Clarity disparity but amplifies Originality disparity; Table 8 reports differences of +18.71 for Originality and −13.32 for Clarity.Human reviews show a marked Clarity gap, while the two groups’ scores are closer on other aspects.Source: Table 8 · p. 13, §5.3.2 · p. 13, Figure 7 · p. 13

5.3.3 Anonymity Analysis

  • Both human and system reviews favor non-anonymous papers across aspects; anonymity’s largest absolute disparity difference is 5.80 and total bias is 28.00, below nativeness’s 18.71 and 43.00.The authors interpret the smaller differences as indicating that the model is less sensitive to anonymity than nativeness.Source: Table 8 · p. 13, §5.3.3 · pp. 13–14

6 Related Work

Related work addresses peer review and review generation from several directions, but the authors distinguish their study through broader diagnostics and extensive evaluation of generated reviews.

Scientific Review Generation

  • Prior scientific review generation includes surface-level fake-review generation by Bartoli et al. (2016) and Wang et al. (2020)’s extraction-and-summarization pipeline.Wang et al. focus mainly on extraction accuracy and assess generated summaries manually on one criterion over 50 papers.Source: §6 · p. 14
  • This paper differentiates itself with diverse review-quality diagnostics, a different summarization method, and extensive evaluation of generated results.The cited contemporaneous pipeline’s generated-summary evaluation is described as precursory.Source: §6 · p. 14

Peer Review

  • Other peer-review research studies assignment, bias, consistency, sentiment, and accept/reject prediction, but does not directly alleviate the review workload.The paper also situates peer review as costly, slow, inconsistent, and subject to bias.Source: §6 · p. 14

7 Discussion and Future Directions

The discussion positions ReviewAdvisor as assistance for human reviewers rather than a replacement, while outlining model, dataset, and evaluation challenges. The authors conclude that scientific review automation is not yet ready for replacement use.

7.1 Machine-assisted Review Systems

  • ReviewAdvisor’s high Summary Accuracy may help reviewers draft paper summaries or help readers grasp the core idea of recent preprints.The discussion proposes use for completing the summary description or quickly understanding a paper’s central idea.Source: §7.1 · p. 14
  • Generated reviews can cover more aspects and provide informative templates and evidence sentences, though their opinions may lack constructiveness.The proposed support may be particularly useful to junior or non-native English reviewers.Source: §7.1 · p. 14

7.2.1 Model

  • Scientific papers commonly exceed 5,000 words, motivating long-document methods; the authors also call for scientific-domain sequence-to-sequence models, structural information, and external knowledge.They report scientific-domain encoders with transformer decoders performed worse than BART on fluency and coherence.Source: §7.2.1 · p. 15

7.2.2 Datasets

  • The authors call for more open, fine-grained review data and more capable scientific-paper parsers, which still struggle with tables and figures.Source: §7.2.2 · p. 15

7.2.3 Evaluation

  • Future evaluation should address fairness and bias, factuality and reliability, confidence in comments, and calibration of review scores.Source: §7.2.3 · p. 15

7.3 Conclusion

  • The authors answer “not yet” to automating scientific review and propose their models, data, and analysis tools as a starting point for human-assisting systems.Source: §7.3 · p. 15

A.1 Reviews of this Paper Written by Our Model

  • The model-written review questions how the paper defines review quality, factual correctness, fairness, and non-factuality, and asks for clearer metrics and result explanations.It also asks for additional explanation of why the system produces non-factual assessment statements and why human reviewers are biased.Source: §A.1 · p. 20

A.3 Details for Evaluation Metrics

The appendix describes how recommendation sentiment, evidence, summary accuracy, and aspect constructiveness are judged, alongside a BERTScore rescaling procedure.

A.3 Details for Evaluation Metrics

  • Reference-review recommendation sentiment is derived from ratings, while two project members label generated-review sentiment and resolve disagreements as neutral.The annotators’ Cohen kappa is 0.5778, described as moderate agreement.Source: §A.3 · p. 20
  • For negative aspects, Informativeness counts whether any evidence is supplied, regardless of whether that evidence is correct.Source: §A.3 · p. 20
  • Paper authors judge Summary Accuracy and support for negative aspects on a 0, 0.5, or 1 scale for disagreement, partial agreement, or agreement.For aspects, agreement means the identified strength or weakness is reasonable or constructive.Source: §A.3 · pp. 20, 22
  • BERTScore is rescaled following Zhang et al. (2019) to make scores more discriminative without changing its ranking ability.Source: §A.3 · p. 22

A.4 Training of Aspect Tagger

The aspect tagger labels review text token by token, using BERT representations and a multilayer perceptron to assign aspect probabilities. Sentence-level inputs identified more aspects than paragraph-level context in the authors’ experiments.

A.4 Training of Aspect Tagger

  • The tagger maps a sequence of review words to a sequence of aspect tags, using BERT token representations as features for token classification.A multilayer perceptron produces probabilities that each token belongs to different aspects.Source: §A.4 · p. 22
  • Sentence-level inputs identified more aspects than paragraph-level context, which produced less satisfying results.The authors report that the model identified fewer aspects when using larger context such as a paragraph.Source: §A.4 · p. 22
  • Training minimizes the negative log-likelihood of the correct token labels.The model was trained on 900 annotated reviews and validated on 100; the selected checkpoint had the lowest validation loss.Source: §A.4 · p. 22

A.5 Heuristics for Refining Prediction Results

This appendix describes post-processing for aspect tags, evaluation of the tagger, long-document BART variants, and sentence extraction for review generation. It also gives an example annotation and the extraction procedure’s keyword and cross-entropy stages.

A.5 Heuristics for Refining Prediction Results

  • Seven sequential heuristics refine predicted aspect spans by smoothing tag boundaries, removing isolated or inconsistent tags, and adjusting summary-span endings.The rules include expanding some non-summary spans and truncating or extending summary spans by at most five words to end with a period.Source: §A.5 · pp. 22–23

A.6 An Example of Automatically Annotated Reviews

  • The example annotation shows the tagger and heuristic rules producing correct polarity labels for each predicted aspect.The example is presented without separate polarity tags to reduce visual burden.Source: §A.6 · p. 23

A.7 Calculation of Aspect Precision and Aspect Recall

  • Precision review lets annotators delete inappropriate predicted spans, while recall review asks annotators to label aspect spans the model missed.The recall procedure compares spans supplied by multiple annotators, including their shared overlaps.Source: §A.7 · pp. 23–24

A.8 Adjusting BART for Long Documents

  • Three long-document BART variants—expanded position encodings, independent windows, and dependent windows with memory—did not produce satisfying fluent, coherent text.The reported problems include shifts between first- and third-person narration, contradictions between consecutive sentences, and fewer opinions.Source: §A.8 · p. 24

A.9 CE Extraction Details

  • Salient-sentence extraction uses keyword filtering followed by a cross-entropy method.The method selects source sentences for the review-generation model’s input.Source: §A.9 · p. 25

A.9.1 Keywords Filtering

  • The keyword stage selects sentences containing any of 48 predefined keywords or their inflections.The appendix provides the keyword list and statistics for sentences selected after filtering.Source: §A.9.1 · p. 25, Table 13 · p. 25

A.9.2 Cross Entropy Method

  • The cross-entropy method samples sentence subsets and updates selection probabilities to maximize a score favoring diverse words.The selected sentences must fit BART’s 1024-token input length; the method iterates until its threshold value stops changing for three iterations.Source: §A.9.2 · p. 25

A.10 Detailed Analysis and Case Study

The case study examines generated aspect text and source attention patterns; the appendix also defines how aspect scores are calculated and averaged for accepted and rejected papers.

A.10 Detailed Analysis and Case Study

  • Generated review language is described as fairly close to that of real reviewers.The case study lists five examples for each aspect mentioned by the model.Source: §A.10 · p. 26
  • Attention patterns link generated summaries to proposal cues, originality to prior work and contributions, substance to experiment details, and comparisons to passages containing “et al.”The authors report that the model attends to what they consider reasonable source locations when generating specific aspects.Source: §A.10 · p. 26

A.11 Calculation of Aspect Score

  • For accepted and rejected papers, the appendix averages aspect scores; an aspect absent from a review receives a neutral score of 0.5.When an aspect appears, its score is based on its positive occurrences.Source: §A.11 · p. 26

A.12 Bias Analysis for All Models

The appendix presents bias analyses across all models using two figures and two tables.

A.12 Bias Analysis for All Models

  • Bias analyses for all models are presented in Figures 10 and 11 and Tables 14 and 15.Source: §A.12 · p. 26

B.1 Dataset Annotation Guideline

The appendix points to the aspect-annotation guideline and presents visuals and tables on sentiment, aspect scores, disparity, and generated-review examples.

B.1 Dataset Annotation Guideline

  • The aspect-annotation guideline is available in the ReviewAdvisor repository.The passage links to the annotation guideline PDF.Source: §B.1 · p. 26
  • Figure 9 maps attention between source documents and generated reviews, with signs marking positive and negative sentiment.Source: Figure 9 · p. 27
  • Figures 10 and 11 show model aspect scores for nativeness and anonymity, respectively.Source: Figure 10 · p. 27, Figure 11 · p. 28
  • Tables 14 and 15 report disparity differences for nativeness and anonymity; each total sums absolute disparity differences.Source: Table 14 · p. 27, Table 15 · p. 28
  • Table 16 gives examples of different aspect mentions in generated reviews.Source: Table 16 · p. 29
Loading 2102.00176v1…