PaperlayerView exact revision on arXiv

Source-linked AI summary

Identifying Aspects in Peer Reviews

Sheng Lu, Ilia Kuznetsov, Iryna Gurevych

arXiv:2504.06910v3cs.CL

TL;DR

Peer-review aspects are important for consistent assessment, yet existing guideline-derived schemata are poorly formalized and often too coarse. The paper derives aspects bottom-up with an LLM, constructs a taxonomy and augmented dataset, and shows that fine-grained aspects support review comparison, specificity assessment, and interpretable LLM-generated review detection.

  • Problem

    Aspect analysis for peer review lacks an operational definition, while existing guideline-derived schemata may be insufficiently comprehensive and granular for NLP applications.

  • Method

    The paper uses an LLM in a semi-automatic, bottom-up process to derive aspects from peer reviews, then builds a multi-level taxonomy and augmented dataset.

  • Results

    Fine-grained, data-driven aspects complement coarse guideline-based aspects and enable nuanced review comparison and interpretable LLM-generated review detection.

  • Takeaways & Limitations

    The resulting aspect framework supports paper and review aspect prediction, specificity-oriented review analysis, and new NLP applications for peer review.

  • Takeaways & Limitations

    The study focuses on NLP and does not construct a cross-domain taxonomy using review data and domain experts from other fields.

Abstract

from arXiv · show

Peer review is central to academic publishing, but the growing volume of submissions is straining the process. This motivates the development of computational approaches to support peer review. While each review is tailored to a specific paper, reviewers often make assessments according to certain aspects such as Novelty, which reflect the values of the research community. This alignment creates opportunities for standardizing the reviewing process, improving quality control, and enabling computational support. While prior work has demonstrated the potential of aspect analysis for peer review assistance, the notion of aspect remains poorly formalized. Existing approaches often derive aspects from review forms and guidelines, yet data-driven methods for aspect identification are underexplored. To address this gap, our work takes a bottom-up approach: we propose an operational definition of aspect and develop a data-driven schema for deriving aspects from a corpus of peer reviews. We introduce a dataset of peer reviews augmented with aspects and show how it can be used for community-level review analysis. We further show how the choice of aspects can impact downstream applications, such as LLM-generated review detection. Our results lay a foundation for a principled and data-driven investigation of review aspects, and pave the path for new applications of NLP to support peer review.

1 Introduction

Peer-review aspects support consistent quality assessment, but existing coarse, guideline-derived schemata lack an operational definition and sufficient granularity. This work derives aspects bottom-up from reviews, builds an augmented dataset and taxonomy, and evaluates applications including aspect prediction and review analysis.

  • The paper addresses three gaps: aspect lacks an operational definition, appropriate granularity is unclear, and fine-grained applications remain underexplored.
  • The authors propose a semi-automatic, bottom-up approach that uses an LLM to derive aspects from a large collection of peer reviews.
  • They develop a multi-level taxonomy and a dataset of peer reviews augmented with their corresponding aspects.
  • The dataset supports paper aspect prediction and review aspect prediction, enabling empirical analysis at the community level.
  • Fine-grained aspect analysis provides a new dimension for comparing reviews, supports nuanced specificity assessment, and enables detection of automatically generated reviews.

2 Related Work

Prior peer-review NLP research uses aspects for review analysis, generation, writing support, and LLM-related assistance, but commonly relies on coarse or incomplete guideline-derived schemata. This paper positions its contribution as a data-driven route to finer-grained and more comprehensive aspects, including practical review-quality analysis.

  • LLMs have created additional peer-review applications, including checklist verification, review writing, and aspect identification.
  • Existing NLP work applies aspects to sentiment analysis, discourse relations, aspect-based review generation, and review-writing support.
  • Review guidelines such as ARR and ACL’23 often list aspects as examples, while major NLP-venue guidelines commonly use coarse-grained and non-comprehensive categories.
  • The paper advances prior work by deriving finer-grained and more comprehensive aspects through a data-driven approach.
  • Fine-grained aspects can compare reviews and assess specificity, addressing the documented challenge that reviews often lack specificity.
  • LLM-generated review detection is motivated by concerns that reviewers may use LLMs to produce reviews entirely, creating ethical risks.

3 Aspect set construction

The paper operationalizes review aspects as paper characteristics judged by reviewers and constructs a multi-level taxonomy from LLM-identified aspects in a diverse review corpus. The resulting dataset preserves fine-grained aspect annotations while documenting aggregation choices and their limitations.

  • 3.1 Operational definition: An aspect is a paper characteristic judged by a reviewer and used to compare manuscripts with one another and with review-guideline standards.The framework allows multiple aspects in one review sentence and represents each review's aspects as a set.
  • 3.2 Method: The corpus contains 350 papers and 1,094 reviews sampled from NLP and machine-learning conferences across multiple time periods, with sentence-level aspect identification.The sampled papers include NLPeer, EMNLP, and NLP-related ICLR papers.
  • 3.2 Method: Post-processing can group unrelated terms, although the authors manually checked results to correct inappropriate groupings; some aspects may also be absent because review forms capture them separately.The paper specifically notes that Improvement can merge Performance Improvement with Improvement Recommendation, and that Reproducibility may be underrepresented in review text.
  • 3.3 Results: GPT-4o initially identified 14,574 unique aspects; filtering removed 9,764 terms, leaving 4,810 aspects that occurred 25,394 times.The remaining frequent aspects include Comparison, Clarity, Performance, Experiments, and Results.
  • 3.3 Results: The taxonomy organizes aspects into 16 broad categories across COARSE, FINE, and raw LLM-annotation levels, while distinguishing paper-agnostic from paper-dependent aspects.A single term may belong to multiple categories, such as Comparison with Related Work appearing under both Comparison and Related Work.
  • 3.3 Results: LLM annotations showed moderate-to-strong consistency across prompts and temperatures, while human annotators agreed with 91% of mapped COARSE annotations.Consistency was measured with exact match, BERTScore, and Jaccard similarity; 45.50% and 67.14% of annotations exceeded 0.9 BERTScore similarity under different settings.

4 Aspect prediction

The paper evaluates paper aspect prediction and review aspect prediction as multi-label classification tasks using conventional supervised models and GPT-4o. Prediction is generally easier for review aspects than paper aspects, and coarse labels outperform fine labels across tasks.

  • 4.1 Task and models: PAP uses paper information to predict paper-dependent aspects, whereas RAP uses supervised models to identify aspects directly from segmented review sentences.Both tasks are modeled as multi-label sequence classification with BoW+RF and RoBERTa; PAP additionally tests GPT-4o.
  • 4.2 Results: PAP models perform poorly overall across input types, with a small BoW+RF advantage when the full paper is used as input.COARSE labels substantially outperform FINE labels, partly because frequent labels such as Methodology occur in nearly all papers.
  • 4.2 Results: GPT-4o outperforms RoBERTa in PAP despite limited few-shot evaluation and tends to capture heuristics better.GPT-4o uses zero-shot evaluation for full-paper or abstract inputs and few-shot evaluation for the remaining settings.
  • 4.2 Results: RoBERTa is the best-performing model for RAP, with COARSE-label models outperforming FINE-label models.Few-shot GPT-4o provides only a small improvement over zero-shot GPT-4o in RAP.
  • 4.2 Results: Across both tasks, COARSE labels yield higher performance, while RAP generally performs better than PAP.RAP extracts aspects from review text, whereas PAP must infer paper-dependent aspects from paper information or heuristics.

5 Practical applications

Fine-grained aspects support cross-track review analysis and reveal greater similarity among LLM-generated reviews, enabling interpretable detection and task-specific granularity choices.

  • 5.1 Aspect analysis: Reviewers emphasize different aspects across tracks: Resources and Evaluation focuses more on Data/Task and Evaluation, while Analysis is most frequent in Computational Social Science and Cultural Analytics.These distributions support comparing reviewing practices across tracks and venues.
  • 5.1 Aspect analysis: Question Answering is most similar in its frequent-aspect profile to Summarization and Information Extraction.Track similarity is based on Levenshtein similarity among the ten most frequent aspects.
  • 5.2 Review comparison: LLM-generated reviews have higher aspect similarity to one another, suggesting they are more generic and less specific than human-written reviews.The comparison uses predicted aspects and Jaccard similarity between review pairs.
  • 5.3 LLM-generated review detection: The FINE aspect set consistently outperforms ACL’18 and SBERT for detecting LLM-generated reviews and remains robust across prompts.Table 6 compares Jaccard-based detection with SentenceBERT across COARSE, FINE, and ACL’18 aspect sets.
  • 5.3 LLM-generated review detection: The detection approach is interpretable but underperforms reference-based and zero-shot methods, motivating better aspect-based detectors.The paper reports 90% accuracy for a cited reference-based approach and leaves higher-performing detectors to future work.
  • 5.4 Recommendations: Coarse labels suit high-level cross-track analysis, whereas fine labels better support nuanced review comparison and LLM-generated review detection.A hybrid strategy can assess coverage with COARSE labels and specificity with FINE labels when evaluating review quality.

6 Conclusion

The paper introduces a data-driven framework for peer-review aspect analysis, combining an operational definition, semi-automatic extraction, a multi-granularity taxonomy, and an augmented dataset. It shows that fine-grained aspects complement guideline-based coarse aspects for nuanced review comparison and interpretable LLM-generated review detection.

  • 6 Conclusion: The paper introduces a data-driven aspect-analysis framework with an operational definition, semi-automatic identification method, taxonomy, augmented dataset, and two review-analysis tasks.The tasks are paper aspect prediction and review aspect prediction.

Limitations

The study identifies limitations involving aspect definitions, taxonomy construction, annotation consistency, model choice, human annotation bias, task formulation, and dataset scope.

  • The operational definition and individual aspect boundaries remain difficult to specify, so future work may refine and expand the definition.Examples include ambiguity in the scope of “Methodology” and polysemy in “Improvement”; manual inspection was used to reduce these issues.
  • The taxonomy was created by one expert, may not be optimal, and would benefit from validation through multiple experts.The authors identify multi-expert fine-grained aspect identification as promising future work.
  • The paper focuses on NLP and uses camera-ready papers for PAP experiments, which may limit generalization to other domains and lower-quality submissions.The authors distinguish this concern from review comparison and LLM-generated review detection, where camera-ready versions are not considered problematic.
  • Annotation quality declined during the questionnaire, and using predefined aspect labels may introduce label bias or automation bias.The authors suggest open-ended annotation and alternative strategies as possible follow-up directions.
  • The study uses GPT-4o annotations and does not compare alternative models because constructing multiple taxonomies would require substantial expert involvement.The authors note that experimenting with open models and alternative aspect schemata is a future possibility.
  • The binary formulation of paper aspect prediction may not reflect real-world differences in aspect importance, motivating ranking or regression alternatives.The paper leaves these formulations for future work.

Ethics Statement

The proposed aspect-prediction task is intended to assist reviewers rather than replace human involvement, and reviewers retain autonomy over suggested aspects.

  • Aspect prediction is designed to recommend aspects for reviewers to consider, while reviewers retain full autonomy over whether to include them.

A Using the proposed aspect set to evaluate this paper

The self-review evaluates the paper using the proposed aspect set across its motivation, definitions, workflow, dataset, experiments, and applications.

  • The paper derives a comprehensive aspect set, creates an augmented review dataset, and evaluates it on two tasks and practical applications.
  • The paper defines aspects operationally and organizes them into a taxonomy that supports analysis at multiple levels of granularity.
  • The paper addresses the lack of comprehensive review aspects by proposing a bottom-up, LLM-assisted identification workflow.
  • The self-review covers evaluation metrics, experimental settings, implementation details, and comparisons across aspect prediction and review detection.
  • The paper reports that LLM-generated reviews are more generic than human-written reviews and that aspect-based models outperform models using previous aspect sets.
  • The dataset contains review sentences augmented with LLM annotations mapped to both COARSE and FINE label sets.

B.2 More on validity check

The validity checks examine annotation consistency across prompts and temperatures and compare human agreement with LLM annotations across annotators and aspects.

  • The consistency analysis varies prompts and temperatures and evaluates raw and COARSE-mapped annotations using exact match, BERTScore, and Jaccard similarity.The corresponding consistency results are reported in Table 10, with examples in Tables 11 and 12.
  • Human validation used three screened Prolific annotators and sampled 100 reviews totaling 1,852 sentences while seeking class balance.
  • 85.19%, 90.11%, and 97.56% are the three annotators’ agreement rates with LLM annotations.The rates represent the percentage of annotations on which each annotator agreed with the LLM.
  • Fleiss’ Kappa indicated substantial agreement for the first 400 questionnaire entries but declined as annotation progressed, suggesting inconsistent quality over time.The entries were randomly sampled, so the authors state that the later items should not have been substantially more difficult.
  • Agreement exceeded 0.9 for most aspects, with Contribution and Ablation strongest and Significance and Presentation weakest.

C More on aspect prediction

The aspect-prediction experiments compare bag-of-words, RoBERTa, and GPT-4o configurations across PAP and RAP, using fine and coarse label sets. Additional tables report model metrics, prompts, and the best RAP classification report.

  • Experimental setup: Models predict aspects on 90% training and 10% testing splits using bag-of-words random forests, RoBERTa, and GPT-4o configurations.The random forest uses 100 estimators; RoBERTa uses batch size 16, learning rate 3e-5, and 10 epochs.
  • Experimental setup: Focal-loss experiments vary α across [0.1, 0.2, 0.3] and γ across [1.5, 2.0, 2.5].
  • Results: Tables report precision, recall, F1 score, and Jaccard similarity for PAP and RAP under fine and coarse label sets and different GPT-4o prompting settings.The reported settings include zero-shot and few-shot evaluation with titles, abstracts, or full papers as inputs.
  • Results: The best RAP model is summarized with a classification report using the coarse label set and weighted-average scores.

D.1 More on aspect analysis

The additional aspect analysis examines similarities and frequencies across EMNLP23 submission tracks, then compares human-written and LLM-generated reviews. It also reports zero-shot GPT-4o performance for review detection.

  • D.1 More on aspect analysis: The 1st–18th EMNLP23 submission tracks have more similar sets of frequent aspects, whereas the 19th–27th tracks are less similar to most other tracks.The first group includes Question Answering, Information Retrieval, and several modeling, application, and resource tracks; the second includes Sentiment Analysis, Machine Translation, Ethics in NLP, and others.
  • D.1 More on aspect analysis: The analysis identifies the five most frequent aspects in each submission track and reports frequencies of Analysis and DDDDEI across tracks.
  • D.1 More on aspect analysis: Figure 6 compares human-written and LLM-generated reviews using prompts from Du et al. (2024), Liang et al. (2024b), and this work.
  • D.1 More on aspect analysis: Table 24 reports GPT-4o’s zero-shot accuracy for detecting LLM-generated reviews.
Loading 2504.06910v3…