Identifying Aspects in Peer Reviews

1 Introduction

• Rising publication volume and review-quality challenges motivate computational support, while form-derived aspects lack an operational definition and may be too coarse for NLP tasks.Source: §1 · p. 1
• The authors define aspects operationally and use a semi-automatic LLM-based method to identify them bottom-up in peer reviews.The method is applied to a large review collection.Source: §1 · pp. 1–2
• The work develops a multi-level aspect taxonomy and a dataset of peer reviews annotated with corresponding aspects.Source: §1 · p. 2
• The dataset supports predicting aspects to focus on given a paper and identifying the aspects covered by a review.The authors use these tasks for a community-level empirical study of aspects.Source: §1 · p. 2
• Fine-grained aspects enable comparisons across reviews, nuanced assessment of review specificity, and detection of automatically generated reviews.Source: §1 · p. 2

2.1 Peer review in the era of LLMs

• This work uses an LLM to identify review aspects, building on prior evidence of LLM performance in aspect identification and summarization.NLP for peer review also targets paper-reviewer matching, efficiency, reproducibility, and dishonest behavior; LLMs have been used to verify checklists and support review writing.Source: §2.1 · p. 2

2.2 Aspect in peer review

• Prior work uses aspects to analyze review sentiment and discourse, generate reviews, and support review writing; this work derives finer-grained, more comprehensive aspects from data.Guidelines such as ARR and ACL’23 present aspects as examples, while major NLP venue guidelines are described as coarse-grained and lacking comprehensiveness.Source: §2.2 · pp. 2–3

2.3 Quality of review writing

• Fine-grained aspects let this work compare reviews and evaluate specificity, addressing a key quality desideratum amid reported lack of specificity in ACL 2023.Review desiderata are often poorly defined or difficult to measure automatically, and simple helpfulness proxies can be biased toward longer reviews or acceptance recommendations (Wang et al., 2021; Goldberg et al., 2023).Source: §2.3 · p. 3

2.4 LLM-generated review detection

• Against existing LLM-generated text detection approaches, this work demonstrates that comprehensive, fine-grained aspects help detect LLM-generated reviews.The approaches described include LLM-as-a-judge, detection models trained on human and generated text, and reference-based similarity methods.Source: §2.4 · p. 3

3.1 Definition of aspect

• An aspect is a paper characteristic judged by reviewers against other manuscripts and guideline standards; a sentence may express multiple aspects, which are modeled as sets.Acceptance Decision is excluded because it is not a characteristic of a paper.Source: §3.1 · p. 3

3.2 Method

• They sampled 350 papers and 1,094 reviews across NLP and ML conference datasets, segmented reviews into sentences, and used GPT-4o for unsupervised sentence-level aspect identification.The ICLR sample covered NLP-related papers from 2020 to 2024.Source: §3.2 · p. 3
• To normalize noisy outputs, they grouped related terms, removed generic terms, omitted unmatched terms occurring fewer than 50 times, manually corrected inappropriate groupings, and allowed terms to belong to multiple categories.For example, Comparison with Related Work appears in both Comparison and Related Work categories.Source: §3.2 · p. 4

3.3 Results

• GPT-4o identified 14,574 unique aspects; 9,764 excluded terms occurred 26,578 times, while 4,810 retained aspects appeared 25,394 times.The most common retained aspects were Comparison, Clarity, Performance, Experiments, and Results.Source: §3.3 · p. 4
• The 16-category taxonomy has COARSE, FINE, and raw LLM-annotation levels, distinguishes paper-agnostic from paper-dependent aspects, and may omit form-scored aspects such as Reproducibility.Table 2 presents the taxonomy.Source: §3.3 · pp. 4–5, Table 2 · p. 4

3.4 Validity check

• 45.50% and 67.14% of annotations across prompt and temperature settings had BERTScore similarity greater than 0.9; coarse-label Jaccard indicated strong consistency, and humans agreed with 91% on average (fair Fleiss’ Kappa).Mapping annotations to COARSE labels increased exact matches across settings.Source: §3.4 · p. 5

4.1 Method

• PAP uses paper-dependent aspects and tests titles, keywords, abstracts, and full papers, with the shorter inputs serving as similarity-based heuristics.Full-paper input may reveal nuanced strengths or weaknesses that are not apparent from the title, keywords, or abstract.Source: §4.1 · p. 5
• PAP is more challenging than RAP: it infers aspects from heuristic or full-paper evidence, whereas RAP extracts them directly from review text.Source: §4.1 · p. 5
• Both tasks use multi-label sequence classification with BoW+RF and RoBERTa (Liu et al., 2019); PAP also tests GPT-4o, while RAP trains supervised predictors on review sentences.Focal loss (Lin et al., 2017) addresses label imbalance; evaluation uses precision, recall, F1, and Jaccard similarity.Source: §4.1 · pp. 5–6

4.2 Results

• PAP models perform poorly overall; BoW+RF has a small advantage with full-paper input, while few-shot GPT-4o outperforms full-data RoBERTa and better captures heuristics.The COARSE label set performs much better than FINE, partly because common aspects such as Methodology occur in nearly all papers.Source: §4.2 · p. 6, Table 3 · p. 6
• Models perform better on RAP than PAP overall, and COARSE labels yield higher performance across both tasks.The results tables report PAP results for the FINE label set and RAP results for the COARSE label set; the prose also describes the COARSE-over-FINE pattern across both tasks.Source: §4.2 · p. 6, Table 4 · p. 6, Table 3 · p. 6
• For RAP, RoBERTa performs best, and GPT-4o few-shot offers a small improvement over zero-shot.The results also report higher performance with COARSE than FINE labels.Source: Table 4 · p. 6, §4.2 · p. 6

5 Practical applications

• The selected RoBERTa model using COARSE labels achieves F1 above 0.75 on 10 of 17 labels, providing predicted aspects for the practical analyses.The model was selected for having the best F1 score.Source: §5 · p. 6

5.1 Aspect analysis

• Across EMNLP23 tracks, review focus is similar for Machine Translation and Multilinguality and Linguistic Diversity, but Resources and Evaluation emphasizes Data/Task and Evaluation more.Analysis is most frequent in Computational Social Science and Cultural Analytics; such comparisons can inform track-specific review forms and guidelines.Source: §5.1 · pp. 6–7, Figure 1 · p. 6, Table 5 · p. 7

5.2 Review comparison

• LLM-generated reviews show greater aspect similarity to one another, suggesting they are more generic and less specific than human-written reviews.The comparison uses Jaccard similarity between predicted aspect sets.Source: §5.2 · p. 7, Figure 2 · p. 7

5.3 LLM-generated review detection

• The FINE aspect set consistently outperforms ACL’18 and SBERT for detecting LLM-generated reviews and remains robust across prompts.The interpretable detector compares intra-paper with inter-paper aspect similarity; although it underperforms reference-based and zero-shot approaches, finer labels reveal distinctions that coarse labels miss.Source: §5.3 · pp. 7–8, Table 6 · p. 8

5.4 Recommendations

• COARSE labels suit high-level analysis across tracks and venues, while FINE labels suit nuanced comparison and LLM-review detection; a hybrid can assess coverage first and specificity second.The optimal granularity depends on the task and user needs.Source: §5.4 · p. 8

• Fine-grained, data-driven aspects complement coarse guideline-derived aspects and enable more nuanced review comparison and interpretable approaches to LLM-generated review detection.Source: §6 · p. 8
• The paper operationalizes the notion of peer-review aspect and develops a semi-automatic, data-driven approach to identifying aspects in reviews.Source: §6 · p. 8
• The authors propose a multi-granularity aspect taxonomy and introduce a peer-review dataset annotated with aspects, supporting paper-aspect and review-aspect prediction tasks.Source: §6 · p. 8

Limitations

• The operational aspect definition and taxonomy may not be optimal: aspect scope and polysemy are difficult, and the taxonomy was built by one expert annotator.Manual inspection was used to reduce categorization issues; the authors suggest refining the definition and studying fine-grained identification with multiple experts.Source: Limitations · p. 9
• The study focuses on NLP; cross-domain analysis would require data and domain experts from other fields to construct a new taxonomy.The authors leave this substantial effort to future work.Source: Limitations · p. 9
• GPT-4o annotations show some prompt and temperature sensitivity, and closed weights limit exact reproducibility despite validity checks suggesting reasonable consistency.The annotations are based on GPT-4o; comparing models would also require multiple taxonomies and expert involvement, while more capable open models may enable future experiments.Source: Limitations · p. 9
• Predefined aspect labels may bias human annotation, while open-ended labeling poses cognitive and consistency challenges and explanations may not fully resolve automation bias.The authors propose exploring alternative experimental strategies to measure and mitigate automation bias.Source: Limitations · p. 9
• The PAP task treats aspect relevance as binary, although ranking or regression may better represent differing aspect importance in practice.The authors leave these alternative formulations for future work.Source: Limitations · p. 9
• Camera-ready papers may limit PAP validity, and acceptance imbalance may skew aspect frequencies; EMNLP23 has only 9 rejected papers, while 69% of NLPeer papers are accepted (Dycke et al., 2023).Overrepresented aspects could bias review forms or guidelines; the authors note more review data could mitigate the limitation. They do not consider camera-ready versions problematic for review comparison or LLM-generated review detection.Source: Limitations · p. 9
• The review-generator study uses commonly used end-to-end general prompts, not paper-specific prompts that require a human to identify aspects after reading the paper.The authors characterize paper-specific prompting as human–AI collaboration and leave its study to future work.Source: Limitations · p. 10

Ethics Statement

• The system recommends aspects for reviewers to consider without replacing human involvement; reviewers decide whether to use them, and the authors report no ethical concerns from processing public, anonymized reviews with commercial LLMs.Source: Ethics Statement · p. 10

A Using the proposed aspect set to evaluate this paper

• The self-review is neutral and illustrative rather than exhaustive, is not LLM-generated, and covers both paper-agnostic and paper-dependent aspects.Source: A Using the proposed aspect set to… · p. 13
• The paper derives a comprehensive aspect set and taxonomy from NLP reviews, introduces an aspect-augmented dataset, and provides an operational definition of aspect.Source: A Using the proposed aspect set to… · p. 13
• The proposed aspects may support better review guidelines and LLM-generated review detection.Source: A Using the proposed aspect set to… · p. 13
• The paper-dependent self-review emphasizes Data/Task and Methodology, and reports analysis, comparisons, two tasks, and evaluation with BERTScore, SentenceBERT, and Jaccard similarity.Source: A Using the proposed aspect set to… · p. 13
• LLM-generated reviews are more generic than human-written reviews, and models trained with the proposed aspect sets perform better than with previous ones.Source: A Using the proposed aspect set to… · p. 13

B More on aspect set construction

• GPT-4o used Table 7’s prompt on EMNLP23 and ICLR reviews without the proposed definition, which was applied later to filter model outputs.Source: B More on aspect set construction · p. 14, Table 7 · p. 14
• 10 hours of expert work went to verifying frequency-filtered LLM annotations; taxonomy construction then took one expert approximately 72 hours.Source: B More on aspect set construction · p. 14

B.1 More on results

• Dataset examples pair each review sentence with an LLM annotation mapped to COARSE and FINE labels; Table 9 reports aspect frequencies by granularity.Source: §B.1 · p. 14

B.2 More on validity check

• The authors tested prompt robustness by replacing “aspect” with “facet” and varying temperature between 0 and 1, comparing consistency for raw and COARSE-mapped annotations.They also presented examples of how text-pair and set-pair comparisons correspond to BERTScore and Jaccard similarities.Source: §B.2 · p. 14, Table 10 · p. 16, Table 11 · p. 17
• Three screened computer-science graduate annotators assessed GPT-4o labels on 1,852 sentences from 100 reviews, providing explanations across 3,032 questionnaire entries.The review sentences were sampled to maintain class balance as much as possible; annotators were required to hold a graduate or doctorate degree in Computer Science, have English as their first language, and have a Prolific approval rate above 90%.Source: §B.2 · p. 14
• 85.19%, 90.11%, and 97.56% were the three annotators’ human–LLM agreement rates; Fleiss’ Kappa declined after the first 400 entries, with overall κ=0.1944.The authors note that entries were randomly sampled and should not have differed substantially in difficulty, so the decline may indicate inconsistent annotation quality over time.Source: §B.2 · pp. 14–15, Figure 3 · p. 15
• Human–LLM agreement exceeded 0.9 for most aspects, was highest for Contribution and Ablation, and was weakest for Significance and Presentation.The comparison is reported across aspects in Table 15.Source: §B.2 · p. 15, Table 15 · p. 17

C More on aspect prediction

• The dataset uses a 90/10 training-test split, and the BoW+RF model was implemented with scikit-learn’s RandomForestClassifier.Source: C More on aspect prediction · p. 18
• Additional tables report highest precision, recall, F1, and Jaccard for PAP-COARSE and RAP-FINE, plus zero-shot GPT-4o results on RAP.For PAP-COARSE, GPT-4o uses full-paper or abstract input in zero-shot and otherwise few-shot; RAP-FINE GPT-4o results are few-shot, while the separate RAP GPT-4o table is zero-shot.Source: C More on aspect prediction · p. 18, Table 18 · p. 18, Table 19 · p. 18
• The RAP classification report covers the best-F1 model trained on COARSE labels and reports weighted-average scores.Source: C More on aspect prediction · p. 18, Table 21 · p. 19
• 0.8694 precision, 0.9430 recall, 0.8909 F1, and 0.7952 Jaccard for BoW+RF, versus 0.8580 precision, 0.7215 recall, 0.7452 F1, and 0.6813 Jaccard for GPT-4o on PAP-FINE.For GPT-4o, the reported evaluation used the abstract in zero-shot and the title in few-shot settings.Source: Table 17 · p. 18, C More on aspect prediction · p. 18

D.1 More on aspect analysis

• The 10 most frequent aspects are more similar across EMNLP23 submission tracks 1–18 than across tracks 19–27.Figure 4 reports Levenshtein similarity; tracks 19–27 are described as less similar to most other tracks.Source: Figure 4 · p. 19, §D.1 · p. 19
• The appendix shows the five most frequent aspects for each EMNLP23 submission track and tabulates the frequencies of Analysis and DDDDEI across tracks.The first summary corresponds to Figure 5 and the frequency table to Table 22.Source: §D.1 · p. 19, Figure 5 · p. 20, Table 22 · p. 21

D.2 More on review comparison

• Table 23 provides the prompts used to generate reviews for the paper’s review-comparison analysis.Source: §D.2 · p. 19, Table 23 · p. 22

D.3 More on LLM-generated review detection

• For detection, reviews were generated only from main-text PDF content, using GPT-4o with temperature 0, seed 2266, and max_tokens 2048.PDFs were converted to text and content after the references was cut.Source: §D.3 · p. 19

D.2 More on review comparison

• Figure 6 compares human-written and LLM-generated reviews using Jaccard similarity across prompts from Du et al. (2024), Liang et al. (2024b), and this work.Source: §D.2 · p. 19, Figure 6 · p. 23

D.3 More on LLM-generated review detection

• 69.23 for Computational Social Science and Cultural Analytics, 68.75 for Linguistic Theories, Cognitive Modeling, and Psycholinguistics, and 62.62 for Commonsense Reasoning are reported in the GPT-4o detection results.These are the track-associated accuracy figures listed in the appendix.Source: §D.3 · p. 21, Table 24 · p. 23
• The detection results also report 60.42 for Linguistic Theories, Cognitive Modeling, and Psycholinguistics and 59.46 for Interpretability, Interactivity, and Analysis of Models for NLP.These are additional track-associated accuracy figures listed in the appendix.Source: §D.3 · p. 21, Table 24 · p. 23