PeerArg: Argumentative Peer Review with LLMs

1 Introduction

• Peer review involves subjective judgments that can be affected by confirmation and first-impression biases.The introduction describes agreement with reviewers’ beliefs and document layout as examples.Source: §1 · p. 1
• Prior NLP systems for review understanding can perform well, but their black-box nature makes their reasoning difficult to understand and trust.The paper positions PeerArg as a response to this interpretability concern.Source: §1 · p. 1
• PeerArg represents reviews with bipolar argumentation frameworks, combines those frameworks, and derives a paper-acceptance decision from the combined framework.The method uses support relations alongside attacks to model reviews and their aggregation.Source: §1 · p. 1, Figure 1 · p. 1
• The best PeerArg hyperparameter combination outperformed the few-shot end-to-end LLM on three review datasets.The comparison covered two conference-review datasets and one journal-review dataset.Source: §1 · p. 1

Understanding Reviews with AI

• AI applications in peer review include review summarization and generating meta-reviews from reviews.Kumar et al. (2023) generate pros-and-cons summaries, while Bhatia et al. (2023) propose MetaGen.Source: §2 · p. 2
• PeerRead provides reviews, decisions, and aspect scores, and has supported work on aspect-score prediction and review sentiment.Chakraborty et al. (2020) report a correlation between aspect-based sentiment distributions and paper acceptance.Source: §2 · p. 2

Argumentation in Peer Review

• Unlike prior peer-review argumentation work cited here, PeerArg considers relations between arguments.The related-work section names Hua et al. (2019) and Fromm et al. (2020) as exceptions that do not consider those relations.Source: §2 · p. 2
• Hua et al. (2019) classify argumentative text in reviews, while Fromm et al. (2020) study whether relevant arguments can inform acceptance decisions.Hua et al. report conference differences in argument trends; Fromm et al. report correct decisions using half of a review.Source: §2 · p. 2

3.1 Bipolar Argumentation

• Bipolar argumentation represents arguments with support and attack relations, and quantitative frameworks assign each argument a base score.A QBAF’s semantics determines each argument’s final strength; the paper names DF-QuAD and MLP-based semantics.Source: §3.1 · p. 2
• DF-QuAD combines supporter and attacker strengths to determine their influence on an argument’s base score.MLP-based semantics instead treats a QBAF as a multilayer perceptron, using edge weights -1 and +1 for attacks and supports.Source: §3.1 · pp. 2–3

3.2 Few-Shot Learning for LLMs

• Few-shot learning supplies task examples so an LLM can produce similarly formatted results without fine-tuning.The examples are called primers, and the task input is called the prompt.Source: §3.2 · p. 3

4 Paper acceptance by end-to-end LLMs

• The end-to-end LLM uses few-shot learning to classify paper acceptance from the paper’s reviews.It uses a quantised 4-bit pretrained Mistral-7B-0.1 model.Source: §4 · p. 3
• The primer contains reviews from four papers with final decisions: two accepts and two rejects.The examples come from the cited review datasets.Source: §4 · p. 3
• The target prompt presents the paper’s reviews without a decision, leaving the LLM to predict the outcome.The input template formats review-decision examples in the primer and omits the decision from the target prompt.Source: §4 · p. 4

5 Peer-Review Enhanced with KR

• PeerArg represents each review as an argumentation framework, combines the frameworks, and aggregates them to determine paper acceptance.The pipeline comprises review QBAF extraction, review QBAF combination, and pre-MPAF aggregation.Source: §5 · p. 4

5.1 Review QBAF Extractor

• Each review is modeled as text arguments linked through aspect arguments to a decision argument, using seven ACL 2016 aspects.Extraction classifies review sentences by aspect, analyzes sentiment, sets argument base scores, and applies QBAF semantics; aspect strengths below 0.5 attack the decision, while strengths at or above 0.5 support it.Source: §5.1 · pp. 4–5

5.2 Review QBAFs Combinator

• After removing text arguments, the system combines reviewers’ aspect and decision strengths into vectors in a pre-MPAF.The pre-MPAF retains the shared aspect and decision arguments, with each argument’s vector containing its strength from each reviewer’s framework.Source: §5.2 · p. 5

5.3 Pre-MPAF Aggregator

• Path 2 converts decision strengths into binary or five-level decisions, then applies majority-voting or all-accept aggregation.Binary majority voting rejects ties; five-level majority voting accepts only when the mapped weights sum to a positive value, while all-accept requires every decision to accept.Source: Figure 6 · p. 6, §5.3 · pp. 6–7
• Path 1 averages reviewer strength vectors to set argument strengths and decision base score, completes support and attack relations, then applies DF-QuAD or MLP-based semantics.The average strength also determines the relation between aspect and decision arguments.Source: §5.3 · p. 6, Figure 7 · p. 7

Experimental Setup

• The evaluation uses PRA, ACL 2017 PeerRead, and medical-field reviews from MOPRD.PRA and PeerRead are conference-review datasets; MOPRD is a journal-review dataset.Source: §6 · p. 7
• Four remaining PeerArg choices are varied: base-score setting, QBAF semantics, decision-strength interpretation, and aggregation semantics.Aspect classification uses a few-shot LLM, while sentiment analysis uses a pretrained model.Source: §6 · p. 8

Experimental Results

• Combination (4) uses sentiment strengths as text-argument base scores, DF-QuAD semantics, uniform five-level decision interpretation, and majority-voting.Table 2 identifies it as the best overall combination.Source: §6 · p. 8, Table 2 · p. 8
• The best overall combination (4) outperformed the end-to-end LLM on all three datasets.The comparison covers PRA, PeerRead, and MOPRD.Source: §6 · p. 8, Figure 8 · p. 8, Figure 9 · p. 8

Conclusion

• PeerArg combines LLMs with computational argumentation to support acceptance prediction from reviews, with greater transparency than the black-box end-to-end LLM.The reported experiments show PeerArg can outperform the end-to-end LLM.Source: §7 · p. 9
• Future work would use argumentation to improve review-aggregation explainability and combine text arguments into pre-MPAFs.The authors also propose exploring uncertainty in aggregation and its effect on acceptance prediction.Source: §7 · p. 9

A Appendix (Technical Details)

• The appendix outlines models involved in PeerArg's argument-mining process.The named components are aspect classification and sentiment analysis.Source: A Appendix (Technical Details) · p. 9
• Aspect classification is one component of the argument-mining process.Source: A Appendix (Technical Details) · p. 9
• Sentiment analysis is also part of the argument-mining process.Source: A Appendix (Technical Details) · p. 9

A.1 Aspect Classification

• Aspect classification uses a quantised 4-bit pretrained Mistral-7B-v0.1 model with few-shot learning.Source: §A.1 · p. 9
• The primer presents seven aspect criteria and ten review-sentence examples with their corresponding aspects.Source: §A.1 · p. 9
• The prompt supplies an unlabeled review sentence for aspect prediction; preprocessing removes newlines, backward slashes, and leading dashes.Figure 11 shows the partial primer-and-prompt template.Source: §A.1 · p. 9, Figure 11 · p. 9

• Sentiment analysis is performed separately for each sentence in a review.Source: §A.2 · p. 9
• A pretrained RoBERTa model trained on Twitter tweets is fine-tuned for sentiment analysis7.Source: §A.2 · p. 9
• For each input text, the model returns a positive, neutral, or negative label and a strength indicating how likely that sentiment is.Source: §A.2 · p. 9