ReviewScore: Misinformed Peer Review Detection with Large Language Models

Empirical motivation

• 26.4% of questions and 15.2% of weaknesses are misinformed in the annotated AI-conference reviews.These findings indicate a considerable presence of misinformed review points.Source: §1 · p. 2

Results

• Eight state-of-the-art LLMs show only moderate human agreement, achieving 0.4–0.5 F1 scores and 0.3–0.4 Kappa scores; models misunderstand review points or predict minority-annotator scores.ADVANCED REVIEWSCORE clearly outperforms BASE REVIEWSCORE, demonstrating the effectiveness of premise-level factuality scoring.Source: §1 · p. 2

REVIEWSCORE criteria

• REVIEWSCORE identifies questions answerable by the paper and weaknesses containing incorrect premises as misinformed review points.Weaknesses are further divided into claims without supporting reasons and arguments with supporting reasons.Source: §1 · pp. 1–2, Figure 1 · p. 1

Method

• The paper reconstructs explicit and implicit premises from arguments before evaluating premise factuality with an automatic argument reconstruction engine.ADVANCED REVIEWSCORE aggregates premise factuality, complementing BASE REVIEWSCORE’s direct weakness-level assessment.Source: §1 · p. 2

Dataset

• The authors construct a human expert-annotated dataset from ICLR reviews to measure the reliability of automatic REVIEWSCORE evaluation.Fifteen experienced graduate students contributed 244 hours of annotation.Source: §1 · p. 2

Review evaluation

• Earlier methods either apply broadly but vague criteria or specific criteria limited to narrow review scopes.The related work contrasts general review-quality rubrics with criteria targeting predefined review types, groundedness, or claims.Source: §2 · p. 3

Factuality evaluation

• Sentence-level factuality does not fully capture the underlying logic of weaknesses, motivating premise-level factuality evaluation.The paper reconstructs explicit and implicit premises before evaluating whether each premise is factually correct.Source: §2 · p. 3

Argument reconstruction

• Unlike prior reconstruction work using synthetic or polished data, this paper reconstructs real-world unpolished peer-review arguments using the full paper context.The reconstruction is designed for arguments whose meaning depends on understanding the target paper.Source: §2 · p. 3

3.1 Definition

• ReviewScore measures whether a review point is misinformed, targeting weaknesses with incorrect premises and questions answerable by the submitted paper.Source: §3 · p. 3, §3.1 · p. 3
• Advanced ReviewScore evaluates arguments by aggregating factuality scores for their explicit and implicit premises, while retaining claim and question scores.Premises are selected from the submitted paper, annotator knowledge, or referred papers.Source: §3.1 · p. 4
• Logical conjunction treats an argument as true only when all premises score 3–5, whereas weighted averaging weights premises by their untrivialness.Untrivialness is judged using common knowledge of CS/AI-majoring undergraduates.Source: §3.1 · p. 4

3.2 Automatic Argument Reconstruction

• The reconstruction engine uses LLM generation, SAT-checked first-order logic validity, faithfulness feedback, and up to 10 regeneration iterations.With Claude Sonnet 3.7, every reconstruction was valid and average faithfulness was 4.5 / 5.Source: §3.2 · p. 5, Figure 2 · p. 5

3.3 Dataset Construction

• The dataset contains 657 review points—143 questions, 92 claims, and 422 arguments—with 1,748 argument premises manually annotated.Fifteen graduate students annotated reviews from 40 ICLR 2021–2023 papers using collaborative curation and rechecking.Source: §3.3 · p. 6
• Human annotation used guidelines, orientation, communication, independent rechecking, and final discussion, achieving median and highest Krippendorff’s Alpha values of 0.489 and 0.663.Source: §3.3 · p. 6

4.2 Main Results

• Most models achieve 0.4–0.5 F1 and 0.3–0.4 Kappa, indicating moderate human-model agreement on ReviewScore evaluation.Source: §4.2 · p. 7, Table 2 · p. 7
• Questions show the highest agreement, arguments follow, and claims show the lowest, with some models reaching zero F1 or nearly zero Kappa on claims.Claims are often unsupported or value-laden, making their intended meaning and factuality difficult to judge consistently.Source: §4.2 · p. 7, §4.4 · p. 8

4.3 Base vs. Advanced ReviewScore

• Advanced ReviewScore consistently outperforms Base ReviewScore; for Gemini 2.5 Flash, F1 is 3.40× higher and Kappa is 3.24× higher.Logical conjunction improves F1 consistently, whereas weighted averaging yields marginal improvements or slight degradations.Source: §4.3 · pp. 7–8, Table 3 · p. 8

4.4 Analysis

• Models’ misunderstanding or incorrect reasoning and predictions matching minority human scores account for nearly all disagreements on arguments and questions.For claims, 36% of disagreements arise from value-laden or unsupported claims, while PDF parsing and missing appendices also contribute.Source: §4.4 · p. 8
• Providing authors’ responses generally increases agreement, but has marginal or negative effects for question unanswerability and claim factuality.Authors’ responses provide cues for argument or premise factuality but can inject bias into question and claim evaluation.Source: §4.4 · p. 8, Table 4 · p. 8

• REVIEWSCORE detects misinformed review points, focusing on questions answerable by the paper and weaknesses containing incorrect premises.Source: §5 · p. 8
• Current state-of-the-art LLMs show moderate agreement with human experts when automatically evaluating REVIEWSCORE.Source: §5 · p. 8
• Premise-level factuality evaluation achieves significantly higher human-model agreement than weakness-level factuality evaluation.Source: §5 · p. 8

Limitations

• Automated REVIEWSCORE evaluation excludes figures and appendices to reduce API costs and is affected by nontrivial incorrect PDF parsing.Source: Limitations · p. 9
• Automatic argument reconstruction is not always perfect because its quality depends on the base model, although upgrading to Claude Sonnet 4 improves output quality considerably.Source: Limitations · p. 9
• Graduate annotators’ varying skillfulness, cognitive load, and manual paper selection introduce unavoidable annotation noise and potential unintended biases.Source: Limitations · p. 9

A Future Impact

• For authors, reconstructed review arguments can clarify reviewer intentions and support rebuttal formulation.Source: A Future Impact · p. 13
• For reviewers and metareviewers, REVIEWSCORE could support self-checking review quality and final decisions, respectively.Source: A Future Impact · p. 13

B Review Quality Criteria Discussion

• Among five low-quality review-point types, answerable questions and misunderstood comments receive agreement from at least two annotators and are therefore selected as criteria.Source: B Review Quality Criteria Discussion · p. 13
• Mixed factual quality within a review point motivates premise-level evaluation, while annotator difficulty weighting parts motivates conjunction and weighted-average aggregation.Source: B Review Quality Criteria Discussion · p. 13

C.1 Implementation details

• The pipeline first extracts verbatim conclusions and reasons, then reconstructs them into premise-conclusion structures with first-order-logic formalizations and deductive proofs.Source: §C.1 · pp. 13–14
• A validation stage uses Z3 to check necessary premises and conclusions, detect circular proofs, and trigger regeneration when reconstructions or generated programs are invalid.Source: §C.1 · p. 14
• The engine translates formal reconstructions back into natural language and judges faithfulness against the original review point, regenerating outputs when they are unfaithful.Source: §C.1 · p. 14

• Perfect validity and nearly perfect faithfulness and pass rate are achieved with feedback, whereas direct reconstruction without feedback performs substantially worse.Source: §C.2 · p. 14, Table 5 · p. 14
• The feedback loop averages 3.09 iterations, indicating active use of feedback signals.Source: §C.2 · p. 14
• Figure 4 provides the score rubric used by human annotators to evaluate reconstruction faithfulness.Source: Figure 4 · p. 14

C.3 Qualitative Results

• Direct reconstruction often produced invalid or unfaithful arguments, whereas the feedback-loop method generated valid and faithful reconstructions.The paper illustrates this comparison with three representative examples in Figures 5–7.Source: §C.3 · pp. 14–15

D.2 Pilot Dataset

• The pilot study used three AI graduate students, five ICLR 2021–2025 papers, and median labels for reviews jointly annotated by non-author annotators.Each annotator reviewed three papers, including one authored by themselves and two shared non-authored papers.Source: §D.2 · p. 19

D.3 Dataset Statistics

• The REVIEWSCORE dataset statistics cover both pilot and main subsets, with the detailed counts reported in Table 6.Source: §D.3 · p. 19, Table 6 · p. 19

D.4 Full List of Papers

• Eight papers were selected by each human-annotator group around a common research interest for the main dataset.The full paper list is reported in Table 7.Source: §D.4 · p. 19

E.1 Human Annotator Expertise

• Annotators averaged 3.93 AI/ML publications and a 4.06 / 5 paper-relevance score, indicating experienced and relevant expertise.No annotator had a paper-relevance score below 3, although expertise differed across groups.Source: §E.1 · p. 19, Figure 8 · p. 19

E.2 Inter-Annotator Agreement

• REVIEWSCORE achieved 0.489 Krippendorff’s Alpha overall, indicating moderate inter-annotator agreement.QScore had the highest agreement, followed by ArgScore, while ClaimScore had the lowest.Source: §E.2 · p. 19
• Groups 2 and 3 had higher agreement than other groups, whereas Groups 1 and 5 had lower agreement.Manual analysis attributed most disagreements to annotators with low paper relevance, which the authors identify as a limitation.Source: §E.2 · pp. 19–20

F.1 Quantitative Results

• Human-model agreement was reported separately for ClaimScore, ArgScore, QScore, and REVIEWSCORE in Tables 10–13.Binary evaluations used Precision, Recall, and F1, while five-point evaluations used Pearson rank correlation, Gwet’s AC2, and Quadratic Weighted Kappa.Source: §F.1 · p. 20

F.2 Qualitative Results

• The qualitative analysis gives two Claude Sonnet 3.7/4 examples for each review-point type, with model scores and justifications shown in Figures 9–14.Source: §F.2 · p. 20, Figure 9 · p. 22, Figure 10 · p. 23
• For an Auto-CoT review, the model identifies that the paper uses “let’s do this step by step” with different prompt samples rather than finding the best prompts.The associated review claims that representative prompts are obvious and that the method is only a small improvement over random sampling.Source: §F.2 · pp. 24–25
• The model’s question example asks whether ReAct could use self-consistency by sampling multiple reasoning traces and selecting the majority action.Source: §F.2 · p. 26

G Model Details

• The evaluation uses eight state-of-the-art LLMs: five proprietary models and three open-source models.Reasoning models are excluded because REVIEWSCORE evaluation does not require longer chain of thought.Source: G Model Details · p. 28

H Prompts

• The prompts cover REVIEWSCORE scoring, review-point classification, argument reconstruction, validity checking, debugging, deformalization, and faithfulness checking.Figures 15–29 document these prompts and their score rubrics or inserted materials.Source: H Prompts · p. 28, Figure 15 · p. 28, Figure 16 · p. 28