PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Imitation: A Framework and Benchmark for LLM-Assisted Peer Review

Rachel S. Y. Teo, Yutaro Yamada, Shashank Kotyan, Yuki Imajuku, Tarin Clanuwat

arXiv:2610.11087v1cs.AI

TL;DR

Peer-review assistance research has largely evaluated whether LLM-generated reviews resemble human reviews, while error detection remains a resource-intensive review task. This paper introduces a contradiction benchmark and a multi-layered review system; author feedback describes the system as beneficial, while the benchmark’s synthetic errors may be easier to detect than natural mistakes.

  • Problem

    Prior LLM review research has focused largely on similarity to human reviews, while detecting substantive errors requires resource-intensive cross-checking.

    Source: §1 · p. 2

  • Method

    The paper builds a benchmark by inserting controlled contradictions into papers and proposes a multi-layered review system that deepens manuscript understanding before generating feedback.

    Source: §1 · p. 2, §2.1 · p. 4

  • Results

    Authors who used the system perceived it as beneficial in their workflows, with the most disagreement concerning weaknesses identified by the agent.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The benchmark offers a systematic way to assess contradiction detection, while the authors regard it as an initial step toward evaluating LLM reviewers’ error-detection capabilities.

    Source: §1 · p. 2, §2.3 · p. 7

    Synthetic contradictions may not resemble genuine mistakes and may make the benchmark easier; review systems perform worse on real mistakes than on benchmark contradictions.

    Source: §2.3 · p. 7

Abstract

from arXiv · show

The rapid growth of scientific publishing has strained peer review, particularly in machine learning, raising concerns about declining review quality and increasing reviewer workload. Large language models (LLMs) have been proposed as automated review assistants, yet their evaluation has focused largely on imitating human-written reviews rather than supporting the core functions of peer review. Here, we introduce a verification-centric perspective on LLM-assisted peer review, emphasizing error detection as a critical and resource-intensive task. We present a scalable benchmark that evaluates review systems' ability to identify logical contradictions, constructed through synthetic insertion of errors into conference papers, yielding unambiguous evaluation targets and enabling systematic comparison. We further propose a Multi-Layered Review (MLR) framework that prioritizes detailed manuscript comprehension before review generation, aligning more closely with human reviewing practices while improving token efficiency. Across evaluations, our approach demonstrates strong alignment with human review scores, achieves high error detection performance, and provides complementary perspectives on reviewer focus. These improvements can be attributed to both the choice of the underlying LLM and the design of our system. At the same time, we corroborate persistent vulnerabilities to adversarial manipulation, underscoring the need for robustness in automated review systems. Our findings highlight the importance of rigorous, error-focused evaluation to guide responsible deployment of LLM-based tools in peer review and other critical scientific workflows.

1 Introduction

The paper reframes LLM-assisted peer review as support for human reviewers, emphasizing contradiction detection alongside review generation. It introduces a knowledge-graph benchmark and a multi-layered review system, addressing prior evaluations centered largely on similarity to human reviews.

1 Introduction

  • The paper aims to augment human reviewers by evaluating contradiction detection, a resource-intensive peer-review task, rather than focusing only on similarity to human-written reviews.The authors also report broad comparisons across review scores, text similarity, contradiction detection, retraction detection, reviewer focus, and manipulation.Source: §1 · p. 2
  • The Contradiction Benchmark uses errors inserted into conference papers to systematically assess whether agentic review systems detect logical contradictions.The authors describe the benchmark as scalable and designed to control error types and severity.Source: §1 · p. 2, §1.1 · p. 4
  • A paper-specific knowledge graph categorizes contradictions by severity and potential review impact, going beyond binary correctness checks.The graph-based pipeline generates contradictions from paper content and assigns categories according to severity.Source: §1 · p. 2, §1.1 · p. 4
  • The Multi-Layered Review framework makes multiple progressively deeper passes over a manuscript before generating feedback.Its central design prioritizes understanding the manuscript before review generation.Source: §1 · p. 2
  • The evaluation compares review scores and text similarity across ICLR, ICML, and NeurIPS, alongside contradiction, retraction, reviewer-focus, and manipulation assessments.The authors state that the conference comparisons are intended to demonstrate conference-agnostic performance and characterize practical strengths and limitations.Source: §1 · p. 2

1.1 Related Work

  • Prior work has assessed generated reviews through human-review similarity and error-finding prompts, while the authors’ benchmark evaluates review systems on controlled, paper-specific contradictions.The cited error-verification studies did not benchmark review systems on their datasets.Source: §1.1 · p. 4

2 Dataset and Benchmark

The paper constructs a contradiction benchmark by generating paper-specific knowledge graphs and inserting contradictions at different graph distances. It evaluates review systems automatically and notes that synthetic contradictions may not reflect natural errors.

2.1 Building the Contradiction Benchmark

  • The benchmark inserts synthetic contradictions into original manuscripts, enabling scalable expansion and systematic control over contradiction types and severity.External information searches cannot directly reveal the inserted errors, though comparing an edited paper with its arXiv original could expose them.Source: §2.1 · p. 4

2.1.1 Data Collection

  • The dataset contains 257 papers from ACL, AISTATS, CVPR, and ICML in 2025 and NeurIPS 2024, with redistribution restricted to permissively licensed papers.The collection aims to balance representation across venues.Source: §2.1.1 · pp. 4–5

2.1.2 Knowledge Graph and Dataset Construction

  • A knowledge graph guides contradiction generation because randomly rewriting sentences could target claims or minor implementation details without distinguishing their importance.The graph uses defined node types and relationships, and nodes include descriptions and exact source quotations.Source: §2.1.2 · pp. 5–6
  • Graph distance from a main-claim node serves as an inverse measure of contradiction severity; the procedure produces 1,164 variants overall.Each graph distance yields a contradiction variant, and the authors report that detection rates decrease with distance.Source: §2.1.2 · p. 6

2.2 Automatic Evaluation

  • The evaluator judges whether a review identifies a contradiction using the review and original and modified paragraphs, with results averaged across 10 runs.The authors selected o3 after testing its behavior on unmodified papers and comparing its benchmark evaluations with other models.Source: §2.2 · p. 6
  • 99.9% accuracy on unmodified papers and 86.8% sensitivity for identifying MLR’s correct contradiction detections support o3’s use as the benchmark evaluator.The authors also report acceptable, conservative performance on the benchmark compared with GPT-4.1 and o4-mini.Source: §2.2 · p. 6

2.3 Limitations

  • 34% of sampled contradictions lacked coherence across adjacent sentences, while 8% sounded obviously AI-generated.The audit rated 50 contradictions; the authors note that inserting contradictions into existing passages may disrupt contextual flow.Source: §2.3 · p. 7
  • Synthetic contradictions may differ from genuine mistakes and make the benchmark easier; the authors call it an initial step toward evaluating error detection.They identify incorporating errors that better resemble naturally occurring mistakes as future work.Source: §2.3 · p. 7

3 Method

The system separates the main text and appendix, optionally adds literature context, and uses a Review Agent to produce the final review. Its review process prioritizes progressively deeper manuscript understanding before synthesis.

3 Method

  • The system separates the main text from the appendix and performs in-depth analysis only on the main text, partly because understanding degrades as input length increases.It retains PDF inputs to preserve mathematical expressions and figures that may be difficult to process accurately as plain text.Source: §3 · p. 7

Appendix Agent.

  • The Appendix Agent summarizes experimental and implementation details, while the optional Literature Review Agent contextualizes novelty and significance before sharing its report with the Review Agent.Literature context is disabled in some evaluations, including WithdrarXiv-Check, where online retraction comments could be found.Source: Appendix Agent · p. 7
  • The Review Agent first outlines core ideas, then develops evidence-supported notes and critiques, and finally integrates selected findings with the other agents’ outputs into a structured review.The final review uses sections for strengths, weaknesses, questions, recommendation, and score.Source: Appendix Agent · p. 8

4 Results

The results section evaluates MLR across contradiction detection, withdrawn-paper review, reviewer focus, review-score alignment, adversarial manipulation, cost, and human evaluation. On the Contradiction Benchmark, MLR detects critical errors more accurately than the baselines, while accuracy declines as node distance increases.

4.1 Contradiction Benchmark

  • Even with a single review, MLR achieves an average accuracy improvement of more than 20% over other review methods.The comparison evaluates four-review ensembles for systems that use them and counts an error as detected if any reviewer identifies it.Source: §4.1 · p. 9
  • MLR detects more than 70% of distance 0 contradictions, a fourfold improvement over the next strongest baseline.The distance 0 errors are identified as the most critical; the figure plots detection accuracy by node distance.Source: §4.1 · p. 9, Figure 3 · p. 10
  • Switching LLM-Review to Claude improves accuracy, especially at distance 0, while MLR yields an additional large gain.The authors attribute MLR’s improvement to both model choice and system design.Source: §4.1 · p. 9
  • Accuracy decreases with increasing node distance, consistent with the knowledge-graph severity classification; distance 6 is an exception attributed to its 18-paper sample.Only six samples exist beyond distance 6; distances 5 and 6 have fewer than 50 papers, so Wilson confidence intervals are reported.Source: §4.1 · p. 9

4.2 WithdraXiv-Check Dataset

  • The WithdraXiv-Check evaluation uses the authors’ protocol and o3 as judge; exact-match review scoring can miss cases where a review identifies a problem without matching the retraction comment.The authors note that Gemini 2.5 Pro is more lenient and sometimes hallucinates.Source: §4.2 · p. 9

Retraction comment

Weakness in review

The section examines LLM-assisted review through weakness detection and human assessments, including how MLR’s review focus relates to judgments of paper quality. It also addresses feedback concerns, manipulation vulnerabilities, and the need for oversight.

Weakness in review

  • For both exact-match and similar-problem evaluation, MLR outperforms every baseline by at least 5%.The similar setting counts a detection when a review and retraction comment describe comparable problems, addressing vague comments that an LLM-judge may not recognize as exact matches.Source: Weakness in review · p. 10, Figure 4 · p. 10, Table 3 · p. 11

4.3 Focus Distribution

  • Simple-prompt reviews align more closely with human focus on strengths, whereas review systems align more closely on weaknesses; MLR emphasizes specific experimental and validity facets, while humans emphasize clarity and novelty.MLR and AgentReview provide fewer general Paper comments than humans, and GPT-5.1 likewise prioritizes validity over novelty in weakness aspects.Source: §4.3 · pp. 11–12, Figure 5 · p. 13, Figure 6 · p. 13

4.4 Evaluation on Conference Submissions

  • The conference-submission evaluation sampled an even split of accepted and rejected papers from ICLR, ICML, and NeurIPS, using MLR’s Appendix and Review agents with web search instead of a full literature review.The study motivated checking that MLR’s distinct review focus did not lead its recommendation to diverge entirely from human judgment.Source: §4.4 · pp. 12–13

4.4.1 Semantic Similarity

  • MLR reviews are semantically different from human reviews, scoring relatively low on most similarity metrics except ROUGE-1, while AgentReview has the strongest semantic consistency among evaluated systems.Human reviews themselves have BertScore values of about 12–18 across conferences, indicating substantial diversity; MLR’s higher ROUGE-1 may reflect shared technical vocabulary despite differing review focus.Source: Figure 7 · p. 14, §4.4.1 · p. 15

4.4.2 Correlation

  • MLR’s predicted scores correlate relatively highly with true human scores across ICLR, ICML, and NeurIPS, while GPT-4.1 predictions from human reviews show Pearson correlations approaching 0.7 to 0.8.Accepted and rejected human-score distributions overlap but have distinct means and medians; MLR’s means remain separated, with a larger gap, and its ICLR rejected-paper scores span a wider range and are lower than other systems’.Source: §4.4.2 · pp. 15–16, Table 7 · p. 16, Figure 8 · p. 16

4.5 Explicit Manipulation

  • All systems were strongly influenced by injected manipulative text, but MLR detected it in 8 of 50 cases and had the smallest mean score change.MLR’s score changes nevertheless had high standard deviations: 2.46 for LLM-judge scores and 1.85 for its own scores; detection was attributed to deep paper comprehension and Claude’s combined image-and-text PDF input.Source: §4.5 · p. 17, Table 8 · p. 18

4.6 Cost Analysis

  • MLR and AI Reviewer each cost about USD 0.50 per review, while MLR uses approximately half as many tokens as AI Reviewer.A single-prompt MLR variant reportedly preserves comparable performance while reducing overall cost by approximately two-thirds; LLM-Review costs less but cannot reliably detect paper errors or distinguish paper quality.Source: §4.6 · pp. 17–18, Table 9 · p. 18

4.7 Human Evaluation and Feedback

  • In a 38-session user study, Beneficiality and Improvement received the highest global ratings (4.25 and 4.04), while weaknesses had the lowest comment acceptance rate at 68%, versus 78% for the To-Do List.Criticality averaged 2.88, near the rubric’s neutral, balanced-tone score of 3; users rated all review sections at least one point above neutral for accuracy and specificity, although weaknesses were least accurate.Source: §4.7 · pp. 18–20, Table 10 · p. 19, Figure 10 · p. 19
  • Criticality weakly correlates with lower Reuse Intention (r = −0.31) and Helpfulness (r = −0.29), while users value actionable To-Do Lists and see MLR as beneficial and accurate.The authors say AI reviewers should balance critique with constructive, improvement-oriented framing.Source: §4.7 · pp. 20–21

5 Discussion

  • The paper introduces a scalable benchmark with controlled, unambiguous error targets, addressing the lack of widely adopted benchmarks centered on peer-review error detection.The authors contrast intentional error insertion with retraction comments that can be vague and difficult to verify.Source: §5 · p. 21
  • MLR prioritizes detailed manuscript comprehension before review generation, improves token efficiency, correlates with human review scores, and achieves the highest error detection rate.Evaluations also found that it differentiates submissions by acceptance quality and offers diverse review-focus perspectives, while experiments showed systems frequently struggle against explicit manipulation.Source: §5 · p. 21
  • The analysis is limited to machine learning and excludes comparison with closed-source review systems that could perform more strongly.The authors describe extension to other scientific domains as an important direction for further study.Source: §5 · p. 21

Broader Impact Statement

  • LLM reviews may be misused by reviewers who do not engage with papers, while error detection could help authors optimize submissions against automated checks.The authors recommend using these systems only to complement careful human review, with transparent use, human oversight, and compliance with venue guidelines.Source: Broader Impact Statement · p. 21

A.1 Contradiction Benchmark

The appendix details how contradictions are injected into papers and how the benchmark is evaluated, including the choice of an LLM judge. It also documents baseline review-system configurations used for comparison.

A.1 Contradiction Benchmark

  • Contradictions are generated by selecting nodes at each distance in a paper-specific knowledge graph, asking GPT-4.1 to contradict the represented point, and replacing the original paragraph.The procedure yields 1,164 variants overall, with about 200–250 data points per conference; some modified LaTeX sources fail to compile.Source: §A.1 · p. 29, Figure 13 · p. 33
  • The authors select o3 as judge because its explanations better distinguish explicit contradiction detection from critiques that merely dispute one claim.They report that o3 was more reliable on outlines, while GPT-4.1’s interpretation of an outline’s “logical inconsistency” critique could be mistaken for detection of the injected error.Source: §A.1 · pp. 29–30
  • 86.80% sensitivity for o3, conditioned on MLR’s correct detections, suggests the judge misses about 13.20% of true positives and may make reported system accuracies conservative.The estimate averages 10 runs per paper over 50 reviews in which MLR correctly detected contradictions.Source: §A.1 · p. 31
  • The knowledge-graph prompt allows “Contradicts,” but only 0.4% of relations use it; removing it largely leaves graph differences within typical LLM-output variation.Relations of this type that do appear would necessarily be affected by removing the option.Source: Figure 12 · p. 32

A.2 Baseline Settings

  • Baseline system implementations retain their described workflows but use selected model and prompt settings, including o4-mini for AI Reviewer and GPT-4o for AgentReview.AI Reviewer uses five review ensembles and up to five reflection rounds; AgentReview uses a benign reviewer and baseline author and area-chair settings.Source: §A.2 · pp. 31–33
  • Baseline comparisons use first-phase reviews, average three reviews for most evaluations, and sample one review for focus-level evaluation.The meta-review is excluded because it summarizes earlier phases rather than following an individual-review format.Source: §A.2 · p. 33
  • The MLR configuration varies by evaluation: some omit literature review and appendix agents, while conference and focus analyses allow the review agent to search the web.The authors also note that the modular literature-review agent could use retrieval frameworks such as PaperQA, left for future work.Source: §A.2 · pp. 33–34, Table 12 · p. 34

A.3 WithdrarXiv-Check Dataset

This appendix describes the WithdrarXiv-Check evaluation and the focus-distribution analysis. It limits the former to shorter papers and checks that the sampled papers preserve human reviewers’ focus distributions.

A.3 WithdrarXiv-Check Dataset

  • The WithdrarXiv-Check evaluation includes 211 papers of 30 pages or fewer from an original 245, excluding longer papers that exceed conference-style systems’ context constraints.The full dataset spans 2 to 136 pages.Source: §A.3 · p. 34
  • The retraction comparison initially requires an exact problem match, then relaxes the criterion to a similar problem because comments can be vague and exact matching can miss related review concerns.The o3 judge defaults to “No” when unsure.Source: §A.3 · p. 34

A.4 Focus Distribution

  • Each review strength and weakness receives both a target and an aspect facet, producing separate focus distributions for strength and weakness targets and aspects.“Prior Research” is available only as a weakness target.Source: §A.4 · p. 35, Table 13 · p. 35
  • 300 of 676 Expert Review papers are sampled, with human-reviewer focus-distribution KL divergences from the full set ranging from 0.0018 to 0.0053.The authors use these low divergences to support comparing system focus distributions against human reviewers using the sample.Source: §A.4 · p. 35
  • The focus evaluation adds newer models while retaining authors’ provided GPT-4o mini data and human reviews.The added models are gpt-5-mini-2025-08-07, gpt-5.1-2025-11-13, gemini-2.5-pro, gemini-3-pro-preview, and claude-sonnet-4-5-20250929.Source: §A.4 · p. 35

A.5 Evaluation on Conference Submissions

This appendix details the similarity and correlation metrics used to compare generated reviews with human reviews, then describes explicit manipulation testing and the high scores on unaltered papers.

A.5.1 Semantic Similarity

  • BLEU-4 combines modified n-gram precision with a brevity penalty, using the closest-length reference when multiple references are available.Modified n-gram counts are clipped to reference counts; corpus calculations aggregate precision counts across texts.Source: §A.5.1 · p. 36
  • ROUGE-1 measures unigram overlap, while ROUGE-L measures longest-common-subsequence overlap; both use maximum matching across references and report F1 scores.Reviews are compared against all human references for each paper; scores are averaged across candidate texts.Source: §A.5.1 · pp. 35–37
  • BertScore compares contextual token embeddings, and for multiple references retains each candidate’s highest candidate-reference score.The final BertScore is the harmonic mean of precision and recall; corpus scores are averaged across candidates.Source: §A.5.1 · p. 37

A.5.2 Correlation

  • Pearson, Spearman, and Kendall’s Tau assess relationships between generated and human review scores, with multi-review scores averaged before correlation.Pearson measures linear dependence, Spearman applies Pearson’s formula to ranks, and Kendall compares concordant and discordant pairs; Kendall is described as more robust to outliers and small samples than Spearman.Source: §A.5.2 · pp. 37–38

A.6 Explicit Manipulation

  • For explicit manipulation tests, malicious text was appended to rejected papers’ conclusions; unaltered arXiv versions already scored above the borderline acceptance score.The authors suggest revisions after review or selective arXiv uploads as possible explanations for the high unaltered-paper scores.Source: §A.6 · p. 38

A.7 Human Evaluations

The appendix documents the human-evaluation questionnaire and shows prompts and examples for several review-generation and assessment steps.

A.7 Human Evaluations

  • The web interface collected feedback when users received an MLR-generated review, with questions at global, section, and comment levels.Comment-level responses were limited to thumbs up, neutral, or thumbs down.Source: §A.7 · p. 38
  • The appendix includes prompts for checking contradiction detection and retraction-comment matches, and for predicting paper scores from reviews.The score-prediction prompt uses scoring rubrics corresponding to the paper’s submission venue.Source: Figure 17 · p. 42, Figure 22 · p. 45, Figure 23 · p. 45
  • Additional figures show prompts for appendix and literature-review agents and for the Review Agent’s successive passes before final review generation.The final-pass prompt incorporates the appendix and literature-review outputs.Source: Figure 18 · p. 42, Figure 19 · p. 42, Figure 20 · p. 43
  • The appendix also documents injected manipulation text and a human rating of whether generated contradictions read naturally and avoid obvious AI phrasing.Raters used a five-point Likert scale, and the figure reports percentages at each response level.Source: Figure 24 · p. 46, Figure 27 · p. 48
  • Global survey questions use five-point Likert scales for Beneficiality, Criticality, Improvement, Helpfulness, and Reuse Intention.Section feedback asks about accuracy and specificity, while comment feedback records agreement, neutrality, or disagreement.Source: Figure 25 · p. 46, Figure 26 · p. 47

B Additional Experimental Results

The appendix introduces additional experimental results spanning benchmark, dataset, conference-submission, and review-focus evaluations.

B Additional Experimental Results

B.1 Contradiction Benchmark

The additional results examine contradiction-detection accuracy across systems and review ensembles, and note a limitation of the synthetic contradictions used in the benchmark.

B.1 Contradiction Benchmark

  • In a small audit of 50 generated contradictions, 34% were judged not to flow naturally, potentially making synthetic errors easier to detect than genuine mistakes.Only 8% were rated as sounding obviously AI-generated; the authors note that systems still struggled with these comparatively easier errors.Source: §B.1 · p. 48
  • Changing LLM-Review from GPT to Claude improves contradiction detection, especially at node distances 0 and 1; MLR adds gains across distances 0–4 and the full dataset.The authors attribute improvement to both model choice and system design, while noting Claude’s high unit cost for other baselines.Source: Table 14 · p. 49, §B.1 · p. 49
  • Table 14 reports system accuracy by node distance and overall, including MLR with four reviews, one review, and a combined three-prompt setting.It also includes an LLM-Review Claude ablation.Source: §B.1 · pp. 48–49, Table 14 · p. 49
  • Ensembling more reviews continues to improve contradiction detection without plateauing, with the largest increase across most node distances occurring from one review to two.The figure plots MLR accuracy by node distance for ensembles of one to six reviews.Source: §B.1 · p. 49, Figure 28 · p. 50
  • Table 15 breaks down MLR accuracy on WithdrarXiv-Check by inclusive page-count range and subject for both Similar and Exact LLM-judge settings.Subjects with fewer than 11 papers are grouped as Others.Source: Table 15 · p. 50

B.2 WithdrarXiv-Check Dataset

MLR’s WithdrarXiv-Check accuracy varies by subject and paper length. Its moderate performance relative to the Contradiction Benchmark is attributed partly to benchmark contradictions’ stylistic artifacts.

B.2 WithdrarXiv-Check Dataset

  • Accuracy tends to decrease as paper length increases, and the authors cite domain, context-length limits, and stylistic artifacts as factors behind moderate performance.They note that benchmark contradictions are easier to detect than genuine errors because of those artifacts.Source: §B.2 · p. 51

B.3 Focus-level Evaluations

The focus-level analysis finds differences between human and LLM reviews in both strengths and weaknesses. OpenAI models are an exception in strength-related communication-clarity emphasis, while Gemini models emphasize methodological weaknesses and humans emphasize novelty.

B.3 Focus-level Evaluations

  • Human reviewers devote more strength-related comments to general paper-level remarks and communication clarity, while LLMs give fewer general comments and more specific feedback.OpenAI models GPT-5 mini, GPT-5.1, and GPT-4o mini align closely with humans on communication clarity.Source: §B.3 · p. 51
  • OpenAI models GPT-5 mini, GPT-5.1, and GPT-4o mini are exceptions to the LLM trend, with communication-clarity emphasis closely aligned with human reviewers.The authors suggest human focus on clarity may reflect limited review time and workload; they also note LLMs can help authors improve clarity and presentation.Source: §B.3 · p. 51
  • Gemini 2.5 Pro and Gemini 3 Pro show a pronounced focus on methodological weaknesses rather than experimental or motivational ones.The authors suggest this may reflect a preference for text-grounded critiques that can be justified with high confidence.Source: §B.3 · p. 51
  • Human reviewers criticize novelty more heavily than LLM reviewers, a difference the authors connect to the system prompt’s paper-only scope.The authors suggest the prompt may omit broader context about how acceptance contributes to knowledge growth.Source: §B.3 · p. 51

B.4 Evaluation on Conference Submissions

Across conference-submission analyses, MLR and AI Reviewer show relatively strong performance across venues, but their relative strengths differ between overall and within-decision score correlations. Combining MLR’s prompts preserves similar score correlations with a small accuracy drop on a benchmark subset, while AgentReview scores show limited spread.

B.4 Evaluation on Conference Submissions

  • Around 3.5% overall accuracy drop occurs for MLR-Combined on a 497-paper Contradiction Benchmark subset when using 1 review, while score correlations remain similar to original MLR.The authors conclude that concatenating the three prompts does not produce a substantial difference, and emphasize guiding deep paper understanding before review generation.Source: §B.4 · p. 52, Table 16 · p. 56
  • MLR and AI Reviewer do relatively well across conferences, while LLM-Review performs strongly on NeurIPS 2024 but not other venues.For ICLR 2025, MLR has higher overall correlation in some cases, whereas AI Reviewer performs better when split by decision outcome.Source: §B.4 · p. 53, Table 17 · p. 57
  • AI Reviewer’s generated scores correlate much more strongly with human scores on NeurIPS 2024 than its LLM-judge predictions, but correlate more poorly on other venues.The authors suggest the full NeurIPS review template may help align its scores with human judgments; MLR’s conference-agnostic template scores correlate as well as or better than predictions despite using NeurIPS rubrics.Source: §B.4 · pp. 53–54, Table 18 · p. 58
  • AgentReview scores cluster between 4 and 6 with limited spread and no clear separation between accepted and rejected papers.The figure uses red for rejected and green for accepted papers, with horizontal lines for category averages; its instructions discouraged scores of 7 or higher before rebuttal except for exceptional papers.Source: §B.4 · p. 54, Figure 33 · p. 56
  • Insufficient score calibration may explain low scores, while low score variance may explain the small score difference under explicit manipulation.The text notes there were no explicit instructions to prevent low scores.Source: §B.4 · p. 55
Loading 2610.11087v1…