PaperlayerView exact revision on arXiv

Source-linked AI summary

AI-Driven Review Systems: Evaluating LLMs in Scalable and Bias-Aware Academic Reviews

Keith Tyser, Ben Segev, Gaston Longhitano, Xin-Yu Zhang, Zachary Meeks, Jason Lee, Uday Garg, Nicholas Belsten, Avi Shporer, Madeleine Udell, Dov Te'eni, Iddo Drori

arXiv:2408.10365v1cs.AI

TL;DR

The paper addresses how to provide scalable, useful, and bias-aware academic reviews while evaluating whether LLM-generated reviews align with human preferences. It builds reviewing and evaluation systems that combine adaptive, multimodal, venue-aware review generation with pairwise preference assessment and error-injection tests. The reported systems deliver fast, consistent reviews, align well with human reviewers in blind human and GPT-4 evaluations, and expose important limitations of LLM reviewing.

  • Problem

    Academic reviewing needs scalable early feedback, quality control, trend analysis, and reduced bias, but LLM reviews can hallucinate, persuade despite inaccuracies, and produce difficult-to-control evaluations.

  • Method

    The paper develops OpenReviewer, Papers with Reviews, and Reviewer Arena, combining adaptive multimodal and venue-aware reviews, error-injection analysis, public review services, and human or LLM pairwise preference evaluation.

  • Results

    LLM-generated reviews align well with human reviewers in blind human evaluation and automatic GPT-4 comparison, while the systems evaluate review quality and detect review limitations.

  • Takeaways & Limitations

    The proof-of-concept systems support faster, more accessible academic review and provide tools for authors, reviewers, editors, and the broader scholarly community.

  • Takeaways & Limitations

    Venue-dependent review materials require yearly updates, and predicting human preferences remains difficult because different people may prefer different reviews.

Abstract

from arXiv · show

Automatic reviewing helps handle a large volume of papers, provides early feedback and quality control, reduces bias, and allows the analysis of trends. We evaluate the alignment of automatic paper reviews with human reviews using an arena of human preferences by pairwise comparisons. Gathering human preference may be time-consuming; therefore, we also use an LLM to automatically evaluate reviews to increase sample efficiency while reducing bias. In addition to evaluating human and LLM preferences among LLM reviews, we fine-tune an LLM to predict human preferences, predicting which reviews humans will prefer in a head-to-head battle between LLMs. We artificially introduce errors into papers and analyze the LLM's responses to identify limitations, use adaptive review questions, meta prompting, role-playing, integrate visual and textual analysis, use venue-specific reviewing materials, and predict human preferences, improving upon the limitations of the traditional review processes. We make the reviews of publicly available arXiv and open-access Nature journal papers available online, along with a free service which helps authors review and revise their research papers and improve their quality. This work develops proof-of-concept LLM reviewing systems that quickly deliver consistent, high-quality reviews and evaluate their quality. We mitigate the risks of misuse, inflated review scores, overconfident ratings, and skewed score distributions by augmenting the LLM with multiple documents, including the review form, reviewer guide, code of ethics and conduct, area chair guidelines, and previous year statistics, by finding which errors and shortcomings of the paper may be detected by automated reviews, and evaluating pairwise reviewer preferences. This work identifies and addresses the limitations of using LLMs as reviewers and evaluators and enhances the quality of the reviewing process.

1 Introduction

AI-based paper reviews are motivated by the need for scalable, timely, quality-controlled, less biased academic reviewing. The paper develops approaches using multiple LLM instances, public review corpora, and human-centered evaluation to address this need.

  • LLM reviewing remains challenging because models may hallucinate or persuade humans despite inaccuracies, making quality and appropriateness difficult to control.
  • Meta-prompting decomposes complex review tasks across expert LLM instances with tailored instructions and can outperform conventional prompting across tasks.
  • AI-based reviews can provide early feedback, support quality control, enable trend analysis, reduce human bias, and help readers identify high-quality papers.
  • Scaling review is increasingly important as the volume of academic papers grows, including a 12.3% increase in arXiv submissions from 2022 to 2023.
  • The paper presents multiple LLM-based reviewing systems and supporting materials, including prompts, review questions, scores, evaluation results, and code.

2 Review Systems

The paper introduces OpenReviewer, Papers with Reviews, and Reviewer Arena as complementary systems for generating, publishing, and evaluating LLM-based academic reviews. Their designs combine paper text and figures, venue-specific criteria, large-scale review access, and anonymous preference comparisons.

  • OpenReviewer: OpenReviewer adapts reviews to venue and paper content while considering text, figures, ethics, and bias to target human-level quality and community-norm adherence.
  • OpenReviewer: OpenReviewer lets users upload papers, receive immediate review feedback and revision instructions, provide feedback, and submit revised versions.
  • Papers with Reviews: Papers with Reviews collects roughly 500 arXiv papers daily and 1,000 open-access Nature papers monthly, then reviews and evaluates them at scale.
  • Papers with Reviews: Papers with Reviews publishes LLM-generated reviews alongside human reviews to support research navigation, author improvement, and broader access to evaluation.
  • Reviewer Arena: Reviewer Arena evaluates human and LLM reviews through direct anonymous pairwise comparisons, using preferences to compute win rates, reviewer scores, and rankings.

3 Review Evaluation Methods

The review evaluation compares human and LLM reviewers through anonymous pairwise preferences, ranks them with win rates and Bradley-Terry modeling, and tests automated preference prediction and review limitations.

  • Anonymous Human and LLM Evaluation: Five expert evaluators compare anonymous reviews from a human, three closed LLMs, an open LLM, and GPT-4 Turbo.Each of 150 papers receives two randomly assigned reviews from the five reviewers.
  • Anonymous Human and LLM Evaluation: Reviewer quality is quantified from pairwise outcomes using win matrices, Bradley-Terry coefficients, and logistic regression.The win matrix records each competitor’s wins divided by total matches against another competitor.
  • Anonymous Human and LLM Evaluation: The Bradley-Terry model estimates relative reviewer strength with a logistic pairwise-win probability and ranks reviewers by descending fitted coefficients.Coefficients are estimated by minimizing binary cross-entropy subject to ξ1 = 0.
  • Automatic LLM Prediction of Human Preference: LLM-based autoevaluation uses synthetic judgments to increase effective sample efficiency while reducing reliance on human-labeled validation data.Prediction-powered inference estimates and corrects synthetic-data bias to improve evaluation precision without sacrificing accuracy.
  • Automatic LLM Prediction of Human Preference: The study fine-tunes an LLM to predict which of two competing LLM responses humans prefer, treating the model as an adjudicator of alignment.The approach targets head-to-head preference prediction for review-quality evaluation.
  • Automatic LLM Prediction of Human Preference: Automatically injecting paper errors and comparing original versus modified reviews maps which shortcomings LLMs detect reliably and which define their review limitations.The analysis compares review scores and content before and after modification.

4 Review Generation Methods

Review generation combines fixed and adaptive question selection with structured summary comparison and persona-based meta-prompting to model the editorial process.

  • Review Question Design: The system supports venue-specific, paper-type-specific, adaptive, and adaptively generated review questions.For adaptive selection, the LLM chooses the top 10 questions from a bank of 40 based on paper content.
  • Review Comparison: GPT-4 extracts structured summary points from human and LLM reviews, then compares human-human and human-LLM overlap using similarity scores.Function calling is used to obtain structured responses and reduce spurious text.
  • Editorial Process Simulation: Meta-prompting simulates the conference editorial process through personas including program chairs, area chairs, reviewers, and authors.The simulated process includes assignments, reviewing, rebuttals, discussions, meta-reviewing, decisions, and notification.

5 User Feedback and Limitations

User feedback evaluates automated review quality and trustworthiness, while the paper identifies venue, preference, and capability boundaries that constrain interpretation and future use.

  • User Feedback: User feedback combines five quantitative review-quality questions with an open-ended question about OpenReviewer outputs.Feedback is collected from users of Papers with Reviews and summarized in Appendix C.
  • Limitations: Venue-dependent context documents make score distributions resemble human distributions and use the full score range, but require yearly updates.The documents include prior-year statistics, reviewer and area-chair guidelines, ethics and conduct codes, and the formal review form.
  • Limitations: Human preference prediction is limited because different people may prefer different reviews, although correcting for behavioral bias partially mitigates this gap.The authors use a Kaggle human-preference dataset as common ground for prediction.
  • Limitations: The paper’s capability and error-testing analysis identifies limits of the current application, especially for understanding how human reviewers should use it.The authors state that these limitations are essential for appropriate application use and plan to study how authors use and trust LLM reviews.

6 Conclusions

The paper presents OpenReviewer as a proof-of-concept system for fast, reliable, large-scale academic reviewing. Its evaluations find substantial alignment with human reviews while revealing task-specific limitations and the need for iterative calibration.

  • LLM-generated reviews align well with human reviewers in blind human and automatic GPT-4 evaluations.
  • The study evaluates both review-quality alignment and adherence to conference guidelines and scientific norms, using human reviews as a baseline.
  • When given previous editorial decisions, the LLM aligns well with human reviewers, but detects overclaiming more reliably than failures to follow expected norms.
  • The authors conclude that iterative design and large-scale empirical evaluation are essential for calibrating LLM applications in reviewing.
  • The systems process tens of thousands of papers and aim to improve review efficiency, literature accessibility, and identification of high-quality papers and emerging trends.

Appendices

The appendices document rapidly increasing conference submission volumes and the venues and journal collections used for open-review data. They include submission trends, venue counts, and Nature journal identifiers.

  • Conference submission counts are reported across CVPR, ICML, NeurIPS, ICLR, and AAAI.
  • The appendices provide tables listing collected papers by venue and mapping Nature journal identifiers to journal names.

C User Feedback on Reviews

User feedback rates the reviews highly overall, praising their depth, organization, personalization, and constructive guidance while identifying opportunities for more detailed and focused critiques. Review-score ablations show that added context changes GPT-4’s recommendation level and variability relative to human reviewers.

  • User Feedback on Reviews: User feedback scores range from 5 to 7, indicating that most reviews were rated very good to exceptional.
  • User Feedback on Reviews: Readers praised the reviews as detailed, well organized, comprehensive, ethical, personalized, and useful for identifying drawbacks and improving papers.
  • User Feedback on Reviews: Feedback also requested more elaboration in sections such as correctness and better balance between suggested additions and concision.
  • Review Scores: P1, P2, and P3 produced recommendation scores above the human average, while P4 reduced the score to 4.61 and P5 brought it near the human level.
  • Review Scores: With all related context, GPT-4’s P5 score distributions resembled human distributions for most dimensions, but confidence remained skewed higher.
  • Evaluating Reviews: The evaluation used a randomized 5% sample and assessed human and P5-written reviews on a 0–5 scale, where zero indicates a content-free review.
  • Evaluating Reviews: The confusion analysis found one high-scored GPT-4 acceptance rejected by humans and four low-scored GPT-4 rejections accepted by humans.

F Blind Human Evaluation

Blind human evaluation found that human-written and context-rich GPT-4 reviews received nearly identical scores for explaining review decisions. The broader evaluation also compared their usefulness for improving papers and content specificity.

  • Human evaluators scored human-written reviews 4.80 and GPT-4 reviews with all related contexts 4.76 for explaining review scores.
  • Reviews written by GPT-4 with all related contexts received the highest author-improvement score when evaluated by humans, at 4.79.
  • Human-written reviews evaluated by GPT-4 received the lowest author-improvement score, 4.14, among the reported conditions.
  • GPT-4 evaluated human-written reviews as more content-specific than human evaluators rated GPT-4-written reviews, with scores of 4.97 and 4.68 respectively.

G Synthesis and Analysis for Mapping Review Capabilities and Limitations

The LLM generally detects injected paper errors by lowering review scores, with especially strong penalties for overclaiming, but detection varies across error types and papers.

  • Human and LLM scores for original papers are close, indicating general alignment before shortcomings are introduced.Figure 15 compares original-paper human and LLM scores with scores after different error injections.
  • Most data points show positive score differences after error injection, indicating that the LLM usually detects errors and shortcomings.The heatmap color intensity represents the magnitude of score changes between original and modified papers.
  • Detection is uneven: technical errors are mostly associated with low score differences, while baseline-comparison and metric omissions vary substantially across papers.Some papers show broad detection across categories, whereas others show predominantly low differences.
  • Adversarial edits delete, insert, or modify paper text to create controlled shortcomings for comparing original and modified LLM reviews.Examples include deleting related work, baselines, or citations; modifying equations; and introducing overclaiming.
  • The LLM markedly penalizes overclaiming and detects different error categories through reduced review scores.The category differences are sorted by magnitude relative to reviews of original papers.

H Examples of Error and Shortcomings and Adversarial Prompts

The paper constructs adversarial paper variants by deleting or modifying content, then compares LLM reviews to assess whether specific shortcomings are detected.

  • H Examples of Error and Shortcomings and Adversarial Prompts: Deleting related work, baseline comparisons, or citations creates modified papers whose example review scores fall from 7 to 5, 4, and 5, respectively.The examples use side-by-side original and modified papers.
  • H Examples of Error and Shortcomings and Adversarial Prompts: Overclaiming edits reduce the example review score from 7 to 3, providing the largest stated penalty among the listed examples.Figures 23–26 show original and modified overclaiming examples.
  • H Examples of Error and Shortcomings and Adversarial Prompts: The proof discussion identifies a wrong theorem inequality: the proof establishes P(ℓa = ℓb) ≤ P(ℓb = ℓoracle), not a lower bound.The text says the proof logic is consistent but the theorem statement has the wrong inequality.
  • H Examples of Error and Shortcomings and Adversarial Prompts: Theoretical mistakes and technical errors are subdivided into five sub-types, while adversarial prompts introduce eight error and shortcoming categories.The classification and prompt examples are summarized in Tables 6 and 7.
  • H Examples of Error and Shortcomings and Adversarial Prompts: Modifying equations by removing square roots and flipping an inequality produces a case where ChatGPT detects the flipped inequality in Theorem 3.The original and modified equations are shown in Figures 21 and 22.

I Automatic Comparisons

Automatic comparison methods measure overlap between human and LLM review points and simulate editorial roles, while venue-specific materials produce score distributions broadly similar to human reviews.

  • I Automatic Comparisons: Human–LLM review summaries have higher overlap than human–human summaries: 3.67 versus 3.05 average overlap.The reported standard deviations are 1.58 for human–LLM overlap and 1.56 for human–human overlap.
  • I Automatic Comparisons: The average weighted Jaccard similarity is 0.214, with Human–LLM similarities higher than Human–Human values.Common overlaps include experimental validation, methodological clarity, and potential real-world applications.
  • I Automatic Comparisons: GPT-4 simulates program chair, senior area chair, area chair, and reviewer roles, reducing the editorial process from human months to machine minutes.The role structure and responsibilities are described in Tables 8 and 9.
  • I Automatic Comparisons: The review-question framework considers conference- or journal-specific fixed questions and paper-type-specific fixed questions alongside other selection approaches.The paper treats question selection as a design choice affecting thoroughness and relevance.
  • I Automatic Comparisons: With all documents, GPT-4 score distributions resemble human distributions for correctness, novelty, significance, and overall recommendation, but confidence remains skewed higher.The overall recommendation distributions have similar means and standard deviations.
  • I Automatic Comparisons: The paper lists preventive actions for ethical and transparent use of LLMs in peer review.These actions are presented in Table 13.

N Predicting Human Preferences: Implementation Details

The preference-prediction system combines open-weight LLMs, quantization, augmentation, tuning, and Bradley–Terry ranking, while retaining humans because of bias and hallucination risks.

  • N Predicting Human Preferences: Implementation Details: Three open-weight models—Gemma-2-9b-it, Llama-3.1-8b, and Mistral-Nemo-Instruct-2407—are quantized to 4 bits for training and inference.The pipeline also applies data augmentation, hyperparameter tuning, and bias correction.
  • N Predicting Human Preferences: Implementation Details: Training explores learning rates from 2e-6 to 2e-4, multiple optimizers, weight decay from 0 to 0.1, dropout from 0.03 to 0.12, and freezing 0, 10, or 16 layers.Inference varies sequence lengths from 512 to 8192 and batch sizes of 2, 4, or 8 across 100 trials per model.
  • N Predicting Human Preferences: Implementation Details: The system augments the competition dataset by paraphrasing prompts while preserving associated responses.Anthropic’s prompt-generation tool and Phi-3-Mini-4k-Instruct are used for paraphrasing.
  • N Predicting Human Preferences: Implementation Details: The system exposes review and reviewer-arena interfaces for online use.The interfaces are shown in Figures 32 and 33.
  • N Predicting Human Preferences: Implementation Details: Bradley–Terry implementation builds win and count matrices, optimizes negative log likelihood, estimates competitor coefficients, and ranks competitors by descending coefficients.The first coefficient is fixed as a constraint during estimation.
  • N Predicting Human Preferences: Implementation Details: The authors retain human involvement because sole LLM evaluation risks bias, favoritism toward similar LLM outputs, misinformation, and hallucinations.The stated goal is to combine humans and LLMs while examining the spectrum between human evaluation and full automation.
Loading 2408.10365v1…