Source-linked AI summary
ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewing
Ryan Liu, Nihar B. Shah
TL;DR
Peer review is strained by growing submissions and reviewer workloads, motivating investigation of LLM assistance. The study evaluates GPT-4 on error detection and checklist verification, finding 7/13 inserted errors and 86.6% checklist accuracy, while revealing failures on literature-dependent reasoning.
Problem
Peer review faces growing submission volumes, substantial reviewer time demands, and frequent passage of flawed papers, motivating LLM assistance.
Method
The study evaluates GPT-4 on deliberately erroneous papers and closed-ended checklist verification, including literature-dependent reviewing cases.
Results
GPT-4 detected errors in 7 of 13 short papers and achieved 86.6% accuracy across 119 checklist examples.
Takeaways & Limitations
The results support using LLMs for selected reviewing tasks while indicating difficulty with errors requiring extraneous knowledge or literature connections.
Takeaways & Limitations
GPT-4 failed to detect a finite-field coding limitation and a false claim about total unimodularity in reviewer-assignment optimization.
Abstract
from arXiv · showhide
Given the rapid ascent of large language models (LLMs), we study the question: (How) can large language models help in reviewing of scientific papers or proposals? We first conduct some pilot studies where we find that (i) GPT-4 outperforms other LLMs (Bard, Vicuna, Koala, Alpaca, LLaMa, Dolly, OpenAssistant, StableLM), and (ii) prompting with a specific question (e.g., to identify errors) outperforms prompting to simply write a review. With these insights, we study the use of LLMs (specifically, GPT-4) for three tasks: 1. Identifying errors: We construct 13 short computer science papers each with a deliberately inserted error, and ask the LLM to check for the correctness of these papers. We observe that the LLM finds errors in 7 of them, spanning both mathematical and conceptual errors. 2. Verifying checklists: We task the LLM to verify 16 closed-ended checklist questions in the respective sections of 15 NeurIPS 2022 papers. We find that across 119 {checklist question, paper} pairs, the LLM had an 86.6% accuracy. 3. Choosing the "better" paper: We generate 10 pairs of abstracts, deliberately designing each pair in such a way that one abstract was clearly superior than the other. The LLM, however, struggled to discern these relatively straightforward distinctions accurately, committing errors in its evaluations for 6 out of the 10 pairs. Based on these experiments, we think that LLMs have a promising use as reviewing assistants for specific reviewing tasks, but not (yet) for complete evaluations of papers or proposals.
1 Introduction
The study explores whether LLMs can assist scientific reviewing amid strained peer review and missed flaws. It evaluates targeted GPT-4 reviewing tasks and concludes that LLMs show promise for specific tasks but are not yet standalone reviewers.
- Motivation: The study asks whether LLMs can help review scientific papers or proposals as submissions increase and human reviewers become overburdened.The motivation also includes evidence that flawed papers can pass human review.
- Pilot study: A pilot compares multiple LLMs and finds GPT-4 to be the only model successful at finding an error in a short paper.The evaluated models include Bard, Vicuna, Koala, Alpaca, LLaMa, Dolly, OpenAssistant, and StableLM.
- Study design: The main study evaluates GPT-4 with targeted prompts on error detection, checklist verification, and choosing the better paper.These tasks move from specific reviewing functions toward a more ambitious comparative judgment.
- Conclusion: The authors conclude that LLMs have considerable promise for specific review tasks but are not yet capable of functioning as standalone reviewers.The conclusion applies particularly to GPT-4 at its current stage.
2 Related work
Related work covers datasets, automation, and emerging language-model applications in peer review. The paper positions its contribution as a small-scale benchmark for evaluating models on reviewing tasks.
- Automation in peer review: Automation research has addressed reviewer assignment, expertise matching, collusion rings, subjectivity, calibration, and submission-guideline checking.Reviewer assignment is described as the most automated part of peer review.
- Large language models: Emerging language-model work studies peer-review roles, review fairness, and peer grading, but the paper investigates direct assistance with reviewing tasks.The cited work includes recommendations for safe LLM use and models assessing textual fluency or review sentiment.
- Peer-review datasets: The paper contributes datasets comprising 13 flawed short papers, 119 labeled checklist question–paper pairs, and 10 abstract pairs for evaluating reviewing models.The flawed papers include inserted gold-standard errors, while the checklist pairs carry labeled answers.
- Peer-review datasets: Existing peer-review datasets include released reviews, reviewer expertise labels, and annotations of reviews or discussion posts.These resources span computer science and other fields.
3 Can LLMs detect errors in computer science papers?
The error-detection experiment tests whether GPT-4 can identify deliberately inserted flaws in short computer science papers. GPT-4 detects many errors, but failures occur when the needed deduction or external knowledge is unavailable.
- Experimental setup: The experiment uses GPT-4 because a pilot found it was the only tested model to detect the inserted error.The experiment focuses on flaws ranging from mathematical mistakes to conceptual fallacies.
3.1 Methods
The methods construct 13 flawed short papers and test GPT-4 with direct, one-shot, and part-by-part prompts. The prompts ask the model to assess claim validity and identify incorrect statements.
- Data construction: The study deliberately inserts mathematical and conceptual errors into 13 short papers and asks GPT-4 to identify them.The experiment uses three prompting strategies.
- Prompt-Direct: Prompt-Direct presents the short paper and asks an expert-reviewer model to check the validity and correctness of its claims.The prompt explicitly asks the model to reason step by step.
- Prompt-OneShot: Prompt-OneShot provides an example paper whose known error should guide the model before it reviews another short paper.The example illustrates a false square-root lower-bound claim for searching a sorted list.
- Prompt-Parts: Prompt-Parts supplies the paper incrementally, asking whether each set of sentences contains an incorrect claim.The parts prompt is designed to provide one set of sentences at a time.
3.2 Summary of results
The study evaluates GPT-4’s error-detection performance using multiple prompts and repeated responses. GPT-4 detects errors in 7 of 13 short papers, though it also produces false positives and misses errors when proofs are incomplete.
- GPT-4 detects incorrect claims in 7 of 13 constructed short papers.The evaluation aggregates three generated responses per prompt and flags a paper when any response detects an error.
- GPT-4 misses all six errors when the papers omit complete proofs and detection requires extraneous knowledge.
- GPT-4 occasionally produces false positives by labeling correct parts of papers as incorrect.
- Other evaluated models fail to identify the errors and sometimes generate unwarranted critiques.The comparison includes Bard, Vicuna, Koala, Alpaca, LLaMa, Dolly, OpenAssistant, and StableLM using Prompt-Direct.
3.3 Paper and response details
The paper presents detailed examples in which the LLM validates some claims but detects logical, statistical, and computational errors in others. Its responses expose both the value and limits of checking claims from abbreviated paper descriptions.
- Bias/fairness: Per-company interview rates are equal across groups, despite unequal aggregate interview counts.The data support equal rates within each company, while the aggregate difference reflects applicant distributions across companies.
- Bias/fairness: The LLM correctly finds that the claimed implication from equal per-company rates to no bias requires careful separation of conditional and aggregate outcomes.
- Non-parametric regression: The LLM correctly identifies that mean squared convergence does not imply exact equality with probability approaching one.The integral squared error can approach zero without becoming exactly zero, so the claimed implication is invalid.
- The examples show that claim verification depends on distinguishing convergence, exact equality, restricted settings, and unsupported generalizations.
- Sorting: The LLM identifies that additional restrictions on sorting inputs do not remove the order R*log(R) worst-case comparison lower bound.The paper does not provide a comprehensive proof for its faster-comparison assertion.
- Sorting: The LLM raises a false alarm about the Fibonacci restriction by overlooking admissible non-prime Fibonacci values greater than 9.
4 Can LLMs verify author checklists?
The study evaluates GPT-4 for verifying author checklist responses using selected paper sections and finds 86.6% accuracy across 119 labeled pairs. Errors often involved questions requiring information unavailable in the provided text, while targeted prompting and majority voting structured the evaluation.
- Experimental design: GPT-4 was evaluated on 16 checklist questions from 15 NeurIPS 2022 papers using manually labeled Yes, No, or N/A answers.The papers included accepted and rejected submissions with author-provided checklists.
- Prompting: Because entire papers exceeded the context limit, each prompt supplied only sections judged most relevant to the checklist question.Questions without directly corresponding material were paired with sections where the material would normally appear.
- Prompting: The prompts asked GPT-4 to answer from a reviewer perspective with yes, no, or N/A and provide a brief justification.The strategy was designed to align with NeurIPS practices encouraging checklist justifications and section labels.
- Summary of results: 86.6% accuracy was achieved across 119 paper–checklist question pairs against hand-labeled ground truth.GPT-4 produced three responses per pair, with the majority label used for evaluation.
- Summary of results: Author-submitted checklists also matched ground truth 86.6% of the time, while GPT-4 and author mismatches showed limited overlap.GPT-4 disagreed with 12 of 16 mismatched author responses, whereas 9 of its 16 incorrect answers had correct author responses.
- Summary of results: Removing questions lacking textual evidence raises accuracy to 92.8%, but the authors regard 86.6% as more representative of deployment conditions.These cases included questions requiring access to figures unavailable to the model.
5 Experiments
The experiments illustrate GPT-4’s checklist reasoning through examples involving citations and societal impacts. Its responses could correctly connect evidence across paper sections, including cases where authors’ checklist answers were wrong.
- Example 1: GPT-4 correctly identified that experimental code, models, datasets, and simulators were accompanied by citations to their creators.The responses referenced numerous sources across the experiments section, including code, likelihood functions, and the SPMe simulator.
- Example 2: GPT-4 correctly rejected an author’s “Yes” answer when the paper did not explicitly discuss potential negative societal impacts.The model distinguished the absence of negative-impact discussion from broader discussion of dataset bias and human-aligned vision.
- Examples: The examples show that GPT-4 could reason over paper contents non-trivially, but could also produce incorrect checklist responses despite identifying relevant evidence.The experiments included cases involving anonymity, sensitive information, and offensive-content disclosures.
5 Conclusion
The conclusion positions LLMs as potentially useful assistants for specific review tasks while highlighting broader concerns about their role in evaluating papers and proposals. It also connects human-aligned DNNs with addressing dataset biases.
- Conclusion: A neural harmonizer is presented as a way to align DNN visual strategies with human perception and potentially improve performance.The authors frame this alignment as relevant to models of human vision and cognitive science.
- Conclusion: The conclusion connects reliable human-like object categorization with possible changes to cognitive science and artificial intelligence.This claim is presented as a broader significance of models that categorize objects like humans do.
- Conclusion: The broader-impact discussion identifies dataset bias and missing facets of human perception as persistent issues for artificial intelligence.It describes human-like visual strategies as a possible scalable path toward correcting such biases.
- Conclusion: The checklist example indicates that the paper did not explicitly discuss potential negative societal impacts, instead emphasizing broader positive impacts.Those impacts included aligning DNNs with human perception and reducing dataset biases.
A.1 Experimental design
The experiment recruited anonymous Amazon Mechanical Turk participants and documented consent, procedure, compensation, risks, benefits, and confidentiality before participation.
- Experimental design: Participants were recruited through Amazon Mechanical Turk, where they remained anonymous and no sensitive personal information was collected.The study prioritized participants with Master or high qualification ratings.
- Experimental design: Before the experiment, participants reviewed consent information covering procedures, completion time, compensation, risks, benefits, and confidentiality.The expected completion time was approximately 5–8 minutes with a $1.4 reward.
A.2 Pruning out uncooperative participants
The study used a three-stage screening process to exclude participants who did not understand the task or failed an expected catch trial. The checklist responses addressed anonymity and sensitive information, but the supplied passages contain differing judgments about whether the discussion was explicit.
- A.2 Pruning out uncooperative participants: Three screening stages tested task understanding, instruction comprehension, and performance on a catch trial before exclusion.Participants who failed any stage were excluded from further analysis.
- A.2 Pruning out uncooperative participants: The checklist response states that AMT participants remained anonymous and that no sensitive personal information was collected.It also mentions a consent form covering study objectives, procedures, risks, benefits, and confidentiality.
- A.2 Pruning out uncooperative participants: Other responses judge that the authors did not explicitly discuss personally identifiable or offensive content, despite noting participant anonymity and apparently non-offensive example images.The supplied responses therefore disagree about whether the checklist requirement was adequately addressed.
5 Can LLMs choose the better of two papers
GPT-4 struggled to choose the better abstract even when one was deliberately superior, making mistakes by overlooking key differences, overvaluing positive results, and being influenced by extraneous factors.
- The LLM made incorrect choices when selecting between abstracts whose superiority was designed to be evident.The evaluation found mistakes arising from missed differences and extraneous influences.
- 5.3.2 Positive result bias: The LLM favored an abstract with a positive p<0.05 result despite the alternative having more participants and nearly identical p-values.Abstract 1 had 342 participants versus 212, while the p-values were 0.050001 and 0.049999.
- The LLM incorrectly preferred Abstract 2 after hallucinating differences, although Abstract 1 covered a broader parameter range.Abstract 1 applied when d ≥2k −2, whereas Abstract 2 applied when d ≥2k −1.
- 5.3.4 Lower bounds: For lower-bound comparisons, the LLM sometimes mistook a smaller bound as stronger, although the supplied analysis states that the tighter larger bound was preferable.The LLM also correctly identified in another case that a smaller upper bound represented a stronger result.
- 5.3.6 Prompt injection attack: The LLM was also reported to be susceptible to prompt injection, acknowledging influence in one of three responses.The paper cautions that the model’s explanation may not faithfully reflect its actual process.
6 Discussion and limitations
The study finds that LLMs may assist with specific reviewing tasks, but current systems are not suitable as standalone reviewers. The authors recommend human verification and identify limitations involving paper length, figures, model quality, and handcrafted evaluation data.
- LLMs can complement human reviewers on tasks such as checklist verification and flaw detection, but mistakes make sole reliance unsuitable.The authors frame the appropriate near-term role as human-AI collaboration.
- Token limits prevented supplying complete papers, while figure-dependent information caused GPT-4 difficulty with related questions.
- Larger-context or multimodal GPT-4 variants may address input and figure limitations, while fine-tuning could target specific reviewing purposes.The authors also note that current open-source models perform substantially worse than proprietary GPT-4.
- Experiments on error detection and paper comparison relied on a limited number of handcrafted papers created to establish ground truth and avoid training-data overlap.The authors advocate larger datasets for future evaluation.
- The study is extending evaluation to other checklists and reviewer-assignment tasks, including comparisons with established reviewer-paper similarity models.
A Pilot experiments
The appendix introduces pilot experiments used to select the model and prompting strategy for the main study.
- The pilot experiments compare candidate models and prompting strategies before the main evaluations.
A.1 Choosing a model
The model-selection pilot found GPT-4 uniquely successful at detecting an inserted error in a fictitious paper, while the accompanying outputs illustrate both confident validation and skepticism about the paper’s claims.
- Only GPT-4 detected the inserted error among the evaluated LLMs in the pilot model comparison.The compared systems included Bard, Vicuna, Koala, Alpaca, LLaMa, Dolly, OpenAssistant, and StableLM.
- The evaluated paper describes sorted-list search, an AI-based algorithm, theoretical guarantees, and simulations as its main claims.
- Several generated evaluations accepted the claims and reported that the simulations supported the theoretical bound.
- Other evaluations noted that the supplied information was insufficient to determine some claims and questioned the comparison with established algorithms.One critique specifically contrasts the claimed sqrt(m) comparisons with Binary Search’s O(log m) complexity.
- The responses also identified missing punctuation and other alleged grammatical issues, showing that model outputs varied in the kinds of problems they flagged.
A.2 Prompting strategies
The prompting pilot found that asking the model to check correctness was more effective for error detection than asking it to write a general review.
- Prompting the LLM to write a review failed to identify errors in the pilot context.
- The successful alternative asked the LLM a targeted question about claim validity and correctness.
- The resulting review praised the claimed sqrt(m) bound and information-theoretic lower bound while noting the absence of technical algorithmic details.
- The example review requests stronger comparisons with existing algorithms and more detail about the lower bound and methodology.
B NeurIPS checklist and prompts
The appendix defines how NeurIPS checklist items were filtered and reformulated into third-person prompts for LLM verification. It retains 16 items with verifiable ground truths and includes conditions alongside each checklist question.
- B.1 Checklist item selection: Item 1(a) was excluded because labeling it required significant field-specific knowledge unavailable for papers sampled across the conference.
- B.1 Checklist item selection: Item 1(d) was excluded because reading ethics guidelines cannot be verified from information submitted to the conference.
- B.1 Checklist item selection: 16 checklist items remained after removing two questions whose answers could not be reliably established from submitted paper information.Item 1(a) required substantial domain expertise, while item 1(d) concerned whether authors had read ethics guidelines, an unverifiable action.
- B.2 Full checklist questions: Each retained item is represented both as an original first-person checklist statement and as a third-person prompt question for the LLM.The prompt questions incorporate the relevant prerequisite condition when one exists.
- B.2 Full checklist questions: Prompt questions were individually input to the LLM as the final component of each user prompt.
- B.2 Full checklist questions: Prerequisite conditions determine when questions apply, such as asking about proofs only when theoretical results are included and reproducibility materials only when experiments are run.
- B.2 Full checklist questions: The retained checklist questions cover theoretical assumptions and proofs, experimental reproducibility materials, training details, error bars, limitations, and societal impacts.
- B.2 Full checklist questions: The experiment asks the LLM to assess reporting of training details and error bars, including data splits, hyperparameters, selection procedures, and repeated-run variability.