PaperlayerView exact revision on arXiv

Source-linked AI summary

PremiseCheck: Efficient Fact-Checking of Human and AI Peer Reviews

Dingjie Song, Zhiling Yan, Ziqian Zhang, Yuxuan Zhang, Lichao Sun

arXiv:2610.21046v1cs.CLcs.AIcs.LG

TL;DR

Review comments can contain factual errors, while checking each one against a long paper is costly and whole-comment decision models struggle with implicit assertions. PremiseCheck uses an LLM to rewrite comments as explicit premises and a decision model to verify them against the paper. Across three fields it approaches GPT-5.5’s accuracy at about half the cost.

  • Problem

    Review comments can contain factual errors, but whole-comment decision models struggle with implicit assertions and a frontier LLM must reread the paper for each check.

    Source: §1 · pp. 1–2

  • Method

    An LLM rewrites each comment as explicit premises without reading the paper, and a decision model checks each premise against the paper.

    Source: §1 · p. 2

  • Results

    Across all three ReviewFact fields, medium-reasoning PremiseCheck stays within 0.3 AUROC points of holistic GPT-5.5 or exceeds it at about half its cost.

    Source: §1 · p. 2

  • Takeaways & Limitations

    PremiseCheck can flag comments and show the premise the paper appears to refute, so a person can check that claim instead of rereading the paper.

    Source: §9 · p. 10

    PremiseCheck targets factual errors about what a paper states, not errors of judgment such as a concern’s importance or the work’s novelty.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Peer review is under strain from the growth of submissions, and review comments written by humans or AI can misstate what a paper says. We introduce ReviewFact, a benchmark that labels such factual errors in review comments under one definition across NLP, machine learning, and the natural sciences, and that measures cost and latency alongside accuracy. A frontier LLM checks comments well but must read the whole paper for each check, while cheap decision models, which score whether a statement holds in a text, fall far short on whole comments that mix several implicit assertions. PremiseCheck is a framework that combines the two. An LLM rewrites each comment as explicit premises without reading the paper, and a decision model checks each premise against the paper. When each comment is checked on its own, PremiseCheck reaches accuracy comparable to GPT-5.5 across all three fields at about half its cost, and on NLP reviews at a quarter of its cost and less than half its latency.

1 Introduction

Factual errors in review comments matter at peer-review scale, but checking each comment is difficult because its implicit claims may require evidence from across a long paper. PremiseCheck separates comment decomposition from paper verification, combining an LLM with a decision model to approach GPT-5.5’s accuracy at lower cost.

1 Introduction

  • Whole-comment decision models fall short because comments mix implicit assertions, although Jev matches GPT-5.5 on single premises written out by human annotators.A frontier LLM must reread the full paper for every comment, multiplying cost across tens of thousands of reviews.Source: §1 · p. 2
  • PremiseCheck has an LLM rewrite each comment as explicit premises without reading the paper, then a decision model verifies them against the paper.The comment receives the score of its weakest premise.Source: §1 · p. 2
  • At about half the cost, medium-reasoning PremiseCheck stays within 0.3 AUROC points of holistic GPT-5.5 or exceeds it on all three ReviewFact fields.On NLP reviews, a decomposer without reasoning reduces cost to a quarter while remaining non-inferior.Source: §1 · p. 2
  • ReviewFact labels factual errors under one definition across NLP, machine learning, and natural sciences, and measures cost and latency alongside accuracy.It evaluates factual claims about a paper, not errors of judgment.Source: §1 · pp. 1–2

2 Related Work

Prior work annotates factual errors in review comments, decomposes claims for verification, and studies cost-aware inference. ReviewFact applies one factual-error definition across fields, while PremiseCheck separates premise writing from verification to avoid sending papers to an expensive LLM.

2.1 Factual Errors in Review Comments

  • ReviewFact differs from prior review-comment datasets by applying one factual-error definition across three fields and measuring cost and latency alongside accuracy.The cited datasets serve broader goals and label factual errors inconsistently, if at all.Source: §2.1 · p. 2

2.2 Claim Decomposition and Grounded Checking

  • PremiseCheck has an LLM write premises without reading the paper, then uses a decision model to check them, unlike approaches that use an LLM verifier.Because decision models accept questions, PremiseCheck can route premises and phrase checks in several ways.Source: §2.2 · p. 3

2.3 Cost-Aware Inference

  • Unlike cascades that still send most comments and their papers to an expensive LLM, PremiseCheck gives the paper only to the decision model.Its decomposer needs no reasoning, contrasting with test-time scaling that spends more output tokens and time for accuracy.Source: §2.3 · p. 3

3 The ReviewFact Benchmark

The ReviewFact benchmark scores whether review comments contain factual errors about their papers, using relabeled expert annotations from three fields. It reports AUROC, cost, and latency, and contains 9,267 comments on 217 papers.

3.1 Task

  • A factual error is a comment claim unsupported by the paper; the task excludes errors of judgment and evaluates AUROC, cost, and latency per comment.AUROC is the probability that an error comment scores below a correct one, with ties counting one half.Source: §3.1 · pp. 3–4

3.2 Labels from Expert Annotations

  • ReviewFact reuses expert comment judgments from ReviewCritique, ReviewScore, and PeerReview Bench, mapping annotations to its factual-error label and excluding out-of-scope comments.For 178 PeerReview Bench comments with rationales, annotators agree with GPT-5.5 labels at κ 0.76 and 0.77; PremiseCheck versus holistic GPT-5.5 differences using annotator labels range from −3.7 to +2.7 and are never significant.Source: §3.2 · p. 4

3.3 Statistics

  • The benchmark contains 9,267 comments on 217 papers, with factual errors in 4.8% of NLP comments and 2.5% of natural-science comments.Median paper lengths are 8.5k, 9.1k, and 14.4k tokens for NLP, machine learning, and natural-science reviews, respectively.Source: §3.3 · p. 4

4 Method

PremiseCheck decomposes a review comment into premises, routes them to paper-based or knowledge-based verification, and aggregates their scores into one comment score. Only the decomposer is an LLM, and it does not read the paper.

Decompose

  • Each comment is decomposed into 1–6 self-contained declarative premises that preserve explicit and implicit assertions without adding facts.The decomposer sees only the comment; its prompt asks it to name referenced methods, tables, or sections.Source: §4 · pp. 4–5

Route

  • Routing classifies each premise as paper-grounded or knowledge-based, using only the premise and a decision model.Premises with a paper-decidability probability of at least 0.5 are treated as paper-grounded.Source: §4 · p. 5

Verify

  • The decision model checks each premise against the paper or general knowledge according to its route; natural-science reviews instead use decomposer flags and a contradiction question.The paper-based and knowledge-based questions ask whether the proposition is true according to the paper or true, respectively.Source: §4 · p. 5

Aggregate

  • The comment score is the lowest premise score, reflecting that a false premise makes the comment factually erroneous.Figure 4 illustrates the lowest premise score as the comment score.Source: §4 · p. 5, Figure 4 · p. 5
  • When no premise is returned, which occurs for fewer than 1% of NLP comments, the system falls back to holistic judgment by the decision model.Source: §4 · p. 5

Why it is cheap

  • Per-comment cost comprises one LLM call on the comment plus a routing and verification call for each premise; excluding the paper from the LLM call keeps it short.Decision-model calls together cost $0.0015 per comment on NLP reviews in the reported implementation.Source: §4 · pp. 5–6

5 Experimental Setup

The experiments use GPT-5.5 as PremiseCheck’s decomposer and Jev as its decision model, and compare per-comment judgments with holistic baselines and a cascade. Non-inferiority is assessed using a pre-specified AUROC margin and paired bootstrap intervals.

Models

  • The implementation uses GPT-5.5 to decompose comments and Jev to decide and verify, with papers truncated to 24k tokens.The main setting checks each comment independently; checking all comments in a finished review is evaluated separately.Source: §5 · p. 6

Baselines

  • Baselines include holistic GPT-5.5 at three reasoning levels, GPT-5.4-mini, Jev, and a cascade that sends the 70% most suspicious comments to GPT-5.5.Source: §5 · p. 6

Metrics

  • Non-inferiority requires the lower bound of the paired AUROC-difference 95% interval to exceed −3, using 2,000 paper-bootstrap resamples.Holistic judgment’s repeated paper input is counted at cached price, favoring that baseline.Source: §5 · p. 6

6 Main Results

With each comment checked independently, medium-reasoning PremiseCheck is close to or above holistic GPT-5.5 across all three ReviewFact parts at about half the cost. Robustness results also show performance with an open-weight decision model and signal beyond comment length.

6.1 Accuracy, cost, and latency

  • Without reasoning, PremiseCheck costs a quarter of the reference on NLP reviews while remaining non-inferior, and takes 45%, 37%, and 22% of reference time across the three parts.Source: §6.1 · p. 7
  • Holistic Jev and GPT-5.4-mini fall 5.7–13.4 points below the reference, while the cascade remains below medium-reasoning PremiseCheck on every part.Source: §6.1 · p. 7

6.2 Robustness

  • With Kev-27B as router and verifier, PremiseCheck reaches Jev’s AUROC; the no-reasoning version is non-inferior at 17.8% of reference API cost, excluding GPU time.With a medium-reasoning decomposer, the interval falls just short of non-inferiority.Source: §6.2 · p. 7

6.1 Accuracy, cost, and latency

  • Within 0.3 AUROC points of the reference or above across all three parts, medium-reasoning PremiseCheck costs about half as much.It is non-inferior on NLP reviews and not significantly different on the other two parts; Figure 5a shows per-comment cost and AUROC on NLP reviews.Source: Figure 5 · p. 7, §6.1 · p. 7

6.2 Robustness

  • Adding PremiseCheck to comment length raises AUROC significantly by 3.3, although none of the reported length-adjusted comparisons reaches non-inferiority.PremiseCheck is not a proxy for comment length, despite longer comments being more often factual errors.Source: §6.2 · p. 7

7 What Makes PremiseCheck Work

PremiseCheck gains accuracy from decomposing comments into premises, while verifier design and phrasing choices shape its cost and performance. Its scope is factual errors, and decomposition and verification errors remain important failure sources.

7.1 Decomposition makes decision models useful

  • Decision models scoring 50 to 57 as holistic judges come within one point of GPT-5.5 as verifiers, while weaker decomposers lose 2 to 6 points.With Kev-27B as verifier, GPT-5.6 Sol, GPT-5.6 Luna, and Nemotron Super 3 are non-inferior at 6% to 17% of holistic-judgment cost; Kev-27B and Perplexity Decider intervals fall just short of non-inferiority.Source: §7.1 · pp. 7–8, Figure 7 · p. 8

7.2 Where the savings come from

  • At low or no reasoning, PremiseCheck is more accurate than per-comment holistic judgment on NLP reviews at less than half its cost; its relative cost falls from 53% to 49% on longer papers.The cost comparison is between the shortest and longest thirds of papers; without reasoning, holistic judgment falls to chance level.Source: §7.2 · p. 8

7.3 Design choices

  • With 16 phrasings, AUROC rises from 63.1 to 65.7; routing and weakest-premise scoring also matter to accuracy.Skipping routing lowers AUROC from 61.8 to 58.8 on common machine-learning premises, and taking the mean instead of the minimum yields 59.4; contradiction questions drive the phrasing gain.Source: §7.3 · p. 8

7.4 Error types and scope

  • On Misunderstanding and Neglect errors, PremiseCheck scores above holistic GPT-5.5, though not significantly; its lowest premise claims an omission for 79% of Neglect errors versus 36% of correct comments.It falls behind holistic GPT-5.5 on labels that also count unimportant concerns or unreasonable requests, while the gap disappears on factual errors alone.Source: §7.4 · pp. 8–9

7.5 Where PremiseCheck fails

  • False alarms arise mostly from decomposition on NLP reviews (58%) and verification on natural-science reviews (45%); NLP misses split between questionable labels (40%) and verification (36%).Holistic GPT-5.5 fails on only 13% and 15% of machine-learning and natural-science false alarms, respectively.Source: §7.5 · p. 9

8 Using PremiseCheck in Practice

Batching allows checks to run after a review is complete, and fusion improves on either batched approach. For flagging, PremiseCheck identifies suspicious comments and the premise a person can inspect.

8.1 Batching and fusion

  • Batched fusion reaches 71.5 at 28% of reference cost, significantly above batched holistic GPT-5.5; the two scores have a rank correlation of 0.56.Batched holistic GPT-5.5 without reasoning reaches 67.8 at 7.7% of reference cost, while batched PremiseCheck reaches 69.0 at 20%; those two do not differ significantly.Source: §8.1 · pp. 9–10

8.2 Flagging comments for a person to check

  • Among the 5% most suspicious comments, PremiseCheck flags errors two to three times as often as random; three top comments per paper find an error in 40% of error-bearing papers, versus 29% for batched holistic GPT-5.5.A flag includes the premise Jev finds false for a person to check.Source: §8.2 · p. 10

9 Conclusion

PremiseCheck combines an LLM that rewrites comments as premises with a decision model that checks them against the paper. It reaches accuracy comparable to GPT-5.5 across ReviewFact fields at lower cost, while remaining limited to factual errors.

9 Conclusion

  • Across every part of ReviewFact, PremiseCheck reaches accuracy comparable to GPT-5.5 at about half its cost when each comment is checked on its own.On NLP reviews, a decomposer without reasoning cuts cost to a quarter while remaining non-inferior.Source: §9 · p. 10
  • PremiseCheck can check comments as they are written or after a review is finished, and can show a person the premise the paper appears to refute.For completed reviews, fusing PremiseCheck with holistic GPT-5.5 gives the best accuracy.Source: §9 · p. 10
  • Errors of judgment, such as unimportant concerns or unreasonable requests, lie outside PremiseCheck and call for different methods.Most false alarms on NLP reviews arise in decomposition.Source: §9 · p. 10

Limitations

PremiseCheck targets factual errors checkable against the paper, while testing further open decision models remains future work.

Limitations

  • PremiseCheck does not target errors of judgment, such as whether a concern is important or novel; detecting them is future work.These are comments that cannot be checked against what the paper states.Source: Limitations · p. 10
  • With Kev-27B, PremiseCheck reaches the same AUROC as with Jev on NLP reviews, but testing further open decision models remains future work.Jev, the main decision model, is served through an API.Source: Limitations · p. 10

Ethical Considerations

This section documents dataset-use safeguards, label verification, and how source annotations and prompts define the review statements being checked. It also examines supplemental analyses and evaluation choices, including decision-model capabilities, alternative verifiers, costs, and accuracy.

Ethical Considerations

  • Dataset licenses restrict evaluation and prohibit training; because verifier errors can unfairly discredit reviewers or lend false authority to incorrect comments, PremiseCheck is meant to direct human attention, not replace human judgment.Because ReviewScore states no license, the machine-learning portion is rebuilt from original data with a script rather than released as data.Source: Ethical Considerations · p. 11

A.1 Source Annotations

  • ReviewFact starts from expert judgments in ReviewCritique (Du et al., 2024), ReviewScore (Ryu et al., 2025), and PeerReview Bench (Kim et al., 2026), with dataset-specific annotation units and factual-error filters.ReviewCritique uses independently labeled review sentences with expert disagreement resolution; ReviewScore uses median ratings of three annotators per premise; PeerReview Bench scientists judge comments, with factual errors determined separately.Source: §A.1 · pp. 14–15

A.2 Label Verification

  • On 178 natural-science comments, PremiseCheck’s AUROC difference from holistic GPT-5.5 ranged from −3.7 to +2.7 with no interval excluding zero; it was +2.7 on 47 jointly factual errors.Judgment and unclear labels were excluded from this evaluation.Source: Table 3 · p. 15, §A.2 · p. 15

B.1 Prompts

  • The NLP decomposer extracts explicit and implicit premises, while the machine-learning variant keeps only asserted or directly presupposed premises and uses fewer of them.The NLP prompt allows 1–6 premises; the machine-learning prompt allows 1–4 and limits general-knowledge premises to at most one.Source: §B.1 · p. 16
  • For natural-science reviews, the decomposer types premises as paper, knowledge, or opinion, and verifies only paper premises unless none are labeled paper.It requires concrete, checkable statements and separates evaluative language into opinion premises.Source: §B.1 · pp. 16, 18

B.2 Decision-Model Questions

  • Decision-model verification uses 16 question phrasings per premise, with separate questions for paper-grounded and knowledge premises; natural-science reviews use a contradiction question and invert its probability.The phrasing subsets pair the corresponding paper-grounded and knowledge questions.Source: Table 4 · p. 17, Table 5 · p. 17, §B.2 · p. 18

C.5 Machine Learning and Natural-Science Reviews

  • On machine-learning reviews, the field-specific prompt shortens premise lists and reduces false alarms; on natural-science reviews, contradiction questions avoid treating unmentioned details as false.Table 10 compares decomposition prompts and verification questions; the natural-science PremiseCheck rows use the natural-science decomposition prompt.Source: §C.5 · pp. 19–20, Table 10 · p. 20

D.1 Capability Map of the Decision Model

  • On the operations PremiseCheck uses, Jev is as accurate as GPT-5.5.Figure 14 plots AUROC for Jev, GPT-5.4-mini, and GPT-5.5 across verification operations.Source: §D.1 · p. 20, Figure 14 · p. 21
  • Jev costs $0.00044 per full-paper premise check and has 216 ms median latency, versus 5.1 s for GPT-5.5; models are close to chance on whole-comment correctness.Jev matches the full paper at least as well and routes premises equally well, but classifies argument types better and claims and questions worse than the comparator.Source: §D.1 · p. 21

D.2 Verifiers

  • MiniCheck-7B lowers PremiseCheck from 63.0 to 54.0, while holistic MiniCheck-7B reaches 59.5 and PremiseCheck exceeds it by +3.4 with one question or +6.1 with 16 phrasings.MiniCheck-7B assigns 25.5% of non-omission premises probability below 0.3, versus 11.2% for Jev; because comments average 3.3 premises and use the minimum score, one false alarm can mark a comment wrong. Using Jev for knowledge premises recovers only 2.8 points.Source: §D.2 · p. 22, Table 13 · p. 22

D.3 Cost Breakdown and Paper Length

  • Holistic GPT-5.5 reads 9,991 input tokens per NLP comment on average, including 8,831 cached tokens, while the decomposer reads only the comment.The passage identifies Table 14 as the per-comment cost breakdown and Table 15 as the cost-by-paper-length comparison, but supplies no additional values for those breakdowns.Source: §D.3 · p. 22, Table 14 · p. 23, Table 15 · p. 23

D.4 Other Decision Models and Other LLMs

  • The comparison evaluates decision models using the same Jev requests and API, while LLMs receive unchanged GPT-5.5 decomposition and holistic prompts.GPT-5.6 Sol and Luna use medium reasoning; other LLMs use no reasoning and temperature 0, except Nemotron Super 3, which uses default sampling.Source: §D.4 · p. 23

D.6 Error Types and Scope

The appendix describes error-type and label breakdowns, failure-analysis procedures, and the setup used to compare batched systems.

D.6 Error Types and Scope

  • Table 19 and Figure 15 break down NLP-review AUROC by error type, while Table 20 compares factual and judgment labels.Table 19 compares each error type with correct comments; Table 20 uses the factual ReviewFact label and broader judgment labels.Source: §D.6 · p. 23, Figure 15 · p. 24, Table 19 · p. 25

D.7 Failure Analysis

  • Failure analysis samples false alarms among the 10% lowest scores and factual-error misses above the median, then assigns the earliest failing stage.Three annotators independently label 164 comments; Fleiss’ κ is 0.45 for stage and 0.47 for code, with discussion resolving 26 comments.Source: §D.7 · pp. 23, 25
  • Tables 16–18 cover alternative models, decomposition choices, and Jev phrasing sets; their captions specify the comparison scopes, not outcomes.The supplied passages give table descriptions but not the corresponding results.Source: Table 16 · p. 24, Table 17 · p. 24, Table 18 · p. 24
  • Table 18 reports 63.1 AUROC for Single question and 63.0 for 8 supporting (Δ −0.0 [−0.4, +0.4]).The table compares phrasings per premise at equal cost.Source: §D.7 · p. 24, Table 18 · p. 24

E Batching and Fusion

  • For batching, both systems process the same comments per review; costs are divided across comments and latency is reported per comment.Batched Jev reads the paper once for up to eight premises, using two questions per premise.Source: E Batching and Fusion · p. 25
  • Figure 16 marks the failing component in four examples, with the lowest premise framed in red.The comment, premises, and expert text are verbatim.Source: Figure 16 · p. 26

F.1 Scored Comments

The examples illustrate both factual errors that PremiseCheck flags and a correct comment it flags, alongside selected machine-learning review cases and author responses.

F.1 Scored Comments

  • A factual error about unclear motivation is split into two premises; Jev assigns 0.04 to the disputed premise, which sets the comment score.Holistic GPT-5.5 also flags the comment, with probability 0.15.Source: §F.1 · p. 26
  • A correct comment about representational similarity analysis is flagged because the paper contains neither that analysis nor linear mappings.The decomposer rewrites “paper” as “the paper,” making both premises false as written; Jev gives each probability 0.03.Source: §F.1 · pp. 26–27
  • Tables 21 and 22 organize sampled failures by failure code and stage, with Table 22 breaking stage counts down by part and case.Table 21 groups codes by stage; Table 22 uses shading for share and bold for stages at or above 40% of a cell.Source: Table 21 · p. 27, Table 22 · p. 27

F.2 Flagged Comments with Author Responses

  • The machine-learning examples include two flagged errors whose authors say the paper contains what the comments request, plus a false alarm about a CLIP citation.For the false alarm, the authors confirm Section 1 lacked the citation; Jev may have missed its location because it reads the whole paper.Source: §F.2 · p. 28

F.1 Scored Comments

  • Figure 18 contrasts holistic Jev’s 0.62 score with PremiseCheck’s 0.07 probability for premise 2, which sets the comment score.The example is a factual error implying that the paper did not test other large-scale language models such as OPT and GPT-3.Source: §F.1 · p. 28, Figure 18 · p. 29

F.2 Flagged Comments with Author Responses

  • Figure 19 presents two flagged machine-learning comments with author responses, with premise routes and probabilities indicated.The caption notes that quoted text is verbatim apart from marked omissions and a corrected typo.Source: Figure 19 · p. 29
Loading 2610.21046v1…