Source-linked AI summary
More Criticism Does Not Make a Better Review: EquiReview-R
Zexing Zhang, Jichao Li, Tianyang Lei, Yude Fu, Yang Kewei
TL;DR
AI review systems can generate many criticisms while missing consequential weaknesses or retaining unsupported allegations. EquiReview-R refines a structured concern set against evidence before complementary omission search and selective stopping, achieving lower overcritique without sacrificing material issue coverage under the stated evaluation conditions.
Problem
AI-assisted review needs to distinguish consequential omissions from unsupported allegations, because these require opposite corrections and aggregate measures obscure the difference.
Method
EquiReview-R revises existing concerns against localized evidence, searches for omissions independently and conditionally from the frozen revised state, and selects stop, continue, or defer.
Results
EquiReview-R meets the prespecified omission non-inferiority criterion and reduces major overcritique by −7.4 percentage points while strict issue coverage changes by only −0.3 percentage points.
Takeaways & Limitations
The results show that reviews can become more concise and better supported without surrendering material issue coverage, while ReviewTrace enables study of revision, disagreement, and provenance.
Takeaways & Limitations
The findings are conditional on recent AI papers, the main-paper view, a fixed external search process, and the stated judgment policy rather than universal completeness.
Abstract
from arXiv · showhide
AI reviewers can now produce many specific criticisms, but more criticism is not necessarily a better review. A review may miss a consequential weakness or retain an allegation that available evidence does not support. These failures require opposite corrections, yet generation-oriented systems and aggregate measures obscure the distinction. We therefore recast AI-assisted review as evidence-guided refinement of a structured concern set, with omission and overcritique treated as separate risks. Building on this formulation, we introduce EquiReview-R, which resolves existing concerns against localized evidence, searches for missing issues from independent and review-conditioned perspectives, and returns stop, continue, or defer. To expose the failure mode that motivates this design, we construct an evidence-linked trajectory corpus. Its retrospective analysis shows why revision must precede further search: nearly all concerns in a high-recall review lack a definitive evidential disposition, while an earlier refinement mechanism cannot revise them. On a frozen cohort of previously unseen papers, EquiReview-R satisfies the prespecified non-inferiority criterion for major omission, reduces major overcritique from 15.5% to 8.1%, and attains a one-sided omission upper bound of 9.9% while stopping on 52.4% of papers. Computation-matched controls, controlled pairs, and ablations show that the gain comes from revision rather than extra inference or shorter output. We release the corpus as ReviewTrace, an evidence-linked resource for studying review revision, disagreement, and provenance.
1. Introduction
AI review quality requires distinguishing missing consequential issues from unsupported allegations. EquiReview-R therefore revises existing concerns against evidence before searching for omissions and making a selective stopping decision.
- AI reviewers can surface issues humans miss but may also overemphasize minor points, shifting the bottleneck toward resolving criticism.
- A missing matched baseline calls for adding a concern, whereas an unsupported data-leakage allegation calls for removal or narrowing.
- Review improvement treats concerns as structured claims linked to paper locations and evidence, allowing the set to expand or contract as issues are assessed.
- 96.3% of concerns in the high-recall review state lacked support, narrowing, or rejection, and an earlier mechanism left every initial concern unchanged.
- EquiReview-R revises unresolved concerns first, then searches independently and conditionally for omissions from the frozen revised state.
- The evaluation uses frozen reviews, independent omission candidates, controlled pairs, matched computation, and ablations to test whether gains arise from revision.
2. Related Work
Prior work expands, organizes, evaluates, and selectively controls AI-generated criticism. This paper instead focuses on which concerns should survive evidential scrutiny before further search or stopping.
- AI-assisted scientific review: Earlier AI-assisted review resources and systems improve review generation through prediction, retrieval, verification, investigation, consolidation, and structured logging.
- AI-assisted scientific review: The paper’s object is the resulting concern set and the question of which concerns should survive evidential scrutiny before further search or a stopping decision.
- Evaluating review content: Review evaluation studies measure score agreement, response usefulness, overlap with observed feedback, concern matching, facet attention, and critique quality.
- Revision, robustness, and selective decisions: Self-critique and tool-interactive critique target model errors, while human-in-the-loop and robustness studies examine correlated errors, manipulation, and deployment risk.
- Revision, robustness, and selective decisions: Selective prediction and conformal risk-control methods provide frameworks for abstention and finite-sample evaluation under prespecified risk constraints.
3. Method
EquiReview-R represents review as a structured, evidence-linked state whose concerns are revised before complementary discovery and selective stopping. Its design separates visible review content from recoverable revision history and evaluates omission and overcritique as distinct risks.
- Problem Formulation: A review state contains visible concerns, typed relations among them, and immutable evidence-and-revision history.
- Problem Formulation: Each concern records an alleged failure, paper location, supporting and countervailing evidence, a resolution condition, materiality, and status.
- Problem Formulation: Validity is assessed before materiality, and primary endpoints use only valid major concerns that could affect claims, central results, or evaluation credibility.
- Problem Formulation: Major omission counts a distinct valid major concern found externally or an unresolved major concern at stopping, while overcritique retains a concern needing removal or material narrowing.
- Coupled operators: The method revises the existing review, searches for omissions from independent and review-conditioned perspectives, consolidates candidates, and returns stop, continue, or defer.
- Evidence-guided revision: Revision builds evidence records and assigns outcomes such as supported, narrowed, refuted, merged, resolved, or unresolved with differing consequence.
- Complementary search: After revision freezes the state, independent and conditioned discovery searches uncovered facets, while consolidation preserves relations and removes duplicates.
- Selective stopping: Selective stopping uses pre-evaluation features including unresolved consequence, evidence completeness, disagreement, relation uncertainty, and discovery yield.
4. Experiments
The experiments separate retrospective diagnosis from confirmation on previously unseen papers, using frozen evaluations and independent omission candidates. They compare EquiReview-R with a computation-matched generation-oriented control and assess omission, overcritique, coverage, output, and judgment reliability.
- Study design: The study uses a retrospective corpus of 380 recent AI papers and a separate frozen cohort of 271 previously unseen papers for confirmation.The confirmation cohort spans machine learning, natural language processing, computer vision, and AI systems, with every system receiving the same main-paper view.
- Baselines and controls: E3-Matched receives essentially the same effective calls and generated tokens as EquiReview-R but repeats generation rather than revising existing concerns.This control isolates the algorithmic contribution from added inference.
- Evaluation protocol: External evaluation freezes each system’s review and stopping decision before independent searches construct missing-concern candidates.The primary omission endpoint combines independent search with a separate review of the frozen state, excluding the shared candidate pool.
- Judgment protocol: Candidates are judged independently by two blinded models, with a third adjudicating categorical disagreements in a separate source-hidden call.Validity, identity, revision action, and materiality are elicited separately, forming a repeatable model-based measurement panel rather than objective scientific truth.
- Outcomes and analysis: The primary tests assess major-omission non-inferiority, major-overcritique reduction, and the omission criterion for stopped papers.The analysis also reports strict coverage, visible concern count, generated tokens, controlled pairs, ablations, relation-policy sensitivity, and subfield heterogeneity.
5. Results
EquiReview-R improves the review error profile by revising existing concerns before searching for omissions, reducing unsupported criticism without materially sacrificing issue coverage. It also provides selective stopping and makes evidence-guided concern outcomes explicit.
- Diagnosis: 16.62 of 17.27 visible concerns per paper remained unresolved in the high-recall initializer, and an earlier revision mechanism changed none across 320 papers.This diagnosis motivates revising unresolved concerns before interpreting additional search.
- Main results: −7.4 percentage points reduced major overcritique relative to the high-recall initializer, while strict issue coverage changed by only −0.3 percentage points.The visible review also contained −4.7 fewer concerns per paper, indicating a changed error profile rather than a recall-for-concision tradeoff.
- Main results: At essentially matched calls and output tokens, E3-Matched retained a larger visible review and more than twice EquiReview-R's major-overcritique rate.The comparison attributes the gain to changing the review state rather than supplying additional inference.
- Selective stopping: 142 of 271 papers were stopped, with empirical omission risk of 5.6% and a one-sided upper bound of 9.9% at 52.4% coverage.Stopping all papers would yield a one-sided upper bound of 16.4%, whereas the remaining papers receive continue or defer decisions.
- Mechanism: Most initially unresolved concerns received definite dispositions, while only 4.0% remained unresolved with high consequence.Supported and narrowed concerns remain visible at evidence-justified scope; refuted, merged, and resolved concerns leave the visible review but remain in the trajectory.
- Mechanism: 95.8% recall of inserted issues accompanied a 3.3% clean-control false-concern rate, compared with 10.8% for E3-Matched.Ablations further link evidence-guided revision to overcritique control and both search perspectives to omission control.
- Robustness: Agreement was 0.74 for validity and 0.52 for materiality, while alternative identity policies and leave-one-system-out pools preserved the coverage ordering.The reduction was not confined to adjudicated cases, although materiality remained the main source of measurement uncertainty.
6. ReviewTrace
ReviewTrace is an evidence-linked corpus built around concern-level revision trajectories, independent judgments, and structured relations. It supports studying how reviews are revised rather than only comparing static review artifacts.
- Corpus design: ReviewTrace records each concern from first appearance through support, narrowing, merging, resolution, or removal, with changes linked to localized evidence.It also preserves two independent judgments with disagreement and adjudication.
- Ablations: Figure 6 indicates that revision primarily controls overcritique, while both discovery perspectives protect omission; review length alone does not reliably determine stopping.Points below the horizontal line satisfy the omission criterion.
- Corpus design: The resource exposes revision trajectories that existing peer-review resources generally do not provide as their primary released unit.The comparison distinguishes trajectories, typed relations among concerns, and independently retained judgments.
- Release: The release contains 1,900 structured states and 35,292 recorded judgments, alongside specifications, evaluation code, hashes, and verification utilities.Its frozen construction record contains 77,929 distinct model calls and a public-price equivalent of $9,700.
- Judgment analysis: Figure 7 reports lowest agreement for materiality, while most overcritique reduction comes from items on which the two primary judges agree.
7. Discussion and Limitations
The paper limits its empirical claims to a specified evaluation setting and emphasizes that deployment requires computation-aware operating points. It also identifies correlated model errors and unresolved judgment noise as boundaries for interpretation and use.
- Scope: The empirical claims are conditional on recent AI papers, the main-paper view, a fixed external search process, and the stated judgment policy.The confirmation results establish reliability under that evaluation condition, not universal completeness.
- Deployment: Deployment should choose an operating point using both stopping coverage and the evidence still required for non-stopped papers.At comparable inference, the matched control indicates that additional generation enlarges rather than improves the visible review.
- Validity and deployment: Errors from Luna, Terra, and Sol may be correlated because they are separate calls from one model family.Blinding, decomposed labels, agreement-subset analyses, and alternative reference policies reduce avoidable circularity but do not create noise-free truth.
8. Conclusion
The paper argues that capable AI review requires deciding which criticisms survive evidence, what remains missing, and when uncertainty should prevent stopping. EquiReview-R combines revision, complementary discovery, and selective risk control to support more concise, better-supported reviews without surrendering material issue coverage.
- Conclusion: The central challenge is determining which criticisms survive evidence, what remains missing, and when uncertainty should prevent stopping.
- Conclusion: EquiReview-R connects evidence-guided revision, complementary discovery, and selective risk control in a reconstructable state.
- Conclusion: The results show that reviews can become more concise and better supported without surrendering material issue coverage.The paper presents this as a foundation for auditable workflows where judgments and responses accumulate as evidence.