Source-linked AI summary
FactReview: Evidence-Grounded Peer Review with Execution-Based Claim Verification
Ling Yue, Chaoqian Ouyang, Hang Xu, Ruijun Huang, Yuchen Liu, Libin Zheng, Wei Liu, Shaowu Pan, Shimin Di, Min-Ling Zhang
TL;DR
LLM reviewers often leave literature- and code-dependent claims difficult to verify. FactReview constructs claim-level evidence records from manuscripts, related work, reference checks, and bounded code execution. On 26 test papers, it achieves 84.3% F1 for claim recovery and preserves human responsibility for acceptance decisions.
Problem
Manuscript-only review leaves literature- and code-dependent claims difficult to verify consistently.
Method
FactReview extracts atomic claims and grounds them in manuscript, literature, reference-integrity, and optional execution evidence with preserved provenance and scope.
Results
84.3% F1 for claim recovery on 26 test papers with 354 human-verified claims.
Takeaways & Limitations
FactReview makes automated assessments inspectable while preserving human responsibility for final decisions.
Takeaways & Limitations
The evaluation is limited to a purposively selected set of machine-learning papers, with one backend, one evidence packet, one reviewer cohort, and bounded execution coverage.
Abstract
from arXiv · showhide
Large language model (LLM)-based reviewing systems typically assess manuscripts in isolation, leaving literature- and code-dependent claims difficult to verify. We present FactReview, an audit pipeline that extracts review-relevant claims, grounds them in related work and reference checks, and, when code is available, executes released artifacts under a fixed repair budget. On 26 paper-disjoint test papers with 354 human-verified claims, FactReview achieves 84.3\% F1 for claim recovery. In a same-backend, evidence-matched comparison, FactReview scores 4.72/5 overall, outperforming a direct LLM reviewer by 0.74 points. Removing execution evidence changes 17.0\% of claim statuses, more than removing any other single evidence source. In a reviewer-assistance study, FactReview reduces mean review time by 58\% while increasing benchmark-claim coverage from 87\% to 99\%. FactReview supports evidence-based claim auditing, with acceptance decisions reserved for human reviewers. The code is public at https://github.com/DEFENSE-SEU/FactReview.
1 Introduction
FactReview addresses the limits of manuscript-only review by auditing literature- and code-dependent claims against linked external evidence. Its claim-level records preserve evidence scope and support human inspection of conservative outcomes.
- Literature search, citation reading, and artifact inspection become difficult to perform consistently as submission volume grows and qualified reviewers remain under pressure.
- Manuscript-only review can leave literature- and code-dependent claims unchecked.
- Execution evidence tests empirical results only when the task, dataset, metric, baseline, and evaluation setting match the paper’s claim.
- FactReview extracts review-relevant claims, gathers manuscript, literature, reference-integrity, and optional execution evidence, and links findings to their sources.
- FactReview assigns four conservative outcomes—Supported, Partially supported, In conflict, or Inconclusive—and leaves acceptance decisions to human reviewers.
2 Related Work
Related work improves review generation, literature grounding, or artifact execution, but these capabilities generally remain separated. FactReview combines external evidence construction with claim-level review synthesis.
- LLM reviewing systems range from direct prompting to retrieval-augmented, multi-agent, and fine-tuned approaches.
- Literature-grounded systems emphasize answers and provenance, while reproducibility studies show that released artifacts contain evidence absent from papers but are difficult to execute reliably.
- FactReview acts earlier than critique-grounding systems by constructing external evidence before synthesizing the review.
3 Method
FactReview builds auditable claim records by extracting atomic targets, routing them to manuscript, literature, reference, and execution checks, and preserving provenance and scope. Execution distinguishes runnable code from claim-aligned result verification.
- FactReview maps manuscripts and repositories to claim-level evidence records spanning extraction, literature positioning, execution-based verification, assessment, and linked outputs.
- 3.1 Document Parsing and Claim-Centered Extraction: The extractor preserves manuscript locations and splits broad claims into atomic targets so local evidence cannot automatically validate broader statements.
- 3.2 Literature and Reference Verification: Novelty and overlap claims are checked against cited methods, baselines, metadata, abstracts, and semantically related work.
- 3.3 Execution-Based Claim Verification: Repository execution enters the evidence record only when observed outputs align with a specific claim’s dataset, metric, and scope.
- 3.3 Execution-Based Claim Verification: Run–Review–Fix permits at most K = 3 repair rounds and restricts repairs to dependencies, paths, launch arguments, and wrappers.
- 3.3 Execution-Based Claim Verification: A repository-level smoke test without claim-linked metrics remains Inconclusive, and fixed-harness comparisons measure backend sensitivity rather than cross-harness robustness.
- 3.4 Claim Assessment and Review Synthesis: Outcomes distinguish Supported, Partially supported, In conflict, and Inconclusive evidence states, while provenance records the source and scope of each judgment.
4 Experiments
FactReview is evaluated on evidence-linked review quality, claim recovery and status assignment, execution verification, reviewer assistance, ablations, case studies, and cost. Across these evaluations, it improves controlled review ratings and claim recovery, while execution evidence, bounded repair, and evidence alignment materially affect outcomes.
- Evaluation scope: The evaluation covers evidence-linked review quality, reference-claim recovery and outcome agreement, claim-aligned code evidence, and reviewer assistance.The benchmark then reports execution diagnostics, reviewer assistance, ablations, harder cases, and pipeline cost.
- Controlled review quality: 4.72 Overall versus 3.98 for the evidence-matched direct reviewer, a 0.74-point difference under matched evidence, backend, token allowance, and output template.FactReview scores 4.79 in Groundedness, 4.74 in Specificity, and 4.62 in Coverage; all confidence intervals exclude zero and Holm-adjusted p-values are below .01.
- Claim-level results: 84.3% F1 measures claim recovery, while 94.7% conditional status agreement measures status assignment among 282 matched claims.FactReview outputs 315 claims, matches 282 of 354 reference claims one to one, and assigns the reference status to 267 matched claims.
- Status assignment: 90.3% accuracy is achieved for reference-conditioned status assignment over 495 claims from the complete 35-paper collection.Class-wise recall is 91.0% for Supported, 92.0% for Partially supported, 82.9% for In conflict, and 76.2% for Inconclusive.
- Execution verification: Paper success rises from 11/20 (55.0%) without repair to 13/20 (65.0%) at K ≤2, while execution-claim pass rate rises from 65/96 (67.7%) to 79/96 (82.3%).The ledger records 20 initial attempts and 26 repair attempts; runtime and token use increase by factors of 1.45 and 1.58 relative to execution.
- Ablations and failure provenance: Paper success falls from 13/20 (65.0%) to 10/20 (50.0%) without planning and to 11/20 (55.0%) without the fix loop; removing alignment or baseline checks yields 12/20 (60.0%) each.Across repair rounds, environment blockers fall from four to three and runtime blockers from two to one, while metric and alignment blockers remain unchanged.
- Evidence sensitivity: Removing execution changes 17.0% of claim statuses, compared with 5.7% for literature search and reading and 8.4% for Semantic Scholar retrieval.Removing all external evidence changes 26.1% of statuses.
5 Conclusion
FactReview organizes automated peer review around explicit claims linked to manuscript, literature, reference, and execution evidence. It makes assessments inspectable while reserving final decisions for human reviewers.
- FactReview links each claim-level judgment to evidence from the manuscript, related literature, reference checks, and code execution.
- 84.3% F1 for claim recovery and 94.7% status agreement were achieved on 26 test papers.
- FactReview obtained the highest mean score in the native-configuration comparison.
- Execution counts as support only when reproduced output matches the claimed dataset, metric, and scope.
- Human reviewers retain responsibility for final acceptance decisions, while the evidence record supports revisiting disputed judgments.
6 Limitations
The evaluation is bounded by a purposively selected machine-learning test set, author-constructed reference inventory, single-backend comparison, one reviewer cohort, and one bounded execution harness.
- The 26-paper test set is purposively selected and limited to machine-learning papers.
- Claim-recovery estimates depend on an author-constructed reference inventory.
- The controlled comparison uses one backend and one evidence packet per paper, while the reviewer study covers one cohort with aggregate outcomes.
- Execution covers 20 artifact-eligible papers with one bounded harness, leaving repeated runs under alternative harnesses outside the evaluation.
- Broader paper and reviewer populations remain part of ongoing evaluation across additional domains and review settings.
A.1 Benchmark Construction
FactReview’s benchmark uses a frozen, paper-disjoint design with human-verified claims and explicit four-way evidence statuses, while its audit and evaluation configurations define the comparison scope.
- Benchmark Construction: The benchmark contains 35 machine-learning papers and 495 human-verified reference claims.
- Benchmark Construction: The remaining 26 papers form a paper-disjoint test set, with related versions, shared repositories, and forks grouped before splitting.
- Benchmark Construction: Candidate inclusion was author-determined, making the benchmark a purposively selected diagnostic collection rather than a random sample.
- Claim inclusion and granularity: A retained claim must be atomic, independently checkable, localizable, assessable against available evidence, and consequential for review assessment.
- Claim inclusion and granularity: The complete inventory contains 178 Supported, 261 Partially supported, 35 In conflict, and 21 Inconclusive claims.
- Annotation and Agreement: Three annotators verified claim selection, wording, and four-way evidence-status labels using a retained pairwise audit design.
- Evaluation Rubrics and Configuration: Groundedness, Specificity, and Coverage use 1-to-5 scales, whose arithmetic mean yields the Overall score.
B.1 Complete Claim-Level Results
On the complete 495-claim diagnostic collection, FactReview assigns statuses with 90.3% accuracy, while most off-diagonal errors involve neighboring evidence-status categories.
- 90.3% overall accuracy was achieved across all 495 reference claims in the complete diagnostic collection.
- 447 of 495 reference claims received the correct predicted status.
- The 48 off-diagonal outcomes comprise the errors in the four-class confusion matrix.
- 19 confusions occurred between Supported and Partially supported, the most frequent label-pair error.
- The native comparison evaluated five AI systems on all 35 papers using 480 source-blinded evaluations from 20 evaluators.
B.3 Reviewer Assistance Results
In the reviewer-assistance study, FactReview reduced review time while increasing coverage of the retained claim set.
- 50.6, 31.6, and 21.3 minutes were the mean review times across the three reported conditions.Time excludes instruction and setup in the exact reviewer-assistance results.
- 86.9%, 97.8%, and 98.9% were the corresponding coverage rates for the retained five-paper claim set.Coverage measures the share of claims addressed in submitted reviews.
B.4 Ablations and Backend Sensitivity
Ablations show that execution evidence substantially affects claim statuses, while workflow components and backend choices influence execution verification under a fixed harness.
- Evidence-source ablation: 17.0%, 5.7%, 8.4%, and 26.1% of claim statuses change when execution, literature search and reading, Semantic Scholar retrieval, or all external evidence is removed.These percentages denote changed claim statuses, not generic accuracy losses.
- Workflow ablation: 82.3% of execution-relevant claims pass in the complete workflow, compared with 67.7% without the fix loop.All variants use the same harness and the same 96 execution-relevant claims.
- Backend sensitivity: The backend comparison measures success rate, runtime, and cost under fixed FactReview orchestration and execution/review harness conditions.It does not test robustness to a different coding agent or repository-execution harness.
- Cost: Table 12 reports average cost for the non-execution stages over the complete 35-paper collection.Runtime is wall-clock time, and the displayed total is computed before row-level rounding.
C.1 Independent Execution Audit
An independent blinded audit compared archived execution evidence with pipeline verdicts and found high agreement, while identifying a small set of false passes and failures.
- Audit procedure: Two independent auditors inspected all 96 execution claims while blinded to pipeline verdicts and repair rounds.They reviewed commands, environments, logs, metric artifacts, claim-alignment records, and repair diffs; a third auditor adjudicated disagreements.
- Agreement: 92/96 claims received auditor agreement before adjudication, with nominal Krippendorff’s α = 0.86.Adjudication marked 76/96 claims and 12/20 papers as passing.
- Validity: Pipeline decisions agreed with adjudication on 91/96 claim verdicts and 19/20 paper outcomes.Four pipeline false passes involved aggregation mismatches, a checkpoint mismatch, and a repair-policy violation; one metric alias caused a false failure.
- Repair compliance: Auditors judged 25/26 repair diffs compliant with the allowlist.The audit separately recorded execution success, metric recovery and comparability, claim alignment, and repair-policy compliance.
C.2 Repair Trajectory and Failure Analysis
FactReview uses bounded repairs to recover executable evidence, analyzes distinct execution failure views, and reports limits on attributing errors to pipeline stages.
- Repair policy: RRF permits dependency, path, launch-argument, and wrapper fixes while excluding changes to architectures, losses, datasets, evaluation logic, and baselines.The repair boundary is intended to preserve fidelity to the released artifact.
- Repair trajectory: The first two repair rounds each recover one paper and seven execution-relevant claims, while the third produces no further gain.After repair, seven earliest blockers remain: three environment, one runtime, two metric, and one alignment failure.
- Failure analysis: Execution failures include 10 environment, 8 runtime, 6 unavailable-metric, 5 missing-baseline, 4 import, and 3 alignment provenance assignments.The provenance view is multi-label, whereas the pre-repair funnel counts each code-available paper once at its earliest blocking stage; the denominators must not be combined.
- Failure boundary: RRF cannot repair missing scientific evidence, even though it can address dependency, environment, path, launch-argument, and wrapper problems.An accessible repository may still fail to yield claim-aligned evidence because metric availability, baseline recovery, imports, or alignment intervene.
- Limitations: The retained aggregate records do not provide an auditable instance-level causal allocation of claim-recovery or status errors across pipeline stages.The paper therefore reports claim recovery, status classification, and execution provenance separately without inferring stage-of-origin breakdowns.
- Illustrative case: The CompGCN example shows that a broad contribution claim can remain partially supported even when a narrower empirical result is reproducible.The example is retained as a favorable qualitative case and does not represent the full range of difficult repository execution.