Source-linked AI summary
PaperDoctor: Evidence-Grounded and Actionable Feedback for Scientific Papers in Progress
Kevin Qinghong Lin, Siyuan Hu, Pan Lu, Yu Chen, Yanzhe Chen, Owen Queen, Yupeng Chen, Jialin Yu, Junchi Yu, Zifeng Ding, Yuanfeng Ji, Sheng Liu, Jindong Gu, Linjie Li, Mike Zheng Shou, Philip Torr, James Zou
TL;DR
AI-assisted and autoresearch writing increases the need for scalable, traceable diagnosis beyond accept-or-reject review. PaperDoctor uses hierarchical screening, typed verification, and prioritized reproduction to produce actionable findings; it reached 70.6% agreement in a human study and yielded more auditable feedback across diverse manuscripts. Its strongest evidence depends on runnable code, while incomplete or uneven pre-submission code limits evaluation of code verification.
Problem
Existing automated reviewers mainly return accept-or-reject decisions, while advisor-style feedback is detailed and traceable but does not scale to growing AI-assisted paper volume.
Method
PaperDoctor combines hierarchical screening, typed claim verifiers, evidence-grounded findings, and prioritized experiment reproduction for pre-submission feedback.
Results
70.6% agreement was achieved in a human study of 30 in-progress papers, while evaluation on 40 diverse manuscripts produced more auditable feedback than human and other agentic reviewers.
Takeaways & Limitations
PaperDoctor shifts automated paper assessment from verdicts toward diagnosis by pairing critiques with evidence and concrete revision suggestions and exposing reproduction gaps.
Takeaways & Limitations
PaperDoctor’s strongest evidence depends on runnable code, and pre-submission code is incomplete and uneven across papers.
Abstract
from arXiv · showhide
Autoresearch agents are reshaping the research ecosystem, but they can also let flawed claims enter the literature at scale. Human advisors catch such issues in drafts through careful, traceable feedback, yet advisor-style assessment requires extensive manual effort and does not scale. To shift automated paper assessment from a judge to a diagnostician, we introduce PaperDoctor, an agent framework for pre-submission feedback with three key innovations. First, a holistic hierarchical framework evaluates writing, layout, references, code, theory, prior work, and experiments through three layers: L1 surface screening, L2 typed verifiers that route each claim to the appropriate evidence, and L3 reproducers that rerun experiments by priority. Second, each finding contains an observation, a pointer to specific evidence such as a sentence, equation, or code line, and a revision suggestion, making critiques auditable and actionable. Third, PaperDoctor selectively rebuilds and reruns experiments based on claim importance and compute budget, surfacing reproducibility gaps and quantitative limitations that are invisible from the manuscript alone. We evaluate PaperDoctor on 30 in-progress papers, yielding 70.6% agreement and all positive holistic scores, and on 40 manuscripts across machine learning, natural science, and social science, covering human- and AI-authored papers with code. Overall, PaperDoctor produces more auditable feedback than human and other agentic reviewers, pairs critiques with concrete suggestions by design, and complements dimensions often overlooked by human reviewers. We also develop an interactive interface that lets authors browse findings grounded in their paper. PaperDoctor reframes automated paper assessment as diagnosis rather than verdict, taking a concrete step toward AI advisors for more rigorous AI-assisted scientific discovery.
1 Introduction
PaperDoctor reframes automated paper assessment as diagnosis: it produces traceable, actionable feedback rather than only accept-or-reject judgments. Its hierarchical pipeline combines evidence-grounded critiques with selective reproduction, achieving 70.6% agreement in a human study and broader coverage across 40 manuscripts.
- Motivation: PaperDoctor targets the gap between scalable automated review and advisor-style diagnosis by pointing to specific problems and prescribing concrete revisions.Its findings are intended to help authors audit and learn from critiques rather than merely receive a submission decision.
- Approach: PaperDoctor combines paper-only screening, typed claim verification, and prioritized experiment reproduction in a holistic hierarchical pipeline.The stages increase verification cost progressively while covering dimensions from writing and layout to code, theory, prior work, and experiments.
- Evaluation: 70.6% agreement was obtained in a human study of feedback on 30 in-progress papers, with all holistic scores positive.The study used junior researcher participants who rated PaperDoctor’s feedback.
- Evaluation: Across 40 manuscripts spanning machine learning, natural science, and social science, PaperDoctor produced more auditable feedback than human and other agentic reviewers.The benchmark included human- and AI-authored papers with accompanying code and data, covering varied writing styles and methodological rigor.
- Impact: Selective code execution exposes published numbers that cannot be re-obtained under the evaluation protocol, revealing gaps invisible to reading-only review.The interactive interface lets authors browse findings anchored directly to locations in their papers.
2 PaperDoctor
PaperDoctor represents findings as evidence-grounded revision units and routes paper claims through specialized checks. Its L1–L3 design separates low-cost manuscript screening, typed verification, experiment-design assessment, and selective reproduction.
- 2.1 Overview: Each finding combines an issue description, evidence tied to a paper location, and a concrete revision suggestion.Evidence may be a sentence, equation, code line, or external URL; findings are classified as errors or warnings according to confidence.
- 2.1 Overview: PaperDoctor parses the paper, rendered pages, code, and bibliography into reusable representations so downstream skills request only the slices they need.The shared representation supports text, visual, code, and reference-specific verification without feeding the entire artifact to one model.
- 2.2 L1 – Paper Screening: The hierarchical L1–L3 pipeline allocates effort according to verification cost, moving from surface checks to claim verification and experiment reproduction.L1 addresses concrete presentation issues, while later levels handle evidence-specific checks and costly execution.
- 2.2 L1 – Paper Screening: L1 checks writing, figures, citations, and extracted claims using targeted evidence such as quoted sentences, rendered pages, resolver URLs, and claim locations.Examples include correcting a typo, rescaling an overflowing figure, verifying a reference, and extracting an experimental assertion.
- 2.3 L2 – Claim Verification: L2 routes claims to specialized verifiers for code, theory, literature, and experiment design rather than treating all claims uniformly.These checks identify paper-code mismatches, unjustified derivation steps, overstated novelty, and unsupported or poorly designed experiments.
- 2.3 L2 – Claim Verification: Experiment-design review creates a declarative reproduction plan covering commands, priorities, feasibility, run modes, and numeric targets before execution.It separately assesses missing experiments, ablations, statistical rigor, baseline recency, fairness, and cherry-picking risk.
- 2.4 L3 – Experiment Reproduction: A reproduced MMLU run yielded 71.2% versus the claimed 78.4%, exposing a 7.2-point discrepancy flagged for protocol or reporting verification.The example illustrates how execution-time checking can identify quantitative gaps that manuscript inspection cannot establish.
3 Results
PaperDoctor received positive author evaluations on pre-submission papers and produced more evidence-grounded, broad, and reproducibility-focused feedback than comparison reviewers. Its reproduction stage also exposed substantial execution barriers and mismatches between reported and reproduced results.
- Author evaluation: 70.6% of 1,299 item-level verdicts accepted PaperDoctor’s Evidence, while all 30 holistic review scores were positive.The mean holistic score was +1.30, with 70% of papers rated somewhat helpful and 30% very helpful.
- Author evaluation: Authors accepted L2 verifiable claims 89% of the time versus 73% for L1 surface checks, while Figure Review had the lowest acceptance at 42%.Experiment-design findings were especially numerous and had an 81% acceptance rate; authors also accepted 71% of Warnings and 66–67% of Errors depending on axis.
- Reviewer comparison: PaperDoctor’s findings always pair Evidence with a Suggestion, unlike referees and the agentic reviewer, but average 50.0 words per finding.Referee findings contained both Evidence and Suggestion 35.9% of the time, compared with 1.5% for the agentic reviewer; PaperDoctor’s design requires both.
- Reviewer comparison: PaperDoctor distributes feedback more evenly across dimensions, allocating 28.4% to experiments, 12.5% to literature, and 11.5% to code.Referees and the agentic reviewer concentrated 90.1% and 89.5% of findings on the main body, respectively.
- Reproduction: Only 53.0% of reproduction plans reached execution; among those, 27.0% produced warnings and 34.5% errors, with incomplete runnable environments the largest blocker at 33.1%.Training reruns were especially difficult, passing 11.3% and erroring 58.5% of the time; high-priority plans passed more often than low-priority plans.
- Reproduction: Reproduction failure patterns differed by paper group: 14.4% of executed ICML plans matched reported numbers, compared with 72.2% for executed Agent4Science plans.ICML plans mainly failed after execution, whereas Agent4Science plans mainly failed before a command ran.
4 Related Work
PaperDoctor differs from judge-oriented peer-review systems by assisting authors with evidence-grounded, actionable checks across papers, code, theory, references, and experiments.
- AI for Autoresearch: PaperDoctor complements autoresearch and peer-review agents by consuming drafts with code and data to locate parts needing revision before submission.It addresses failure modes including hallucinated citations, inflated novelty, paper-code mismatches, and unverified empirical statements.
- Paper Verification: PaperDoctor treats papers as verifiable systems, routing informal arguments and claims toward formal, executable, citation, or reproduction checks.Its verification perspective spans citations, theoretical claims, and experiment reproduction.
- Comparison: Table 1 compares research agents across reviewer reports, grounded evidence, revision suggestions, multimodal inputs, code audits, reproduction, and support for in-progress papers.The comparison distinguishes full, partial, and absent capabilities.
- Paper Peer Review: Unlike judge-oriented reviewers optimized for verdict agreement, PaperDoctor grounds each finding in a concrete artefact and pairs it with an actionable revision.Its success is framed around author acceptance and action on findings rather than score agreement.
5 Discussion
PaperDoctor reframes automated assessment as diagnosis for authors, combining traceable findings with selective verification while preserving human judgment over research value and direction.
- Core contribution: PaperDoctor shifts assessment from a judge to a diagnostician by pairing each finding with evidence, a location, and a revision suggestion through cost-scaled verification.The pipeline combines manuscript checks, typed claim verifiers, and selective experiment reruns under a priority budget.
- Human judgment: PaperDoctor automates checkable feedback such as cross-checking numbers, aligning prose with code, and rerunning experiments, while leaving questions of value and direction to humans.The intended benefit is to redirect human time toward taste and research direction.
- Verifiability: Every feedback item can be audited through an anchor to a sentence, equation, code line, or reference, making claims with runnable code and traceable numbers cheaper to check.Claims lacking such support remain expensive to trust.
- Future work: Future work targets paper-to-code reproduction without released code and larger advisor-level studies grounded against reviewer and AI feedback.The paper notes that synthesized implementations would yield lower reproduction rates than the current system.
A PaperDoctor Interface
The PaperDoctor interface links each finding to the exact paper span and corresponding code, then displays the observation and revision suggestion for auditing and action.
- Interface: The interface overlays numbered, color-coded findings on the paper and shows corresponding implementation evidence in a code viewer.A finding card presents the observation and concrete revision suggestion for the selected highlight.
B Case Studies
The case studies show PaperDoctor diagnosing citation, code, theory, literature, and reproduction problems through reason, evidence, and suggested action. These examples expose errors that reading-only or standard checking can miss.
- Case-study format: Each representative case is organized around why a finding matters, where its quoted evidence appears, and how the paper could be revised.The case-study format explicitly follows reason, quote evidence, and suggestion.
- L1 Citation Check: A citation verifier flags GPT-4.5’s cited 2024 release date as chronologically impossible because it was announced in 2025.The verifier uses a web search, unlike standard citation checkers that only accept correctly formatted entries.
- L2 Code Verification: A code verifier finds that the implementation uses all-MiniLM-L6-v2 with 384-dimensional vectors instead of the paper’s SciBERT and 768-dimensional specification.The mismatch is documented across three repository files and concerns both model identity and dimensionality.
- L2 Theory Verification: Theory verification identifies an approximation bound whose symbols, constants, exponents, and proof are not introduced or established in the paper.The checked expression includes W, L, α, β, C1, C2, and C3, alongside the approximation term.
- L2 Literature Check: Literature verification challenges a novelty claim about DeepONet and FNO by consulting their original method papers.The cited FNO work demonstrates applications involving parametric PDEs with varying initial conditions and forcing terms.
- L3 Reproduction: PaperDoctor’s L3 reproduction finds that a claimed “factor of 2” recall increase is actually 1.14×.The analysis rebuilds results from shipped source data and reads the recall values directly from Figure 2d.