DeepReviewer 2.0: A Traceable Agentic System for Auditable Scientific Peer Review

1 Introduction

• Fluent critiques can remain hard to verify and act on when they do not identify manuscript locations, supporting evidence, or concrete fixes.The introduction frames usable reviews as both auditable and actionable.Source: §1 · p. 2
• The system builds a manuscript-only claim–evidence–risk ledger, then uses it to create a verification agenda and write anchored critiques with concrete repair actions.Novelty checks compare closely related prior work under a matched-setting gate, and final synthesis is gated on minimum retrieval, structure, and anchored-feedback requirements.Source: §1 · p. 2
• Its review package combines a structured report, page- and paragraph-anchored annotations, a prioritized repair plan, and a novelty/value assessment.Traceability is a required output rather than a best-effort behavior.Source: §1 · p. 2
• DeepReviewer 2.0 enforces process constraints rather than semantic truth, so it is positioned as an assistive tool rather than a replacement for peer review.Its evidence links and export requirements aim to improve auditability and execution even when judgments remain fallible.Source: §1 · pp. 2–3
• DeepReviewer 2.0 ranks first among automatic systems in its pool, improves strict major-issue coverage over Gemini-3.1-Pro, and is preferred against a human review committee.The evaluation uses strict issue coverage, anonymous system ranking, and blind comparison under a shared rubric.Source: §1 · p. 2

2 Related work

• Earlier automated-review work ranges from semi-automated infrastructure and neural review generation to LLM systems adding fluency, rubrics, role decomposition, or stronger evaluation resources.Other adjacent work studies review aggregation and audits failure modes under realistic conditions.Source: §2 · p. 3
• Process-oriented systems study reviewer interactions, staged retrieval and verification, or multidimensional review quality; DeepReviewer 2.0 makes anchored annotations and export-time traceability mandatory.The cited examples include AgentReview, DeepReview, and ReviewEval.Source: §2 · p. 3
• Evidence-grounded assistance, contribution-level novelty comparison, and document localization inform DeepReviewer 2.0’s goal of combining diagnosis, novelty checking, and actionable revision guidance.The paper frames its target as an auditable peer-review artifact rather than fluent review generation alone.Source: §2 · p. 3

3 DeepReviewer 2.0

• DeepReviewer 2.0 uses a staged cognitive chain to produce an auditable review package rather than a single free-form review.The workflow parses the paper, builds a claim–evidence–risk ledger and investigation agenda, performs agenda-driven verification, and gates final synthesis before export.Source: §3 · p. 3, Figure 2 · p. 4

3.1 Task formulation and notation

• The package separates a structured final report, paper-location-anchored annotations, a prioritized repair plan, and a novelty/value assessment.The assessment makes comparisons explicit and records uncertainty when evidence is thin.Source: §3.1 · p. 5

3.2 Stage I: Global diagnostic review

• Stage I builds a manuscript-only claim–evidence–risk ledger and investigation agenda before external verification.Issues are marked confirmed when supported by a direct internal contradiction or a core claim lacks supporting anchors after a second pass; otherwise they remain suspected and enter the agenda.Source: §3.2 · p. 5

3.3 Stage II: Verification-oriented annotation and synthesis

• Stage II retrieves candidate papers for agenda questions and checks novelty/value claims and methodological choices.Prior work counts as direct overlap evidence only when task, dataset or benchmark, and primary metric match; novelty tags default to UNCLEAR when evidence is insufficient or contradictory.Source: §3.3 · pp. 5–6
• Stage II annotations pair a paper-location anchor and issue category with severity, scientific risk, and an actionable repair suggestion.Page-and-span anchors are the stable reference; highlight rectangles are derived afterward and may be best-effort when layout data is unavailable.Source: §3.3 · p. 6

3.4 Process constraints and auditability guarantees

• Export requires a structured report, minimum literature-search and distinct-question budgets, and a minimum number of anchored annotations.Reported minimums are α = 3, β = 3, and γ = 10; readiness checks also validate anchor integrity and annotation completeness.Source: §3.4 · pp. 6–7
• The export gate enforces process properties, not semantic truth or perfect novelty judgments, so the system is intended as assistance rather than an oracle.The paper also notes risks from hidden-content prompt injection and from treating pre-review screening as a substitute for peer review.Source: §3.4 · p. 7

4.1 Data, protocol, and metrics

• On 134 ICLR 2025 submissions, strict coverage used a human-derived issue list, while ranking used valid judge chains after fixed export and parsing checks.Ranking metrics used a subset with valid comparison chains; validity exclusions can understate end-to-end deployment brittleness.Source: §4.1 · p. 7

4.2 Results

• 37.26% major-issue coverage, versus Gemini-3.1-Pro’s 23.57%, with DeepReviewer 2.0 also highest in strict overall coverage (35.74%).The strict benchmark counts missing outputs as uncovered and uses binary issue matching.Source: §4.2 · p. 7, Table 2 · p. 8
• 66.86% of non-tied comparisons favored DeepReviewer 2.0 over its hardest automatic-system opponent, Gemini-3.1-Pro.The result indicates wins were not limited to weaker baselines.Source: Figure 3 · p. 8, §4.2 · p. 9
• 2057.17 Elo (95% CI: [1990.55, 2140.81]) exceeded Gemini-3.1-Pro’s 1915.11; uniform average non-tie win rate was 92.34% and tie-aware average rank was 1.889.The multiple aggregates support first place under the fixed judge and tie rule, but are not a direct measure of factual correctness.Source: Figure 4 · p. 9, Figure 5 · p. 9, §4.2 · pp. 9–10
• 71.63% of micro-averaged blind comparisons preferred DeepReviewer 2.0 over the human review committee, with 16.28% ties and 12.09% losses.Wins were strongest in Constructive Value (84.50%) and Communication Clarity (86.05%); Technical Accuracy and Analytical Depth had smaller margins.Source: Table 3 · p. 10, §4.2 · p. 10
• Ethics strict coverage was 0.00% for DeepReviewer 2.0, while another system reached 50.00% under this protocol.The category had few issues in this benchmark, but the passage identifies ethics-sensitive checks as important for deployment risk.Source: §4.2 · p. 10
• DeepReviewer 2.0 led in 4/8 issue categories—Novelty, Clarity, Presentation, and Other—but was not best in every category.Other systems remained stronger in Soundness/Experiments and ethics-sensitive checks; the issue list was imbalanced, including 225 Experiments issues and 138 Soundness issues.Source: §4.2 · p. 10, Figure 6 · p. 11

• DeepReviewer 2.0 makes review judgments checkable through a review-package interface and staged cognitive chain with traceability budgets and gated synthesis.Source: §5 · p. 11
• Across strict issue coverage, anonymous system ranking, and blind comparison with a human committee, the resulting packages are preferred under the paper's protocol.Source: §5 · p. 11
• Strict coverage remains incomplete and ethics coverage can fail sharply, so the system is positioned as an assistive reviewer under human oversight and venue-specific checklists.Source: §5 · p. 11

6 Limitations and Ethical Risk Considerations

• Process guarantees do not ensure semantic truth, and incomplete issue coverage, judge-dependent preferences, and a filtered evaluation pool constrain what the results establish.Ethics coverage can fail sharply; the authors recommend treating high-stakes or normatively loaded domains as out of scope unless dedicated safeguards are added.Source: §6 · pp. 11–12

A Evaluation instruction templates

• The appendix presents judge instructions as evaluation contracts, specifying inputs, decision rules, and outputs to make the protocols reproducible.Source: A Evaluation instruction templates · p. 14

A.1 Strict issue-coverage evaluator (Step-2)

• Strict issue coverage uses a canonical human-derived issue list and binary labels, counting an issue only when the review clearly covers it with concrete semantic overlap.The evaluator returns one structured entry per issue; missing, vague, or weakly related coverage receives label 0.Source: §A.1 · p. 14

A.2 Anonymous ranking judge (Step-3)

• The anonymous ranking judge compares anonymized reviews through weakness-by-weakness analysis and ranks five aspects, converting score gaps of 1 or less to ties.Its assessments consider correctness, materiality, evidence quality, repairability, and actionability.Source: §A.2 · p. 14

• The appendix details metric definitions and computation rules for strict coverage (Step-2) and anonymous ranking (Step-3).Source: B Evaluation metrics and computation… · p. 14

B.1 Notation and dataset filtering

• 134 ICLR 2025 submissions form the evaluated paper set, while both ranking protocols use the same subset with parseable Step-3 judge outputs.Source: §B.1 · p. 14
• For each evaluated paper, the canonical issue list assigns every issue a major-or-minor severity tag and one of eight category labels.Source: §B.1 · p. 14

B.2 Step-2 coverage metrics (strict policy)

• Missing or unparseable output is counted as uncovered under the strict policy.Source: §B.2 · p. 15
• Severity-split coverage separates each paper’s major and minor issue subsets.The supplied passages identify those subsets but do not provide a complete formula.Source: §B.2 · p. 15
• Category coverage is computed over the issues in each paper that fall into a given category.The supplied passage identifies the category-specific issue set but does not provide a complete formula.Source: §B.2 · p. 15

B.3 Step-3 ranking metrics

• Average rank, strict pairwise win fraction, and uniform average win rate summarize system rankings, with ties excluded from strict wins.Ties correspond to score gaps within the judge’s 1–10 resolution; the authors exclude them rather than count them as half-wins.Source: §B.3 · p. 15

B.4 Bradley–Terry Elo estimation and bootstrap confidence intervals

• Non-tie comparisons are fitted with a Bradley–Terry model by maximum likelihood and a minorization–maximization iteration, then mapped to Elo.The ability scale is normalized so its geometric mean is 1; 95% confidence intervals use 1,000 paper-level bootstrap resamples with the model refit for each resample.Source: §B.4 · p. 16

B.5 DeepReviewer 2.0 vs human committee percentages

• DeepReviewer 2.0’s comparison with the human committee is reported as per-aspect rank win, tie, and loss percentages, with a micro-average across aspects.The percentages are normalized per row, and the micro-average aggregates counts across aspects.Source: §B.5 · p. 16

C Document parsing and anchored annotation

• The parsing and annotation representation is designed to let critiques be traced to concrete page regions, including when PDFs are noisy.Source: C Document parsing and anchored… · p. 16

C.1 From PDF to page-indexed semantic text

• Each PDF is normalized into a page-indexed semantic text stream and a layout trace; stable page-and-line anchors preserve traceability even when bounding boxes are missing or noisy.Available boxes and per-page reference scales support consistent highlight placement; text supports reasoning and comparison.Source: §C.1 · pp. 16–17
• Bounding boxes use page-local coordinates normalized by each page’s observed extents to map highlights back onto the source PDF.Figure 7 shows MinerU-returned block boxes on the first four pages of an example paper.Source: §C.1 · p. 17

C.2 Implementation pseudocode (MCP tool loop)

• The backend pipeline makes agent interactions explicit: the agent reads pages and writes annotations through MCP tools, while export checks explicit counters and artifacts.Source: §C.2 · p. 17

C.3 Annotation objects and anchor resolution

• Each annotation couples a page-and-line location with a diagnosis and repair action, alongside a short summary and longer comment.The described example marks an issue major, explains its scientific risk, and specifies a concrete revision requirement.Source: §C.3 · p. 17

C.4 Overlay rendering and continuation pages

• Annotations render as translucent highlights linked to right-margin callouts; comments that do not fit without overlap move to a continuation sheet linked back to the source highlight.The source page retains a compact index card so it remains readable while the critique keeps its detail.Source: §C.4 · p. 17

D Case study: an anchored review package

• The global report separates summary, strengths, weaknesses, and recommendation, including distinct novelty, methodological-transparency, and statistical-reporting concerns.Its novelty judgment turns on whether “first” claims survive direct comparison with related autonomous-scientist systems under matched settings.Source: D Case study: an anchored review package · pp. 19–20
• Each anchored critique diagnoses a passage’s role, states a defect and its scientific risk, then specifies an executable revision, sometimes with a proposed rewrite.Continuation sheets preserve detail while keeping source pages readable, leaving a trail humans can check, dispute, and use to revise the paper.Source: Figure 9 · p. 18, Figure 10 · p. 19, D Case study: an anchored review package · p. 20
• The report says DeepScientist surpasses human SOTA methods by 183.7%, 1.9%, and 7.9% across Agent Failure Attribution, LLM Inference Acceleration, and AI Text Detection, respectively.It flags that 183.7% is a relative gain from 12.07% to 29.31% (17.24 percentage points absolute), and that variance or confidence intervals are absent.Source: D Case study: an anchored review package · pp. 23, 25
• The critique says the “first” claims need precise dimension-bounding against related systems, while missing promotion criteria, valuation definitions, and failure-handling details undermine reproducibility.It also questions missing UCB hyperparameter selection and sensitivity analysis.Source: D Case study: an anchored review package · p. 23
• The report recommends major revisions, calling for bounded claims, completed methodological details, and appropriate controls for causal claims.It says the paper lacks ablations or matched-capacity controls to separate proposed mechanisms from computation, implementation quality, tuning, and selection effects.Source: D Case study: an anchored review package · pp. 23, 25

Overstated Efficiency Claims

• The review questions the efficiency comparison because it assumes full evaluation of all 5,000 ideas, although hierarchical filtering avoids expensive validation for most.The counterfactual number of tests random sampling would require is unestablished; the review also flags the 1.9% throughput improvement against computational investment.Source: D Case study: an anchored review package · pp. 25, 28

Novelty Claim Overreach

• The review finds that “first” claims lack defined dimensions, while retrieved papers demonstrate comparable end-to-end autonomous discovery capabilities.It recommends bounding novelty claims to specific dimensions where the work is genuinely novel.Source: D Case study: an anchored review package · p. 26

Missing Reproducibility Details

• The review notes that the method omits promotion thresholds, valuation-vector definitions, and failure handling, leaving key reproduction details unspecified.It calls for explicit criteria, definitions, and retry and error-logging policies.Source: D Case study: an anchored review package · p. 26

Misleading Improvement Metrics

• The review says the claimed 183.7% improvement uses a low baseline (12.07% to 29.31%), while the absolute gain is 17.24 percentage points and variance is unreported.It recommends reporting absolute and relative gains and adding variance or confidence intervals across multiple runs.Source: D Case study: an anchored review package · p. 26

Unsupported Causal Attribution

• The review says the attribution of improvements to A2P’s counterfactual reasoning lacks a controlled ablation, so confounders remain possible.It recommends matched-capacity ablations and cautious causal wording until controls are added.Source: D Case study: an anchored review package · p. 26

Human Evaluation Independence Unclear

• The review questions human-evaluation independence and depth because reviewer independence is unestablished, reviews lasted 55 minutes, and only five papers were evaluated.It recommends clarifying reviewer independence and instructions and discussing the small sample size.Source: D Case study: an anchored review package · pp. 26–27

Bounded novelty

• The proposed novelty claim is that DeepScientist is the first system to achieve SOTA-surpassing discoveries across three AI research domains within a month-long evaluation window.The claim is bounded to the number of domains and the evaluation timeline.Source: D Case study: an anchored review package · p. 29

Motivation

• The framing identifies a core limitation of prior AI Scientist systems: they can generate novel findings but often lack focus on human-defined challenges.The problem is presented as a motivation for a goal-oriented approach.Source: D Case study: an anchored review package · p. 29

Method

• DeepScientist’s proposed discovery process formalizes research as Bayesian Optimization with hierarchical hypothesize-verify-analyze evaluation and cumulative Findings Memory.The method outline describes Findings Memory as balancing exploration and exploitation.Source: D Case study: an anchored review package · p. 29

Results

• Across 20,000 GPU hours and 5,000 generated ideas, DeepScientist surpassed human SOTA by 17.24 percentage points (Agent Failure Attribution), 3.65 tokens/second (LLM Inference Acceleration), and 0.063 AUROC (AI Text Detection).The proposed results summary names three AI tasks and gives one improvement for each.Source: D Case study: an anchored review package · p. 29

Discovery process

• The discovery-process outline reports 5,000 ideas, 1,100 validated ideas, and 21 innovations, alongside near-linear scaling.The supplied passages state these as process insights and a scaling relationship.Source: D Case study: an anchored review package · p. 30

Case study results

• Reported results include gains in AI-text detection and throughput, plus stronger human-evaluation AUROC than the baseline.The MBPP passage reports 29.31% (handcraft) and 47.46% (algorithm-gen) vs 12.07%/16.67% baseline; ACRA reports 193.90 vs 190.25 baseline; human evaluation reports 0.863 AUROC and 60ms latency vs 0.800/117ms baseline.Source: D Case study: an anchored review package · p. 33

Evidence limitations

• Evidence is limited by unreported variance, missing mechanism ablations, small human-evaluation samples, and uncertain out-of-domain or cross-domain generalization.The human evaluation used five DeepScientist papers and three ICLR-experienced reviewers; its reviewer independence was not fully established.Source: D Case study: an anchored review package · p. 33

Automated evaluation against AI Scientist systems

• The reported average rating was 5.00 versus ICLR 2025’s 5.08 average, while DeepScientist’s acceptance rate was 60% versus 0% for others.The passages say the papers approach human quality and are superior to prior AI Scientists, while cautioning that the curated set may not represent typical output.Source: D Case study: an anchored review package · p. 33

Selection strategy ablation

• Success rates were ~0% with random sampling and 1–3% with UCB selection, supporting the claim that intelligent filtering is essential.The comparison uses 100 ideas per task; confidence intervals were not reported.Source: D Case study: an anchored review package · p. 33

Research-theme gap diagnosis

• The research-theme diagnosis questions whether gains arise from claimed mechanisms and whether single-run improvements are stable and reproducible.It highlights absent ablations that isolate counterfactual reasoning or long-term memory, and the lack of variance, confidence intervals, or significance tests.Source: D Case study: an anchored review package · p. 34

Proposed research experiments

• Planned evaluations compare A2P full, A2P without counterfactual, and All at Once; run five seeds per task with different initializations; test A2P, ACRA, and PA-Detect on benchmark OOD variants; and evaluate DeepScientist at 1K, 5K, 10K, and 20K GPU-hour budgets.Source: D Case study: an anchored review package · p. 35

Goal-Oriented Bayesian Optimization Formulation

• DeepScientist models scientific discovery as goal-driven Bayesian optimization, using iterative exploration and expanding research memory to manage uncertainty.The review ledger describes a Strategize–Implement–Analyze hierarchy, Findings Memory, and UCB-based selection for sample-efficient search over costly hypotheses.Source: D Case study: an anchored review package · p. 36

SOTA-Surpassing Discoveries Across Three Domains

• 183.7% (Accuracy), 1.9% (Tokens/second), and 7.9% (AUROC) are the reported margins by which DeepScientist exceeds respective human SOTA methods across three AI research tasks.The tasks are Agent Failure Attribution, LLM Inference Acceleration, and AI Text Detection.Source: D Case study: an anchored review package · p. 36

First Large-Scale Empirical Demonstration

• The claim of a first large-scale demonstration should be bounded to specific dimensions, since Kosmos [3] is also described as large-scale and spans four or more science domains.The passages identify DeepScientist’s scale as 20K GPU hours, 5K ideas, three domains, and a month-long evaluation, and recommend avoiding a categorical first claim.Source: D Case study: an anchored review package · pp. 36, 39, 42

SOTA-surpassing on three AI domains

• Its distinction from AI Scientist-v2, Robin, and CodeScientist [2] is the reported SOTA-surpassing work across three AI research domains within a month-long evaluation.The novelty assessment describes the prior systems’ reported accomplishments as narrower in task or domain scope.Source: D Case study: an anchored review package · p. 38

D Case study: an anchored review package

• The Bayesian optimization framing for scientific discovery is shared with HypoAgents and related to Kosmos’s structured world model.The passage characterizes this overlap as partial, distinguishing it from DeepScientist’s specific hierarchical approach.Source: D Case study: an anchored review package · p. 43
• DeepScientist’s claimed novelty is integrating Bayesian optimization with LLM hypothesis generation and hierarchical filtering.Its specific approach uses a three-stage hierarchy with UCB-based selection and LLM-driven hypothesis generation.Source: D Case study: an anchored review package · p. 43

• 20,000 GPU hours and 5,000 generated ideas underpin experimentation the reviewer calls a meaningful contribution.Source: Final Score: 5.5/10 · p. 44
• Reproducibility is compromised by missing operational definitions for promotion criteria, valuation components, and failure handling.Source: Final Score: 5.5/10 · p. 44
• The review flags relative-only improvement reporting without absolute gains or variance measures, and a lack of ablations isolating claimed mechanisms.Source: Final Score: 5.5/10 · p. 44
• The reviewer values autonomous SOTA-surpassing discovery across three AI domains using a BO-framed hierarchical exploration system.Source: Final Score: 5.5/10 · p. 45
• The reviewer says modest absolute gains, including 3.65 tokens/second for 20K GPU hours, and AI/ML-only scope constrain impact.The cited scope excludes physical sciences.Source: Final Score: 5.5/10 · p. 45
• The reviewer recommends major revision, judging the current paper unacceptable because of reproducibility gaps and overbounded novelty claims.Source: Final Score: 5.5/10 · pp. 44–45