DeepReview: Improving LLM-based Paper Review with Human-like Deep Thinking Process

• The released models are DeepReviewer-7B and DeepReviewer-14B.Source: Resources · p. 1
• The code repository is zhu-minjun/Researcher.Source: Resources · p. 1
• The project homepage is ai-researcher.net.Source: Resources · p. 1
• The DeepReviewer demo is available at ai-researcher.net/deepreviewer.Source: Resources · p. 1

• As research submissions grow, LLMs are used to support reviewers; ICLR 2025 introduced an LLM-based assistance system [Blog 2024].Prior approaches include review-simulating agents such as AI-Scientist [Chris et al., 2024] and AgentReview [Jin et al., 2024a], and open-source models trained on platform reviews such as ReviewMT and CycleReviewer [Weng et al., 2025].Source: §1 · p. 2
• Existing systems provide superficial feedback, lack evidence-based justifications and actionable suggestions, and can produce inaccurate evaluations under prompt engineering.The introduction identifies scarce structured review datasets and LLM limits in domain knowledge, dynamic updating, and verification as challenges to systematic, reliable assessment.Source: §1 · p. 2
• DeepReview structures review into novelty assessment, multi-dimensional evaluation, and reliability verification, with retrieval and ranking, self-verification, and self-reflection in its data synthesis pipeline.The pipeline is intended to ensure the soundness and robustness of generated suggestions and expose review reasoning and decision-making.Source: §1 · p. 2
• DeepReview-13K contains research papers, structured intermediate review steps, and final assessments; DeepReview-Bench has 1.2K samples for quantitative evaluation and review generation.DeepReviewer-14B offers Fast, Standard, and Best inference modes to balance efficiency and response quality.Source: §1 · p. 2
• 44.80% ↑ Rating MSE, 6.04% ↑ Rating Spearman, and 1.80% ↑ Selection Accuracy improvements are reported against CycleReviewer-70B, GPT-o1, and Deepseek-R1, alongside an 80% win rate against GPT-o1 and Deepseek-R1.DeepReviewer also shows resilience to adversarial attacks without explicit robustness training; test-time scaling analysis finds performance can improve by adjusting reasoning paths and response lengths.Source: §1 · p. 2

Reasoning in LLMs

• Reasoning research progressed from chain-of-thought to self-reflection and test-time-compute models, but targets general problem-solving; Review-with-Thinking adapts it to peer review.The cited progression includes chain-of-thought [Akyürek et al., 2022, Nye et al., 2021, Wei et al., 2022], self-consistency [Wang et al., 2023], self-verification [Weng et al., 2023], self-reflection [Madaan et al., 2024], and test-time compute [Yao et al., 2024, Guan et al., 2025].Source: §2 · p. 3

Reliable Scientific Literature Assessment

• AI scientists [Chris et al., 2024], multi-agent frameworks [Ghafarollahi and Buehler, 2024, Rasal and Hauer, 2024, Su et al., 2024], and review systems [Weng et al., 2025] advance scientific assessment, but often lack structured reasoning.The passage characterizes the resulting feedback as unreliable.Source: §2 · p. 3

Robust LLM-based Paper Review

• Paper-review systems span roleplaying-agent generation, meta-review synthesis, bias detection, and human-AI workflows, yet struggle with complex assessments and adversarial attacks.The literature also covers evaluation benchmarks and ethical analyses; the passage identifies explicit reasoning as a need. The cited approaches include roleplaying agents [D’Arcy et al., 2024, Gao et al., 2024, Yu et al., 2024, Weng et al., 2025], meta-review synthesis [Santu et al., 2024, Li et al., 2023, Zeng et al., 2024], bias detection [Liang et al., 2024, Tyser et al., 2024, Tan et al., 2024a], and hybrid workflows [Jin et al., 2024b, Zyska et al., 2023].Source: §2 · p. 3

3 Data Collection

• DeepReview-13K is a training dataset designed to capture intermediate reasoning processes in academic paper reviews.It addresses the scarcity of high-quality, structured review datasets and standardized evaluation frameworks.Source: §3 · p. 4

3.1 DeepReview-13K

• From 18,976 submissions across two ICLR cycles, the final dataset contains 13,378 valid samples with review text, rebuttal discussions, scores, meta-reviews, and acceptance decisions.Reviews include overall ratings (1–10) and fine-grained Soundness, Presentation, and Contribution scores (1–4); papers were converted to Markdown using MinerU [Wang et al., 2024a].Source: §3.1 · p. 4

3.2 DeepReview-Test

• A random 10% sample (1.2K) of the dataset forms DeepReview-Bench for evaluating quantitative scores and qualitative review generation.The benchmark’s quantitative tasks are rating prediction, paper quality ranking, and pairwise paper selection.Source: §3.2 · p. 4
• For qualitative evaluation, the framework uses Gemini-2.0-Flash-Thinking for pairwise review comparisons rather than relying on text-similarity metrics such as ROUGE and BLEU.This choice follows concerns that similarity metrics fail to capture review capabilities and is motivated by recent findings [Li et al., 2024a].Source: §3.2 · p. 4

4 Methodology

• The framework decomposes review into novelty verification (z1), multi-dimensional evaluation (z2), and reliability verification (z3).These steps assess originality through literature review, synthesize multiple expert perspectives, and examine internal consistency and logical coherence.Source: §4 · p. 4

4.1 Task Definition

• The task generates a review pair comprising a qualitative meta-review and a score, with scoring formulated over sequential reasoning chains.The formulation models dependencies between reasoning steps and intermediate states to support reliable evaluation.Source: §4.1 · p. 5

4.2 Structured Reasoning Process

• Novelty verification generates three research questions, analyzes the paper, and retrieves approximately 60 relevant papers before reranking to the top 10 for novelty analysis.Qwen-2.5-72B-Instruct generates questions, Gemini-2.0-Flash-thinking analyzes the paper, and Semantic Scholar retrieval is built on OpenScholar Asai et al. [2024]; Qwen-2.5-3B-Instruct creates search keywords.Source: §4.2 · p. 5
• The review-reconstruction stage combines reviews with author rebuttals to turn criticisms into technically detailed, actionable suggestions while preserving professional tone and citations.It uses rebuttal evidence about experimental results, theoretical proofs, and implementation details.Source: §4.2 · p. 5
• Reliability verification checks methodology and experiments, assigns confidence levels to evidence-supported comments, and integrates verification outcomes into a new meta-review.The step identifies key weaknesses and produces evidence-based analysis and constructive suggestions.Source: §4.2 · p. 5
• Automated quality control checks reasoning-chain consistency and data completeness, flagging and removing samples that fail either check.Qwen-2.5-72B-Instruct checks for logically coherent, non-contradictory reasoning and missing or empty fields.Source: §4.2 · pp. 5–6

4.3 Model Training

• DeepReviewer is trained from Phi-4 14B on DeepReview-13K for 23,500 steps, with a 256K extended context window and 40K context during training.Training used 8 H100 80G GPUs, DeepSpeed with ZeRO3, batch size 16, and learning rate 5e-6; overlength samples were randomly truncated.Source: §4.3 · p. 6
• Inference offers Fast, Standard, and Best modes that execute the final assessment directly, core steps z2 and z3, or the full z1–z3 chain, respectively.All modes share the architecture; Best-mode novelty verification uses Semantic Scholar and OpenScholar for literature review and citation checking.Source: §4.3 · p. 6

5.1 Experimental setting

• The experiments compared prompt-based AI Scientist and AgentReview with five LLM backbones, and fine-tuned CycleReviewer-8B and CycleReviewer-70B baselines.Inference used temperature 0.4, maximum input length 100K tokens, and maximum output length 16,384 tokens to support complete text processing.Source: §5.1 · p. 6

5.5 Test-Time Scalability Study

• 3,000-token Fast mode outperformed 6,000-token CycleReviewer across Decision Accuracy, Rating MSE, and fine-grained Spearman correlations for Soundness, Presentation, and Contribution.The authors report that Fast mode retains key-content identification and critical-flaw analysis while using fewer output tokens.Source: Table 3 · p. 7, §5.5 · p. 9

5.2 Main Results

• 65.83% average Rating MSE reduction versus prompt-based baselines and 44.80% versus CycleReviewer-70B accompany 64.06% Decision Accuracy and 0.6307 F1 on ICLR 2024.The evaluation included prompt-based AI Scientist and AgentReview systems using several LLM backbones, plus CycleReviewer-8B and -70B; inference used temperature 0.4 and maximum input/output lengths of 100K/16,384 tokens.Source: §5.2 · pp. 6–7, §5.1 · p. 6
• 0.3559 and 0.4047 Rating Spearman on ICLR 2024 and 2025, respectively, accompany 0.62 and 0.64 pairwise accuracy and Soundness MSE 0.1578 with MAE 0.3029.Rating Spearman improved over CycleReviewer-70B by 6.04% and AI Scientist (DeepSeek-R1) by 25.02%; the Soundness errors improved over CycleReviewer-70B by 33.58% and 22.09%, respectively. DeepReviewer was marginally below AI Scientist (Gemini-2.0-Flash-Thinking) in Contribution and Soundness accuracy but performed strongly across dimensions.Source: §5.2 · p. 7, Table 3 · p. 7

5.4 Defend Attacks Analysis

• 0.31-point overall-rating increase under attack (5.38 to 5.69), versus 4.26 points for Gemini-2.0-Flash-Thinking and 1.41 for DeepSeek-V3, demonstrates stronger robustness.DeepReviewer’s Soundness score rose 0.12 points, compared with 1.10 for Claude-3.5-Sonnet and 1.38 for Gemini; the authors attribute robustness to multi-stage content understanding, novelty verification, and reliability checks, while noting room for improvement through adversarial training.Source: Figure 2 · p. 8, §5.4 · p. 9

5.3 Review Text Quality

• 99.02% constructive-value and 99.01% analytical-depth win rates over AgentReview (GPT-4o) accompany an 88.21% overall win rate against AI Scientist (GPT-o1) on ICLR 2024.Against AI Scientist (Gemini-2.0-Flash-Thinking), Gemini favored DeepReviewer in constructive value and analytical depth 53.47% of the time, but preferred its own reviews in technical accuracy 20.79% of the time.Source: Table 4 · p. 7, §5.3 · p. 8

5.5 Test-Time Scalability Study

• 8.97% higher Rating Spearman (0.326 to 0.355) resulted from Fast-to-Best Reasoning Path Scaling, while additional reviewers consistently improved scoring tasks.Reasoning Path Scaling provides Fast, Standard, and Best modes with approximately 3,000, 8,000, and 14,500 output tokens; Reviewer Scaling varies simulated reviewers from R=1 to R=6. Reviewer Scaling results varied especially when R ≠ 4, likely because training focused on four reviewers; scoring gains exceeded ranking gains, which exceeded selection gains.Source: Figure 3 · p. 8, §5.5 · p. 9

6 Conclusions

• DeepReviewer uses Test-Time Scaling to adapt reasoning depth to diverse paper-review needs.The conclusion identifies Reasoning Path and Reviewer Scaling as strategies for this scaling.Source: §6 · pp. 9–10
• DeepReview-13K provides detailed annotations to support systematic and deep paper evaluation, alongside the training of the DeepReviewer model.Source: §6 · p. 10
• The authors report comprehensive validation of DeepReviewer’s superiority in both objective and subjective assessments.Source: §6 · p. 10

Limitations

• DeepReview-13K’s synthetic data may not fully capture the complexities of human review, despite quality controls and real-world ICLR review data with structured reasoning annotations.The authors state that the inherent limitations of synthetic data persist.Source: Limitations · p. 10
• The “Best” mode’s complete reasoning chain and external knowledge retrieval can be computationally intensive, while “Fast” and “Standard” modes trade thoroughness for lower computational cost.These modes are offered to accommodate diverse application needs.Source: Limitations · p. 10
• DeepReviewer shows robustness against adversarial attacks but is not completely immune, leaving security and reliability as areas for further research.Source: Limitations · p. 10

Ethical Considerations

• DeepReviewer may amplify training-data or architectural biases, disadvantage underrepresented or unconventional research, and contribute to reviewer deskilling if relied on without oversight.The authors describe these as risks even though DeepReview-13K was synthetically generated to be representative and fair.Source: Ethical Considerations · p. 10
• The authors recommend treating DeepReviewer as decision support under expert oversight, with open-source scrutiny, user guidance, accountability measures, and ongoing bias audits.They describe a synthetic dataset and generation pipeline designed to model expert reasoning and diverse perspectives, user disclosures of affiliation, personal information, and intended use, and continued benchmarking across datasets and review scenarios.Source: Ethical Considerations · p. 10

A Responsible Use and Recommendations for DeepReviewer

• DeepReviewer should augment, not replace, human reviewers, whose expertise, nuanced judgment, and final authority must remain central.The recommendations frame its role as providing supplementary insights and streamlining some aspects of review under human oversight.Source: A Responsible Use and Recommendations… · p. 15

A.1 Enhanced Author Self-Assessment and Manuscript Refinement

• Authors can use DeepReviewer before submission to identify perceived strengths and weaknesses in soundness, clarity, and contribution, then decide revisions using their own judgment.Its feedback is a preliminary signal for self-improvement; authors may also seek trusted colleagues’ feedback, and the tool should not encroach on formal peer review.Source: §A.1 · pp. 15–16

A.2 Preliminary Assistance for Human Reviewers in Initial Paper Scoping

• Reviewers handling many submissions may optionally use DeepReviewer for a rough initial overview and possible workload prioritization, but must independently read and critically analyze each paper.The output should remain an extremely rough initial signal and must not influence or diminish reviewers’ own detailed engagement.Source: §A.2 · p. 16

A.3 Author-Facing Pre-Review Feedback via Deployed Model

• A deployed author-facing service can provide pre-submission automated feedback for authors’ information and improvement, not for formal submission or academic decisions.The feedback should be explicitly identified as neither a substitute for nor representative of human peer review, and cannot be a basis for acceptance or rejection.Source: §A.3 · p. 16

• The three core tasks assess LLMs’ ability to perceive, judge, and differentiate paper quality using a point-wise paradigm inspired by LLM-as-a-judge research.Source: B Evaluation Tasks and Metric · p. 16
• Score Task measures predicted paper ratings against expert averages using MSE and MAE, and evaluates Accept/Reject decisions with accuracy and F1.Source: B Evaluation Tasks and Metric · p. 16
• Ranking Task ranks papers by predicted quality scores and evaluates ranking accuracy with the Spearman coefficient.Source: B Evaluation Tasks and Metric · p. 17
• Selection Task measures whether the model identifies the highest-scoring paper in each non-overlapping two-paper batch, averaging accuracy across all pairwise matches in the Test dataset.Source: B Evaluation Tasks and Metric · p. 17
• Review-comment comparisons use Gemini-2.0-Flash-Thinking to judge randomized review pairs as win, lose, or tie across five dimensions: constructive value, analytical depth, plausibility, technical accuracy, and overall judgment.Source: B Evaluation Tasks and Metric · p. 17

C Data Collection Permissions

• OpenReview supplies the original papers and review comments, while some papers come from ArXiv, whose materials may carry several different licenses.OpenReview data is distributed under CC BY 4.0; ArXiv paper licenses may include CC BY 4.0, CC BY-SA 4.0, CC BY-NC-SA 4.0, and CC Zero.Source: C Data Collection Permissions · p. 17
• Because the original papers were not modified, their use is stated to comply with their original agreements; the dataset claims no copyright and retains authors’ names.Source: C Data Collection Permissions · p. 17

D Case Study: Analysis of DeepReviewer’s Meta-Review

• The case study examined DeepReviewer-14B in Best mode on the ICLR 2025 CycleResearcher submission, which was excluded from training, against four independent human reviews.The authors frame the comparison as a test of whether the system could emulate a holistic meta-reviewer.Source: D Case Study: Analysis of… · p. 17
• DeepReviewer’s summary captured CycleResearcher’s research-lifecycle framework, its CycleResearcher and CycleReviewer components, SimPO, and the Review-5k and Research-8k datasets, echoing human reviewers’ summaries.Its identified strengths also matched human praise for the framework, preference data, automation, and datasets.Source: D Case Study: Analysis of… · pp. 17–18
• DeepReviewer consolidated reviewers’ concerns about data bias, dataset diversity, computational resources, misuse safeguards, methodology clarity, and generalizability into weaknesses and suggestions.The authors describe this aggregation as a nuanced and comprehensive meta-analysis.Source: D Case Study: Analysis of… · pp. 19–20
• 6.0 was DeepReviewer-14B’s overall rating prediction, aligning closely with the average human rating in the CycleResearcher case study.The authors also report alignment in overall sentiment and identified strengths and weaknesses.Source: D Case Study: Analysis of… · pp. 17, 20
• The authors caution that DeepReviewer’s technical understanding may fall short of a specialist human meta-reviewer’s and that its synthesis may miss implicit or nuanced concerns.They nevertheless characterize its meta-review as coherent, insightful, and critically aligned.Source: D Case Study: Analysis of… · p. 20

E Information About Use Of AI Assistants

• The authors state that DeepReviewer-14B reviewed the article and that they revised it according to the review comments.Source: E Information About Use Of AI Assistants · p. 20