Source-linked AI summary
DeepReview: Improving LLM-based Paper Review with Human-like Deep Thinking Process
Minjun Zhu, Yixuan Weng, Linyi Yang, Yue Zhang
TL;DR
Existing LLM-based paper-review systems struggle with comprehensive flaw detection, evidence-based justification, structured data, and reliable scientific assessment. DeepReview introduces a structured multi-stage framework and trains DeepReviewer-14B on DeepReview-13K, with strong reported performance against established models and configurable reasoning depth. The authors also identify synthetic-data limitations, computational cost in Best mode, and ethical risks including bias amplification and reduced reviewer accountability.
Problem
Existing LLM review systems lack comprehensive flaw detection, evidence-based justifications, actionable suggestions, and reliable evaluation foundations.
Method
DeepReview structures review into novelty verification, multidimensional evaluation, and reliability verification, supported by retrieval, self-verification, and self-reflection.
Results
44.80% Rating MSE reduction, 6.04% Rating Spearman improvement, and 1.80% Selection Accuracy improvement are reported against existing systems.
Takeaways & Limitations
DeepReview-13K and DeepReview-Bench provide structured reasoning data and multidimensional evaluation resources for LLM-based paper review.
Takeaways & Limitations
The synthetic DeepReview-13K data may not fully capture the complexities and nuances of genuine human paper review.
Abstract
from arXiv · showhide
Large Language Models (LLMs) are increasingly utilized in scientific research assessment, particularly in automated paper review. However, existing LLM-based review systems face significant challenges, including limited domain expertise, hallucinated reasoning, and a lack of structured evaluation. To address these limitations, we introduce DeepReview, a multi-stage framework designed to emulate expert reviewers by incorporating structured analysis, literature retrieval, and evidence-based argumentation. Using DeepReview-13K, a curated dataset with structured annotations, we train DeepReviewer-14B, which outperforms CycleReviewer-70B with fewer tokens. In its best mode, DeepReviewer-14B achieves win rates of 88.21\% and 80.20\% against GPT-o1 and DeepSeek-R1 in evaluations. Our work sets a new benchmark for LLM-based paper review, with all resources publicly available. The code, model, dataset and demo have be released in http://ai-researcher.net.
Resources
The paper provides open-source models, code, a homepage, and a demo, while positioning DeepReview as assistance for human peer reviewers rather than replacement.
- DeepReviewer-7B and DeepReviewer-14B are provided as the paper’s models.
- The code repository is zhu-minjun/Researcher.
- The project homepage is ai-researcher.net.
- A DeepReviewer demo is available at ai-researcher.net/deepreviewer.
- The work explores LLM assistance in peer review rather than advocating replacement of human reviewers.
1 Introduction
The introduction frames LLM-assisted peer review as important but currently limited by shallow, weakly justified evaluations and insufficiently structured review data. DeepReview addresses these gaps with structured reasoning, new resources, and a model that performs strongly across evaluation settings.
- Peer review supports scientific progress by evaluating research novelty, reliability, and rigor before publication.
- Existing systems often miss submission flaws, lack evidence-based justifications and actionable suggestions, and remain vulnerable to prompt engineering.
- DeepReview combines novelty assessment, multidimensional evaluation, reliability verification, retrieval, self-verification, and self-reflection in a structured framework.
- DeepReview-13K contains raw papers, intermediate review steps, and final assessments, while DeepReviewer-14B offers Fast, Standard, and Best inference modes.
- 44.80% Rating MSE reduction, 6.04% Rating Spearman improvement, and 1.80% Selection Accuracy improvement are reported against existing systems.
- The work presents DeepReview-Bench and reports resilience to adversarial attacks alongside substantial improvements over existing approaches.
2 Related Work
Related work spans LLM reasoning techniques, automated scientific research, and robust paper-review systems. These lines of work include agent collaboration, open-source review models, human-AI workflows, and bias-detection mechanisms.
- Reasoning in LLMs: LLM reasoning research includes scratchpads, chain-of-thought, self-consistency, self-verification, and self-reflection.
- Reliable Scientific Literature Assessment: Automated scientific research uses AI scientists, multi-agent frameworks, and reinforcement learning for collaborative or review-oriented reasoning.
- Robust LLM-based Paper Review: Robust paper-review research covers roleplaying agents, meta-review synthesis, bias detection, and hybrid human-AI workflows.
3 Data Collection
DeepReview-13K is constructed as a structured training and evaluation resource from real conference and repository data, combining review text, rebuttal discussions, scores, and intermediate reasoning. DeepReview-Bench samples 1.2K examples and evaluates both quantitative prediction tasks and qualitative review generation.
- DeepReview-13K: DeepReview-13K captures intermediate reasoning processes in academic paper reviews to address shortages of structured datasets and standardized evaluation frameworks.
- DeepReview-13K: 18,976 submissions from two ICLR conference cycles were collected from OpenReview and arXiv repositories.
- DeepReview-13K: Each review set combines textual assessments, rebuttal-phase discussions, and standardized scores.
- DeepReview-Bench: DeepReview-Bench randomly samples 10% of the dataset, yielding 1.2K evaluation examples.
- DeepReview-Bench: Quantitative evaluation covers rating prediction, paper quality ranking, and pairwise paper selection using classification, error, and correlation metrics.
- DeepReview-Bench: Qualitative review generation is evaluated through LLM-as-a-judge pairwise comparisons rather than simple text-similarity metrics.
4 Methodology
DeepReview decomposes paper review into novelty verification, multi-dimensional evaluation, and reliability verification, supported by structured data construction and quality control. Its inference modes trade intermediate reasoning for computational efficiency according to test-time requirements.
- Structured Reasoning Process: The review process is decomposed into novelty verification z1, multi-dimension evaluation z2, and reliability verification z3.These stages assess originality through literature review, synthesize multiple expert perspectives, and examine internal consistency and logical coherence.
- Data Construction: The data construction pipeline generates supervised fine-tuning samples containing complete reasoning paths represented as z1, z2, and z3.The pipeline is designed to capture the full sequence of reasoning stages for training.
- Novelty Verification: Novelty verification combines question generation, paper analysis, and literature review to identify research gaps, innovative directions, and methodological breakthroughs.Qwen-2.5-72B-Instruct generates three key research questions based on each paper.
- Multi-dimension Review: Multi-dimension review reconstructs reviews and author rebuttals into instructive suggestions covering experimental results, theoretical proofs, and implementation details.The pipeline synthesizes multiple reviews into comprehensive perspectives.
- Reliability Verification: Reliability verification analyzes evidence through methodology, experimental, and comprehensive checks, requiring supporting paper evidence and assigning confidence levels.A new meta-review is generated by integrating the verified analysis with the original review content.
- Quality Control: Samples are removed when automated quality control detects logical inconsistency, incompleteness, or missing reasoning-path and evaluation components.Qwen-2.5-72B-Instruct checks coherence between z1, z2, z3 and the final evaluation (s, a).
- Inference Strategy: Fast mode directly generates final results and reports, while Standard mode executes core evaluation steps including z2 and z3 to adjust test-time efficiency.The modes are created by cropping reasoning paths from DeepReview-13K samples.
5 Experiments
DeepReviewer outperforms prompt-based and fine-tuned baselines across rating, ranking, selection, fine-grained assessment, and review-text quality, while supporting robust and scalable inference.
- 65.83% average Rating MSE reduction and 15.2% average Decision Accuracy improvement over prompt-based baselines.
- 44.80% lower Rating MSE than CycleReviewer-70B, with 64.06% Decision Accuracy and 0.6307 F1 on ICLR 2024.
- Rating Spearman reaches 0.3559 on ICLR 2024 and 0.4047 on ICLR 2025, while pairwise selection accuracy reaches 0.62 and 0.64.
- On ICLR 2024 Soundness, DeepReviewer-14B achieves 0.1578 MSE and 0.3029 MAE, improving over CycleReviewer-70B by 33.58% and 22.09%.
- DeepReviewer wins 88.21% against AI Scientist (GPT-o1) and 98.15% against AgentReview (GPT-4o), with strongest advantages in constructive value and analytical depth.
- Under attack, DeepReviewer’s overall rating increases only 0.31 points, from 5.38 to 5.69, indicating stronger robustness than baseline systems.
- Reasoning Path Scaling improves Rating Spearman by 8.97%, from 0.326 to 0.355, while Reviewer Scaling generally benefits scoring tasks but varies across tasks.
- Fast mode uses approximately 3,000 output tokens yet outperforms CycleReviewer with 6,000 tokens across several rating and fine-grained metrics.
6 Conclusions
The paper presents DeepReviewer as a structured framework for reliable LLM-based paper evaluation, supported by a reasoning dataset, trained model, and test-time scaling strategies.
- DeepReviewer is introduced to enhance the reliability of LLMs in research paper evaluation.
- The work contributes DeepReview-13K, the DeepReviewer model, and validation across objective and subjective assessments.
- Test-Time Scaling is demonstrated through Reasoning Path Scaling and Reviewer Scaling strategies.
Limitations
DeepReview is limited by reliance on synthetic training data, computational demands in its most thorough mode, and incomplete resistance to adversarial attacks.
- Synthetic DeepReview-13K data may not fully capture the complexity and nuance of genuine human paper review.The dataset uses automated construction and quality controls, while incorporating real ICLR review data and structured reasoning annotations.
- DeepReviewer’s Best mode can be computationally intensive because it uses the complete reasoning chain and external knowledge retrieval.Fast and Standard modes provide alternatives that trade thoroughness against computational cost.
- Complete immunity to adversarial attacks has not been achieved despite demonstrated robustness.The authors identify enhanced security and reliability as ongoing research needs.
Ethical Considerations
DeepReviewer raises risks of bias amplification, reviewer deskilling, and reduced transparency and accountability, so its use is framed as assistance requiring human oversight.
- Automating peer-review aspects may amplify bias, deskill human reviewers, and erode transparency and accountability.Potential disadvantages are noted for underrepresented research, unconventional methods, and less mainstream topics.
- The development process attempts to mitigate ethical risks through synthetic-data design that models expert reasoning and incorporates diverse perspectives.The stated goal is to minimize unintended bias in the dataset and generation pipeline.
- DeepReviewer is intended to enhance rather than replace human peer review, with final judgment remaining under human oversight.The system is positioned as a supplementary tool that provides insights and streamlines selected review activities.
- Authors may use DeepReviewer before submission to obtain preliminary feedback on manuscript strengths and weaknesses.This application is described as formative self-improvement rather than part of formal review.
- Reviewer-facing use is limited to optional, high-level initial scoping and must not replace independent reading or critical analysis.Reviewers are instructed to engage deeply with the paper itself and apply their own expertise.
- Author-facing deployment provides automated feedback for information and improvement without entering formal submission or decision-making processes.The service is designed to avoid burdening human reviewers at this preliminary stage.
B Evaluation Tasks and Metric
DeepReview evaluates paper-review systems through point-wise Score, Ranking, and Selection tasks, alongside pairwise LLM-as-a-judge comparison of review comments.
- Quantitative Evaluation: The evaluation uses three tasks to test independent quality assessment, quality ranking, and identification of the best paper from a small candidate pool.These tasks target different abilities to perceive, judge, and differentiate paper quality.
- Score Task: Score Task predicts a scalar quality rating for each paper and compares it with average expert ratings using MSE and MAE.The task measures accuracy in independent paper assessment scenarios.
- Ranking Task: Ranking Task orders papers by predicted quality, with ranking accuracy evaluated using the Spearman coefficient.Models first predict scores for a collection and then produce a descending quality order.
- Selection Task: Selection Task chooses the highest-quality paper from non-overlapping batches, using m = 2 and averaging pairwise-selection success across the test set.The model’s choice is compared with the paper having the highest actual review score.
- Qualitative Evaluation: Review-comment quality is assessed through randomized pairwise comparisons judged by Gemini-2.0-Flash-Thinking, which outputs win, lose, or tie.Each comparison includes the original paper and reviews containing summaries, strengths, weaknesses, and suggestions.
- Data Provenance: DeepReview-13K is built from OpenReview data, with some papers originating from arXiv and OpenReview review comments licensed under CC BY 4.0.The cited source and licensing information describe the provenance of data used in dataset construction.
D Case Study: Analysis of DeepReviewer’s Meta-Review
The CycleResearcher case study finds that DeepReviewer’s Meta-Review captures core contributions, aligns with human-identified strengths and weaknesses, and synthesizes concerns across reviews, while retaining technical-depth limitations.
- Summary and Strengths: DeepReviewer accurately summarizes CycleResearcher’s framework, components, SimPO-based iterative reinforcement learning, and Review-5k and Research-8k datasets.These elements correspond closely to the central aspects highlighted by four human reviewers.
- Evaluation Setup: Figure 4 shows the system prompt used to guide Gemini-2.0-Thinking-Flask in judging generated review comments.The case study’s evaluation also uses a separate prompt to improve review comments by incorporating author responses while preserving context.
- Summary and Strengths: The Meta-Review’s identified strengths mirror positive aspects recognized by the human reviewers.The case study specifically connects its description of an innovative approach with reviewers’ praise of the framework.
- Weaknesses and Suggestions: DeepReviewer synthesizes weaknesses and suggestions by consolidating concerns about training-data bias, dataset diversity, and cross-domain generalizability.The analysis presents this aggregation as evidence of nuanced meta-analysis across multiple reviews.
- Limitations and Rating: The case study reports that technical understanding and subtle human-review perspectives may be weaker in specialized areas such as reinforcement learning or AI ethics.It also reports an overall rating prediction of 6.0 that closely aligns with the average human rating.
E Information About Use Of AI Assistants
The paper describes AI-assisted research and review applications alongside concerns about generated-content quality, model scaling, ethics, and accountability. It also reports resources, training approaches, safeguards, and a positive review assessment.
- DeepReviewer-14B was reviewed and the paper was revised accordingly based on its review comments.
- The reported CycleResearcher framework uses policy and reward models to generate papers and simulate peer reviews through iterative reinforcement learning.
- Reviewers identified hallucinations, unsupported claims, and missing citations in generated papers, raising concerns about their factual reliability.
- CycleResearcher-12B reportedly achieved a higher acceptance rate than its 72B and 123B counterparts, but the paper does not sufficiently explain this counterintuitive scaling result.
- The discussion raises ethical concerns involving bias, accountability, misuse, research integrity, and effects of automation on researchers and peer reviewers.