OpenReviewer: A Specialized Large Language Model for Generating Critical Scientific Paper Reviews

• Llama-OpenReviewer-8B is fine-tuned on 79,000 expert reviews and trained to follow standard review templates and guidelines.Source: §1 · p. 1
• The open-source demo combines PDF processing with the specialized model to generate comprehensive reviews from paper submissions.Source: §1 · p. 1
• OpenReviewer reviews align considerably better with human expert reviews than GPT-4o and Claude-3.5-Sonnet, whose recommendations tend to be overly positive.Source: §1 · p. 1
• OpenReviewer offers rapid, structured pre-submission feedback that may help authors identify and address manuscript weaknesses.Source: §1 · p. 2

• Top venues receive over 10,000 submissions annually, increasing reviewer workload while authors lack quick access to preliminary expert feedback.Source: §2 · p. 2
• General-purpose LLMs can miss field-specific conventions, inadequately evaluate technical contributions, and lack the depth, specificity, critical perspective, or structure expected in reviews.Source: §2 · p. 2

3.1 Demo Interface

• The demo converts uploaded PDFs to editable markdown, accepts an editable review template, and streams generated reviews; users can also paste markdown directly.The free demo uses dynamically allocated ZeroGPU resources, which can affect speed and snappiness.Source: §3.1 · p. 2

3.2.1 Training Data

• The model is fine-tuned on roughly 79,000 high-confidence reviews from ICLR and NeurIPS papers from 2022 onward, after filtering the collected papers and reviews.The collection began with 36K papers and 141K reviews; Marker converts PDFs to markdown and was selected for its per-page accuracy and ability to convert equations and tables.Source: §3.2.1 · pp. 2–3

3.3 Prompt Design

• A system prompt sets the reviewer role and guidelines, while the input includes the full paper text and a venue-specific review template.Source: §3.3 · p. 3

3.4 Training

• Llama-3.1-8B-Instruct is fully fine-tuned for three epochs with a 128k-token maximum sequence length.Training used an effective batch size of 64 and took approximately 34 hours on 64 NVIDIA A100 80GB GPUs.Source: §3.4 · p. 3

4 Evaluation

• The evaluation treats similarity to human reviews as quality, an assumption the authors caution may be imperfect because human-written reviews have limited quality control.Source: §4 · p. 4
• The evaluation used 400 held-out NeurIPS 2024 and ICLR 2025 papers, comparing OpenReviewer with two Llama models, Claude-3.5-Sonnet, and GPT-4o.The authors note that review quality lacks a single objective measure because expert judgments can differ and free-form reviews express the same criticism in varied ways.Source: §4 · pp. 3–4

4.1 Matching Recommendations

• 55.5% of OpenReviewer recommendations matched at least one human reviewer, with 0.96 average error, versus GPT4o’s 23.8% and 2.34.The recommendation scores were normalized from 1 (strong reject) to 10 (strong accept).Source: Table 1 · p. 4, §4.1 · p. 4
• Other LLMs usually gave more positive recommendations: their average scores were 6.9 or higher, compared with OpenReviewer’s 5.4.Llama-3.1-8B-Instruct averaged 8.1, which the authors say would mean an accept for most test papers.Source: §4.1 · p. 4

4.2 Review Arena

• 60%–76% of the time, GPT-4o’s judge preferred OpenReviewer over competing LLMs when comparing reviews against human expert reviews.The win rates ranged from 60% against GPT4o to 76% against Llama-3.1-70B-Instruct.Source: Figure 2 · p. 5, §4.2 · p. 5

Review Quality and Criticism Level

• OpenReviewer’s average rating of 5.4 matches the human reviewer distribution exactly, while general-purpose LLMs average 6.9–8.1 on a 10-point scale.The evaluation reports better alignment with human judgments than general-purpose LLMs.Source: §5 · p. 5

Use Cases and Limitations

• OpenReviewer is intended to help authors identify weaknesses before submission, not replace human peer review; its ML/AI-focused training may limit effectiveness elsewhere.The discussion recommends using it alongside other forms of feedback.Source: §5 · p. 5

Ethical Considerations

• Automated review systems raise risks of authors gaming them or conferences using them in place of human reviewers, making clear use guidelines important.The discussion emphasizes maintaining review quality and preventing misuse.Source: §5 · p. 5

Technical Tradeoffs

• The 8B model balances performance with accessibility and computational requirements; the authors suggest specialized training may matter more than scale, while scaling could further improve reviews.The authors also note that larger models might achieve better results.Source: §5 · p. 5

6.1 AI-Assisted Peer Review

• Earlier tools addressed reviewer-paper matching, review quality, bias, meta-reviews, or manuscript revision; ReviewRobot (Wang et al., 2020) predicts scores and generates comments using knowledge graphs.Most existing systems target isolated aspects rather than comprehensive reviewing support.Source: §6.1 · pp. 5–6

6.2 Review Generation with Large Language Models

• LLM review studies report technical-depth and consistency limits for GPT-4 (Liu and Shah, 2023) and a focus on writing over technical contributions (Liang et al., 2023); OpenReviewer fine-tunes on review data.Reviewer2 (Gao et al., 2024) uses aspect-focused prompts before review generation, while SEA (Yu et al., 2024) supports review standardization, evaluation, and analysis.Source: §6.2 · p. 6

6.3 Datasets and Resources

• PeerRead (Kang et al., 2018) covers ML reviews, NLPeer (Dycke et al., 2023) computational linguistics, and MOPRD multiple disciplines; OpenReviewer uses a larger, newer collection.The training collection is from top ML conferences.Source: §6.3 · p. 6

6.4 Evaluation of Review Quality

• Prior measures cover review soundness (Shah et al., 2018), helpfulness (Wang and Shah, 2019), and structured quality; this study uses automated evaluation without humans-in-the-loop.Zhou et al. (2024) use question answering to evaluate LLM-generated reviews and find weak critical feedback.Source: §6.4 · p. 6

6.5 Ethics and Bias in Peer Review

• Research identifies institutional, gender (Hofstra et al., 2020), and strategic-manipulation bias; NLP automation also raises fairness, transparency, and accountability concerns (Ye et al., 2024).Source: §6.5 · p. 6

Conclusion

• OpenReviewer is an open-source system for generating high-quality peer reviews of ML/AI papers.Source: §7 · p. 6
• Specialized training on expert reviews produced more realistic and critical reviews than general-purpose LLMs, addressing their tendency toward overly positive assessments.The paper identifies this as its key contribution.Source: §7 · p. 6

Future Directions

• The authors propose expanding training data across venues and domains, adding literature search and citation graphs for novelty assessment, improving review-quality metrics, and building collaborative human-AI interfaces.Source: §7 · pp. 6–7

Conclusion

• The system is intended to provide preliminary feedback and complement, rather than replace, human peer review.Source: §7 · p. 7

Data

• The training data covers papers from 2022 onward at ICLR and NeurIPS in ML/AI, and partial use of non-anonymized papers may introduce information leakage.The time restriction is described as helping maintain contemporary relevance.Source: Limitations · p. 7

Technical

• The system depends on PDF-to-markdown conversion accuracy and has a relatively modest 8B parameters; limitations also include figure handling and incorporating background from references.Source: Limitations · p. 7

Evaluation

• The evaluation covers 400 test papers, relies primarily on automated metrics rather than detailed human analysis, and compares against a limited set of baselines.The authors attribute these evaluation constraints to commercial LLM usage costs and resource constraints.Source: Limitations · p. 7

Practical

• Practical constraints include limited hardware, possible false confidence from automated feedback alone, and potential bias toward particular conferences, templates, and ML/AI domains.Source: Limitations · p. 7

Ethics and Broader Impact Statement

• OpenReviewer is designed to complement, not replace, human peer review, although misuse could automate reviewing and compromise scientific rigor.The authors recommend using it alongside other forms of feedback.Source: Ethics and Broader Impact Statement · p. 7
• Training data drawn primarily from top ML/AI conferences may encode field biases and disadvantage underrepresented perspectives or methodological approaches.Researchers with better computational resources might also gain an unfair advantage in preparing submissions.Source: Ethics and Broader Impact Statement · p. 7
• OpenReviewer could broaden access to high-quality feedback for researchers without large peer networks or in resource-constrained environments.The authors say early feedback could help improve submissions and potentially support higher-quality publications and more efficient use of reviewer time.Source: Ethics and Broader Impact Statement · p. 7
• The authors list open availability, documentation of limitations, and clear use-case disclaimers as responsible-use measures.The statement also flags potential misuse and fairness concerns as issues for careful consideration.Source: Ethics and Broader Impact Statement · p. 7
• The authors call for research-community guidelines for ethical deployment of automated review systems.They also recommend research into bias detection and mitigation.Source: Ethics and Broader Impact Statement · p. 7

A.1 Prompts

• Figure 3 presents OpenReviewer’s system prompt, with brace-enclosed fields serving as placeholders.The prompt sets a reviewer role, guidelines, markdown format, and a venue-specific template.Source: Figure 3 · p. 11
• Figure 4 presents OpenReviewer’s user prompt, with brace-enclosed fields serving as placeholders.The user prompt asks for a review of the following paper and supplies the full paper text.Source: Figure 4 · p. 11
• Figures 5 and 6 show the system and user prompts used for GPT-4o as the judge.The prompts are referenced as the exact instructions for the judge.Source: §A.1 · p. 10, Figure 5 · p. 11, Figure 6 · p. 11

A.2 Example Outputs

• Figure 7 gives an example of GPT-4o judging a comparison between an OpenReviewer review and a GPT-4o review.The judge compares each review section with corresponding sections in expert reviews before choosing A, B, or Tie.Source: §A.2 · p. 10, Figure 7 · p. 12

A.3 Training Hyperparameters

• The configuration enables Liger plugin options and sets gradient accumulation and micro-batch size to 1.It also specifies pad_token <|finetune_right_pad_id|> and eos_token <|eot_id|>.Source: §A.3 · p. 13
• The training configuration specifies sequence_len 131072, 3 epochs, and learning_rate 2e-5.It also lists AdamW, a cosine learning-rate scheduler, and 50 warmup steps.Source: §A.3 · p. 13
• Figure 8 presents the training hyperparameters used for Llama-OpenReviewer-8B with the axolotl library.The configuration names meta-llama/Llama-3.1-8B-Instruct as the base model.Source: §A.3 · pp. 10, 13, Figure 8 · p. 13