ReviewEval: An Evaluation Framework for AI-Generated Reviews

ReviewEval

• ReviewEval evaluates AI-generated reviews for human alignment, factual accuracy, analytical depth, actionable insights, and adherence to reviewer guidelines.These dimensions target semantic coverage, errors or hallucinations, depth beyond generic commentary, constructive suggestions, and conformity with conference criteria.Source: §1 · p. 2

ReviewAgent

• ReviewAgent dynamically aligns reviews with conference-specific criteria through guideline-derived prompts mapped to relevant paper sections.The alignment process retrieves and preprocesses venue guidelines, converts them into prompts, and supports one-to-many mappings from criteria to sections.Source: §1 · p. 2
• ReviewAgent uses a self-refinement loop to improve prompts and intermediate review outputs, plus an external improvement loop driven by ReviewEval.A Supervisor LLM provides feedback on prompt clarity, consistency, guideline alignment, and completeness; the external loop uses evaluation metrics to improve final reviews.Source: §1 · p. 2

Evaluation

• ReviewAgent was evaluated against AI-Scientist, MARG, and expert-written reviews on 120 papers spanning NeurIPS, ICLR, and UAI and multiple domains.The dataset included computer science, biology, social science, finance, physics, and engineering papers.Source: §1 · p. 2

Results

• 6.78% and 47.62% gains in actionable insights over existing AI baselines and expert reviews respectively were reported for ReviewAgent.The paper also reports gains in analytical depth and guideline adherence, but this point highlights the stated actionable-insights comparison.Source: §1 · p. 2

AI-based scientific discovery

• AI-based research automation has expanded from predefined optimization and evaluation tasks toward systems covering ideation, experimentation, manuscript generation, and peer review.These approaches remain connected to human-defined search spaces or evaluation metrics in earlier settings.Source: §2 · p. 2

AI-based peer-review

• Existing peer-review systems include criterion-based autonomous review pipelines and multi-agent architectures that divide section review, aspect assessment, and feedback synthesis.AI Scientist uses structured paper understanding, criterion-based evaluation, and final synthesis, while MARG uses worker, expert, and leader agents.Source: §2 · p. 2

Evaluation framework for AI-based peer-review

• Evaluation studies use aspect coverage, ROUGE, BERTScore, BLANC, manual analysis, and recall, precision, and Jaccard similarity against human reviews.These measures capture coverage, lexical or semantic similarity, informativeness, and alignment between AI and human comments.Source: Figure 2 · p. 3
• Prior evaluation metrics emphasize AI-human review similarity while overlooking other quality dimensions and relying on opaque end-to-end LLM judgments.ReviewEval is presented as introducing more interpretable metrics to address these gaps.Source: Table 1 · p. 3, §2 · p. 3

Iterative Refinement of Large Language Models

• Iterative refinement methods use self-generated feedback, learned feedback with simulated annealing, or external proxy metrics to improve factuality, coherence, and task performance.The cited approaches include Self-Refine, LLMRefine, and ProMiSe.Source: Table 1 · p. 3

3 ReviewEval

• ReviewEval evaluates AI-generated reviews across expert alignment, factual correctness, constructiveness, analytical depth, and reviewer-guideline adherence.The framework is implemented under an LLM-as-a-Judge paradigm with consistent model specifications for each metric.Source: §3 · p. 3, §3.1 · p. 3, §3.2 · p. 4

3.1 Comparison with Expert Reviews

• Semantic similarity and topic coverage compare AI-generated reviews with expert reviews to assess alignment and breadth of addressed topics.Reviews are embedded for cosine similarity, while extracted topics are matched using an LLM-based similarity threshold τ = 2 before computing coverage.Source: §3.1 · pp. 3–4

3.2 Factual Correctness

• ReviewEval converts review claims into verification questions, retrieves supporting context, generates rebuttals, and scores whether claims are substantiated.The factual-correctness pipeline decomposes claims into sub-questions and compares the review against an evidence-based rebuttal.Source: §3.2 · p. 4

3.3 Constructiveness

• Constructiveness measures actionable criticism, methodological feedback, and improvement suggestions using specificity, feasibility, and implementation-detail criteria.An insight is actionable when Sact,i = σi + ϕi + ζi exceeds 1, and review constructiveness is the percentage of actionable insights.Source: §3.3 · pp. 4–5

3.4 Depth of Analysis

• Analytical depth averages normalized scores for literature comparison, logical gaps, methodological scrutiny, results interpretation, and theoretical contribution.Each dimension receives a score from 0 to 3, with higher Sdepth indicating more comprehensive and critical engagement.Source: §3.4 · p. 5

3.5 Adherence to Reviewer Guidelines

• Guideline adherence scores reviews against extracted subjective and objective venue criteria, using 0–3 ratings for qualitative alignment and binary scoring for objective requirements.The resulting adherence score quantifies compliance with the prescribed reviewing guidelines.Source: §3.5 · pp. 5–6

4 ReviewAgent

• ReviewAgent aligns review generation with conference-specific criteria while operating across ReviewEval’s multidimensional quality metrics.The framework is designed to support thorough reviews under venue-specific evaluation requirements.Source: §4 · p. 6

4.1 Conference-Specific Review Alignment

• Conference guidelines are retrieved, converted into instructional prompts, and mapped one-to-many onto relevant paper sections for independent section-wise evaluation.The section-level outputs are later aggregated into the final review.Source: §4.1 · p. 6

4.2 Iterative Refinement Loop

• A Supervisor LLM iteratively evaluates prompts against their guidelines and revises them for clarity, consistency, alignment, and comprehensiveness.The same refinement mechanism is applied to section-wise reviews; the current implementation runs one iteration.Source: §4.2 · p. 6

4.3 Improvement using ReviewEval

• ReviewEval scores draft reviews for constructiveness, analytical depth, factual accuracy, and guideline adherence, then supplies feedback to generate an enhanced final review.The improvement loop currently runs once; real-world submissions without expert comments rely on the four objective criteria.Source: §4.3 · p. 6

Results for FullCorpus-120

• ReviewAgent produces near-top actionable insights on FullCorpus-120, with MARG-v1-qwen narrowly ahead of ReviewAgent-qwen.The near-match indicates that ReviewAgent generates concrete, actionable feedback.Source: §5 · pp. 7–8
• ReviewAgent achieves the highest depth of analysis on FullCorpus-120, surpassing expert reviewers and all baseline models.ReviewAgent-Qwen leads this metric, while Improvement and Reflection variants perform especially strongly in additional evaluations.Source: §5 · pp. 7–8
• ReviewAgent-Qwen achieves approximately 0.85 guideline adherence and ReviewAgent-DeepSeek approximately 0.75, exceeding MARG and AI-Scientist.The adaptive system dynamically adjusts to review guidelines rather than using a fixed format.Source: §5 · p. 8

Results for CoreCorpus-30

• On the CoreCorpus-30 dataset, ReviewAgent-Qwen-Improvement-Reflection leads, closely followed by the Improvement variant, and both outperform all MARG and AI-Scientist baselines.The evaluation uses binary Improvement and Reflection settings, with suffixes indicating enabled features.Source: Table 3 · p. 7, §5 · p. 8
• ReviewAgent-Qwen-Improvement achieves the highest guideline adherence on CoreCorpus-30, with the Improvement and Reflection variants also exceeding all baselines.The results highlight dynamic prompt generation, evaluation-driven calibration, and iterative refinement.Source: Table 3 · p. 7, §5 · p. 8

Metric-level findings

• ReviewAgent-3.7-sonnet maintains balanced topic coverage, ReviewAgent-Qwen-Improvement best matches expert phrasing, and Reflection achieves the strongest analytical depth among variants.A MARG variant has the top single-model coverage, while iterative reflection is associated with deeper critique.Source: §5 · p. 8

Results for FullCorpus-120

• ReviewAgent variants remain among the highest in factual correctness, while DeepSeek- and Haiku-based versions perform best on this dimension.The models generally stay close to source material and avoid fabrication.Source: §5 · p. 8

6 Conclusion

• ReviewEval evaluates alignment, factual accuracy, analytical depth, constructiveness, and adherence to reviewer guidelines.The framework is presented as a multidimensional evaluation approach for AI-generated reviews.Source: §6 · p. 8
• ReviewAgent combines conference-specific alignment, iterative self-refinement, and external improvement through ReviewEval.These components target review quality across the framework’s evaluation dimensions.Source: §6 · p. 8
• Experiments show that ReviewAgent matches expert quality and surpasses existing AI frameworks.The conclusion summarizes the comparative evaluation reported in the paper.Source: §6 · p. 8

8 Limitations

• The evaluation samples papers only from AI conferences, which may limit generalizability to other domains, disciplines, or larger datasets.Although the dataset includes AI-plus-X topics, its conference scope remains restricted.Source: §8 · p. 8
• LLM-as-a-Judge dependence introduces potential hallucinations, bias, prompt sensitivity, temporal instability, and black-box behavior.These weaknesses constrain interpretability, reliability, transparency, and trust.Source: §8 · p. 9
• Interdependence among semantic similarity, factual accuracy, and analytical depth can introduce subjectivity and constrain model flexibility.The limitation concerns overlap among evaluation dimensions.Source: §8 · p. 9
• Iterative refinement, external improvement, and retrieval augmentation impose nontrivial computational expense.The paper notes that caching, batching, and chunk optimization could reduce this cost.Source: §8 · p. 9
• Purely automated metrics may miss qualitative properties such as tone, subtle criticism, and readability.These aspects are not fully captured by the reported automated evaluation.Source: §8 · p. 9

Method

• The analysis establishes a six-metric baseline, removes each metric individually, and compares adjusted and original unified scores.Positive changes indicate the removed metric generally scored lower than the others, while negative changes indicate higher scores.Source: §A.1.1 · p. 10
• Mean absolute change, relative percentage change, and percentage contribution are aggregated across models with mean and standard deviation values.The resulting plots visualize metric influence through removal effects and contribution shares.Source: §A.1.1 · pp. 10–11

Results

• Removing actionable insights causes the largest positive mean change, 0.0654 or +9.00%, in the unified score.This suggests actionable insights typically score below the other metrics’ average.Source: §A.1.1 · p. 10, Table 6 · p. 11
• Removing factual correctness causes the largest negative mean change, -0.0492 or -6.96%, in the unified score.Its removal decreases the score most because factual correctness typically scores above the other metrics’ average.Source: §A.1.1 · p. 10
• Factual correctness contributes 22.47% of the unified score on average, followed by depth at 19.14% and semantic similarity at 17.34%.Actionable insights have the lowest average contribution at 9.17%.Source: §A.1.1 · p. 11, Table 6 · p. 11

• Pearson coefficients were computed for every metric pair, with p-values used to assess statistical significance.Source: §A.1.2 · p. 11
• The correlation results are presented in a matrix and visualized through a heatmap and scatter plots.Source: §A.1.2 · pp. 11–12, Table 7 · p. 12, Table 8 · p. 12
• Adherence Score and Depth Score had the strongest positive correlation (r ≈ 0.426, p < 0.001).Source: §A.1.2 · p. 11, Table 7 · p. 12, Table 8 · p. 12
• Coverage showed a statistically significant negative correlation with Actionable Insights, indicating a possible trade-off between breadth and actionability.Source: §A.1.2 · p. 12, Table 7 · p. 12, Table 8 · p. 12
• Factual Correctness had very weak and non-significant correlations with the other five metrics.Source: §A.1.2 · p. 12, Table 7 · p. 12, Table 8 · p. 12

A.2 Examples of question generation and decomposition in factual correctness pipeline

• The factual-correctness pipeline converts a review point into a question and decomposes it into independently answerable sub-questions using LangChain and a custom RAG pipeline.The example derives a question from a PeerRead review (Kang et al., 2018) and decomposes it into comparison and discussion checks.Source: §A.2 · pp. 12–13

A.3 Examples of three dimensions of constructiveness

• Constructiveness examples distinguish actionable recommendations addressing privacy costs in private-data use and computational-cost transparency through stage-wise GPU-hour breakdowns from generic criticism or requests.Concrete issue-and-remedy recommendations receive score 1, whereas generic statements or requests receive score 0.Source: §A.3 · p. 13

A.4 Prompts Used

• The prompts extract reviewer guidelines from HTML while filtering noise, preserving relevant surrounding text and tables, selecting the first paper type when multiple types appear, and enforcing conference-specific output formats.Separate prompts then instruct section review around clarity, relevance, logical consistency, evidence, originality, contribution, and conference requirements, before a final formatter rewrites the complete review without changing its content.Source: §A.4 · p. 14

A.5 Example Reviews

• The accepted ReviewAgent example reports a unified path-gradient framework with 1.3–8× runtime improvements, higher effective sample size, and coverage of coupling, continuous, and implicitly invertible flows.The review also cites 97.4% ESS improvement for GMMs and 17.5× faster runtime for implicit flows, while noting missing statistical rigor and domain restrictions.Source: §A.5 · p. 15, Strengths · p. 15, Weaknesses · pp. 15–16

Constructive Feedback

• The accepted example recommends clearer explanations, stronger evidence integration, statistical reporting, explicit domain limitations, and analysis of runtime–tolerance trade-offs before acceptance with revisions.Its soundness assessment supports the technical method and empirical benchmarks but identifies unaddressed limitations on non-physical tasks.Source: Weaknesses · p. 16, Constructive Feedback · p. 16

Weaknesses

• The rejected AlloNet example recognizes biologically motivated integration and qualitative subitization relevance but criticizes novelty overstatements, missing reproducibility details, limited theoretical justification, narrow task scope, and weak statistical comparisons.The review specifically notes that subitization is the sole demonstrated task and that broader robotics or spatial-navigation claims are unsupported.Source: Strengths · p. 17, Weaknesses · p. 17
• The work reuses established components, including the Hammel model and ring attractors, without sufficient novelty justification, while broader applicability claims remain speculative.The review also notes undefined presentation jargon, unclear figures, missing error bars and labels, and no dedicated Contribution section.Source: Weaknesses · p. 17

Questions for Authors

• The review asks authors to justify predictive adaptation and Gaussian synaptic weights, compare against related models, validate human-like error rates, and address slow reaction times and manual tuning.It also notes ignored recent SNN literature, including Rapu & Ganguli (2024), and requests discussion of potential biases in AI systems mimicking human cognition.Source: Questions for Authors · pp. 17–18

Additional Comments

• Recommended revisions include adding a Contribution section, contrasting recent work, releasing code and hyperparameters, and demonstrating broader tasks such as spatial navigation.These changes target the review’s concerns about novelty, reproducibility, and the scope of the applicability claims.Source: Additional Comments · p. 18