ReviewRobot: Explainable Paper Review Generation based on Knowledge Synthesis

• ReviewRobot targets vague, generic, and inconsistent reviews by providing knowledgeable comments with detailed evidence.Source: §1 · pp. 1–2
• The framework constructs target-paper, related-work, and background knowledge graphs, then compares them to extract category-specific evidence.Source: §1 · p. 2
• Manually selected constructive review sentences are generalized into templates that convert predicted scores and structured evidence into natural-language comments.Source: §1 · p. 2
• 71.4% overall-recommendation accuracy nearly matches 72.2% inter-human agreement, while appropriateness and impact each reach 100% accuracy.Source: §1 · p. 2
• Domain experts judged up to 70.5% of generated comments valid, with generated comments outperforming human-written ones 20% of the time.Source: §1 · p. 2
• The benchmark contains 8K paper-review pairs, 473 sentence-review pairs, and a background knowledge graph built from 174K papers.Source: §1 · p. 2

2.1 Overview

• The system constructs knowledge graphs, compares knowledge elements across papers, and uses the resulting evidence to predict scores and generate category-specific comments.Source: §2.1 · p. 3
• ReviewRobot evaluates categories including summary, appropriateness, clarity, novelty, soundness, meaningful comparison, and potential impact.Source: §2.1 · p. 3

2.2 Knowledge Graph Construction

• It builds target-paper, related-work, and pre-publication background graphs to represent techniques and field knowledge.Source: §2.2 · p. 4
• Knowledge-graph nodes represent six entity types, while edges encode seven scientific relations such as Used-for, Feature-of, Compare, and Evaluate-for.Source: §2.2 · p. 4

2.3 Evidence Extraction

• Evidence is extracted by comparing differences among the constructed knowledge graphs for each review category.Source: §2.3 · p. 4

2.4 Score Prediction

• Category-related sentences and extracted evidence are encoded and concatenated to predict a quality score from 1 to 5.Source: §2.4 · pp. 4–5

2.5 Comment Generation

• A classifier trained from 200 manually annotated paper-review pairs selects constructive review sentences, achieving 85.25% binary classification accuracy for template generalization.Source: §2.5 · p. 5
• Template-based generation describes evidence-graph relations for summary and novelty, uses score-controlled polarity, and recommends recent uncited papers for related work.Source: §2.5 · p. 5

3.1 Data

• The corpus contains 8,110 paper-review pairs, while the background knowledge graph uses 174,165 papers from an open research corpus.Source: §3.1 · p. 5

3.2 Score Prediction Performance

• 71.43% accuracy for overall recommendation closely matches 72.2% human agreement, while the model achieves the lowest MSE across all categories.Source: §3.2 · p. 5
• 71.43% novelty accuracy exceeds 28.57% from abstract-only methods, and newer background knowledge lowers measured novelty after 2017.Source: §3.2 · pp. 5–6, Figure 3 · p. 5

3.3 Comment Generation Performance

• Experts judged 70.5% of Summary, 44.6% of Novelty, and 41.7% of Meaningful Comparison comments constructive and valid.Source: §3.3 · p. 6
• For 20% of papers, ReviewRobot supplied constructive missing-related-work comments when human reviewers did not, although human comments could remain unsupported by references.Source: §3.3 · pp. 6–7

3.4 Remaining Challenges and Limitations

• ReviewRobot is mainly limited by scientific-literature Information Extraction and lacks salience weighting, deep soundness reasoning, and support for formulas, tables, and figures.Source: §3.4 · pp. 7–8
• Constructing high-quality human-review training data is time-consuming and expensive because selected review sentences must be constructive and evidence-supported.Source: §3.4 · p. 8

Paper Acceptance Prediction

• ReviewRobot extends paper acceptance prediction from classification to review-category scores and comments supported by detailed knowledge-element evidence.Source: §4 · p. 8

Paper Review Generation

• Earlier paper-review generation used a neural generator trained on 48 papers, whereas ReviewRobot uses a larger dataset and produces more concrete, explainable category-specific reviews.Source: §4 · p. 8

Review Generation in other Domains

• Automatic review generation in music, restaurants, and products commonly uses sequence-to-sequence models with attention to aspects and attributes.Source: §4 · p. 8
• Paper review generation is more challenging than these domains because it requires deep paper understanding, cross-paper knowledge-graph comparison, and evidence synthesis.Source: §4 · p. 8

Controlled Knowledge-Driven Generation

• Knowledge-driven generation spans knowledge description and knowledge synthesis, including table-to-text, graph-to-text, dialogue, comment generation, paper generation, and summarization.Source: §4 · p. 8

• ReviewRobot’s designed evidence types are limited to NLP, ML, and related areas, so they do not directly apply to domains such as biomedical science and chemistry.Source: §5 · p. 9
• Reviewers should use ReviewRobot as an assistant for constructive comments and note comparison, not submit its output without reading the paper carefully.Source: §5 · p. 9
• Editors may use ReviewRobot to help filter very bad papers during screening, but should not make decisions from its output alone.Source: §5 · p. 9
• Authors can use ReviewRobot for initial feedback on writing, missing references, and potentially novel points, but should not revise merely to exploit its scoring features.Source: §5 · p. 9
• Researchers can use ReviewRobot for literature surveys, discovering papers, and validating novelty, within the system’s domain and evidence scope.Source: §5 · p. 9
• Authors should not deliberately cite all related papers or add irrelevant terms to increase ReviewRobot scores.Source: §5 · p. 9

• ReviewRobot predicts scores and generates detailed comments for each review category to assist human reviewers and authors polishing their papers.Source: §6 · p. 9
• Its key innovation is comparing knowledge graphs from the target paper and in-domain background papers to summarize paper-level pros and cons with detailed evidence.Source: §6 · p. 9
• Future work will improve knowledge reasoning with a taxonomy over the background KG and multimodal analysis of formulas, tables, figures, and citation networks.Source: §6 · p. 9