Source-linked AI summary
Automated Peer Reviewing in Paper SEA: Standardization, Evaluation, and Analysis
Jianxiang Yu, Zichen Ding, Jiaqi Tan, Kangyang Luo, Zhenmin Weng, Chenghua Gong, Long Zeng, Renjing Cui, Chengcheng Han, Qiushi Sun, Zhiyong Wu, Yunshi Lan, Xiang Li
TL;DR
The growing volume of research strains traditional review mechanisms, while publication quality varies. SEA standardizes reviews, fine-tunes an LLM to generate reviews, and evaluates consistency; experiments report better review quality, comprehensiveness, and consistency than existing methods across eight datasets.
Problem
Growing research volume strains traditional review mechanisms, and variable publication quality motivates automated reviews with strong evidence to help authors improve their work.
Source: §1 · p. 1Method
Results
Across eight diverse datasets, SEA significantly outperforms existing methods in review quality, comprehensiveness, and consistency.
Source: §1 · p. 2Takeaways & Limitations
Automated peer reviewing can provide timely feedback that enhances research quality and accelerates scientific development.
Source: §5 · p. 9SEA has limited expansion across academic disciplines and insufficient alignment with human standards.
Source: Limitations · p. 10
Abstract
from arXiv · showhide
In recent years, the rapid increase in scientific papers has overwhelmed traditional review mechanisms, resulting in varying quality of publications. Although existing methods have explored the capabilities of Large Language Models (LLMs) for automated scientific reviewing, their generated contents are often generic or partial. To address the issues above, we introduce an automated paper reviewing framework SEA. It comprises of three modules: Standardization, Evaluation, and Analysis, which are represented by models SEA-S, SEA-E, and SEA-A, respectively. Initially, SEA-S distills data standardization capabilities of GPT-4 for integrating multiple reviews for a paper. Then, SEA-E utilizes standardized data for fine-tuning, enabling it to generate constructive reviews. Finally, SEA-A introduces a new evaluation metric called mismatch score to assess the consistency between paper contents and reviews. Moreover, we design a self-correction strategy to enhance the consistency. Extensive experimental results on datasets collected from eight venues show that SEA can generate valuable insights for authors to improve their papers.
1 Introduction
SEA addresses shortcomings in automated paper reviewing with a three-module framework; experiments on eight diverse datasets report better review quality, comprehensiveness, and consistency than existing methods.
1 Introduction
- Growing publication volume strains traditional feedback mechanisms, motivating automated reviews that provide constructive, evidence-supported feedback for authors.The passages link publication growth to pressure on peer review and concerns about publication quality.Source: §1 · p. 1
- Prompted LLM reviews can be generic, while fine-tuning on a single review risks biased or partial feedback across varied formats and criteria.The cited approaches also lack a self-correction mechanism for less appealing generated reviews.Source: §1 · p. 2
- SEA-S uses GPT-4 to integrate multiple reviews into a unified format with constructive content, then distills that capability into Mistral-7B.The integrated reviews form an instruction dataset for supervised fine-tuning.Source: §1 · p. 2
- SEA-E is fine-tuned on parsed papers, standardized reviews, and human-crafted prompts to generate reviews that analyze papers.The pipeline parses papers into text and LaTeX codes before assembling the fine-tuning data.Source: §1 · p. 2
- SEA-A estimates mismatch scores and supports self-correction, prompting SEA to rethink and regenerate reviews inconsistent with parsed papers.The stated aim is to promote more constructive, paper-consistent reviews.Source: §1 · p. 2
- Across eight diverse datasets, SEA significantly outperforms existing methods in review quality, comprehensiveness, and consistency.The framework is presented as a way to generate constructive reviews for authors.Source: §1 · p. 2
- SEA is intended to provide timely researcher feedback, not to recommend paper acceptance or rejection.The authors frame the feedback as support for improving research work and moving to subsequent projects.Source: §1 · p. 2
2 Related Works
Related work covers long-context language models and automated scientific reviewing. The paper selects Mistral-7B for its stated context capacity, while prior reviewing efforts range from early small-model studies to LLM-based approaches.
2.1 Long-context Large Language Models
- Mistral-7B is selected based on RULER results and is reported to handle at least 16K tokens, enough for most academic papers' input requirements.Long-context work includes attention modifications and positional encoding methods such as ALiBi, xPOS, and RoPE variants.Source: §2.1 · p. 2
2.2 Automated Scientific Reviewing
- Early automated reviewing studies examined paper fluency and peer-grading reliability, while later LLM work used prompted GPT-4, academically trained LLaMA2-70B, and question-guided review generation.The passage notes that full-paper scientific reviewing had not been studied before LLMs because earlier models struggled with long contexts.Source: §2.2 · p. 3
3 SEA
SEA comprises Standardization, Evaluation, and Analysis modules. It standardizes peer reviews, uses them to train a paper-reviewing model, and estimates paper–review consistency to guide self-correction.
3.1 SEA-S: Standardization
- SEA-S integrates each paper’s reviews into a unified format, distilling GPT-4’s standardization capabilities into an open-source model.The unified reviews are intended to reduce redundancy and error while emphasizing major strengths and weaknesses.Source: §3.1 · pp. 3–4
3.2 SEA-E: Evaluation
- SEA-E uses parsed papers and SEA-S standardized reviews for fine-tuning; Nougat converts formulas to LaTeX, and Mistral-7B handles long contexts.The standardized reviews contain evidence for review arguments.Source: §3.2 · p. 4
3.3 SEA-A: Analysis
- SEA-A defines ground-truth mismatch from reviewer ratings relative to their confidence-weighted paper average, with smaller absolute scores indicating greater review–paper consistency.The score’s sign may indicate whether a review emphasizes strengths or is critical.Source: §3.3 · p. 4
- SEA-A estimates mismatch from parsed paper–review pairs using a lightweight regression model trained against ground-truth mismatch scores.The model uses long-context text representations and learned query and key vectors.Source: §3.3 · p. 4
- When estimated mismatch exceeds a threshold, SEA-EA adds the score to the prompt and regenerates the review to improve consistency.The self-correction strategy analyzes reviews generated by SEA-E.Source: §3.3 · p. 5
4 Experiments
Across the tested scenarios, SEA outperforms the baselines, with particularly notable ROUGE Recall gains and strong cross-domain performance. Its results also support the value of standardized review data and self-correction, while human and GPT evaluations favor SEA-E and SEA-S.
4.1 Experimental Details
- The experiments cover in-domain NeurIPS and ICLR data and cross-domain CONLL, ACL, COLING, and ARR datasets, with structurally formatted reviews and PDF papers.The baseline comparison includes direct Mistral-7B inference, SFT using a randomly selected review, GPT-3.5-standardized SFT, and REVIEWER2 on a smaller test set.Source: §4.1 · pp. 5–6
4.2 Main Results
- SEA outperforms baseline models across all testing scenarios, with notable ROUGE Recall gains; SEA-EA also surpasses SEA-E in every case.The authors report strong performance on cross-domain datasets and attribute SEA-E’s advantage over M-7B-3.5 to SEA-S’s standardized data.Source: §4.2 · p. 6
4.3 Comparison of Standardized Results
- SEA-S generates a significantly larger percentage of exclusive content than Mistral-7B and GPT-3.5, supporting richer standardized reviews, and slightly more exclusive content than GPT-4.Using regular-expression matching based on instruction formats, SEA-S generated 100% correctly formatted data; around 10% of GPT-4 data did not fully comply with instructions.Source: §4.3 · p. 7
4.4 Mismatch Score in SEA-A
- SEA-EA consistently outperforms other models on mismatch scores across datasets, with SEA-E the runner-up.The authors interpret these results as showing greater consistency between generated reviews and their corresponding papers.Source: §4.4 · p. 7, Figure 5 · p. 7
4.5 Quantitative Score Analysis
- SEA’s methods outperform baseline models in practically all reported score-analysis cases, and SEA-EA improves over SEA-E in most cases.The comparison uses MSE against confidence-weighted reference scores; SEA-E generates valid scores 100% of the time versus 86% for M-7B-3.5, despite SEA-E’s Presentation MSE being 0.02 larger on ICLR-2024.Source: §4.5 · p. 8, Table 3 · p. 8
4.6 Qualitative Decision Analysis
- SEA-EA has the largest reported decision accuracy and BERTScore, and on ICLR-2024 its accuracy exceeds SEA-E’s by over 4%.The authors note that M-7B’s roughly 97% recall accompanies poor precision, while SEA performs better in precision and F1-score.Source: §4.6 · p. 8, Table 4 · p. 8
4.7.1 Review Quality (SEA-E)
- SEA-E significantly outperforms other models on ten review-quality questions in both human and GPT-4o evaluations, and experts preferred it in 90% of cases.The human evaluation used 20 experts, while GPT-4o scored reviews using the same ten questions.Source: §4.7.1 · p. 9, Table 5 · p. 9
4.7.2 Standardized Content Quality (SEA-S)
- Human evaluators particularly favored SEA-S for relevance, logical coherence, and conciseness, while GPT-4o generally assigned higher scores across models.SEA-S consistently stood out against the other models in both evaluation methods.Source: §4.7.2 · p. 9, Table 6 · p. 9
5 Conclusion
SEA combines standardized review data, fine-tuned review generation, and a metric for consistency between papers and generated reviews. The authors report that its feedback aligns with human reviews.
5 Conclusion
- SEA presents an automated paper-reviewing framework that combines standardized review data, fine-tuned review generation, and a consistency metric.The metric measures consistency between papers and generated reviews.Source: §5 · p. 9
- Comprehensive experiments demonstrate that SEA can generate feedback that aligns with human reviews.Source: §5 · p. 9
- The authors anticipate that SEA will help researchers improve their work and contribute to automated scientific reviewing.Source: §5 · p. 9
Limitations
The authors identify limits in SEA's disciplinary reach and human-standard alignment, and note that natural-language-guided self-correction training and rebuttal assistance remain future work.
Limitations
- SEA has been tested within machine learning but has not yet been expanded to disciplines such as physics and mathematics.The authors propose investigating its review quality in other disciplines.Source: Limitations · p. 10
- SEA-E has not been trained with natural-language guidance based on mismatch scores during self-correction.The authors plan to collect guided self-correction data to improve instruction following and alignment with the paper.Source: Limitations · p. 10
- Methods to assist authors in preparing effective rebuttals are reserved for future research.The authors describe rebuttals as a stage where authors can correct misunderstandings, clarify the paper, or provide additional supporting information.Source: Limitations · p. 10
Ethical Considerations
The authors describe potential misuse of generated reviews in formal conference review and plan license clauses to prohibit it.
Ethical Considerations
- The framework uses long-context LLMs and supervised fine-tuning to align generated reviews with human reviews and assist authors in improving papers.Source: Ethical Considerations · p. 10
- Authors could receive model-generated reviews in formal conference processes without knowing they were generated by a model.The authors warn that this could affect fairness and transparency and raise trust and authenticity concerns.Source: Ethical Considerations · p. 10
- The authors plan to add license clauses that strictly prohibit misuse of the system.They intend the system to serve as a beneficial tool in academia.Source: Ethical Considerations · p. 10
A.1 SEA-S
SEA-S standardizes multiple reviews; the examples contrast its output with Mistral-7B and GPT-3.5 on completeness and redundancy.
A.1 SEA-S
- SEA-S produces rich, concise standardized reviews without redundant information in the examples.Source: §A.1 · p. 13
- Mistral-7B's example lacks a complete format and has sparse content.The figure highlights missing parts in orange.Source: §A.1 · p. 13
- GPT-3.5's example largely extracts original review text and retains redundant phrasing such as repeated uses of “Lack of”.The figure marks the excessive repetition in red.Source: §A.1 · p. 13
A.2 SEA-E
SEA-E and SEA-EA produce clearer viewpoints and broader paper-content coverage than the compared Mistral-7B variants, with SEA-EA focusing more on paper details. The examples also show distinct weaknesses in the baseline outputs.
A.2 SEA-E
- Mistral-7B raises broad, general issues, splits a complexity issue across two Strengths points, and gives Weaknesses that do not match the paper decision.Source: §A.2 · p. 13
- SEA-E and SEA-EA provide clearer viewpoints and more extensive content coverage, while SEA-EA focuses more on paper details.Source: §A.2 · p. 13
- Mistral-7B-Random produces shorter, less detailed reviews, while Mistral-7B-GPT-3.5 produces duplicates attributed to insufficient SFT-stage instruction-dataset standardization.Source: §A.2 · p. 13
A.3 SEA-A
The authors report that SEA-A assigns larger mismatch scores to randomly paired papers and reviews than to corresponding pairs, which they interpret as evidence of consistency discrimination. Applying SEA to this paper yielded a review that aligned with some official feedback and added suggestions.
A.3 SEA-A
- Because SEA-A is trained mostly on low-scoring samples, the reported mismatch-score values are not substantial; the authors show them visually for intuition and provide specific values in Table 8.Source: §A.3 · p. 13, Table 8 · p. 13
- Every random-pair mismatch score in Table 7 exceeds its corresponding-pair value in Table 8, which the authors interpret as SEA-A distinguishing paper-review consistency.Source: Table 7 · p. 13, Table 8 · p. 13, §A.3 · p. 13
- On this paper, SEA-generated feedback aligned with some official Strengths, Weaknesses, and Questions while adding constructive improvement suggestions.Source: §A.3 · p. 14
B Compare with REVIEWER2
On a smaller test set, SEA-E and SEA-EA perform excellently relative to REVIEWER2, whose ROUGE results are not ideal and BERTScore results are unstable. The authors attribute REVIEWER2's weaker results to short outputs lacking valuable information.
B Compare with REVIEWER2
- SEA-E and SEA-EA show excellent performance, while REVIEWER2's ROUGE results are not ideal and its BERTScore results are unstable.Source: B Compare with REVIEWER2 · p. 14
- 100 samples per dataset were randomly selected for the comparison, or all samples when a dataset contained fewer than 100; this subset was smaller than the paper's test set.Source: B Compare with REVIEWER2 · p. 14
- REVIEWER2 often produces relatively short outputs that lack valuable information, which the authors give as the reason for its metric results.Source: B Compare with REVIEWER2 · p. 14
C Human and GPT evaluation
The evaluation materials describe questionnaires for review quality and standardized-review quality, alongside prompts specifying how reviews should be consolidated and formatted. The review rubric covers substantive and presentation criteria, while the consolidation instructions require a structured review with integer scores.
C Human and GPT evaluation
- The review-quality questionnaire contains 10 questions, including an added human-evaluation choice of the best review among model outputs.Source: C Human and GPT evaluation · p. 15
- The SEA-S standardized-content questionnaire has seven questions about consolidation, objectivity, usefulness, repetition, contradictions, professionalism, and overall quality.Source: C Human and GPT evaluation · p. 15
- The SEA-E review rubric asks about research importance, originality, methodological strengths and weaknesses, useful comments, constructiveness, paper-based evidence, interpretation, objectivity, language, and overall quality.Source: C Human and GPT evaluation · p. 15
- The consolidation prompt specifies nine review sections and requires integer scores for Soundness, Presentation, Contribution, and Rating, plus an Accept/Reject decision with reasons.Source: C Human and GPT evaluation · p. 20